Feature #4267
closeddeleteHidden.py
0%
Description
I wrote a script to help manage unwanted data. There may already be something like this in place, but I didn't see it. The script is called deleteHidden.py, and it erases frames or leginon data that are hidden or rejected. In tests, it cut down our data usage per session by 1/3. It's an appionloop, so it has all the usual options. The key ones are:
--dryrun Just show files that will be deleted, but do not
delete them.
--leginondata Delete hidden image data
--framesdata Delete frames data from hidden images.Updated by Gabriel Lander almost 8 years ago
Very nice script, we're using it extensively, freeing up TBs of data.
I noticed that if an aligned image is trashed, the corresponding unaligned image & frames aren't removed, and vice-versa. Added code to do both.
Should change the original functionality that Scott intended, but please test with the dryrun option selected.
Updated by Gabriel Lander almost 8 years ago
correction - should NOT change the original functionality!
Updated by Anchi Cheng almost 8 years ago
- Target version set to Appion/Leginon 3.3
Gabe, Could you commit your change as an option but not default ? If a user do multiple alignment runs and trash one of the run, it would be bad if it trashes the uncorrected frames when other run is the one he/she want to keep.
Comment: There was indeed a script for this a while ago but was rotten because people started to trash while the experiment was still running. It is probably not the kind you want to run with "wait for more images".
Updated by Gabriel Lander almost 8 years ago
okay I added a "--propagate" option. Without this, the script works exactly like Scott's original script.
Note that the original doesn't remove aligned frames in the ddstack directory. Currently this will only happen with the "--propagate" option set to True.
Updated by Anchi Cheng over 6 years ago
- Tracker changed from Bug to Feature
- Status changed from New to Closed