Removing data older than x
Added by Christopher Lilienthal over 7 years ago
Hello,
I was wondering how you clean up the files/folders/sessions/projects/references/etc once they have aged sufficiently to warrant removal? For example, let's say your retention policy is 60 days how do you go about cleaning up everything that is 61 days or older? You could just remove the files but then you'd still have entries in the database that point to non-existent files and folders.
Thanks
Replies (3)
RE: Removing data older than x - Added by Anchi Cheng over 7 years ago
Our current policy is to clean up files, not database entries. It is possible to extract a part of database, usually by project and start propagate from that. It has only been done once for production, so we know it works but you may need a bit of debug on your system. See Issue #2915
RE: Removing data older than x - Added by Anchi Cheng over 7 years ago
On the other hand, if you really want to, you can clean up table by table with a mysql script. All the tables you care have a timestamp. The tables that accumulates fast are also not essential to keep after the user no longer care to see or analyze through Appion and already know the metadata such as pixel size, exposure dose, reference image (in the case of direct detector movies). Here are some examples:
Used only data collection:
EMTargetData
QueueData
Used by the web image viewer to link images together to display images and targets: Keep until neither need to see in the web imageviewer nor use Appion.
AcquisitionImageStatsData
AcquisitionImageTargetData
PresetData
Metadata of the image: Keep until image metadata is written in other form. (Magnification, defocus range, number of images taken, length of session etc.) Might be needed at the time of the publication.
CameraEMData
ScopeEMData
AcquisitionImageData
ImageCommentData
To keep Leginon running, it is easiest just to keep the smaller tables intact.
RE: Removing data older than x - Added by Anchi Cheng almost 7 years ago
There is also a delete hidden Appion feature. See Issue #4267 and is available in beta and 3.3