SEMC file storage policy¶

1. Raw frames are transferred off camera to buffer server with rsync using rawtransfer.py which also removed the finished frame stack on the camera to make room for more to come.
2. A script (Available upon request) compresses the frames and transfer the compressed files to gpfs primary storage without deleting the frame stacks from buffer server. bzip or bzip2 are commonly used by the users to compress the files. Since the raw frames are integers, it can be easily compressed to 4-10 times smaller.
3. User perform frame alignment of the frame stacks on the buffer server which saves the aligned sum image on primary storage. Aligned frame stack is not saved.
4. The raw frames are needed if default frame processing does not give optimal results. We encourage users to save these for their own. We keep these on buffer server for 2 weeks and then deleted. The compressed frame stack on primary storage is kept for 30 days. It is user's responsibility to archive this on external drives.
5. User will use the aligned sum image saved in the Leginon database typically as "-a" images for post processing. 5% of the projects find it necessary to realign the frames with a different parameters or a different programs.