DDD raw frame file transfer¶

Introduction:¶

Due to the limited disk space on the camera host computer, we wish to transfer the raw frame data to a location with more storage space. The following assumes the camera host is a Windows system and the destination is a linux system. For best performance, the linux system should be directly attached to the destination storage rather than over a network. The "rawtransfer.py" script will run continuously in the background on the linux system, independent from Leginon. It will periodically check the Leginon database to see if any new raw frames have been acquired by Leginon, then pull them over the network from the camera host.

In addition, the script changes the directory naming (DE) or stack file naming (K2 Summit) to reflect the integrated image filename transmitted to Leginon. Therefore, even if you have network it so that the frames are directly saved to the file server, you should make modification of rawtransfer.py and use it to do this filename synchronization.

Initial setup of your systems:¶

Camera host (Windows): the folder where your raw frames are stored must be configured as a shared folder (without a password in our example). Remember the share name, as you will need it when mounting from linux.

Destination host (Linux): The samba-client package must be installed in order to mount the shared Windows folder

Mount the shared folder¶

From the linux host, mount the shared folder on the camera host. For example, the windows system host name is "decamera" and the shared folder is "D:\RawFrames\DE12\", but has a share name of "DE12". The mount command would be:

mount //decamera/DE12 /example_mount_point

(replace example_mount_point with a directory you have created for this purpose).

Start rawtransfer script in the background¶

The script "rawtransfer.py" is provided in the leginon package, usually installed in your python site-packages folder (referred to as PYTHONSITEPKG below). In the future, this script will be wrapped up in a proper init script such as you would find in /etc/init.d. That would allow for automatically starting it when the system boots, but for now you must launch it manually. The script will check the environment variable RAW_FRAMES_PATH for the location of the shared folder mount point. The following command line will set that variable and then launch rawtransfer.py in the background:

RAW_FRAMES_PATH=/example_mount_point nohup $PYTHONSITEPKG/leginon/rawtransfer.py > rawtransfer.log &

The "nohup" ensures that it will continue running even when you log out or the terminal session is closed.

The example generates the log file "rawtransfer.log" in whatever directory you run this from. There is no automatic log rotation, so you must occasionally check that this file does not grow out of control.

The script is currently set up to check for new frames every 30 seconds. If it finds new files, it will check the database to see if they are connected to a Leginon image. If it is from Leginon, then it will rsync the folder to the rawdata folder for that session. The name of the folder will be something like: 11nov28_00001sq_00001_hl_00001_en.frames (based on the same name as the Leginon mrc image). Then it will remove the original folder. It looks like it can copy one frame per second, but this could vary depending on network traffic. Any frames folders found that are not connected to a Leginon image are not transferred and not removed, so you may want to periodically clean up those extra frames.