DDD raw frame file transfer¶

Introduction:¶

Due to the limited disk space on the camera host computer, we wish to transfer the raw frame data to a location with more storage space.
The following assumes the camera host is a Windows system and the destination is a linux system. For best performance, the linux system should be directly attached to the destination storage rather than over a network. The "rawtransfer.py" script will run continuously in the background on the linux system, independent from Leginon. It will periodically check the Leginon database to see if any new raw frames have been acquired by Leginon, then pull them over the network from the camera host.

Initial setup of your systems:¶

Camera host (Windows): the folder where your raw frames are stored must be configured as a shared folder (without a password in our example). Remember the share name, as you will need it when mounting from linux.

Destination host (Linux): The samba-client package must be installed in order to mount the shared Windows folder

Mount the shared folder¶

From the linux host, mount the shared folder on the camera host. For example, the windows system host name is "decamera" and the shared folder is "D:\RawFrames\DE12\", but has a share name of "DE12". The mount command would be:

mount //decamera/DE12 /example_mount_point

(replace example_mount_point with a directory you have created for this purpose).

Start rawtransfer script in the background¶

It is currently set up to check every 30 seconds if any new files have been stored on the Direct Electron PC shared folder. If it finds new files, it will check the database to see if they are connected to a Leginon image. If it is from Leginon, then it will rsync the folder to the rawdata folder for that session. The name of the folder will be something like: 11nov28_00001sq_00001_hl_00001_en.frames (based on the same name as the Leginon mrc image). Then it will remove the original folder. It looks like it can copy one frame per second, but this could vary depending on network traffic.

The script is in leginon directory. Use a command like this:

RAW_FRAMES_PATH=/your_base_directory/dde nohup $PYTHONSITEPKG/leginon/rawtransfer.py > rawtransfer.log &