DDD raw frame file transfer » History » Revision 13
Revision 12 (Anchi Cheng, 10/30/2013 05:32 PM) → Revision 13/20 (Anchi Cheng, 12/19/2013 05:11 PM)
h1. DDD raw frame file transfer
h2. Introduction:
Due to the limited disk space on the camera host computer, we wish to transfer the raw frame data to a location with more storage space. The following assumes the camera host is a Windows system and the destination is a linux system. For best performance, the linux system should be directly attached to the destination storage rather than over a network. The "rawtransfer.py" script will run continuously in the background on the linux system, independent from Leginon. It will periodically check the Leginon database to see if any new raw frames have been acquired by Leginon, then pull them over the network from the camera host.
In addition, the script changes the directory naming (DE) or stack file naming (K2 Summit) to reflect the integrated image filename transmitted to Leginon. Therefore, even if you have network it so that the frames are directly saved to the file server, you should make modification of rawtransfer.py and use it to do this filename synchronization.
h2. Initial setup of your systems:
*Camera host (Windows)*: the folder where your raw frames are stored must be configured as a shared folder (without a password in our example). Remember the share name, as you will need it when mounting from linux.
*Destination host (Linux)*: The samba-client package must be installed in order to mount the shared Windows folder
h2. Mount the shared folder
From the linux host, mount the shared folder on the camera host. For example, the windows system host name is "decamera" and the shared folder is "D:\RawFrames\DE12\", but has a share name of "DE12". The mount command would be:<pre>mount //decamera/DE12 /example_mount_point</pre>(replace example_mount_point with a directory you have created for this purpose).
h2. Start rawtransfer script in the background
The script "rawtransfer.py" is provided in the leginon package, usually installed in your python site-packages folder (referred to as PYTHONSITEPKG below). In the future, this script will be wrapped up in a proper init script such as you would find in /etc/init.d. That would allow for automatically starting it when the system boots, but for now you must launch it manually.
*myami-2.2*
The script will check the environment variable RAW_FRAMES_PATH for the location of the shared folder mount point. The following command line will set that variable and then launch rawtransfer.py in the background:<pre>RAW_FRAMES_PATH=/example_mount_point nohup $PYTHONSITEPKG/leginon/rawtransfer.py > rawtransfer.log &</pre>The "nohup" ensures that it will continue running even when you log out or the terminal session is closed.
The example generates the log file "rawtransfer.log" in whatever directory you run this from. There is no automatic log rotation, so you must occasionally check that this file does not grow out of control.
*myami-3.0*
You need to pass in several options
<pre>
--method=METHOD method to transfer, e.g. --method=mv or --method=rsync
--source_path=PATH Mounted parent path to transfer, e.g. --source_path=/mnt/ddframes
--camera_host=CAMERA_HOST Camera computer hostname in leginondb, e.g. --camera_host=gatank2
--destination_head=PATH Specific head destination frame path to transfer if
multiple frame transfer is run for one source to frame paths not all mounted on the same computer, e.g. --destination_head=/data1
</pre>
For example,
<pre>
nohup $PYTHONSITEPKG/leginon/rawtransfer.py --method=rsync --source_path=/example_mount_point --camera_host=scope1cam1 --destination_head=/data > rawtransfer.log
</pre>
* mv method is used when the frame files are saved directly onto network drive. No transfer is needed. However, since the filename is only timestamped, this script will change the filenames and organize them by sessions.
* In this version, files are saved under frames directory and organized by session below. With the destination_head in the example, the resulting frames.st file will be in
<pre>
/data1/frames/your_session/rawdata/image_name.st
</pre>
For example,
<pre>
nohup $PYTHONSITEPKG/leginon/rawtransfer.py --method=rsync --source_path=/example_mount_point --camera_host=scope1cam1 --destination_head=/data > rawtransfer.log
</pre>
* If you have another frame-saving camera on scop2cam1 then you start another on with different -source_path and --camera_host values.
The script is currently set up to check for new frames every 30 seconds. If it finds new files, it will check the database to see if they are connected to a Leginon image. If it is from Leginon, then it will rsync or move the folder to the rawdata folder for that session. The name of the folder will be something like: 11nov28_00001sq_00001_hl_00001_en.frames (based on the same name as the Leginon mrc image). Then it will remove the original folder. It looks like it can copy one frame per second, but this could vary depending on network traffic. Any frames folders found that are not connected to a Leginon image are not transferred and not removed, so you may want to periodically clean up those extra frames.