SOLVED: How does rawtransfer.py decide where images need to go?
Added by Patrick Goetz over 6 years ago
We have a working leginon install which was set up for a single research group. Now we've added 2 new research groups, one of which has image security concerns, so I'm having to re-arrange things to accommodate multiple independent raw image areas on the storage pool.
My original configuration was:
Image directory:
/cryodata/leginon
/etc/leginon.cfg:
[Images]
path: /cryodata/leginon
rawtransfer systemd command line:
ExecStart=/usr/local/lib/python2.7/dist-packages/leginon/rawtransfer.py --method=rsync -- source_path=/local/talos-k2-frames --camera_host=talos-k2 --destination_head=/cryodata 2>&1 >/ var/log/rawtransfer-talos-k2.log
I'm attempting to change destination head to /EM, but now when I start the rawtransfer service, I get the following error message:
Jul 19 09:46:34 kraken rawtransfer.py[5101]: **running /local/talos-k2-frames/20180627_18193023.mrc Jul 19 09:46:34 kraken rawtransfer.py[5101]: frames_path = /cryodata/frames/18jun27a/rawdata Jul 19 09:46:34 kraken rawtransfer.py[5101]: Destination frame path does not starts with /EM. Skipped Jul 19 09:46:34 kraken rawtransfer.py[5101]: **running /local/talos-k2-frames/20180627_18200924.mrc Jul 19 09:46:34 kraken rawtransfer.py[5101]: frames_path = /cryodata/frames/18jun27a/rawdata Jul 19 09:46:34 kraken rawtransfer.py[5101]: Destination frame path does not starts with /EM. Skipped Jul 19 09:46:34 kraken rawtransfer.py[5101]: **running /local/talos-k2-frames/20180627_18204825.mrc Jul 19 09:46:34 kraken rawtransfer.py[5101]: frames_path = /cryodata/frames/18jun27a/rawdata Jul 19 09:46:34 kraken rawtransfer.py[5101]: Destination frame path does not starts with /EM. Skipped Jul 19 09:46:34 kraken rawtransfer.py[5101]: **running /local/talos-k2-frames/20180627_18212826.mrc
Even after changing every instance of cryodata in the configuration settings above to EM. Now I'm wondering where this:
frames_path = /cryodata/frames/18jun27a/rawdata
is set?
Replies (2)
RE: How does rawtransfer.py decide where images need to go? - Added by Anchi Cheng over 6 years ago
The frame path is determined during imaging session creation. It is determined by the leginon sum base image path specified by the user in SetupWizard and default to what is in leginon.cfg. The frame_path is saved in your leginondb.SessionData.
If you have a group of people who want to save their data under /EM/frames instead /cryodata/frames, the simplest way is to set them up with their local leginon.cfg in their linux home directory (See Configure_leginoncfg), with the following:
[Images] path: /EM/leginon [References] path: /common/leginon_ref
The "Images" section defines the default base sum image directory for them as /EM/leginon, and by extension, the base frames directory as /EM/frames.
The "Refereneces" section is something that other users with the same concern brought up, and I happen to have a solution already implemented. See How does leginon find the gain reference - through naming.... This section defines the base path for special sessions that contains gain/dark images that every user need accessing so they don't need to repeat the effort. /common/leginon_ref will be a disk that all users have read and write access to.
RE: How does rawtransfer.py decide where images need to go? - Added by Patrick Goetz over 6 years ago
Thanks. As soon as I posted that and walked over to the lab I realized that this information must be stored in the database by way of the parameters set in start-leginon.py. Will mark as solved.