rawtransfer.py issues
Added by Tom Houweling over 11 years ago
rawtransfer.py (svn rev=17647) creates a mess with frames:
User "yuanhe" ran a session collecting data in /k2data/leginon/yuanhe using myami trunk on a Gatan K2:
During the session Digital Micrograph stored frames in /k2data/leginon/frames:
A subsequent run of rawtransfer.py (RAW_FRAMES_PATH="/k2data/leginon/frames") caused frames to be moved to:
/k2data/frames
instead of
/k2date/frames/yuanhe/13jun15b
as expected.
See screen log for one frame volume:
sent 484185550 bytes received 31 bytes 107596795.78 bytes/sec
total size is 484126344 speedup is 1.00
chown -R 55425:55425 /k2data/frames
cd /k2data/leginon/frames
chown -R 55425:55425 /k2data/frames
**running /k2data/leginon/frames/20130616_14563317.mrc
frames_path = /k2data/frames
DEST /k2data/frames/13jun15b_00039sq_v01_00003hl_00004en.frames.mrc
mkdirs /k2data/frames
rsync -av --remove-sent-files /k2data/leginon/frames/20130616_14563317.mrc /k2data/frames/13jun15b_00039sq_v01_00003hl_00004en.frames.mrc
sending incremental file list
20130616_14563317.mrc
Replies (5)
RE: rawtransfer.py issues - Added by Jim Pulokas over 11 years ago
Tom,
The latest revision of rawtransfer.py (r17706) should be smarter. This does not fix what is already done, so you would have to write a script to fix them or manually edit the database and move files around.
Let me know if this works for you now.
Jim
RE: rawtransfer.py issues - Added by Harry Kao over 11 years ago
Can rawtransfer.py be run alone on the image storage server which has no Legion/Appion installation?
Thank you.
Harry
RE: rawtransfer.py issues - Added by Anchi Cheng over 11 years ago
It does need to access database information thru sinedon and use leginon configuration and need some library we created.
My recommendation for minimal installation (or as a local copy that you include in $PYTHONPATH)
leginon (You can work out what minimal set is in there if you want to)
pyami (specifically weakattr.py and fileutil.py in there)
sinedon (You will need majority of this small subpackage)
On the supporting package side, you should only need
python
MySQL Python client
sinedon.cfg is certainly needed. You may need leginon.cfg set up as well even just to make it run.
The server where you run rawtransfer.py on also need permission to read from the leginondb database.
Anchi
RE: rawtransfer.py issues - Added by ShinBo Chen over 8 years ago
We also have problem
How to make sure that raw transfer.py work well?
I mount the shared folder from DD camera host (windows system) and I could see the session created at images path: /tmp/emdata2/frames/sessionname/rawdata/******.mrc.
Before the **.mrc. written, the DD camera host: //DE20PC/RawFrames/ has written two folders includes *.tif data. (attachment)
We mount //de20-pc/RawFrames /tmp/DE20
We run the script after above.
#nohup $PYTHONSITEPKG/leginon/rawtransfer.py --method=rsync --source_path=/tmp/DE20/ --camera_host=de20-pc --destination_head=/tmp/emdata2/frames > rawtransfer.log
Finally, we didn't see any image_name.st at "images path: /tmp/emdata2/frames/sessionname/rawdata/".
Does it works ?? What are the **.mrc .
| rawtransfer.log (837 Bytes) rawtransfer.log | |||
| IMG_20160225_115517.jpg (2.28 MB) IMG_20160225_115517.jpg |
RE: rawtransfer.py issues - Added by Anchi Cheng over 8 years ago
Leginon get a sum image returned to its interface and save it as mrc images under the session image path. Normally, the image paths are named /tmp/emdata2/leginon/sessionname/rawdata, not frames.
Movie frames saved on Windows PC (the tif files in the time-stamped folder in your screenshot). These are the ones that should have been transferred. When it works, you get FOLDERS in /tmp/emdata2/frames/sessionname/rawdata that contains the tif files. What we do in rawtransfer.py is simply rsync and then a folder name change so that instead of timestamp, you get a matched filename with the sum image.
Chceck rawtransfer.log to see if there is error message or simply stalled.
You should not get *.st from rawtransfer.py