Users tried processing with 2 different leginon paths, now rawtransfer is stuck

Added by Patrick Goetz about 6 years ago

We have 2 microscopes, and are transitioning from one leginon folder location to another (an obvious recipe for distaster).

- Rawtransfer for the krios microscope is set up to use /cryodata as the destination head (the old path)
- Rawtransfer for the talos microscope is set up to use /EM as the destination head (the new path)

User accounts A, B use the default leginon.cfg, which has image path /cryodata/leginon
User accounts C, D use local leginon.cfg files, which have image path /EM/lab[C,D]/leginon

I'm out of the office for a couple of weeks and left things in this transitional state so people could still get work done while I was gone. I still don't know exactly what happened, but it appears that they attempted to use the leginon system with talos using both the A and the C accounts. Now if I run rawtransfer for talos using /EM as the destination head I get these messages in the log:

   Destination frame path does not starts with /EM. Skipped

And if I restart it using /cryodata as the destination head, I get these messages in the log file:

   Destination frame path does not starts with /cryodata. Skipped

In both cases, no images are processed. It seems to be completely stuck because of the fact that images were collected with different leginon.cfg files with different image paths.

Is there any way to get out of this?

Replies (2)

RE: Users tried processing with 2 different leginon paths, now rawtransfer is stuck - Added by Anchi Cheng about 6 years ago

Patrick,

If  where rawtransfer.py runs can access both /cryodata and /EM, then you do not need to specify destination_head, just source_path and camera_host.  That may resolve your problem.

As far as what you described as stuck, it does not sound possible. rawtransfer.py checks all files in the source_path each time it runs and only skips the ones not having the destination_head in somthing like this:

for name in all_filenames:
  my_destination_head = .....find it in detabase
  if my_destination_head != rawtransfer.py_option_destination_head:
    continue

When restrict it to /EM, it will skip all the ones not going there, but should still process the /EM ones. You just have to run another rawtransfer.py that restricts it to /cryodata to clear those. Both will complain half of the time, but should still do the work for their own half.

RE: Users tried processing with 2 different leginon paths, now rawtransfer is stuck - Added by Patrick Goetz about 6 years ago

Thank you, that works! This begs the question why would anyone ever include a destination_head on the rawtransfer command line? Here is my new and improved systemd rawtransfer service file:

[Unit]
Description=rawtransfer.py service for Talos
After=network.target

[Service]
Type=simple
Restart=on-failure
User=root
Group=root

ExecStart=/usr/local/lib/python2.7/dist-packages/leginon/rawtransfer.py --method=rsync --source_path=/local/talos-k2-frames --camera_host=talos-k2

[Install]
WantedBy=multi-user.target

Maybe it would be helpful have the Wiki instructions/example have a default of no destination_head and then just add a note that you need to add a destination_head to the command line when ...

(I don't know when that is)

(1-2/2)

Project

General

Profile

Leginon