Leginon

When the image and movie was saved, the database records what reference was used in correcting the sum. During "Create frame stack", it looks up the record and find the reference from the database and then read into the program and do so.

If a specific reference need to be used, an image that uses the references intended to be used is specified by its id number by adding "--refimgid=" followed by the id number.

If you have moved the references, This may be a problem. I don't have an easy solution for you. The location of the references is obtained in the same way as that used in getReferences.py

A similar question from my side:

For one of my datasets the transferscript was not setup and I manually moved the data (D:/frames) from the K2 computer. 
Is there a simple way to run the script? 
I tried to run rawtransfer.py giving the source path as (where my frames are now) and the method mv to a destination_head and camera_host-gatancustomer (our computer name) 
but than I get the error message that destination does not starts with " /data_leginon/destinationfolder "

I only want to apply the proper gain correction on the frames

cheers
and many thanks
Arne

Arne,

I am not sure about the purpose of running the rawtransfer.py script in the question. I assume that it is to change the filenames from time-stamp to Leginon style, in order to use Appion to apply proper gain corrections.

I will answer in two scenarios:

1. You do not intend to use Appion to do gain correction and any other things after. Then, you can use getReferences.py which is now part of the repository in leginon folder, to find them, apply it with your favorite program. You just have to be aware that there may be rotation or flip between the frames and references. Hopefully you have not reaquire gain references too many times in the session, because that would mean that you will need to know when it was changed.

2. You intend to let Appion to do gain correction for you, and/or want to change the filenames. This means you will need to start rawtransfer.py separately since your source path is different. The mv option you used was correct, but I think maybe you have already moved the files into the normal destination path of /data_leginon/frames/session_name/rawdata. This caused confusion to the script since it was cautious about overwriting things already there. The solution is then first move the frames "out" of the final destination so that it can move back in by the script. "mv" on the same drive is just a change in its link, so it should be very fast. Here is how to do it assuming your leginon session images goes into /data_leginon/leginon/session and frames need to go to the parallel /data_leginon/frames/session:

1. move the session frames to under, say, /data_leginon/tmp_folder/
2. start rawtransfer.py with options --method=mv --source_path=/data_leginon/tmp_folder --destination_head=/data_leginon

Hey Anchi,

thanks for the response not sure I understood correctly - as it does not seem to work.
I think the problem is that my frames are named for example , while the leginon entry would be 16apr18e_00003sq_v08_00043hl_00002ed2.mrc

I would like to run the gain corrections within leginon and will setup the rawtransfer script etc very soon so option 2 would be what I prefer - currently I did not have a GPU machine available etc so I thought I would simply collect the data to D:frames and move it later- and did not realize that these images would not be gain corrected.

when I run your correctframes.py script on a test image (e.g 20160418_12370000.mrc) I get a Fatal error image could not be found - even when I give it an absolute path.

cheers &
many thanks
Arne

Programs like correctframes.py or getReferences.py use leginon database to find references. The first thing the script does is to take the name you give, assuming that it is a leginon sum image filename such as 16apr18e_00003sq_v08_00043hl_00002ed2.mrc and search for an image record in the database. The timestamped file is not an image file name in the database, but a special field entry to help with the transfer process. That is why it said it could not find it.

You can still run option two. It just change filenames, that way you no longer have to deal with matching the filenames.

If you don't want to change the time-stamped name (20160418_12370000.mrc), then we will need to add a search pattern for it in correct frames.py You may have to point me to where you get the file so I can make the change there for future users.

mmh -thats what I thought and that is obviously a problem if you already started hiding images while data collection

I have no problem to change the name I just dont exactly know which one is which

what do you need ?
a set of files -or filenames?

Arne

For running what script ?

In general. my scripts assumes the filename you see in Leginon image viewer.

for both scripts - I think - especially for rawtransfer

I understood that rawtransfer would assign a real leginon name to the frame name that was given by the gatan software - in my case the frames were moved to a different folder to free up space on the gatan computer - they do not have a leginon-like name and are not gain corrected - However when I run this script I get the error "destination does not starts with " /data_leginon/my_folder"

There may be a mixup with the database or some other problems - and I would be happy if I could simply apply gain corrections as the rawtransferscript will be setup soon I simply only want to keep the first few datasets that I have already collected. I tried the correctframes script. I reasoned that I could run this in batch mode with a list of files to correct - but it also does not work

I made the gain corrections in Appion and the summed frames in the image viewer are gain corrected - but when I run getReferences.py I only get filename as output- (I also ran it with a session name as variable in this case I get image not found) - so could that be the problem?

cheers
Arne

Show me an example of the rawtransfer.py script that fails with the options supplied and the full path of the time-stamped files and leginon full path. Use pseudo names if you need to.

Yes, getReferences.py gets you the directory name and filename for one image only. It is meant to be used if you have your own script for gain correction and just want to know where to find the references.

rawtransfer.py is a daemon program. It checks the source path for movies belong to Leginon and transfer them all as they appear.

After a skype call, here is the situation and solution.

Arne's system saves leginon data with this pattern as "image path" in leginon.cfg

[Images]
path: /data_leginon

While our assumption is that there is another directory at the head of it which is the destination-head referred in rawtransfer.py

For example,

[Images]
path: /data_leginon/leginon

Without the head, the path parsing become problematic. During Leginon new session creation, it probably generate the frame path and saved that to the database at a place where the user has no control of.

The solution is to reorganize leginon path. In this case, we did these:

1. mkdir /data_leginon/leginon
2. mkdir /data_leginon/frames
3. mkdir /data_leginon/appion
4. mv /data_leginon/16* /data_leginon/leginon

Now that the image files are in /data_leginon/leginon/sessionname/rawdata. We can change database. In the downloaded myami folder dbschema, there is a script called session_path_change.py

Run it as described in Issue #2995, i.e.

python session_path_change.py session /data_leginon/session/rawdata

For each session, do it as follows since the frames are not there. The script was written for moving existing frame data saved in the new directory, not for before rawtransfer.py is done.

mkdir /data_leginon/frames/session
mkdir /data_leginon/frames/session/rawdata
python session_path_change.py session /data_leginon/frames/session/rawdata frame

Anchi Cheng wrote:

When the image and movie was saved, the database records what reference was used in correcting the sum. During "Create frame stack", it looks up the record and find the reference from the database and then read into the program and do so.

If a specific reference need to be used, an image that uses the references intended to be used is specified by its id number by adding "--refimgid=" followed by the id number.

If you have moved the references, This may be a problem. I don't have an easy solution for you. The location of the references is obtained in the same way as that used in getReferences.py

Hi Anchi - how do I retrieve the reference image ID? We ran into a problem where a user wants to specify a different gain reference instead of the one associated with their leginon session.

Thanks!
mike

Two possibilities:

1. Go to any myamiweb image viewer, hover your cursor on the "i" information tool above the image whose references you would like to use if your browser is set to show the url link. The link ends with the image id of the image.

2. Click on the "i" information tool, the image report page has the imageId under the section Image Information.