Leginon

Question 1:

Yes. Just change the parent frame directory. This info is in Using_Gatan_K2_Summit_in_Leginon but not very obvious. I now also added it in DDD_raw_frame_saving
You don't need to change setup in DM. The plugin is accessing the function at lower level than the gui.

The rawtransfer.py should still be run if you want to have a correspondance of filename and process with Appion for gain correction and alignment as we do at NRAMM. Just set --method=mv instead of rsync. It runs very fast as it just change the link between data and file path.

Question 2:

We've found the same thing last week. See Issue #2670 for solution

Hello Anchi,

Thanks for your help. Just a quick question, after I download the new dmsem.py and replace the old file in the pyscope folder, do I have to compile the new dmsem.py file again?

-Long

python compiles on the fly. It will detect that the file has changed and recompile. If you worry, remove its .pyc file first before you start the client.

Anchi,
Here we have to change the cameraid in order for dmsem.py to recognize the Orius and K2 which in our system are 1(0) and 1(2) respectively. I downloaded the new version of dmsem.py and modified it following the instructions, plus the cameraid. Now, we still get the same error about the size mismatch. Are there other parameters that we need to worry about?

Thanks

Mario

Hello Anchi,

Thank you for your help. I am still testing the motion-correction software from Yifan Cheng's group. These raw frames are currently saved in local disk drive D:\frames and their sizes seem to be correct. However, after I copied these raw frames and run the motion-correction software on the GPU cluster. The program returns a lot of 0 value and the final aligned mrc file is completely black when I opened it via ImageJ MRC-writer plugin. Have you encountered such problem before? I have downloaded the newest dmsem.py, set the DM2.3 value to TRUE and replaced the old ones.

Thanks again,
Long

Mario,

Please use a seperate thread for your problem.

Long,

A local user here made a naive mistake which lead to discovery of Issue #2673. The symptom seems the same as yours. You can either apply the fix or just use the workaround.

Hello Anchi,

I am sorry to bother you again. If I understand correctly, does that mean we have to set the exposure time in "tomo" preset longer than the the exposure time in Leginon Node "Correctrion"? But I have just checked that and the exposure time in both of "tomo" and "Correction" is the same at 200 ms.

BTW, the frame exposure time in Gatan DigitalMicrograph software is also 200 ms.

I have attached our settings as the attached images. Could you take a look at it and give me a more detailed instruction?

Thanks a lot,
-Long

200 ms exposure for 200 ms frame time should be enough. You just need to be multiple of it. What caused Issue #2673 was that the user put 100 ms exposure time when 200 ms was the frame time. The algorithm in Leginon did floored the division and made the number of frames recorded to say 0.

Your preset settings looks fine and the known problem should only occur if you did your dark correction at exposure time shorter than the frame time.

If your problem was not solved by reacquiring dark image with an exposure at multiple of the frame time, we will have to find another possible cause.

Hello Anchi,

The dark correction for the K2 camera has been done about 3 weeks ago in Leginon/Calibration steps. I have checked my notes and we have finished the dark correction at 2000 ms exposure time, 5 images to combine. But we have not checked "Save frames" when we did it last time. Did we have to make sure that "Save Frames" is ON when we collected the dark reference?

Or, I am thinking, the problem could be also possible due to the alignment software so it might be worthwhile to check the raw frames. I have tried opening it via MRC reader in ImageJ but ImageJ did not read the raw frames. Do you have any suggestion for checking these raw .mrc frames file?

Thanks a lot,
Long

Long,

You should not have to select "Save frames" when you acquire the dark/bright images. Issue Issue #2673 is a bug.

If MRC reader in ImageJ does not read the raw frames, it is likely have NaN in there.

I attach here a python script I often use for checking frame gain/dark correction. This script is generally useful and should run with the same environment that you run leginon and appion. This gives back integrated frame(s) with dark/gain correction for either K2 or DE detectors.

Usage: correctframes.py sessionname start_frame nframe image
  sessionname (str): leginon sessionname used to identify dd models
  start_frame (int): first frame to use (minimal=0)
  nframe (int): number of frames to sum up
  image (str or int): imagefilename or imageid

I added to it some test printing which will print information that I can use to see if you have the same problem. Please run it on one of your dd image with frames and post here the last part of the output starting with the line "corrected_frame min, max:"

Hello Anchi,

Sorry for the late reply. We are getting busy in preparing new sample and freezing new grids. Luckily now we are getting good sample and grid for cryoEM.

For the still existing problems about DDD frames alignment. I have downlowned your .py file but the program exit with a error: "No module named appionlib" because I have not installed Appion on the computer yet. So I think it might be better if I could upload one of the .mrc files so it would be easier for you to check the raw frams.

And I have run the motion-correction software from Yifan's group again, and I have noticed at the beginning of the running process it shows:

Input Stack: Nx(3710) Ny(3838) Nz(10)

Crop Image: Offset(0 0) Dim(3710)
Allocate host memory: 0.925194 Gb
Use GPU: #0 GeForce GTX 690
GPU memory: free:1655Mb total:2048Mb

Read stack:
Error when reading #000
......Read and sum frame #000 mean:0.000000
Error when reading #001
......Read and sum frame #001 mean:0.000000
Error when reading #002
......Read and sum frame #002 mean:0.000000
Error when reading #003
......Read and sum frame #003 mean:0.000000
Error when reading #004
......Read and sum frame #004 mean:0.000000
Error when reading #005
......Read and sum frame #005 mean:0.000000
Error when reading #006
......Read and sum frame #006 mean:0.000000
Error when reading #007
......Read and sum frame #007 mean:0.000000
Error when reading #008
......Read and sum frame #008 mean:0.000000
Error when reading #009
......Read and sum frame #009 mean:0.000000

It seems that the motioncorr program could not read the raw frames at the beginning. Could you think it might be due to some errors about Leginon saving raw frames?

Many thanks!

-Long

Hi, Long,

It works fine here. It gave this initial output from our 2.0 version of dosefgpu_driftcorr

Input Stack: Nx(3710) Ny(3838) Nz(10)

Crop Image: Offset(0 0) Dim(3710 3838)

Allocate host memory: 1.061430 Gb
Use GPU: #0 Tesla K20c
GPU memory:  free:4364Mb    total:4800Mb

Read stack:
......Read and sum frame #000   mean:1.663643
......Read and sum frame #001   mean:1.663393
......Read and sum frame #002   mean:1.664553
......Read and sum frame #003   mean:1.665679
......Read and sum frame #004   mean:1.664467
......Read and sum frame #005   mean:1.665513
......Read and sum frame #006   mean:1.664886
......Read and sum frame #007   mean:1.665484
......Read and sum frame #008   mean:1.664857
......Read and sum frame #009   mean:1.665035

and the movie was aligned. I could see that there was a defect column. You should redo gain correction in linear mode where such defect would be handled. If it remains a problem, mark it as a bad column, then Leginon will handle it when it does its software gain correction.

attached is the aligned image

Hello Anchi,

Thank you for helping us check the raw frames. At least the raw data from Leginon seems okay. So next step I will check the motion correction program (from Yifan's group) to make sure it could read the raw frames.

Another problem is the defect column you mentioned in the final aligned .mrc file. Your suggestion is to redo gain correction in "linear" mode. But I have finished Gain/Dark reference in K2 "counted" mode under Leginon/Calibrations and all the raw frames are also collected in "counted" mode. Do I still need to redo gain correction in "linear" mode? Another thing is that it might be over 1 month since we took the camera reference in Gatan DM software. Do you think that these "old" camera reference also contribute to the defect column?

Thanks again for your help!

Long

The defect column might have surfaced within last month after your "old" DM side references. This happens. You should ask Gatan about it if you'd like to know why.
Either marking it on Leginon side as bad column or let DM redetermining them will work. At the moment, it may be better to do it on Leginon side until DM 2.31.

So, just forget my suggestion about redoing gain correction in "linear" mode for now.

Hello Anchi,

We have fixed this program by updating the alignment software to the newest version. We used to apply the motion correction program downloaded from the Nature Methods website and it could not read our raw frames data set. I send an email to the author of the paper and they are very nice to share us the v2.0 motion correction program. Now the programs works well and could align the raw frames very quickly and suit for the on-the-fly processing.

So my next question is, if we plan to set up the similar enviroment as Scripps (when the leginon collects raw frames, the alignment program could run automatically in the background and upload the aligned image to Leginon.) do we have to install Appion too? Is this function dependent on Appion?

Thanks a lot,
Long

Yes, it is dependent on Appion but you don't need the full installation.

You need only a partial installation of appion. Think Appion as a package that helps you link up different software packages. You can install a minimal parts to do frame alignment and upload.

You need these parts to get transfer/alignment/upload going:

1. Setup rawtransfer.py for your raw frame transfer to match their name to Leginon naming and to network drive.DDD_raw_frame_file_transfer
2. Enable processing database creation (ap*) privilege as in step 11 of Database_Server_Installation
3. Create a processing database for your project Create_a_Processing_Database
4. Install like Leginon processing server on the gpu capable computer Processing_Server_Installation
5. Install appion python scripts as in Install_Appion_Packages under headings "Install the appion python package"
6. Make sure motion correction program is in your path as dosefgpu_drift_corr
7. Edit directly config.php in your webserver/myaiweb location to uncomment the processing plugin addition.

addplugin("processing");

The step 3 is the most time consuming part. But you've already know how to do it.

Hello Anchi,

Just another quick question about the defect column showed in the final aligned image.

Today when I collected the data with saving raw frames, I could clear see this defect column. However, after I turned off the saving raw frames function and just saved the recorded images at a exposure time ~ 2.0 seconds, the defect column disappeared. So this defect column only shows when Leginon saves the raw frames.

Does that indicate that when Leginon saves raw frames, it uses different Gain/Dark references with Leginon saving simply recorded images? Have you seen this kind of situation before?

One of the potential reason might be from the Gatan DigitalMicroscopy software. We have not done the camera gain calibration on the Gatan Camera for ~1 month. I will try to do the camera calibration tomorrow and keep you updated.

Thanks,
Long

First,

Did you add the bad column to the correction plan in Leginon or not ? If it is added, you should not see the defect in the sum image shown in leginon whether you save the frames or not. On the other hand, the raw frames are saved without correction. However, when you later run makeDDRawFrameStack.py in appion, it includes Leginon bad column correction
before the alignment, so you will be fine.

If you did not save the bad column as bad column in Leginon correction node, then somehow your DM 2.3.0 turned its correction off when it is in dose fractionation mode. That would be odd since it is not supposed to happen until 2.3.1.

If this is indeed the behavior, find the bad column in the image and add it to Leginon bad column correction plan so that it will correct the raw frames.