Feature #4512
closedAdd MotionCor2 from UCSF to appion
Added by Anchi Cheng over 7 years ago. Updated over 6 years ago.
0%
Description
Align movies with it and upload both non-dose-weighted and dose weighted aligned images.
Updated by Anchi Cheng over 7 years ago
- Related to Task #4379: break up apDDprocess.py to separate camera selection and alignment program selection added
Updated by Anchi Cheng over 7 years ago
- Related to Feature #4511: Use MotionCorr2 program to flip and rotate gain reference added
Updated by Anchi Cheng over 7 years ago
- Subject changed from Add MotrionCorr2 from UCSF to appion to Add MotionCorr2 from UCSF to appion
Updated by Dmitry Lyumkis over 7 years ago
Anchi,
We're reprocessing old data, and notice that the gains are now applied internally in MotionCor2. This is causing an inappropriate gain normalization in the images. Was there a difference between how they were saved (in DM) 2 years ago vs. now? Looks like there's a rotation and flip? I tried physically rotating the norm & bright by -90°, which helped, but not sure if there is also a flip that is necessary. Is it possible to skip gain normalization in MotionCor2 and perform it internally in Appion, as it was done before?
Dmitry
Updated by Anchi Cheng over 7 years ago
committed option --force_cpu_flat. Give it a try.
Updated by Anchi Cheng over 7 years ago
- Status changed from Assigned to In Test
- Assignee changed from Anchi Cheng to Dmitry Lyumkis
Updated by Anchi Cheng over 7 years ago
Also check CameraEMData of your old images to see what is saved under "frame flip" and "frame rotate". If it is really old, these might not be recorded yet. The value recorded should corresponds to the required operation between frames and the references.
Updated by Dmitry Lyumkis over 7 years ago
- Assignee changed from Dmitry Lyumkis to Anchi Cheng
A couple things.
First, I was able to fix some of the gain issues with the latest MotionCor2 binary (updated on Agard's website 10/19/2016). The internal gain normalizations performed within MotionCor2 now look ok.
I had to fix some nomenclature errors assigned to temp_aligned_sum files, so I fixed those in the code, and it's working ok for me now. I committed the minor fixes.
However, the new option --force_cpu_flat, does not work for MotionCor2, as it is applying gains twice (first using CPU, then using MotionCor2). The reason is that the dark/gain normalization is hard-coded in appionlib/apDDFrameAligner.py, and it adds the command regardless. The simplest solution, in my opinion, would be to check whether or not the dark/gain normalization was already performed inside the function def setGainDarkCmd(). I've kinda lost track of how the specific variables/parameters are passed between the classes, so it's not obvious to me how to best do this. Would seem like the function has to query whatever boolean is set in "use_frame_aligner_flat" (apDDprocess.py), but I can't figure out if that gets passed to the MotionCor2_UCSF class ... The "alignparams" dict does not contain relevant information. Anchi, my guess is that you'd have an easy fix?
Updated by Dmitry Lyumkis over 7 years ago
correction: the MotionCor2 distribution that I'm using is coming from Appion's distribution, updated 9/19, not the recent distribution updated 10/19
Updated by Anchi Cheng over 7 years ago
- Assignee changed from Anchi Cheng to Dmitry Lyumkis
Dmitry, hasn't been able to test your changes, yet. I assume you have a reason to want to include the text gpuid in the temp file name. If so, I will have to change the name in catchUpDDAlign.py
Updated by Dmitry Lyumkis over 7 years ago
The program was giving me errors about not being able to find a file, which was named without the gpuid. I thought this was going to be standard naming convention, but maybe you want to keep with / without the "gpuid" name for the gpu / non-gpu processes, respectively? In that case, the only part that should be changed in in apDDMotionCorrMaker.py.
Updated by Michael Cianfrocco over 7 years ago
- Assignee changed from Dmitry Lyumkis to Anchi Cheng
I'm just getting to testing out MotionCor2 now within Appion, and I'm getting a weird error where the command is generated by Appion, but then get an error due to the appion script progressing without executing the motioncor2 command.
If I copy and paste the generated motioncor2 command, the command works so it's a problem with the execution of the motioncor2 command.
Is this a specific problem to our local setup or something others have seen ?
Beginning Main Loop
... 16nov09b_05gr_03sq_02hl_02en locked by another parallel run in the rundir
... Pixel size: 0.6
rawframe_basepath /data/microscopes/frames/16nov09b/rawdata
... Setting new camera info....
... Reference comes from current image
... Reference comes from current image
... Reference comes from current image
... Reference comes from current image
... Setting new camera info....
... Reference comes from current image
... Reference comes from current image
... Reference comes from current image
!!! WARNING: From Bright Reference 16nov09a_09152739_00_7676x7420_bright_0
!!! WARNING: Save Norm Reference 16nov09a_09153151_05_7676x7420_norm_0 to /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc
!!! WARNING: MotionCor2 does not handle dark reference. Assumes zero
... Gain Dark Command Option: -Gain /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc
!!! WARNING: dose not available for this image, try another image
... Setting new camera info....
... Reference comes from current image
... Reference comes from current image
flip rotate True 0
... Setting new camera info....
... Reference comes from current image
... Reference comes from current image
... Reference comes from current image
... Reference comes from current image
... link /data/microscopes/frames/16nov09b/rawdata/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en.frames.mrc to /data/microscopes/appion/16nov09b/ddstack/ddstack5/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en_st.mrc.
... Running: motioncor2 -InMrc /data/microscopes/appion/16nov09b/ddstack/ddstack5/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en_st.mrc -OutMrc tempaida.gpuid_0_sum.mrc -Bft 100 -Iter 7 -Tol 0.500 -Patch 5 5 -Group 1 -MaskSize 1.000 1.000 -Gain /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc -FlipGain 1 -RotGain 0 -Gpu 0
Traceback (most recent call last):
File "/data/software/repo/myami/trunk/bin/makeDDAlignMotionCor2_UCSF.py", line 128, in <module>
makeStack.run()
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/appionLoop2.py", line 88, in run
results = self.loopProcessImage(imgdata)
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/appionLoop2.py", line 146, in loopProcessImage
return self.processImage(imgdata)
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDStackMaker.py", line 127, in processImage
self.otherProcessImage(imgdata)
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDAlignStackMaker.py", line 162, in otherProcessImage
self.alignFrameStack()
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDAlignStackMaker.py", line 248, in alignFrameStack
self.framealigner.alignFrameStack()
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDFrameAligner.py", line 89, in alignFrameStack
self.writeLogFile(output)
File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDFrameAligner.py", line 398, in writeLogFile
midshx = shifts[midval][0]
IndexError: list index out of range
Updated by Anchi Cheng over 7 years ago
The execution is done through subprocess. You can check with a test script using this extraction from appion/appionlib/apDDFrameAligner.py
# run as subprocess
apDisplay.printMsg('Running: %s'% cmd)
self.proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
(stdoutdata, stderrdata) = self.proc.communicate()
If this works, then an output should be generated along with the log that should be parsed by the part that failed.
Updated by Anchi Cheng over 6 years ago
- Subject changed from Add MotionCorr2 from UCSF to appion to Add MotionCor2 from UCSF to appion
- Status changed from In Test to Closed