Appion

A couple things.

First, I was able to fix some of the gain issues with the latest MotionCor2 binary (updated on Agard's website 10/19/2016). The internal gain normalizations performed within MotionCor2 now look ok.

I had to fix some nomenclature errors assigned to temp_aligned_sum files, so I fixed those in the code, and it's working ok for me now. I committed the minor fixes.

However, the new option --force_cpu_flat, does not work for MotionCor2, as it is applying gains twice (first using CPU, then using MotionCor2). The reason is that the dark/gain normalization is hard-coded in appionlib/apDDFrameAligner.py, and it adds the command regardless. The simplest solution, in my opinion, would be to check whether or not the dark/gain normalization was already performed inside the function def setGainDarkCmd(). I've kinda lost track of how the specific variables/parameters are passed between the classes, so it's not obvious to me how to best do this. Would seem like the function has to query whatever boolean is set in "use_frame_aligner_flat" (apDDprocess.py), but I can't figure out if that gets passed to the MotionCor2_UCSF class ... The "alignparams" dict does not contain relevant information. Anchi, my guess is that you'd have an easy fix?

Dmitry, hasn't been able to test your changes, yet. I assume you have a reason to want to include the text gpuid in the temp file name. If so, I will have to change the name in catchUpDDAlign.py

I'm just getting to testing out MotionCor2 now within Appion, and I'm getting a weird error where the command is generated by Appion, but then get an error due to the appion script progressing without executing the motioncor2 command.

If I copy and paste the generated motioncor2 command, the command works so it's a problem with the execution of the motioncor2 command.

Is this a specific problem to our local setup or something others have seen ?

Beginning Main Loop
 ... 16nov09b_05gr_03sq_02hl_02en locked by another parallel run in the rundir
 ... Pixel size: 0.6
rawframe_basepath /data/microscopes/frames/16nov09b/rawdata
 ... Setting new camera info....
 ... Reference comes from current image
 ... Reference comes from current image
 ... Reference comes from current image
 ... Reference comes from current image
 ... Setting new camera info....
 ... Reference comes from current image
 ... Reference comes from current image
 ... Reference comes from current image
!!! WARNING: From Bright Reference 16nov09a_09152739_00_7676x7420_bright_0
!!! WARNING: Save Norm Reference 16nov09a_09153151_05_7676x7420_norm_0 to /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc
!!! WARNING: MotionCor2 does not handle dark reference. Assumes zero
 ... Gain Dark Command Option:  -Gain /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc
!!! WARNING: dose not available for this image, try another image
 ... Setting new camera info....
 ... Reference comes from current image
 ... Reference comes from current image
flip rotate True 0
 ... Setting new camera info....
 ... Reference comes from current image
 ... Reference comes from current image
 ... Reference comes from current image
 ... Reference comes from current image
 ... link /data/microscopes/frames/16nov09b/rawdata/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en.frames.mrc to /data/microscopes/appion/16nov09b/ddstack/ddstack5/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en_st.mrc.
 ... Running: motioncor2 -InMrc /data/microscopes/appion/16nov09b/ddstack/ddstack5/16nov09b_CSBNucl_00005gr_00003sq_00003hl_00001en_st.mrc -OutMrc tempaida.gpuid_0_sum.mrc -Bft 100  -Iter 7 -Tol 0.500 -Patch 5 5  -Group 1 -MaskSize 1.000 1.000  -Gain /data/microscopes/appion/16nov09b/ddstack/ddstack5/norm-aida-0.mrc -FlipGain 1  -RotGain 0  -Gpu 0
Traceback (most recent call last):
  File "/data/software/repo/myami/trunk/bin/makeDDAlignMotionCor2_UCSF.py", line 128, in <module>
    makeStack.run()
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/appionLoop2.py", line 88, in run
    results = self.loopProcessImage(imgdata)
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/appionLoop2.py", line 146, in loopProcessImage
    return self.processImage(imgdata)
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDStackMaker.py", line 127, in processImage
    self.otherProcessImage(imgdata)
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDAlignStackMaker.py", line 162, in otherProcessImage
    self.alignFrameStack()
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDAlignStackMaker.py", line 248, in alignFrameStack
    self.framealigner.alignFrameStack()
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDFrameAligner.py", line 89, in alignFrameStack
    self.writeLogFile(output)
  File "/data/software/repo/myami/trunk/lib/python2.7/site-packages/appionlib/apDDFrameAligner.py", line 398, in writeLogFile
    midshx = shifts[midval][0]
IndexError: list index out of range