Appion

The shutdown was due to incorrect plots being created as output from CTFfind and ACE2 runs, as well as improper correction. The latest version of ACE2 appears to estimate the CTF properly, and the output graphs look ok.

Reviewed and tested r17161, and at least for 12apr05b, the 2d power spec overlay image shows up. I'll do some more testing on other sessions as well to see if there are any issues.

Everyone, this is a major issue that needs to be resolved before we can branch 3.0. The primary concern is that someone could do their CTF estimation with the old standard using 2.1 or 2.2, and then do CTF correction with the current trunk (or 3.0 if we released without fixing this).

1. Revert to using a negative angle in the trunk - installations that are not currently using the trunk are entirely unaffected moving forward, for those of us that are using the trunk,
2. Write a DB update script to change the angles to old standard
3. For Ace2 ctf-correction, if the difference in the astig angles are large the results could still be wrong. Add a warning to the CTF correction page for Ace2 that asks the user if they really want to use the estimation run.

Jim also suggested adding a column to the DB table that identifies which convention is being used for the estimation, then handling the angle appropriatly in the code.

Any other suggestions?
Also, Scott, Gabe and Neil, it would be helpful to know to what extent this compatibily issue is affecting your labs. It is a big deal for us since users here often switch between versions during processing of a single session.

Hi Amber,

Old versions of ACE2 gives an invalid astig angle, so there is no reason to convert them, you would have to re-run the program to get a valid angle. ACE1 does not do astig, and CTFFIND did not change.

The main issue is the conversion from negative underfocus to positive underfocus. I think we fixed this, fingers crossed.

All new estimates have several new columns, so we could use any of those to test if it is a new or old version.

Neil

Amber, other than having incorrect CTF plots, this is not affecting my lab too much because we use ACE1 primarily. As I understand it, the current version puts the ACE1 calculated confidence values in the database, and doesn't change the sign of the ACE estimated defocus.

Thanks guys.

Scott, are you saying that the trunk code still adds the Ace1 values as negative underfocus values, while Ace2, ctffind and xmipp are inserting positive underfocus values?

1. In ctfdb.py, there are several places where the apCtfData is retrieved. Put this query into a function that will change the sign of the retrieved defocus values for each entry where the values in the new colums are all NULL.
2. When getting the best defocus values for an image for doing ctf correction, add a check to ignore Ace2 runs from the older version. This is to avoid incorrect astig values. Another option here is to only ignore it if there is a large enough difference between the 2 angles to really affect the outcome. If I understand Anchi correctly, if the difference is small enough, it won't matter and it may not be worth re-running the estimation.

Thoughts?

To the best of my knowledge, ACE1 has always put positive defocus values in the database.

OK. I'm seeing your response to #2016 as well is along this line. So...I should NOT change the sign of the defocus values if they are from an older Ace1 run, only if it from an older ctffind or Ace2 run. Does this seem right?
But if the CS stays positive in Ace1, would we still have to flip something somewhere to make the results correct in Appion?

Well, we taking the absolute value in the code whenever we pull the CTF, so we don't really need a database update, it could help us distinguish the old values from new values in future.

Ok, I've sifted through the code to confirm if everything is using the absolute value of the defocus. It looks like much of it does. There are a few places that are not, but I'm not sure what effect these have on the final results or display.

1. I believe the def1 and def2 values are always either both positive or both negative and if not something is wrong. If this is true, I won't worry about things like sqrt(defocus1*defocus2) - which I think will barf if the answer is a complex number, and ctfDisplay.py, which only takes the absolute value if BOTH defocus1 and defocus2 are negative. Please confirm.
2. Our code is often forcing the defocus value to be negative with things like

   -1*abs(defocus1)

   . Is this the correct thing to do? Why?

1. non-absoluted defocus is used in apMatlab.py for ace correct command. I'm not sure if Ace is going to abs it or not.
2. For makestack phase flipping, it looks like it is possible to send a non-absoluted defocus value to EMAN. This stems from the fact that getBestCtfValueForImage() in ctfdb.py is not taking the abs() of the defocus. Again, not sure how EMAN handles this.

If none of these matter, then in theory there should not be a backward compatibility issue, however in practice the issue has been observed, see #2101.
Perhaps the issue is that there are a couple of places that are currently not but should be using abs()? Or is it incorrect to use abs() in these instances?

My guess is that some ctf correction program like EMAN's applyctf only takes negative values. ctfit gui program they wrote have the user slide in positive number but the parameter output is negative. Empirically, I noticed that our defocus parameter sent to EMAN applyctf is always negative and with no astigmation. Here is an example:

[CTF run]   method: ctffind | runname ctffindrun6
[CTF param] def1: 1.69e-06 | def2: 1.84e-06 | angle: -42.9 | ampcontr 0.07 | defratio 1.085
[CTF stats] conf_30-10: 0.744 | conf_5peak: 0.794 | res_0.8: 14.2A | res_0.5 14.2A
[CTF stats] conf: 0.794 | conf_d: 0.590
 ... phaseflipping entire micrograph with defocus -1.765 microns
EMAN: applyctf /ami/archive/12/leginon/09sep03a/rawdata/09sep03a_00024ma_0.mrc /ami/data00/appion/09sep03a/stacks/stack8/09sep03a_024ma_0-ctfcorrect.dwn.mrc parm=-1.765459,200,1,0.070,0,17.4,9,1.53,120,2.0,2.169850 setparm flipphase

example of negative estimation in database

[CTF run]   method: ctffind | runname ctffindrun4
[CTF param] def1: -1.68e-06 | def2: -1.84e-06 | angle: -43.1 | ampcontr 0.07 | defratio 1.094
[CTF stats] conf: None | conf_d: 0.495
 ... phaseflipping entire micrograph with defocus -1.76 microns
EMAN: applyctf /ami/archive/12/leginon/09sep03a/rawdata/09sep03a_00024ma_0.mrc /ami/data00/appion/09sep03atemp/stackstemp/stack10anchitest/09sep03a_024ma_0-ctfcorrect.dwn.mrc parm=-1.759523,200,1,0.070,0,17.4,9,1.53,120,2.0,2.169850 setparm flipphase

So, I think historically, ACE1 was positive, ACE2 was made to be negative, when CTFFIND was added to the pipeline it was positive, but flipped to conform to ACE2.

According to Ludtke's (http://dx.doi.org/10.1006/jmbi.2001.5133, page 7) 2001 paper, EMAN1 was changed to have underfocus be positive:

"In addition, we are now using the more common convention that the defocus is positive for underfocus."

So, I am surprised that it takes negative values.

Thanks for looking into this.

Anchi, the test results that you show above may not be accurate because we were printing out the last ctfvalues examined by the loop rather than the best values returned by the function. In my testing, I am finding that a command like this:

makestack2.py --single=start.hed --fromstackid=102 --invert --normalized --xmipp-normalize=4.5 --phaseflip --flip-type=emanpart --maskassess=manualrun4 --boxsize=240 --bin=2 --description="test positive defocus" --ctfmethod=ctffind --projectid=303 --preset=en --session=zz07jul25b --runname=stack104 --rundir=/ami/data00/appion/zz07jul25b/stacks/stack104 --no-rejects --no-wait --commit --limit=100 --continue --expid=8556 --jobtype=makestack2

[CTF run]   method: ctffind | runname ctffindrun43
[CTF param] def1: -1.16e-06 | def2: -1.21e-06 | angle: -74.3 | ampcontr 0.07 | defratio 1.041
[CTF stats] conf: None | conf_d: 0.494
---DEFOCUS in getDefocusAmpConstForImage----:  1.185594e-06
Traceback (most recent call last):
  File "/home/amber/workspace/myami/appion/bin/makestack2.py", line 1165, in <module>
    makeStack.run()
  File "/home/amber/workspace/myami/appion/appionlib/appionLoop2.py", line 70, in run
    results = self.loopProcessImage(imgdata)
  File "/home/amber/workspace/myami/appion/appionlib/appionLoop2.py", line 117, in loopProcessImage
    return self.processImage(imgdata)
  File "/home/amber/workspace/myami/appion/appionlib/apParticleExtractor.py", line 439, in processImage
    total_processed_particles = self.processParticles(imgdata,partdatas,shiftdata)
  File "/home/amber/workspace/myami/appion/bin/makestack2.py", line 76, in processParticles
    self.boxedpartdatas, self.imgstackfile, self.partmeantree = self.boxParticlesFromImage(imgdata,partdatas,shiftdata)
  File "/home/amber/workspace/myami/appion/bin/makestack2.py", line 324, in boxParticlesFromImage
    imgstackfile = self.phaseFlipParticles(imgdata, imgstackfile)
  File "/home/amber/workspace/myami/appion/bin/makestack2.py", line 468, in phaseFlipParticles
    self.checkDefocus(defocus, shortname)
  File "/home/amber/workspace/myami/appion/appionlib/apParticleExtractor.py", line 222, in checkDefocus
    apDisplay.printError("defocus is positive "+str(defocus)+" for image "+shortname)
  File "/home/amber/workspace/myami/appion/appionlib/apDisplay.py", line 57, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception: 
 *** FATAL ERROR ***
defocus is positive 1.185594 for image 07jul25b_013gr_05sq_013hl_03en

I'm planning to take the abs value of defocus in getDefocusAmpConstForImage() in apParticleExtractor.py (line 151) unless anyone objects.

Neil, Anchi is nearly positive that the command line for EMAN applyctf requires a negative defocus value.

Opps I mean it fails if the defocus value is negative in the database.

Neil, Scott, I've checked in a fix for eman applyctf, now there is one more place in the code where the defocus is passed on without taking the absolute value.

In apMatlab.py, runAceCorrect()

    ctfvalues, conf = ctfdb.getBestCtfValueForImage(imgdict)

    ctdimname = imgname
    ctdimpath = os.path.join(params['rundir'],ctdimname)
    print "Corrected Image written to " + ctdimpath

    #pdb.set_trace()
    acecorrectcommand=("ctfcorrect1('%s', '%s', '%.32f', '%.32f', '%f', '%f', '%f');" % \
        (imgpath, ctdimpath, ctfvalues['defocus1'], ctfvalues['defocus2'], ctfvalues['angle_astigmatism'], voltage, apix))

I know there is some confusion over ace1 right now, but can anyone tell me what sign this command SHOULD have for underfocus images. Or will Ace just use an absolute value when it runs making the sign irrelevant?

I am pretty sure the above code is NOT used in the pipeline. We do not (nor have we ever consistently) use ACE1 to do image correction.

We should trace where and if the runAceCorrect() command is being called and probably remove it.

Ok, I'll track it down.