Appion

Neil, you are probably right. I got a reverse-ordered defocus1 and defocus2 in Appion database value when I ran ace2 and ctffind on the same image that has a small positive angle of astigmatism. Both method got the same angle but opposite defocus1/defocus2.

When ctffind is committed, the angle_astigmatism is -float(ctffind_output) and defocus1 and defocus2 are only multiply by -1, implying that Appion definition and ctffind definition are only different by handness. Was that the original design? Does anyone know what the definition was meant to be?

I am unifying the definition of the CTF across several programs and papers. After reviewing 51 papers that contain a CTF equation (no joke, I have lots of undergrads), I came up with a consensus set of equations and conventions, which matches the definition by ctffind. I put my preliminary results on wikipedia:

http://en.wikipedia.org/wiki/Contrast_transfer_function

Right now, I am working on figuring out what is going into the database and matching it with the convention.

I was reviewing the ACE2 code in ace2.m and Ellipse.m, but not being an objective-C programmer it was hard to follow. It appears the astig angle comes form the Ellipse class in Ellipse.m, but where it comes from was hard for me to track down. The defocus variable are kept in the main ace2 part. My guess is that it is comes from the trig functions here, lines 284-288 in Ellipse.m. Like it should be an atan2() function instead of just atan(). I can have my students do the math and test it out.

source:trunk/programs/ace2/Ellipse.m#L284

HI, Neil,

That's a lot of paper and math. If you can, check against the definition for the exchange format. I've confirmed that what they meant by contour-clockwise means + direction on x-axis towards + direction on y-axis and that defocusU is the value at ellipse major axis, and therefore, the defocus closer to zero.

Forgot the link.

http://i2pc.cnb.csic.es/emx/CargarDictionaryFormat.htm?type=LabelList

I like that you used the arcsin simplification, that helped me a lot to understand the influence of the amplitude contrast. Your final derivation virtually the same as the one I ended up going with...

I also have a derivation when I was still trying to create a least-squares optimizable equation for fitting the CTF. My conclusion was that CTF fitting would always require multiple initializations (for each possible integer assignment of the first detected zero), but optimizing by least squares within each possible local minima would still be more robust then the current non-linear fitting ACE2 uses.

IIRC, I checked the ACE2 results by making sure the generated CTF's correctly matched the 2D simulated ones... It possible they are both using different conventions and doing the translations both ways. It's also possible that ACE2 is using one convention internally, but outputs a different one, likely because of trying to match something else's convention, most likely ACE1. In any case, shouldn't swapping the defoci be the easiest way to switch between either angle convention?

Oh, except for conventions that are based on the angle being dependent on which defocus angle is larger/smaller, what a pain.

The x-axis with counter-clockwise positive, is used by all papers. But the kicker, as Craig pointed out, the direction also depends upon which defocus is greater than the other.

The Xmipp says defocusU is the major axis (or longer axis) implying less defocus, which move the zero further from the center. So, |defocusU| < |defocusV| or |defocus1| < |defocus2|, which is what we decided on. Thank goodness. Ugh, it is not like you can post a random website somewhere and say this is the standard.

They need some sort of definition for the amplitude contrast, pretty much everyone (EMAN1/2, ACE1/2, Ctffind, maybe SPARX) uses the same definition, expect SPIDER and Xmipp.

I wasn't aware that amplitude contrast wasn't essentially the same by all definitions. Don't they all work by phaseshifting the CTF by up to 90º?

These are the three major definitions for the amplitude contrast that we have found:

• Common: sqrt(1-A^2)*sin() + A*cos()
• Xmipp: 1*sin() + A*cos()
• SPIDER : (1-A)*sin() + A*cos()

I mean they are similar, but not insignificant in the location of the first few peaks.

crikey, what are the objective basis for the differences? I always thought it was somewhat odd how far ACE2 would fudge the amplitude contrast even when hitting 7+ zeros. I always assumed perhaps it was partially compensating for errors in other parameters, but maybe the conventional ctf formulation isn't quite as accurate as the others? Dunno.

We switched the sign of the ACE2 astigmatism to match what was being output from CTFFIND.
Now the new version of CTFFIND flips its angle of astigmatism to match the MRC processing package. So this is going to be a real mess.
http://emlab.rose2.brandeis.edu/ctf

Gabe, to my knowledge, the new version of Ctffind only flips the tilt angle NOT the astig angle.

The only reason for the amp contrast difference is convention and historical reasons. Some people like the look of the formula in one particular way. Richard Henderson and Nigel Unwin were the first to use the sqrt(1-A^2), Pawel and Frank liked the (1-A). It has really little effect on the fitting of the data other than being slightly off.

Ah yes. Neil you're totally right, I misread that. We should probably start keeping track of the CTFFind version that was used for ctf estimation, so that the per-particle CTF correction is performed correctly.

The exchange format convention is here:

http://i2pc.cnb.csic.es/emx/CargarDictionaryFormat.htm?type=Convention

Looks like it is the common Amplitude contrast definition.

have you checked the correction on an actual image rather than just verifying that the numbers are consistent? I used to apply an astigmatic CTF to a synthetic image, even a photograph I knew well, then estimate, correct and check the final image to make sure it was as close to the original as possible. Like I mentioned, I'm still not sure if it isn't doing conversions internally that make it appear as if the numbers are inconsistent. What was the final fix you implemented?

Now ace2 provides the same astig angle as ctffind.

I applied a CTF to three images with ace2correct at 0, 35, and 70 degree angles. Estimated the angles with ace2.exe and then applied a 2nd CTF to them using the *.ctf.txt file using ace2correct and they overlapped. There was an error in the way ace2correct reads ace2's angle, so I made the change to ace2correct in r16778.

Here is my new favorite CTF test image:

http://lsbr.niams.nih.gov/bsoft/howto/bsoft_ctf_fan.html

In my test on a Leginon collected image, whose angle_astig is in 0<a<45, the ace2 results are identical as before after your change, as expected. However, since its database value has -5.5 e-7 in defocus1 and -3.67 in defocus2, this angle seems to be read from y-axis to first-ellipse axis and therefore different from what ctffind gives in Appion Database. The latter is read from x-axis to first-ellipse axis. Neil's change in ace2correct may make ace2 result right, it will make ctffind result wrong. Therefore,
here is again my general question for all of you who worked on this part before: What is the definition of defocus1, defocus2, angle_astigmatism in Appion database?

Note about the image I post: It came from an image during coma-free alignment, so it has a strong beam-tilt. Ace2 would not fit the image perfectly but should pick up the shape of the astigmatism o.k. even if it comes from coma-effect.

Oh, crap you are right. I fixed it wrong. I want it such that |def1| < |def2|. I think the angle will be right if you swap the defocus values.

I cannot get back to it until Friday; I am closing on a new house tomorrow!

Also, after I fix it, I'll have my students run new ACE2 and CTFFIND on a big dataset and plot the results.

Oops, r16782 goes with this too.

Alright, I created new ace2 .ctf.txt file with standard units. Ran new ace2correct on test image, estimated with both old ace2 and new ace2, and corrected with new ace2correct. In each case, the correction worked (unless of course the rare case where confidence value was low from ace2)

Can you summarize: How broken was it before? Astigmatic images? Only under certain angles, or when one defocus was larger than the other? Or where the fixes just to consolidate on a standardized format?

Craig, the fix is r16776. You used atan(y/x) instead of atan2(y,x), which basically means that the output of ACE2 was the astig angle mod 90 (ang%90) and did not cover the full 180 range. So, an astig angle of 90 and 0 degrees would get the same angle of 0 degrees from ACE2.

As you can see from my mysql queries above the ACE2 was limited between -45 < ang < 45 and the defocus angle never switch position.

http://en.wikipedia.org/wiki/Atan2

Cool, how embarrassing. I was just lecturing Lauren a couple months back on how one should only ever use atan2. Hubris....

As a far as I am concerned, this bug is fixed. Now that question is, what do we do with all of the old ACE2 data?

I feel for you Craig. Last semester, I graded a question on a biochemistry exam wrong that most of the class got right.

You mean in the database? I'd say leave it as is. It's historical. If someone is concerned about an old reconstruction they can always reprocess the dataset. I'd be super surprised if the fix made a noticeable difference anyways.... the high res stuff is done in ice, and ice has poor contrast which leads too more tenuous CTF estimations anyways. To be affected by this you'd need a very high resolution dataset, with lots of contrast, with moderately high astigmatism, that happened to have an astigmatism angle in the bad angular range.

Updated makestake2 for change parameter file, though it would have worked before, because ace2correct would detect the old version of the file and read radians instead of degrees.

I updated ace2.exe & ace2correct.exe today and it appears to be applying the inverted defocus. Are the defocus values supposed to be positive in the mrc.ctf.txt files now?
Final Defocus (m,m,deg): 1.854514e-06 1.872291e-06 -19.750656
My ACE2-corrected particles all look like an overfocus CTF has been applied. I compiled with gobjc-4.4 on Ubuntu 10.04.4

Yes, I flipped the sign to conform to EMX standards, ace2correct is supposed to read the file ctf.txt like this:

If units are included, defocus should be positive and degrees, if not = negative and radians:

Final Defocus (m,m,deg): 1.854514e-06 1.872291e-06 -19.750656

Final Defocus: -1.854514e-06 -1.872291e-06 -0.344713

I ran a several tests in the program to check to make sure the program we reading the values, and then I also setup a synthetic test image set to test the entire estimation and correction process, with both new ace2estimate and old ace2estimate. Everything passed my tests. Old ace2correct, should fail when given a new ace2estimate.

Gabe, would you mind sending me a couple of test images.

For some reason I can't upload the files to redmine, I'm sending them to your email.
I have a micrograph "image.mrc" & the output from the new ace2.exe "ctf.txt"
when I run the command:

ace2correct.exe -ctf ctf.txt -apix 4.62 -img image.mrc -out image.corrected.neg.mrc -wiener 0.1