Appion

Based on test data set of Anke's published 30S ribosomes 09oct22c, which has a CTF out to 5 Angstroms. I determined that the sign for the CTF in the visualization part was flipped (i.e., overfocus was underfocus and vice-verse). This was done earlier to match Scott's problem, which may have been overfocused.

So, as of r17144, I believe CTFFIND is making good estimates and those estimates match the CTF display.

Okay, so as of r17150. CTFFIND and ACE2 give the same results for the defocus and astig angle. Both method go to 5 Angstroms with perfect fits to Anke's data. The only change I made was to the confidence and resolution calculations as well as the corresponding images, so there were no changes to how the program runs and or to the four key parameters (def1, def2, angle_astig, and amp_contrast).

It also appears that SPIDER and EMAN phase flipping is working properly. But ACE2 phase flipping is broken, I will look at that now.

It looks like the confidence values for this ACE2 r17144 are much lower than for r17010. The average confidence for a run previously was ~0.9, now it's ~0.7
Do we need to change the default 0.8 confidence cutoff to something lower?

ACE2 was not working at r17144, try r17156, I believe it should all work now. I am doing final tests.

Okay, I am calling it fixed as of r17156. I ran CTFFIND estimation and all four corrections types (ACE2 wiener, ACE2 phaseflip, SPIDER phaseflip, EMAN phaseflip) and all 2 phaseflips gave exact same results, wiener filter looked good as well. Then in a separate session, I used ACE2 estimation and all four corrections types, same results. Both ACE2 and CTFFIND gave approx. same estimates for all CTF parameters and the data went out to 5 Angstroms. I ran a quick particle alignment on the wiener filter corrections and the result was a nice looking 30S ribosomal subunit.

I am so happy to have this working and I apologize to all of you for the agony to you it may have caused. Gabe, if it is still not working, I will buy you a green flash beer at the cryo-course.

Unfortunately ACE2 estimation is doing a worse job with r17158. Images that had a confidence of 0.9 before now are below 0.6
Maybe it's not compiling properly on our machines? I get this output when I compile:

[glander@caviale ace2]$ make
gcc -c -O3 -D'VERSION="svn revision 17158"' ctf.m -o ctf.o
ctf.m: In function ‘fitCTF’:
ctf.m:485: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
ctf.m: In function ‘estimateDefocus’:
ctf.m:732: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
ctf.m:736: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
ctf.m: In function ‘ctfNormalize’:
ctf.m:869: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
gcc -c -O3 -D'VERSION="svn revision 17158"' ace2.m -o ace2.o
ace2.m: In function ‘main’:
ace2.m:217: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
ace2.m:305: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
ace2.m: In function ‘createRadialAverage’:
ace2.m:804: warning: format ‘%s’ expects type ‘char *’, but argument 3 has type ‘int’
gcc -o ./ace2.exe -O3 -D'VERSION="svn revision 17158"' ace2.o cvtypes.o Array.o MRC.o PGM.o util.o Image.o Ellipse.o geometry.o getopts.o ctf.o -lfftw3 -lgsl -lgslcblas -lobjc -lm -static -fopenmp
gcc -c -O3 -D'VERSION="svn revision 17158"' ace2correct.m -o ace2correct.o
gcc -o ./ace2correct.exe -O3 -D'VERSION="svn revision 17158"' ace2correct.o cvtypes.o Array.o MRC.o PGM.o util.o Image.o Ellipse.o geometry.o getopts.o ctf.o -lfftw3 -lgsl -lgslcblas -lobjc -lm -static -fopenmp
gcc -o testf.exe -O3 -D'VERSION="svn revision 17158"' testf.o cvtypes.o Array.o MRC.o PGM.o util.o Image.o Ellipse.o geometry.o getopts.o ctf.o -lfftw3 -lgsl -lgslcblas -lobjc -lm -static -fopenmp

Here's the ACE2 output:
Program: /archive/glander/myami/appion/bin/ace2.exe in source ace2.m, svn revision 17158,
which was compiled on Sep 29 2012 at 14:17:13.

Reading Image /leginon4/taylor/12sep26a/rawdata/12sep26a_00017gr_00002sq_v01_00002hl_v01_00002en.mrc: DONE in 0.28 seconds
Processing image with 4096 4096 pixels
APIX: 1.154504 KV: 120.000000 CS: 2.200000
Binning image by 2 pixels... DONE in 0.14 seconds
Blurring 100 edge pixels... DONE in 0.05 seconds
Generating power spectrum... DONE in 15.74 seconds
post-binning by 2...
Blurring edges by 10.0 pixels...
Finding edges for ellipse fitting... DONE in 0.44 seconds (16192 edges)
Using RANSAC to find ellipse parameters...
RANSAC GAVE UP... Could not find a model better than 538.000000 of 16192 (0.03)
DONE in 5.41 seconds
Creating 1D radial average...
Calculated CTF signal for ellipse: 2.298299e+06, circle: 2.169564e+06, ratio: 1.059337e+00
Using Elliptical Average
DONE in 0.14 seconds
Estimating initial defocus...
Found 125 125 extrema for estimate
Using limits 47.288497 236.442487 for estimation
Estimated Defocus is: -1.802562e-06
Fitting noise and envelope...
Iterations: 3706
Noise: -1.47e-01 3.20e-01 5.41e-06 -1.45e+02 6.31e-01 1.93e-01 1.18e-01 6.33e-06 5.73e+02
DONE in 2.17 seconds
Fine-tuning defocus...Normalized....
Minimizer used 1587 iterations (0.01%)
Writing CTF Fitting results to: 12sep26a_00017gr_00002sq_v01_00002hl_v01_00002en.mrc.norm.txt

Calculating confidence score from 47.288497 to 236.442487
            DONE, Total Time: 24.65
    Final Params:
    Defocus: 1.78894 1.89288 -83.29301
    Amplitude Contrast: 50.00%
    Voltage: 120 kV
    Spherical Aberration: 2.20mm
    Angstroms per pixel: 1.15
    Confidence Score: 0.59
 ... ace2 completed in 25.18 sec
 ... reading log file 12sep26a_00017gr_00002sq_v01_00002hl_v01_00002en.mrc.ctf.txt
 ... Reading new ACE2 defocus
 ... ============
 ... Defocus: 1.789 x 1.893 um (-5.65 percent astigmatism)
 ... Angle astigmatism: -83.29 degrees
 ... Amplitude contrast: 50.00 percent
Final confidence: 0.589 < 0.921

Hi Gabe, I did not need to make any changes to ace2.exe today, only ace2correct.exe, I just recompiled them together, so there should be no difference from before in the estimation.

Hi Gabe, I see your RANSAC is failing. That is even before any CTF assumptions take place, maybe you need to adjust your blur and edge threshold. Try maybe 8 and 0.01

Can we have a summary of what is supposed to be working now and what is not. From the redmine trail I thought I saw Neil say that everything was working and Gabe say that it was not.
Can we have a summary of what works for estimation (ACE1, ACE2, CTFFIND, CTFTILT, anything else?) and what works for correction (EMAN by stack, EMAN by whole image, what else?)
I do not think that ACE2 correction ever worked right and I think we should take that one out of the pipeline - it for sure should not be the default (as I think it once was).
Can all concerned parties please test the release Neil just filed and get back to us ASAP as to what they think is still an issue.
We are determined to resolve this issue in the next few days.

All things should be working, though I never tested ACE1. I tested all possible pairs of ACE2/CTFFIND estimation -> ACE2/EMAN/SPIDER correction on Anke's 30S data. If others can test, please report back.

Michael Cianfrocco and I spent this morning checking ACE2 & CTFfind estimation and then using all the CTF correction options. We compared the phases of all the outputs and everything seems like it's behaving as well as before the overhaul.
No Matlab here, so I don't know about ACE1.

ACE1 still has problems. I think it's putting the correct defocus in the database, but it's not simulating the CTF properly and thus not calculating the confidence correctly.

Thanks for checking things Gabe. Scott, I am sorry about ACE1, I will have to look into it more. The current problems with the CTF occurred when I tried to make everything more compatible with ACE1.

Well, in the meantime, the raw confidence scores from ACE1 are being inserted into the database and when you choose to phaseflip, it still only uses the program-based (not Appion-based) confidence values. So, everything should function normally until I can get this worked out.

I am teaching 4 classes this semester (3 new preps), but next semester I have all repeat courses, so I will have more time to work on this.