Bug #3494
open
difficult image for new ctfdisplay.py to do noise subtraction
Added by Anchi Cheng over 9 years ago.
Updated almost 8 years ago.
Affected Version:
Appion/Leginon 3.2
Description
This image attached has unusually high defocus and a bad drift correction.
Cs=2.0
pixelsize=1.47 A
ctfestimate.py --runname=ctffindrunanchitest3 --rundir=/emg/data00/appion/acheng/15aug31a/ctf/ctffindrunanchitest3 --commit --projectid=1 --session=15aug31a --no-rejects --no-wait --continue --ampcarbon=0.15 --ampice=0.07 --fieldsize=512 --medium=ice --bin=2 --resmin=100 --resmax=10 --defstep=1000 --numstep=25 --dast=500 --expid=626 --jobtype=ctfestimate --mrclist=15aug31a_example1.mrc
Final params: def1: 5.81e-06 | def2: 5.93e-06 | angle: -52.5 | defratio 1.02
... Defocus Astig Percent Diff 1.02 -- 5.815e-06, 5.934e-06
Computing power spectra in 1024x1024 blocks...................................
... Computing median of power spectra series
... Compute PSD with fieldsize 1024 and 35 images complete in 20.99 sec
... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
... Number of available peaks is 250
... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
!!! WARNING: Not enough points (0) in section 3 to do background subtraction
Files
|
15aug31a_example1.mrc (54.3 MB)
15aug31a_example1.mrc |
|
Anchi Cheng, 08/31/2015 08:58 PM
|
|
|
16mar16b_05fcn_1-plots.png (532 KB)
16mar16b_05fcn_1-plots.png |
|
Neil Voss, 05/24/2016 10:25 AM
|
|
|
15aug31a_example1_1-plots.png (921 KB)
15aug31a_example1_1-plots.png |
|
Neil Voss, 05/24/2016 10:34 AM
|
|
|
17apr12a_00019en.mrc (64 MB)
17apr12a_00019en.mrc |
|
Anchi Cheng, 04/12/2017 09:02 PM
|
|
|
old_plot.png (357 KB)
old_plot.png |
|
Anchi Cheng, 04/12/2017 09:02 PM
|
|
|
new_plot.png (328 KB)
new_plot.png |
|
Anchi Cheng, 04/12/2017 09:02 PM
|
|
|
simulated_data.png (361 KB)
simulated_data.png |
|
Neil Voss, 04/13/2017 11:08 AM
|
|
|
17apr12a_neil_fit.png (208 KB)
17apr12a_neil_fit.png |
|
Neil Voss, 04/13/2017 11:08 AM
|
|
|
ctffind4_fit.png (242 KB)
ctffind4_fit.png |
moved data to excel |
Neil Voss, 04/13/2017 11:08 AM
|
|
|
Screen Shot 2017-04-13 at 9.09.57 AM.png (60.7 KB)
Screen Shot 2017-04-13 at 9.09.57 AM.png |
|
Neil Voss, 04/13/2017 11:09 AM
|
|
|
17apr12a_019en_1-pow_avrot.png (72.1 KB)
17apr12a_019en_1-pow_avrot.png |
|
Neil Voss, 04/13/2017 11:09 AM
|
|
|
hires_ctffind4_fit.png (211 KB)
hires_ctffind4_fit.png |
|
Neil Voss, 04/13/2017 11:19 AM
|
|
|
pixelsize1.1_2d.png (1.66 MB)
pixelsize1.1_2d.png |
|
Neil Voss, 04/13/2017 11:54 AM
|
|
|
pixelsize1.1_1d.png (274 KB)
pixelsize1.1_1d.png |
|
Neil Voss, 04/13/2017 11:54 AM
|
|
|
pixelsize1.1_ctffind4.png (166 KB)
pixelsize1.1_ctffind4.png |
|
Neil Voss, 04/13/2017 11:59 AM
|
|
- Related to Bug #3438: noise subtraction in ctfdisplay normalization error added
- Related to Bug #3942: Create a CTF test dataset added
Hi Anchi, Can I get the voltage for this micrograph, I want to include it in a test dataset.
- Priority changed from Normal to High
- Related to Feature #3950: Create CTF test dataset within Docker added
- Status changed from Assigned to In Test
- Assignee changed from Neil Voss to Anchi Cheng
this one works for me now.
close to and far from focus test images
This is still a problem. I am attaching a new example image 120 kV, 0.963 A/pixel, Cs 2.2 mm, Ctffind4 and Ctffind3 gave almost identical results. 0.79 um, 0.81 um, astig angle 36.3 and two 1d plots, from old code and new code.
Hmm, I see that in the new plot it fit the curve just fine, but the generated CTF signal did not match. I will download the image and test it, probably Friday.
Hi Neil,
I thin this needs to be fixed ASAP or everyone will just not use it anymore and that would be sad. It is too slow and not giving good results now and so the users will vote with their feet and move on to something else. In my experience it is very hard to get them back after they leave!
Let us know if you want to chat.
Bridget
I have not changed anything to it in months, so what happened? There is a slight difference in the defocus values and at that close to focus it is often enough to shift the CTF oscillations off at high resolution, I see 0.786/0.808 um in the new and 0.784/0.812 um in the old.
I understand the need to be quick, but the CTF fitting is very touchy, so I do not like to make a modification that works with one image and then breaks the rest. I just got the kids to bed, so I will do some initial tests now, but unfortunately, my time is mostly booked for the next 24 hours. I have a quiz and 5 hours of lecture to prepare for tomorrow.
OK, get those kids bedded down happily and don't fret too much! I am a big fan of your method and want to protect its use but it is not an emergency so do not lose any sleep over it or neglect those kids or that quiz. Let's just keep it on the high priority list and see what we can do to restore confidence....
B
never mind, I messed up the pixel size when I uploaded it
Something is wrong. According to the CTFFIND4 2d images it easily fits 7 Thon rings. Looking at my 2d images, it shows green for only 4 Thon rings, but the 2D image fits the 7 Thon rings despite them being red. When we look at the 1D graph it shows a different story.
The 1D graph and 2D graph are not in agreement. I am looking into this now.
I just tested the current code on Joachim Frank's simulated images from the CTF challenge and the resolution went out to 2 Angstroms (max) for all of the images.
Are you sure that the pixel size and spherical aberration is correct for the image you uploaded? Should I try this on some similar images. Based on the fit it appears the things are getting out of sync once the spherical aberration term kicks in, which leads me to question the correctness of the pixel size and spherical aberration.
Pixel Size is correct. Your CTF estimation is way off. For this, the nominal defocus was 0.8 um. resmax I used was 3 Å, and resmin was also lowered to 30 Å, At such low defocus, defstep of 1000 is also not reasonable, so I was using 200. This is the typical setting for Phase plate data collection even though this one does not use phase plate, so I am not making it estimate phase shift. Low defocus, high resolution estimation is common now.
By the way, the CTFFind4 I used was 4.1.5
What was the CTFFIND 4 resolution it was reporting. I ran several run at low defsteps.
Here is CTFFind4 output captured by Appion
Input image file name : 17apr12a_019en.mrc
Output diagnostic image file name : 17apr12a_019en-pow.mrc
Pixel size : 0.963
Acceleration voltage : 120.0
Spherical aberration : 2.2
Amplitude contrast : 0.15
Size of amplitude spectrum to compute : 1024
Minimum resolution : 30.0
Maximum resolution : 4.27
Minimum defocus : 2844.7
Maximum defocus : 12844.7
Defocus search step : 200.0
Do you know what astigmatism is present? : no
Slower, more exhaustive search? : no
Use a restraint on astigmatism? : yes
Expected (tolerated) astigmatism : 273.8
Find additional phase shift? : no
Do you want to set expert options? : no
Summary information for file 17apr12a_019en.mrc
Number of columns, rows, sections: 4096, 4096, 1
MRC data mode: 2
Bit depth: 32
Pixel size: 0.963 0.963 0.963
Bytes in symmetry header: 0
Working on micrograph 1 of 1
Estimated defocus values : 8082.36 , 7862.55 Angstroms
Estimated azimuth of astigmatism: -53.71 degrees
Score : 0.14389
Pixel size for fitting : 1.401 Angstroms
Thon rings with good fit up to : 4.5 Angstroms
Did not detect CTF aliasing
and my appion script
ctffind4.py --ampcontrast=0.07 --fieldsize=1024 --resmin=30 --resmax=3 --defstep=0.02 --numstep=25 --dast=0.05 --min_phase_shift=10 --max_phase_shift=170 --phase_search_step=10 --runname=ctffind4run3 --rundir=/data/appiondata/17apr12a/ctf/ctffind4run3 --preset=en --commit --projectid=18 --session=17apr12a --best-images --no-wait --continue --bestdb --expid=9311 --jobtype=ctfestimate
higher res ctffind4 fit. The shape of the curves is concerning to me, it gets very saw tooth. I have to stop right now, but I will look further.
The plot thickens. I changed the pixel size of the image to 1.1 A (instead of 0.96 A). Ctffind4 still fits the data well:
Estimated defocus values : 10379.04 , 10092.08 Angstroms
Estimated azimuth of astigmatism: -47.05 degrees
Score : .08995
Thon rings with good fit up to : 4.7 Angstroms
[CTF run] method: unknown | runname ctffind4run2
[CTF param] def1: 1.009 um | def2: 1.038 um | angle: 43.0 | ampcontr 0.15 | defratio 1.028
[CTF param] additional phase shift: 0.0 degrees
[CTF stats] conf_30-10: 0.954 | conf_5peak: 0.967 | res_0.8: 5.1A | res_0.5 4.8A
and now my system in agreement with the high resolution. The same effect can be achieved by adjusting the Cs value. So, I am not sure what to think.
Also available in: Atom
PDF
