Bug #3494
opendifficult image for new ctfdisplay.py to do noise subtraction
Added by Anchi Cheng about 9 years ago. Updated over 7 years ago.
0%
Description
This image attached has unusually high defocus and a bad drift correction.
Cs=2.0
pixelsize=1.47 A
ctfestimate.py --runname=ctffindrunanchitest3 --rundir=/emg/data00/appion/acheng/15aug31a/ctf/ctffindrunanchitest3 --commit --projectid=1 --session=15aug31a --no-rejects --no-wait --continue --ampcarbon=0.15 --ampice=0.07 --fieldsize=512 --medium=ice --bin=2 --resmin=100 --resmax=10 --defstep=1000 --numstep=25 --dast=500 --expid=626 --jobtype=ctfestimate --mrclist=15aug31a_example1.mrc
Final params: def1: 5.81e-06 | def2: 5.93e-06 | angle: -52.5 | defratio 1.02 ... Defocus Astig Percent Diff 1.02 -- 5.815e-06, 5.934e-06 Computing power spectra in 1024x1024 blocks................................... ... Computing median of power spectra series ... Compute PSD with fieldsize 1024 and 35 images complete in 20.99 sec ... Preform a rotational average and remove spikes... PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE ... Number of available peaks is 250 ... Determine and subtract noise model PART 2: BACKGROUND NOISE SUBTRACTION !!! WARNING: Not enough points (0) in section 3 to do background subtraction
Files
15aug31a_example1.mrc (54.3 MB) 15aug31a_example1.mrc | Anchi Cheng, 08/31/2015 08:58 PM | ||
16mar16b_05fcn_1-plots.png (532 KB) 16mar16b_05fcn_1-plots.png | Neil Voss, 05/24/2016 10:25 AM | ||
15aug31a_example1_1-plots.png (921 KB) 15aug31a_example1_1-plots.png | Neil Voss, 05/24/2016 10:34 AM | ||
17apr12a_00019en.mrc (64 MB) 17apr12a_00019en.mrc | Anchi Cheng, 04/12/2017 09:02 PM | ||
old_plot.png (357 KB) old_plot.png | Anchi Cheng, 04/12/2017 09:02 PM | ||
new_plot.png (328 KB) new_plot.png | Anchi Cheng, 04/12/2017 09:02 PM | ||
simulated_data.png (361 KB) simulated_data.png | Neil Voss, 04/13/2017 11:08 AM | ||
17apr12a_neil_fit.png (208 KB) 17apr12a_neil_fit.png | Neil Voss, 04/13/2017 11:08 AM | ||
ctffind4_fit.png (242 KB) ctffind4_fit.png | moved data to excel | Neil Voss, 04/13/2017 11:08 AM | |
Screen Shot 2017-04-13 at 9.09.57 AM.png (60.7 KB) Screen Shot 2017-04-13 at 9.09.57 AM.png | Neil Voss, 04/13/2017 11:09 AM | ||
17apr12a_019en_1-pow_avrot.png (72.1 KB) 17apr12a_019en_1-pow_avrot.png | Neil Voss, 04/13/2017 11:09 AM | ||
hires_ctffind4_fit.png (211 KB) hires_ctffind4_fit.png | Neil Voss, 04/13/2017 11:19 AM | ||
pixelsize1.1_2d.png (1.66 MB) pixelsize1.1_2d.png | Neil Voss, 04/13/2017 11:54 AM | ||
pixelsize1.1_1d.png (274 KB) pixelsize1.1_1d.png | Neil Voss, 04/13/2017 11:54 AM | ||
pixelsize1.1_ctffind4.png (166 KB) pixelsize1.1_ctffind4.png | Neil Voss, 04/13/2017 11:59 AM |
Updated by Neil Voss about 9 years ago
- Related to Bug #3438: noise subtraction in ctfdisplay normalization error added
Updated by Neil Voss almost 9 years ago
- Related to Bug #3942: Create a CTF test dataset added
Updated by Neil Voss almost 9 years ago
Hi Anchi, Can I get the voltage for this micrograph, I want to include it in a test dataset.
Updated by Neil Voss almost 9 years ago
- Related to Feature #3950: Create CTF test dataset within Docker added
Updated by Neil Voss over 8 years ago
- Status changed from Assigned to In Test
- Assignee changed from Neil Voss to Anchi Cheng
this one works for me now.
Updated by Neil Voss over 8 years ago
- File 16mar16b_05fcn_1-plots.png 16mar16b_05fcn_1-plots.png added
- File 15aug31a_example1_1-plots.png 15aug31a_example1_1-plots.png added
close to and far from focus test images
Updated by Anchi Cheng over 7 years ago
- File 17apr12a_00019en.mrc 17apr12a_00019en.mrc added
- File old_plot.png old_plot.png added
- File new_plot.png new_plot.png added
- Assignee changed from Anchi Cheng to Neil Voss
This is still a problem. I am attaching a new example image 120 kV, 0.963 A/pixel, Cs 2.2 mm, Ctffind4 and Ctffind3 gave almost identical results. 0.79 um, 0.81 um, astig angle 36.3 and two 1d plots, from old code and new code.
Updated by Neil Voss over 7 years ago
Hmm, I see that in the new plot it fit the curve just fine, but the generated CTF signal did not match. I will download the image and test it, probably Friday.
Updated by Bridget Carragher over 7 years ago
Hi Neil,
I thin this needs to be fixed ASAP or everyone will just not use it anymore and that would be sad. It is too slow and not giving good results now and so the users will vote with their feet and move on to something else. In my experience it is very hard to get them back after they leave!
Let us know if you want to chat.
Bridget
Updated by Neil Voss over 7 years ago
I have not changed anything to it in months, so what happened? There is a slight difference in the defocus values and at that close to focus it is often enough to shift the CTF oscillations off at high resolution, I see 0.786/0.808 um in the new and 0.784/0.812 um in the old.
I understand the need to be quick, but the CTF fitting is very touchy, so I do not like to make a modification that works with one image and then breaks the rest. I just got the kids to bed, so I will do some initial tests now, but unfortunately, my time is mostly booked for the next 24 hours. I have a quiz and 5 hours of lecture to prepare for tomorrow.
Updated by Bridget Carragher over 7 years ago
OK, get those kids bedded down happily and don't fret too much! I am a big fan of your method and want to protect its use but it is not an emergency so do not lose any sleep over it or neglect those kids or that quiz. Let's just keep it on the high priority list and see what we can do to restore confidence....
B
Updated by Neil Voss over 7 years ago
never mind, I messed up the pixel size when I uploaded it
Updated by Neil Voss over 7 years ago
Something is wrong. According to the CTFFIND4 2d images it easily fits 7 Thon rings. Looking at my 2d images, it shows green for only 4 Thon rings, but the 2D image fits the 7 Thon rings despite them being red. When we look at the 1D graph it shows a different story.
The 1D graph and 2D graph are not in agreement. I am looking into this now.
Updated by Neil Voss over 7 years ago
I just tested the current code on Joachim Frank's simulated images from the CTF challenge and the resolution went out to 2 Angstroms (max) for all of the images.
Are you sure that the pixel size and spherical aberration is correct for the image you uploaded? Should I try this on some similar images. Based on the fit it appears the things are getting out of sync once the spherical aberration term kicks in, which leads me to question the correctness of the pixel size and spherical aberration.
Updated by Anchi Cheng over 7 years ago
Pixel Size is correct. Your CTF estimation is way off. For this, the nominal defocus was 0.8 um. resmax I used was 3 Å, and resmin was also lowered to 30 Å, At such low defocus, defstep of 1000 is also not reasonable, so I was using 200. This is the typical setting for Phase plate data collection even though this one does not use phase plate, so I am not making it estimate phase shift. Low defocus, high resolution estimation is common now.
By the way, the CTFFind4 I used was 4.1.5
Updated by Neil Voss over 7 years ago
What was the CTFFIND 4 resolution it was reporting. I ran several run at low defsteps.
Updated by Anchi Cheng over 7 years ago
Here is CTFFind4 output captured by Appion
Input image file name : 17apr12a_019en.mrc Output diagnostic image file name : 17apr12a_019en-pow.mrc Pixel size : 0.963 Acceleration voltage : 120.0 Spherical aberration : 2.2 Amplitude contrast : 0.15 Size of amplitude spectrum to compute : 1024 Minimum resolution : 30.0 Maximum resolution : 4.27 Minimum defocus : 2844.7 Maximum defocus : 12844.7 Defocus search step : 200.0 Do you know what astigmatism is present? : no Slower, more exhaustive search? : no Use a restraint on astigmatism? : yes Expected (tolerated) astigmatism : 273.8 Find additional phase shift? : no Do you want to set expert options? : no Summary information for file 17apr12a_019en.mrc Number of columns, rows, sections: 4096, 4096, 1 MRC data mode: 2 Bit depth: 32 Pixel size: 0.963 0.963 0.963 Bytes in symmetry header: 0 Working on micrograph 1 of 1 Estimated defocus values : 8082.36 , 7862.55 Angstroms Estimated azimuth of astigmatism: -53.71 degrees Score : 0.14389 Pixel size for fitting : 1.401 Angstroms Thon rings with good fit up to : 4.5 Angstroms Did not detect CTF aliasing
Updated by Anchi Cheng over 7 years ago
and my appion script
ctffind4.py --ampcontrast=0.07 --fieldsize=1024 --resmin=30 --resmax=3 --defstep=0.02 --numstep=25 --dast=0.05 --min_phase_shift=10 --max_phase_shift=170 --phase_search_step=10 --runname=ctffind4run3 --rundir=/data/appiondata/17apr12a/ctf/ctffind4run3 --preset=en --commit --projectid=18 --session=17apr12a --best-images --no-wait --continue --bestdb --expid=9311 --jobtype=ctfestimate
Updated by Neil Voss over 7 years ago
- File simulated_data.png simulated_data.png added
- File 17apr12a_neil_fit.png 17apr12a_neil_fit.png added
- File ctffind4_fit.png ctffind4_fit.png added
- File Screen Shot 2017-04-13 at 9.09.57 AM.png Screen Shot 2017-04-13 at 9.09.57 AM.png added
my images
Updated by Neil Voss over 7 years ago
Updated by Neil Voss over 7 years ago
- File hires_ctffind4_fit.png hires_ctffind4_fit.png added
higher res ctffind4 fit. The shape of the curves is concerning to me, it gets very saw tooth. I have to stop right now, but I will look further.
Updated by Neil Voss over 7 years ago
- File pixelsize1.1_2d.png pixelsize1.1_2d.png added
- File pixelsize1.1_1d.png pixelsize1.1_1d.png added
The plot thickens. I changed the pixel size of the image to 1.1 A (instead of 0.96 A). Ctffind4 still fits the data well:
Estimated defocus values : 10379.04 , 10092.08 Angstroms Estimated azimuth of astigmatism: -47.05 degrees Score : .08995 Thon rings with good fit up to : 4.7 Angstroms
[CTF run] method: unknown | runname ctffind4run2 [CTF param] def1: 1.009 um | def2: 1.038 um | angle: 43.0 | ampcontr 0.15 | defratio 1.028 [CTF param] additional phase shift: 0.0 degrees [CTF stats] conf_30-10: 0.954 | conf_5peak: 0.967 | res_0.8: 5.1A | res_0.5 4.8A
and now my system in agreement with the high resolution. The same effect can be achieved by adjusting the Cs value. So, I am not sure what to think.