Project

General

Profile

These attachments cannot be bulk downloaded because the total file size exceeds the maximum allowed size (100 MB)
Actions

Bug #3494

open

difficult image for new ctfdisplay.py to do noise subtraction

Added by Anchi Cheng about 9 years ago. Updated over 7 years ago.

Status:
In Test
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
08/31/2015
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 3.2
Show in known bugs:
No
Workaround:

Description

This image attached has unusually high defocus and a bad drift correction.
Cs=2.0
pixelsize=1.47 A

ctfestimate.py --runname=ctffindrunanchitest3 --rundir=/emg/data00/appion/acheng/15aug31a/ctf/ctffindrunanchitest3 --commit --projectid=1 --session=15aug31a --no-rejects --no-wait --continue --ampcarbon=0.15 --ampice=0.07 --fieldsize=512 --medium=ice --bin=2 --resmin=100 --resmax=10 --defstep=1000 --numstep=25 --dast=500 --expid=626 --jobtype=ctfestimate --mrclist=15aug31a_example1.mrc

Final params: def1: 5.81e-06 | def2: 5.93e-06 | angle: -52.5 | defratio 1.02
 ... Defocus Astig Percent Diff 1.02 -- 5.815e-06, 5.934e-06
Computing power spectra in 1024x1024 blocks...................................
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 1024 and 35 images complete in 20.99 sec
 ... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 250
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
!!! WARNING: Not enough points (0) in section 3 to do background subtraction

Files

15aug31a_example1.mrc (54.3 MB) 15aug31a_example1.mrc Anchi Cheng, 08/31/2015 08:58 PM
16mar16b_05fcn_1-plots.png (532 KB) 16mar16b_05fcn_1-plots.png Neil Voss, 05/24/2016 10:25 AM
15aug31a_example1_1-plots.png (921 KB) 15aug31a_example1_1-plots.png Neil Voss, 05/24/2016 10:34 AM
17apr12a_00019en.mrc (64 MB) 17apr12a_00019en.mrc Anchi Cheng, 04/12/2017 09:02 PM
old_plot.png (357 KB) old_plot.png Anchi Cheng, 04/12/2017 09:02 PM
new_plot.png (328 KB) new_plot.png Anchi Cheng, 04/12/2017 09:02 PM
simulated_data.png (361 KB) simulated_data.png Neil Voss, 04/13/2017 11:08 AM
17apr12a_neil_fit.png (208 KB) 17apr12a_neil_fit.png Neil Voss, 04/13/2017 11:08 AM
ctffind4_fit.png (242 KB) ctffind4_fit.png moved data to excel Neil Voss, 04/13/2017 11:08 AM
Screen Shot 2017-04-13 at 9.09.57 AM.png (60.7 KB) Screen Shot 2017-04-13 at 9.09.57 AM.png Neil Voss, 04/13/2017 11:09 AM
17apr12a_019en_1-pow_avrot.png (72.1 KB) 17apr12a_019en_1-pow_avrot.png Neil Voss, 04/13/2017 11:09 AM
hires_ctffind4_fit.png (211 KB) hires_ctffind4_fit.png Neil Voss, 04/13/2017 11:19 AM
pixelsize1.1_2d.png (1.66 MB) pixelsize1.1_2d.png Neil Voss, 04/13/2017 11:54 AM
pixelsize1.1_1d.png (274 KB) pixelsize1.1_1d.png Neil Voss, 04/13/2017 11:54 AM
pixelsize1.1_ctffind4.png (166 KB) pixelsize1.1_ctffind4.png Neil Voss, 04/13/2017 11:59 AM

Related issues 3 (1 open2 closed)

Related to Appion - Bug #3438: noise subtraction in ctfdisplay normalization errorClosedGabriel Lander08/26/2015

Actions
Related to Appion - Bug #3942: Create a CTF test datasetClosedNeil Voss02/10/2016

Actions
Related to Appion - Feature #3950: Create CTF test dataset within DockerAssignedNeil Voss02/15/2016

Actions
Actions #1

Updated by Neil Voss about 9 years ago

  • Related to Bug #3438: noise subtraction in ctfdisplay normalization error added
Actions #2

Updated by Neil Voss over 8 years ago

  • Related to Bug #3942: Create a CTF test dataset added
Actions #3

Updated by Neil Voss over 8 years ago

Hi Anchi, Can I get the voltage for this micrograph, I want to include it in a test dataset.

Actions #4

Updated by Neil Voss over 8 years ago

  • Priority changed from Normal to High
Actions #5

Updated by Neil Voss over 8 years ago

  • Related to Feature #3950: Create CTF test dataset within Docker added
Actions #6

Updated by Anchi Cheng over 8 years ago

200 kV, Cs = 2.0 mm

Actions #7

Updated by Neil Voss over 8 years ago

  • Status changed from Assigned to In Test
  • Assignee changed from Neil Voss to Anchi Cheng

this one works for me now.

Updated by Anchi Cheng over 7 years ago

This is still a problem. I am attaching a new example image 120 kV, 0.963 A/pixel, Cs 2.2 mm, Ctffind4 and Ctffind3 gave almost identical results. 0.79 um, 0.81 um, astig angle 36.3 and two 1d plots, from old code and new code.

Actions #10

Updated by Neil Voss over 7 years ago

Hmm, I see that in the new plot it fit the curve just fine, but the generated CTF signal did not match. I will download the image and test it, probably Friday.

Actions #11

Updated by Bridget Carragher over 7 years ago

Hi Neil,
I thin this needs to be fixed ASAP or everyone will just not use it anymore and that would be sad. It is too slow and not giving good results now and so the users will vote with their feet and move on to something else. In my experience it is very hard to get them back after they leave!
Let us know if you want to chat.
Bridget

Actions #12

Updated by Neil Voss over 7 years ago

I have not changed anything to it in months, so what happened? There is a slight difference in the defocus values and at that close to focus it is often enough to shift the CTF oscillations off at high resolution, I see 0.786/0.808 um in the new and 0.784/0.812 um in the old.

I understand the need to be quick, but the CTF fitting is very touchy, so I do not like to make a modification that works with one image and then breaks the rest. I just got the kids to bed, so I will do some initial tests now, but unfortunately, my time is mostly booked for the next 24 hours. I have a quiz and 5 hours of lecture to prepare for tomorrow.

Actions #13

Updated by Bridget Carragher over 7 years ago

OK, get those kids bedded down happily and don't fret too much! I am a big fan of your method and want to protect its use but it is not an emergency so do not lose any sleep over it or neglect those kids or that quiz. Let's just keep it on the high priority list and see what we can do to restore confidence....
B

Actions #14

Updated by Neil Voss over 7 years ago

Ignore this comment

Actions #15

Updated by Neil Voss over 7 years ago

never mind, I messed up the pixel size when I uploaded it

Actions #16

Updated by Neil Voss over 7 years ago

Something is wrong. According to the CTFFIND4 2d images it easily fits 7 Thon rings. Looking at my 2d images, it shows green for only 4 Thon rings, but the 2D image fits the 7 Thon rings despite them being red. When we look at the 1D graph it shows a different story.

The 1D graph and 2D graph are not in agreement. I am looking into this now.

Actions #17

Updated by Neil Voss over 7 years ago

I just tested the current code on Joachim Frank's simulated images from the CTF challenge and the resolution went out to 2 Angstroms (max) for all of the images.

Are you sure that the pixel size and spherical aberration is correct for the image you uploaded? Should I try this on some similar images. Based on the fit it appears the things are getting out of sync once the spherical aberration term kicks in, which leads me to question the correctness of the pixel size and spherical aberration.

Actions #18

Updated by Anchi Cheng over 7 years ago

Pixel Size is correct. Your CTF estimation is way off. For this, the nominal defocus was 0.8 um. resmax I used was 3 Å, and resmin was also lowered to 30 Å, At such low defocus, defstep of 1000 is also not reasonable, so I was using 200. This is the typical setting for Phase plate data collection even though this one does not use phase plate, so I am not making it estimate phase shift. Low defocus, high resolution estimation is common now.

By the way, the CTFFind4 I used was 4.1.5

Actions #19

Updated by Neil Voss over 7 years ago

What was the CTFFIND 4 resolution it was reporting. I ran several run at low defsteps.

Actions #20

Updated by Anchi Cheng over 7 years ago

4.5 Å

Actions #21

Updated by Anchi Cheng over 7 years ago

Here is CTFFind4 output captured by Appion

Input image file name                              : 17apr12a_019en.mrc
Output diagnostic image file name                  : 17apr12a_019en-pow.mrc
Pixel size                                         : 0.963
Acceleration voltage                               : 120.0
Spherical aberration                               : 2.2
Amplitude contrast                                 : 0.15
Size of amplitude spectrum to compute              : 1024
Minimum resolution                                 : 30.0
Maximum resolution                                 : 4.27
Minimum defocus                                    : 2844.7
Maximum defocus                                    : 12844.7
Defocus search step                                : 200.0
Do you know what astigmatism is present?           : no
Slower, more exhaustive search?                    : no
Use a restraint on astigmatism?                    : yes
Expected (tolerated) astigmatism                   : 273.8
Find additional phase shift?                       : no
Do you want to set expert options?                 : no

Summary information for file 17apr12a_019en.mrc
Number of columns, rows, sections: 4096, 4096, 1
MRC data mode: 2
Bit depth: 32
Pixel size: 0.963 0.963 0.963
Bytes in symmetry header: 0

Working on micrograph 1 of 1
Estimated defocus values        : 8082.36 , 7862.55 Angstroms
Estimated azimuth of astigmatism: -53.71 degrees
Score                           : 0.14389
Pixel size for fitting          : 1.401 Angstroms
Thon rings with good fit up to  : 4.5 Angstroms
Did not detect CTF aliasing

Actions #22

Updated by Anchi Cheng over 7 years ago

and my appion script

ctffind4.py --ampcontrast=0.07 --fieldsize=1024 --resmin=30 --resmax=3 --defstep=0.02 --numstep=25 --dast=0.05 --min_phase_shift=10 --max_phase_shift=170 --phase_search_step=10 --runname=ctffind4run3 --rundir=/data/appiondata/17apr12a/ctf/ctffind4run3 --preset=en --commit --projectid=18 --session=17apr12a --best-images --no-wait --continue --bestdb --expid=9311 --jobtype=ctfestimate

Actions #25

Updated by Neil Voss over 7 years ago

higher res ctffind4 fit. The shape of the curves is concerning to me, it gets very saw tooth. I have to stop right now, but I will look further.

Updated by Neil Voss over 7 years ago

The plot thickens. I changed the pixel size of the image to 1.1 A (instead of 0.96 A). Ctffind4 still fits the data well:

Estimated defocus values        : 10379.04 , 10092.08 Angstroms
Estimated azimuth of astigmatism: -47.05 degrees
Score                           : .08995
Thon rings with good fit up to  : 4.7 Angstroms

[CTF run]   method: unknown | runname ctffind4run2
[CTF param] def1: 1.009 um | def2: 1.038 um | angle: 43.0 | ampcontr 0.15 | defratio 1.028
[CTF param]   additional phase shift: 0.0 degrees 
[CTF stats] conf_30-10: 0.954 | conf_5peak: 0.967 | res_0.8: 5.1A | res_0.5 4.8A

and now my system in agreement with the high resolution. The same effect can be achieved by adjusting the Cs value. So, I am not sure what to think.

Actions

Also available in: Atom PDF