Appion

Can you give a bit more info on how you're doing CTF estimation & correction?
I noticed this phenotype if you run CTFFind to estimate, and then use ACE2correct, since ACE2correct requires that defocus1 be smaller than defocus2. Are you sure you're running the latest version of myami? This should be fixed now with r16912

Yeah, it looks the fix from r16912 from Wednesday, but it is tough to say.

I've been using CTFFIND for the CTF estimation and Ace2 weiner filter to apply. I've just re-run makestack2.py on longboard/myami and longboard/betamyami and I"m still encountering the same problem. However, comparing the two stacks, it seems that the way the CTF estimation is applied is different as different particles have the compromised phenotype between the two stacks.

If I try to run EMAN2 phaseflip it crashes in both myami and betamyami. Here's the part of the logfile and the error from longboard/myami

Beginning Main Loop

Starting image 1 ( skip:316, remain:171 ) id:1874378, file: 12jun26y_12jun18b_c_030gr_059sq_09hl_03ed_st

... Pixel size: 1.424
... Found 103 particles
... CTF run info: runname='ctffindrun3', confidence=0.245, defocus=-2.060 um
... phaseflipping entire micrograph with defocus -2.06 microns
EMAN: applyctf /ami/data00/leginon/12jun26y/rawdata/12jun26y_12jun18b_c_00030gr_00059sq_v01
... .424000 setparm flipphase
... boxing 102 particles into temp file: /ami/data00/appion/12jun26y/stacks/stack20/12jun26y_12jun18
... gr_059sq_09hl_03ed_st.hed
... writing stack to disk from memory: /ami/data00/appion/12jun26y/stacks/stack20/12jun26y_12jun18b_c_030gr_059sq_09hl_03ed_st.hed

... wrote 102 particles to header file
... finished in 1.25 sec
... reading stack from disk into memory: 12jun26y_12jun18b_c_030gr_059sq_09hl_03ed_st.hed
... read 102 particles equaling 25.5 MB in size
... finished in 86.34 msec
missing acecutoff
None
Traceback (most recent call last):
File "/opt/myami-2.2/bin/makestack2.py", line 964, in ?
makeStack.run()
File "/opt/myami-2.2/lib/appionlib/appionLoop2.py", line 70, in run
results = self.loopProcessImage(imgdata)
File "/opt/myami-2.2/lib/appionlib/appionLoop2.py", line 117, in loopProcessImage
return self.processImage(imgdata)
File "/opt/myami-2.2/lib/appionlib/apParticleExtractor.py", line 386, in processImage
total_processed_particles = self.processParticles(imgdata,partdatas,shiftdata)
File "/opt/myami-2.2/bin/makestack2.py", line 68, in processParticles
self.boxedpartdatas, self.imgstackfile, self.partmeantree = self.boxParticlesFromImage(imgdata,partdatas,shiftdata)

• FATAL ERROR ***
  Standard deviation for particle in image 12jun26y_12jun18b_c_030gr_059sq_09hl_03ed_st

Hi Melody (and others),
In my opinion we should not in general be using the Wiener filter in Ace2. Can you just use the phase flip option in Ace2? The Wiener filter has issues in that it needs to basically divide by zero near the CTF zeroes so any misestimation of the ctf amplifies up the noise. I am betting that this is why some particles are looking bad and others not.
Bridget

Melody - could you attach a new screen shot of the particles from betamyami? In the first screenshot you attached some of the particles were clearly inverted, definitely not a result of ACE2 wienerfilter.
Should we switch the default option in makestack2 webform from ACE2 wienerfilter to phaseflip?

Does the myami (stable) website, use the new version of ACE2?

As Gabe pointed out, there was a bug (not fixed until Wednesday) that did a overfocus correction instead of underfocus.

You could switch to the "classic" EMAN phaseflip by particle. I would like to see the entirety of the applyctf command, maybe there is another overfocus problem creeping in.

"applyctf /ami/data00/leginon/12jun26y/rawdata/12jun26y_12jun18b_c_00030gr_00059sq_v01
... .424000 setparm flipphase"

Before I made any of these CTF changes, there was already a mixture of negative and positive (ACE1 vs. ACE2) defocus values in the database, so I made no changes to makestack2, because I thought it handled the nega/positive switching with absolute values. Maybe this was wrong. I mean the getBestDefocus returns -abs(defocus), so I am lost.

I don't mean to distract from Melody's issue, but the question of storing positive vs. negative defocus values had also come up in the cubicle discussions. Is there a reason why we are now displaying only positive values?

Hi Melody,
you shouldn't have to rerun CTFfind, the fixes in the last revision should work. Could you also quickly make the same stack of particles without any CTF correction? What you uploaded definitely looks like what I was getting before the fixes in r16912 - are you completely sure that betamyami has the updated ACE2correct.exe and running the latest code?

Hi Dmitry, personally I prefer negative values, but the Xmipp people (and others?) setup the CTF standards and decided underfocus is positive, so I thought it was best to adhere to standards. ACE1 always has had positive underfocus, ACE2 changed that. Just trying to play nice.

http://i2pc.cnb.csic.es/emx/CargarExampleTests.htm?type=CTF

Just want to mention that the emx is meant for exchange. It does not mean that we have to display in that way, but surely it is convenient to be consistent with a standard.

Just to add - SPARX & EMAN2, SPIDER, FREALIGN, & I believe BSOFT all use positive values for defocus values. I'm personally on board with this - it's always irked me that our DEfocus values were shown as negative. A negative focus value is underfocus, a negative DEfocus value is technically overfocus.
Regardless of semantics, Melody's issue needs to be worked out.

I agree with Gabe. I was just saying that we had NO standard before and lots of abs() functions; and now we have one, though it appears to be screwed up...

So this is sorta embarrassing and I'm really sorry for all the trouble-shooting you’ve had to do, but I think ultimately what the problem was was the fact that the images I’ve been processing are the non-square 4k x 3k DDE images. I didn’t think that this was an issue because using the ace2 phaseflip weiner filter seemed to work correctly for all the stacks I made between June 28th and July 1st in this same session (the first time it failed was July 4th) but maybe, ironically, those were the imges that hadn’t had the CTF correctly applied. After talking to Anchi about it and going through the image correction step by step we’ve realized it is the image dimensions. I’ve made a stack now using the EMAN phaseflip by boxed stack per image and everything looks fine too, as the boxes are still square.

sorry for not mentioning the atypical image dimensions earlier, and thanks for all the help!

Are we absolutely 100% sure that this is due to the rectangular images? It seems that (i) this worked a few weeks ago before the current frenzy of activity on the CTF modules; (ii) The correct whole image is being done using an EMAN module (at least hat is my understanding) and it seems unlikely to me that EMAN does not know how to correct a rectangular image?

I thought ACE2 never worked on rectangular images? I think it is more that my changes has shown how really hacked together our CTF system has been. Ugh. I will download the rectangular image set and start experimenting.