Appion

I like this idea, Scott. Also, when I do it, I use a large boxsize for the frealign stack than the EMAN stack because it is not CTF corrected and the CTF is known to spread the data out. This causes some problems with missing particles in one stack to another due boxing on the edges of the image.

Initially, I wrote prepFrealign.py with the requirement that python is not installed on the cluster, so it could run on Solaris boxes for example, but this is probably not necessary, because I think everyone has some form python of on their cluster. A design flaw I regret now due to the extra complexity.

Neil

---

On Jan 12, 2011, at 12:19 PM, Scott Stagg wrote:¶

OK I have lots of comments:

Bridget, yes the make a stack from another stack thing works, but not if you want one-to-one correspondence with stacks of focal pairs, which is why I wrote the script I mentioned. I only mentioned the new script in the previous email because I thought it might be a useful tool for synchronizing pairs of stacks. Say for example you had figured out how to eliminate a bunch of particles after you had already made your frealign stack.

Arne, I agree that the workflow for running frealign is overly complicated and prone to errors. I really think the best solution will be to split up some of the things that prepfrealign does into other smaller programs. I think prepfrealign should only setup the run directory, create the parameter file, and prepare the stack and volume. A separate program should take care of setting up the frealign run. The latter is what I tried to do with runFrealign.py. The advantage of separating the two is that you can specify your frealign run parameters in a separate step from where you do all the Euler conversions, stack modifications, etc. Also, I wrote it so that you can change frealign parameters every iteration, much like we do with EMAN. Also, the frealign run part is where one typically makes the most mistakes. You don't want to run prepfrealign every time you want to change a single parameter.

Moving forward, I propose a work flow like this:
Make a CTF corrected stack
Run EMAN
Run prepfrealign
Arne, incorporating your suggestion, prepfrealign would at this point make a new stack of non-corrected particles from the EMAN run
It would actually box particles out like makestack but without CTF correcting etc.
Also, it would not upload the stack to the db because everything would point back to the original particles.
Transfer stuff to cluster
Run runFrealign.py to execute frealign
Transfer back to RAID
Upload results

Regarding the handedness flip. This is a very subtle but important bug. Before refinement, the model has the correct handedness but the Eulers do not. After refinement, the volume will have the correct handedness because frealign will refine the incorrect Eulers to the handedness of the initial model, but the Eulers are incorrect to begin with. The only way you will notice this is by using the reconstruct without refining option in Frealign (option 0 or something like that). Way back when I noticed this, Gabe suggested that it might have something to do with using an imagic stack instead of an MRC stack. I never followed up on this, but if you read in an image from the wrong direction, it can flip the handedness of the particle. If Frealign reads in Imagic images in a different way than EMAN does, it could result in the observed bug. Regardless, the most important point is you have to check this BEFORE doing the Frealign refinement.

OK, that is a lot of stuff. Please let me know if anything is unclear or if you disagree with my proposed workflow. If anyone would like to chat with me about this, I would be more than happy to talk with you over the phone.

Thanks,
Scott

---

On Jan 12, 2011, at 9:27 AM, Bridget Carragher wrote:¶

I agree that highest possible resolution is always the goal.
I thought we had solved the one-to-one correspondence issue between eman and frealign long ago? correct me if I am wrong. I also thought that we could make any new stack using an existing stack so we should always be able to get a corresponding stack?

---

On Jan 11, 2011, at 3:13 PM, Scott Stagg wrote:¶

I disagree that the preparing EMAN alignment shouldn't try to get to the highest resolution possible. The better the starting Eulers, the better chance Frealign has in its refinement. Also, you can use all the other sorting and junk filtering features to get the best particles possible before running frealign. The trick is having two stacks with one-to-one correspondence between the particles but where one is corrected and one is not. A possible solution is to use the focalPairSync program that I recently prototyped. It takes two stacks with different particles and the same picking run and outputs two new stacks with one to one correspondence. I wrote it with focal pairs in mind, but it could really be used for any pair of stacks. It is still very much a prototype as it doesn't yet upload the particles back to the db, but it should work.

Other comments:
You say that the handedness is correct. Did you fix bug #399? I'm curious how you got around that.
The initial model should be white on black even though the particles are black on white

Scott

---

On Jan 11, 2011, at 5:44 PM, Arne Moeller wrote:¶

Sure Imagic needs white on black as well.
That's why we want to flip it directly before Frealign - the preparing Eman alignment is not meant to produce highest possible resolution anyway so you probably do not need the Ctf correction.
I think both ways will work -we just have to make clear which way to go - so far it is obviously not clear.
What is with the intial model? Is this flipped in the prep frealign process?

---

On Jan 11, 2011, at 2:27 PM, Neil Voss wrote:¶

I agree but SPIDER does not work with black density on white background, but FREALIGN does not like CTF corrected stacks. So, you almost have to make a separate stack anyway. I am not sure about EMAN, but I think it requires white density as well. If you are doing alignment it is best to do a CTF correction first. If you are not, I would reconsider what you are doing.

So, you can do it that way, but make sure you are not using a CTF corrected stack.

My workflow is create "white" CTF corrected stack, filter particle in alignment pipeline, create new "black" stack not CTF corrected for FREALIGN.

My 2c,
Neil

---

On Jan 11, 2011, at 4:18 PM, Arne Moeller wrote:¶

Hi all,

Lauren, Amber and I agree that the problem is: The way Appion is set up right now is that the input stack is white density on black background. Frealign does not like that. The result is an inverted map (because Frealign flipped the output). If we flip it back everything looks fine (including correct handedness).
Since we want to be able to run a rough eman alignment on the same stack that is going to be used for Frealign we decided to flip that stack (and model) directly before Frealign starts.
We are changing that right now...

Arne

---

On Jan 11, 2011, at 2:08 PM, Neil Voss wrote:¶

Whenever you apply a CTF correction, it flips the contrast. The ACE2 Weiner filter does the CTF correction and an inversion. FREALIGN prefers to work with dark on light and do they correction with no inversion.

Neil

---

On Jan 11, 2011, at 4:01 PM, Bridget Carragher wrote:¶

Hmmmm..... so is this just a problem of contrast? Seems unlikely. I am cc'ing Scott, Neil and Gabe too as they may have some insights.

---

On Jan 11, 2011, at 1:38 PM, Niko Grigorieff wrote:¶

Hi Arne,

Thanks for the good wishes. I hope you also had a good start into the new year!

I am not sure I fully understand your question. Frealign does not work with class averages. But there is a contrast flip between input particle images and output 3D density: On input, Frealign expects dark protein density on light background and on output it will produce light protein density on dark background. The reason is the CTF correction. Remember that the CTF goes negative at low resolution. Therefore the flip in the contrast after CTF correction.

You can also always check the Frealign forum on our web page for questions like this:

http://emlab.rose2.brandeis.edu/frealign/forum

Also, you can post your own questions if you do not find the answer you are looking for. This has the advantage that answers will be visible to everybody (you have to register with the web page though).

Looking forward to seeing you all later in the year!

Best wishes,

Niko.

---

On Tue, 11 Jan 2011, Arne Moeller wrote:¶

Hi Niko!
Happy new Year!

Frealign is running nicely in Appion, with one exception.
The densities of the resulting volumes and classaverages are flipped.
A quick fix is to flip it back after each iteration. However I am concerned that this is a symptom for something being messed up within the code and that Frealign might not run correctly.
At this point we can't tell if this is an Appion issue, or if that is something Frealign does.
That's why I want to ask, if you have an idea and/or if that happened to you before?

We actually want to release Appion as soon as possible (with Frealign included) but are hesitant as we need this bug to be fixed beforehand.

Currently we are using Frealign version 8.08.

Best Arne

r15260 added a note on the web form explaining the requirements of the stack and redirects the user to the wikipage for more information and the workflow for creating the stack. This is just a quick fix for the release. I will continue to work with Arne and Anchi on implementing the automatic stack creation option within prepFrealign.

I ran frealign on a GroEl black on white stack and the reconstructions look fine to me, i.e. the handedness is not flipped (http://cronus3.scripps.edu/betamyamiweb/processing/reconreport.php?expId=8304&reconId=25). I actually do not fully understand the handedness issue. How do the initially incorrect Eulers affect the final volume if Frealign corrects the handedness when it refines the Eulers to the model? And why are the Eulers incorrect to begin with? Is this just something we should note and move on, or is it actually a bug that needs to get fixed? Would someone who better understands this issue be willing to add some information about it to the wikipage?