Appion

I haven't used frealign in ages, but I do remember something like this, and I know others in the lab here who use frealign have seen this sort of thing when converting their imagic volumes to mrc or spi. It probably has something to do with Imagic's definition of xyz being totally ass backwards compared to all the other packages. I think just putting a proc3d "yflip" in the conversion is in order.

The problem is not the volume. The problem is the Eulers. If you simply convert the Eulers from EMAN then reconstruct the volume with frealign without doing the Euler refinement, the resulting volume is mirrored.

This is definitely a big problem. I have not looked into it too closely. Gabe wrote the Euler conversion part and I have not modified it. I think Scott has a good test case for this below. How does our Euler angle conversion compare to EMAN2?

From Gabe:

hold on now I'm confused - the Imagic volume that prepFrealign creates has the expected
handedness - do you mean without importing eulers from EMAN?

From Scott:

If, for instance you use prepFrealign.py to generate the frealign Euler particle parameters from
an EMAN iteration, then use the 0 option in frealign to generate a volume from the Eulers
generated from prepFrealign, the resulting volume is mirrored. This doesn't have to do with the
initial volume; it has to do with the initial Eulers.

Add warning that the handedness if flipped in GUI

I'm just moving a couple emails here to keep everything in the issue...

On Mar 2, 2011, at 4:47 PM, Bridget Carragher wrote:

Hi y'all,
At the Appion meeting today I head that locally no one uses Frealign because it gives bad results. I though t I remembered that we had at least once gotten a ~5A GroEL map out of Frealign here. The team thinks we might be using is wrong so I wondered if any of your guys have been using it a lot and if so maybe we can chat more about how to set it up etc.
Hope you are all doing well,
Bridget

From Scott:

Hi all,
Yes, I have frealign working here, but the appion implementation does not work properly. The biggest problem (as I keep saying) is that there is a handedness flip associated with how frealign reads our imagic stacks and the imagic volume. That problem (and many others) will be solved if we go back to using mrc stacks for frealign. I have proposed many other changes in redmine and have implemented a few of them, but they are really major changes. I don't want to pull the trigger for completely revising the appion frealign code without the input from everyone else. Also I don't have the time to do it properly. I can just hack stuff together and hope that the group will help refine.

Scott

From: Gabriel Lander
Date: March 2, 2011 2:11:26 PM PST

If an imagic stack is going to be used for frealign, you need to use the "excopy.e" command with the "REVERSE" option to create a flipped copy of the stack. Otherwise it won't work at all. This I'm sure of. I thought it was fixed in appion? I guess not.

There is a workaround for the handedness flip. Arne, please update this issue with the workaround we are using here.
We can discuss this again in next meeting.

Neil, Scott, Gabe,

As I am working on the stack format conversion. Could you give me the specific commands for convert from our eman Imagic stack as boxed from default makestack of white on black box to mrc format (stack) that freealign will take in correctly? Appion only warns about contrast flipping but not handedness flip. My conversion script will make it into whatever you specify

For example, I can't find excopy.e in our installed packages. Where is it from?

From Neil:

My virus CNV structure showed improvements (6.5 -> 5.8) awhile ago, but I setup it up to use a flipped model initially, because of said problem.

Neil

excopy.e is an IMAGIC function, it should be installed in the "incore" directory your imagic directory:
/path/to/imagic-version#/incore/excopy.e

I strongly advocate that we go back to using mrc stacks for frealign. If you look way back at the first version of runfrealign.py that I started, you will see my code for creating an mrc stack (http://emg.nysbc.org/projects/appion/repository/revisions/9000/entry/trunk/appion/bin/runfrealign.py). In addition to not having to worry about the handedness flip, you can use a mrc volumes instead of imagic volumes. That way, you can look at the volumes with chimera directly during refinement instead of always having to convert to mrc first.

Please also see http://emg.nysbc.org/issues/397 for my proposal for revising the frealign workflow.

Gabe's solution can not be implemented in Appion since this running FreAlign should not require IMAGIC installation since the latter is not free, we can not force Appion users to have that just to do this. I looked at Scott's code with Jim just now. We can use it. However, I wonder if we can use proc2d to do this more efficiently? I am a bit worry about memory usage of reading the whole stack image array by python in order to use mrc.write(). Also, where is the handedness flip in this conversion of format. Is it that Frealign itself assumed opposite handeness in its code when IMAGIC file was read?

Hi Anchi,

I have a nice IMAGIC stack reader and writer python class: apImagicFile.processStack() that I use primarily for creating Xmipp "stacks" apXmipp.breakUpStack(), but I see you used it for your beam tilt correction. This class would be ideal for this task. It reads the available memory and only reads data that will fit into memory and it is quite fast.

Anchi, yes there is no explicit handedness flip. I think Frealign just defines a different origin or something, so it reads the images from bottom to top instead of top to bottom or something like that.

Scott, if it is as you say that there is no flip in MRC stacks, but there is a flip in IMAGIC stacks, should we check we Nico to see if this is a FREALIGN bug, because he may be reading the files wrong? I dunno.

I don't think there is any need to open that can of worms quite yet. Let's just see if using mrc stacks works like I think it will and see if that solves the handedness problem.

I need to organize my thought here.

1. For the same image, different viewers give horizontal mirror according to the origin location when it reads the image.
2. EMAN1 boxer boxed image as saves the stack in Imagic Format that reads starting from higher row to lower row number of an mrc image. Therefore the image itself is a horizontal flip
3. EMAN2 proc3d does a horizontal flip when converting Imagic stack to mrc stack(3D map).
4. Jim's pyami imagic2mrc.py which is used in prepRefineFrealign simply add mrc header on top of Imagic data block. Therefore it gives the "flipped" image stack relative to ones created by proc3d.

We now have an array of transformation in appion, mainly in apEulerCalc.py They all seem to make the assumption that the Imagic file is flipped from the others. This means that we do have to do a reverse when converting from and to Imagic. I will start an issue to have Jim work on it. It is a shame because the appending method is very fast for large stack.