Appion

This revision flips my image density - anyone else see this?

Is this a brick wall or can we just continue using ctffind? What does the earlier email from Gabe re the accuracy of ctffind (good) vs. ace2 (bad) mean for past and present processing? Should we just use ctffind or try to figure out the issues with ace and ace2?
P.S. Not to say that I think we should not fix the bugs with ace2 but if this method is really far less accurate hen ctffind then ?????

At this point I have no clue what acetwo phaseflipping is doing - the previous revision (r17010) doesn't flip the density, but obviously doesn't include this fix for the amp contrast. I wouldn't use ACE2 phaseflipping at this point, but Neil should weigh in on this.
As far as the accuracy, ace2 estimation from r17010 is far worse than CTFFIND (see Bug #2053).
So for now I suggest only using ctffind for estimation, and any of the non-ace2 phaseflipping methods or use FREALIGN.

I just want to weight in here too to say that ACE1 still works for CTF estimation, and it makes excellent estimates. There is still an unresolved bug (please see Bug #2016) where new CTF makes it look like ACE1 is giving a poor estimate when it's actually quite good, both in the plots that it makes and the fact that ACE1 estimates in my hands always give higher resolution reconstructions than ctffind (4.5 Å for ACE1 vs ~5.0Å ctffind). I also want to point out that using EMAN1 to do the phase flipping also works just fine.

Hi all, I will take a look at this on Friday afternoon and into the weekend. It is a major issue and I apologize for the slow response.

On this particular bug, I found that ACE2 was using the wrong definition for the amp contrast and needed to be fixed or we could not continue with CTFFIND and ACE1 values (See Bug #2016). I looked at ACE2Correct when uses different internal functions and it did not need to be changed, but maybe it did.

On the underfocus check warning, I measure the confidence for both over and underfocus and if the conf_under > conf_over, I print out the message, but I may be like 0.998 > 0.997.

The underfocus check gives very different results for me, despite an overall confidence of 0.85

PART 5: CTF FIT AND CONFIDENCE
1/30A - 1/10A confidence is 0.386 (overfocus 0.888)
!!! WARNING: Image is possibly over-focused
5 peak confidence is 0.486 (overfocus 0.926)
!!! WARNING: Image is possibly over-focused

Hi Gabe, this may mean that ACE2 is fitting overfocus and so the overfocus value comes out better even though it is not overfocused. I am now using CTFFIND as the benchmark (instead of ACE2), so I will look into it. That is exactly why I added the check.

1. Ace1 - #2016, works well, but the results display is not working correctly and is making the results look bad even though they are not.
2. Ace2 - #2053, #2021, there is an issue with Ace2correct that is making results bad
3. ctffind - works, but does not work as well as Ace1, at least for Scott.

It also looks like Neil has some ideas for fixing Ace1 and Ace2, but will not have time to do so in the near future.

Is there anything else?

It seems to me that getting the Ace1 display working should be a priority and Ace2 may be able wait. Thoughts?
Neil, do you think the Ace1 display issue is something that I could resolve with a little guideance from you? I'd like to have this issue finalized before branching off 3.0 in a couple(ish) weeks.

I do not have MATLAB, so I cannot test ACE1 directly. I always happy to help, but the math on this bug is fairly involved, whenever I adjust something, something else breaks.

Are there any updates on this? Does this mean that ctf estimation or correction is broken in Appion? Or is there a workarround that is working for everyone one way or another. If CTF est or correction is really broken then I think this should be the most urgent priority we have.

We have been trying to make CTF-corrected stacks through Appion in the cubicle. Currently, the only option that is working is "EMAN phase-flip by boxed stack per image". ACE2 phaseflipping is giving inverted image density, "EMAN flip by whole micrograph" is returning a standard deviation=0 for particle X error, and "spider phase flip" is also returning an error, stating that the corrected SPIDER image was not generated. I have not looked into this issue in depth, but it seems to me that the CTF correction methods need to be thoroughly tested with the new CTF estimation methods and parameter extraction from the database.

ACE2 is broken. I am waiting for someone to tell me if ACE1 is broken. Scott was using an old version in #2016. As I posted in #2016, SPIDER and EMAN phase flipping worked fine for me, but needs confirmation.

I need to find some time to do this, but I really have none. I think it may be an overfocus/underfocus issue. But it may be related to the amplitude contrast. I do not think reverting will work.

I've also not had any time to test ACE1. I just finished giving a talk I had been working on and should be able to give ACE1 some attention after class on Monday.

S

The plots and confidence values that are output for both CTFFIND and ACE2 seem to be completely off in the displays, showing a far greater confidence for overfocus than underfocus values. We don't have a Matlab license, so I can't speak for ACE1.
Frustrated with the dozens of complaint emails I'm receiving every day from my lab members every day in regards to this, I've completely taken down the entire Appion processing site from our webserver and told everyone to use my command line scripts for running CTFfind, making stacks, performing 2D classification, and running Frealign reconstructions.
This is not ideal.
Please thoroughly test any changes made to such a crucial part of the pipeline in the future before committing.

Hi All, I apologize for the problems. I test everything thoroughly and then these individual problems crop up that uncover huge problems.

I've asked Heather to watch the kid tomorrow, so I dedicate some time to this problem.

I will focus on check that CTFFIND results to make proper phaseflip using both SPIDER and EMAN. Then secondary, I will get ACE2 estimation back to the same values. Finally, I will work on ACE2 correction with CTFFIND values.

We need to have at least some working solution, since ACE2 is the hardest to work with I will focus on the others before moving to ACE2. A fix maybe temporarily disabling ACE2, I need to check.

Perhaps in the future such a large overhaul could be made in a branch instead of trunk. That way we can all test stuff out before merging into the trunk.

I think that it is absolutely essential that these kind of overhauls be made in a branch and not transferred the to a main trunk before adequate testing. This kind of major mess is the sort of thing that just moves people off to other solutions (as in the case of Gabe's group) and then it is hard to persuade them to come back. Neil please get this as far to working or back to the way it was as possible and then let the rest of the group know where you are at. I am going to ask Amber, Jim and Anchi to drop anything else they can next week and try to get this properly working. But as they did not do it in the first place they will need whatever input you can give them to get started.