Appion

Let's break them up:
1. Extend image assessment to its sibling. Do you mean that if you reject en, then ef is marked rejected? If this is done in manualpicker.py I recommend doing it in the same script then the web interface will be easier.

2. Paired particle picks. I've noticed that alignDefocalPairs.py stands alone and makestack2.py need its
database entry to find the corresponding position. I think this one should be done at the webviewer level by using that database entry, not to create a whole set of new particle data records.

It is o.k. to do the python part and assign the web part to others as long as instruction on what error checking is needed and how to link the two are clear, very clear. You should also assume the responsibility of testing it.

Could you open new issue for each and assign the first to yourself to do the python part and once that is completed it can be assigned to Eric or Amber to do the web part.

For the second issue, Eric has agreed to make a web interface for alignDefocalPairs.py and the particle mapping. You should give some instruction about the former. I will help him with finding the components I learned from recent experience for the latter.

Scott,

You are right about the correspondance issue in the stacks. I think we will need to have both display of sibling picks in the imageviewer. makestack can make either sibling already but a table for paired particles in stacks is needed if query through ef picks and defocal pair shift is too inefficient.

Neil, how does RCT deal with correspondence issue in the stack of tilted and untilted particles now? I imagine the problem is similar.

I don't think you'd need a new table for paired particles. If you made a stack of en particles and another stack of ef particles all from the same pick run, then the corresponding particles will have the same pick parent in the database. All you would need to do then is find the common particles and make two new stacks from the original ones where the new stacks will have one to one correspondence. Does that make sense?

Scott, for RCT I have to insert a particle for both images, because the tilt transform is not perfect due to distortions in the grid height, for defocal pairs I assume the x-y shift and zoom transform is all you need to perfectly reproduce the defocal particle pair, so two entries is not necessary.

On another note, I set up a "defocal pair" system for Anna-Clare to test her direct detector data. She has high dose and low does images rather than defocal pairs, but she is using the same system. To this end, Appion's FREALIGN allows you to use one stack for alignment (high dose or defocus) and a second stack for backprojection (low dose or defocus).

What I have her do is create two stacks from a single particle picking run, one from each defocus. I know that works fine. I am not exactly sure what you are trying to implement, so I cannot comment further.

Scott,

Do you mean then you want to make two stacks one en one ef but to make sure that the 1256th particle in en stack to be picked from the equivalent position in the 1256th particle in ef stack? In other words to throw out particles in ef stack that do not correspond? I guess it is either a substack job or a specialized makestack. The latter may be better because we still have an unsolved problem of substack that could not take a very long list of particle exclusion.

Yes, that was the problem. We had no good way to make sure the particle were perfectly in sync for each stack that was then required for FREALIGN. Anna-Clare situation was simple, because the shift was always (0,0) it is the same image, so no additional particles were thrown out in one preset and not the other. But that is the major problem.

In RCT, the database solves the problem for me. I do not care what order the particles are in the stack I just look up the corresponding particle in the database. Something to think about.

Neil, I'm a little lost. I don't understand why your solution for Anna-Clare would work. There is no guarantee that the particles would be the same in the en or ef stacks.

Anchi, yes, that is what I'm suggesting. I think adding an option in makestack might be a little unwieldy. My suggestion would technically produce substacks, but they aren't necessarily required. They could be separate independent stacks, I think.

Scott, sorry if I was confusing. You are right there is NO guarantee the particles will match up.

In her case, the images are exactly same location it is just the number of frames going into image that make them different. So, in that case, we usually had no problems, because the particle picking run was the same and the assessment IS INHERITED between defocal pairs in makestack (it fails for speed and clarity reasons in assess_summary.php). So, in her case, they were matched up. The only time we had problem is when the defocal shift calculation failed to find the peak and inserted an almost random shift, but that was fixed.

We need a mechanism to guarantee that they match up. We could have makestack produce two stacks, one from each preset, simultaneously and force matching.

Another option that would be easier to implement, but its major drawback is that it creates extra data.

If you look at my program tiltStackSync.py it is designed to take two existing stacks from the same tiltpicker run and produce two new stacks with the particles exactly sync'd. Gabe and Ed have been using this to export RCT stacks outside of the pipeline for custom SPIDER processing.

Are you saying that makestack already will already extend assessment between defocal pairs? In other words with the current makestack, if I reject an ef image its corresponding en image would be skipped in a makestack run on en images?

What you are describing with tiltStackSync is exactly what I have been suggesting. I think it might be worth using the extra disk space to have a program like this for defocal pairs instead of making yet another option for makestack. Makestack is so complicated already.

Sorry, no you have to assess the images for each preset. makestack2 uses the appionLoop2 image rejector, which calls the apDatabase.getImgCompleteStatus(), which has no defocal pairs component. There is a function, apDatabase.getSiblingImgAssessmentStatus() that does this, but probably needs updating.

In our case, we uploaded images, so they were all good, but if you only pick particles on the correctly assessed images you should be good, but this is definitely not a fool-proof system.

OK. Based on all the input. I am going to do two things.
1) I will make a new option for makestack called --ea (for extend assessment) which will use the sibling's assessment data to determine whether to skip an image

2) I will make a new program called focalPairSync.py that will take an en stack and an ef stack with the same picking run and create two new stacks with a one to one correspondence between particles in the two stacks.

I will make new issues for each of these items. If anyone has any comments or problems with that, please let me know now.

P.S. Redmine is awesome for facilitating this discussion. I feel so much more in the loop now!

Sounds like a good plan. Looking forward to the new feature.