Gain/Dark correction of the raw frame with or without drift correction¶

This procedure can perform two things:

Gain/Dark Corrected Frame Stack Creation¶

This procedure performs the flat field correction of the movie frames obtained from direct detection camera (DD).

Motion correction¶

As an option, the motion between frames can be corrected by shifting one against the other. The resulting aligned sum is uploaded into Leginon as an image under a new preset. If original preset is "ed", then the new preset is named "ed-a".

This is based on doi:10.1038/nmeth.2472

General Workflow:¶

1. Make sure that appropriate run names and directory trees are specified. Appion increments names automatically, but users are free to specify proprietary names and directories.
2. Select the leginon preset corresponding to the images you'd like to process. This has to be a preset that movies were collected. Images without frame saved will be skipped.
3. Enter a run description.
4. Specify image looping parameters:
   • Select the appropriate preset of images to search (default is en if you collected with leginon).
   • Check boxes allow the option to run the frame stack making concurrently with data collection. "Wait for more images" will wait until collection times out before stopping processing. The "Limit" box allows restrictions on the number of images to process, but is generally unneccessary
   • Radio buttons under "Images to Process" allows a level of pre-processing image filtering. Images that were rejected, hidden, or made exemplars in the image viewer can be included or excluded.
   • Radio buttons under "Image Order" sets the order in which images are processed and radio buttons under "Continuation" gives the option of continuing or rerunning a previous run.
   • Make sure that "Commit to Database" box is checked. (For single image test runs in which you do not wish to store results in the database this box can be unchecked).
5. Activate "align" check box if frame alignment is to be performed (gpu required if "defer gpu processing" is not checked).
6. Click on "Just Show Command" to obtain a command that can be pasted into a UNIX shell. See Notes below on how to improve throughput.
   

Notes, Comments, and Suggestions:¶

Parallelization of the running:¶

It is necessary to parallelize the frame stack making and alignment if it is used during the data collection to create quicker feed back since each process takes about two times longer than the rate of data collection.

Running in parallel, each instance of the run will lock the image it is currently processing so that other instances will take on another image available to it. You can run these in different ways. Here is what works best for us:

We have a computer with 4 gpus, and other computers with reasonable cpu speed. To get maximal parallelization, we use computers without gpu cabacity to do gain/dark correction, deferring the actual alignment and then run 4 alignment instance using the script catchUpDDAlign, each on a different gpu (for example add --gpuid=1)

Therefore, once the data collection is on the way, and rawtransfer.py has started moving the frames on to file server available to these processing computers, we

1. Copy the makeDDRawFrameStack.py command generated above to a terminal without gpu with "align" and "defergpu" activated and execute it.
2. Repeat this on another terminal, either on the same host or not, depending on how much memory and number of cpu there is on the computer.
3. You can launch this to as many cpu processes you like until it is limited by I/O and data collection speed. It is advisable to run these parallel process in different image order. If you always run in "Foward" direction, the most recent image is unlikely to get a chance to be processed. If you always run in "Reverse" direction, most processes would sit and wait for
   the newest frames to be moved to the file server. We recommend run 1 out of every 3 in "Reverse" order. Default is Forward, add " --reverse" in the command makes it run in the reverse order.
4. Once it is underway, start multiple Catch up DD frame alignment, each assigned to different gpuid.

If you have a powerful computer with all the goodies, you can run everything there without defergpu.

---

<Direct Detector Frame Processing | Catch up DD frame alignment when frame stack making defers the alignment >

---