Appion

Rough web pages for each of the above scripts now exist, are strung together, and are linked to from a selection page under the Tomography side-header. TODO: plotcorr2.py plots, summary page, and tiltseries selection menu in Protomo Preparation page.

1. You need to include $extraHTML as the second parameter to the constructor
2. You had a typo in the key values for the validations. grid_limit -> gridsearch_limit and grid_step -> gridsearch_step so that they match the key values of the addParam() function.

The individual tilt series processing and webpages (File Preparation, Coarse Alignment, Coarse Alignment Summary page, Refinement, Refinement Summary page, and Reconstruction) are functional and nearly finished. I'm now working on the tilt series batch processing page. Batch processing will allow the user to apply the protomo parameter files created during individual processing to all of the tilt series in the session. The batch processing step will (hopefully) be embarrassingly parallelized from within python. This should go smoothly because all of the parameters have already been determined by the user by this point in the workflow.

There are a few issues I need to figure out:

1. How to plot raw xy-data from a file within a php/html webpage.
2. How to detach the table of fourier transforms from the default function buildCommand table on the refinement summary page.
3. How to cite multiple citations on an Appion page.

Within a week I'll commit a functional set of files for others to test=)

myamiweb/processing/inc/particledata.inc function getAlignLogShiftFromDDAlignedImageId has this example for reading data from a file and put them into arrays

    if (is_file($align_log)) {
      $text = file_get_contents($align_log);
      $bits = explode('Sum Frame',$text);
      if (count($bits) > 1) {
        $shiftlines = explode("\n",$bits[1]);
        foreach($shiftlines as $shiftline) {
          $shiftbits = explode('with xy shift:',$shiftline);
          if (count($shiftbits) > 1) {
            $shifts = explode(' ',trim($shiftbits[1]));
            $data[] = array('x'=>((float) $shifts[0])*$apix,'y'=> ((float) $shifts[count($shifts)-1])*$apix);
          }
        }
      }
    } else {return $align_log;}
    return $data;

myamiweb/dddriftgraph.php then use this array ($data) to produce a jpeg image of an xy plot

You can then display the plot in html by enclosing the image making php script with <img src='your_script.php'>
like the way it is used in getdddriftgraph.php

<img src="dddriftgraph.php?id=<?php echo $aligned_imgid?>&expId=<?php echo $sessionid?>&s=<?php echo $size?>">

By the way, the data array making example above had to search for data line that have specific text as you might have figured out already.

Maybe I'll finally release something once I get parallelization working! For now, here's what's changed since my last update.

Added: Functionality allowing users to reconstruct from any refinement iteration on the Reconstruction page.
Added: When going back one step in the workflow the user is presented with a set of remove commands to clean up failed alignments.
Added: All log files and protomo output files are now moved into the rundir after running any of the protomo2*.py commands.
Added: Gifs now include frame numbers.
Added: All values that can be defaulted are defaulted.
Added: Hover-over information text has been added to each subsection of the Coarse Alignment and Refinement webpages.
Added: Hover-over information text has been to the Refinement Summary webpages to help the user understand what they are looking at and what to look for in good reconstructions.
Added: Much of the hover-over text has been edited and expounded upon based on the Protomo user manual and direct questions posed to Protomo's developer.
Added: protomo2batch.py now exists and batch processing now works, but it's not parallelized yet (my top priority).
Added: The Refinement Summary webpage now has a button that will show all summary information on the current page. By default this section is collapsed. Pushing the 'View All Alignments' expands the section.
Added: Users can now enter multiple ranges of tilt series to be processed into the batch processing webpage and protomo2batch.py will parse the values into an array of iterables.
Added: protomo2reconstruction.py now creates and populates a folder, ready_for_batch, in the rundir with the three required param files for batch processing.

Fixed: Coarse Alignment options now only include options that coarse alignment uses.
Fixed: Values that you would want to be in angstroms (eq. filters) now are. Conversion into Protomo units is done in Python scripts.
Fixed: Discrete options on the webpages are now pull-down menus.
Fixed: Some default parameter values have been changed after consulting with Protomo's developer.
Fixed: Removed a few parameters from the webpage that the user never needs to change.
Fixed: Moved the protomo2prep.py functionality into apProTomo2Prep.py (prepareTiltFile) and prepended the Protomo Preparation webpage onto the Protomo Coarse Alignment webpage.
Fixed: The table of per-iteration correlation peaks on the Refinement Summary webpage is now properly formatted.
Fixed: Some bug fixes.

To do: Extend applicable parameters into additional columns for multiple rounds of refinement iterations with different parameters.
To do: Parallelize batch processing.
To do: Add correlation plots to alignment summary pages.
To do: Add tilt series gifs to alignment summary pages, if possible.
To do: Add scalebars to gifs.
To do: Determine if Appion supports HTML5 video. Currently the gifs that are produced (default size: 640x480x200) each take up about 30MB of space. This can cause webpage strain and it uses valuable disk space. HTML5 video, which uses mp4 encoding, might reduce this by an order of magnitude. Should I look into this? All current browsers support HTML5 video using mp4 containers: http://www.w3schools.com/html/html5_video.asp
To do: Figure out how to clear ram after using apTomo.getGlobalShift to get image shifts during the initial file setup. This step currently consumes ~7GB of ram, which I can't seem to forcibly clear using del or gc.collect(). This will be a big problem once batch parallelization is working.
To do: Add 'upload to Appion' functionality after the reconstruction and batch reconstruction webpages.
To do: General code cleanup. But who has time for that?
To do (longer-term): Add dual-tilt functionality (I've never even tried using Protomo on a dual tilt series, so I don't know what will be required).

Discussed during Appion call. Alex made lots of progress. The last thing to do is to get parallelization working in Python. Gabe suggests looking at runjob module using threading. This is in Dogpicker and Template picker. Neil suggests looking at apthread. Otherwise apfindem.py.

I'm in the process of trying to commit about 20 files (this is the 1st time I've used svn commit so I'm making sure I don't mess anything up in the process). Parallel batch processing is finally working, but is not included in this release. Here are the changes since the last update:

Added: Automatic Coarse Alignment and Refinement restarting. Alignment runs sometimes fail partway through because the cross correlation search area is too large and goes out of bounds, usually because one or more raw images is shifted too far. Now options are available to the user to let the python script automatically reduce the search area size by a user-given percentage for a user-given number of retries before giving up. Protomo restarts on the image that previously failed, so essentially no time is wasted and a lot of manual work and guess work is saved.
Added: Rounds have been introduced into the Refinement Form and python script. The user can now set different parameters for each round for applicable parameters. Defaults have been filled in for 3 rounds. This addition doubled the number of parameters in all of the scripts (protomo2RefineForm.php, protomo2aligner.py, help.js, and there are about 20 more in apProTomo2Prep.py).
Added: Switches have been introduced into the Refinement Form and python script. The user can now toggle most true/false parameters at a specific iteration.
Added: Individual Refinement Summary pages for each iteration. Click on a correlation peak on the Refinement Summary page to get there. The layout can be improved, but I'm no artist.
Added: Restart functionality in the Refinement Form and python script. Now users can run a refinement, look at the results, and decide that they would like to revert back to a particular iteration and run more refinements from there.
Added: Correlation plots to Refinement Iteration Summary pages.
Added: Tilt series gifs to alignment summary pages. (Still not shifted in the right direction?)
Added: Scale bars to tilt series gifs and reconstruction gifs.
Added: The option to symlink raw images instead of copying. Symlink is now default. Linking significantly reduces overhead, disk accessing, and processing time.
Added: Depiction images and gifs are now made in the background using python multiprocessing during individual tilt series alignments (not during batch because that would be redundant). Now correlation peak gifs, correlation images, tilt series gifs, and reconstruction gifs are made concurrently in parallel. This significantly reduces processing time (e.g. Coarse alignment and summary page creation using default settings now takes half the time).

Fixed: Enter Parameters/Next Step prompts should now show properly in all situations (made the conditionals more robust).
Fixed: Refinement Summary page was broken when number of iterations was less than 4 (4 is the width of the correlation peak table).
Fixed: The peaksearch option in the coarse param file was not there, now it is.
Fixed: The ability to change the sampling of the depiction reconstruction failed because if it is not equivalent to the sampling value then protomo barfs. I fixed this by adding a second binning stage during image processing.

Updated: The files that need to be removed if the user wants to re-do a refinement.
Updated: Moved multi-use code to apProtomo2Aligner.py library. protomo2aligner.py is now a lot cleaner.

To do: Add a parameter output log.
To do: Add a runtime output log.
To do: Images in tilt series gifs might not be shifted properly? I haven't taken time to see if this is a problem.
To do: Find a way to not load content beneath a button (Refine Summary page). Scrapped?
To do: Finish parallelizing batch processing.
To do: Add 'upload to Appion' functionality after the reconstruction and batch reconstruction webpages.
To do: Remove the 'Just Show Command' button from Summary webpages.
To do: Add docpops to iteration summary pages (where are they imported from?)
To do: General code cleanup. But who has time for that?
To do (longer-term): Add dual-tilt functionality (I've never even tried using Protomo on a dual tilt series, so I don't know what will be required).

I've committed all of my protomo files to svn. These include:

appion/appionlib/data/protomo.param, protomo_coarse_align.param
appion/appionlib/apProTomo2Aligner.py, apProTomo2Prep.py
appion/bin/protomo2aligner.py, protomo2batch.py, protomo2reconstruction.py
processing/protomo2RefineIterationSummary.php, selectAlignTiltSeries.php, selectBatchAlignTiltSeries.php
processing/js/help.js
processing/img/protomo.png, other_software.png
processing/inc/menuprocessing.php
processing/inc/forms/protomo2BatchForm.inc, protomo2ReconstructionForm.inc, protomo2BatchSummary.inc, protomo2RefineForm.inc, protomo2CoarseAlignForm.inc, protomo2RefineSummary.inc, protomo2CoarseAlignSummary.inc

The individual tilt series alignment processing that you get to from the menuprocessing.php sidebar >> 'Align Tilt Series' should work. I am still working on the 'Batch Align Tilt Series' py and php scripts, so no guarantees there (it currently works just with a single processor and without the round parameters).

Report any issues you have with the individual tilt series alignment workflow to this issue. Thanks!

I missed one thing in massive list of files I committed. Line 353 in appion/appionlib/apProTomo2Prep.py should be un-commented and line 354 should be removed. I committed this change to svn this morning, but I'm not sure if it will propagate through because the servers are moving.

While I'm here I should mention that the files I committed require the following to be in your paths:

protomo 2.4.1
i3
pylab
PIL
imagemagick

This is a non-committed update because I haven't put everything back together yet. The main features to report on are that batch parallel processing works, depiction videos will be moving from gifs to html5 video (html5 will be default, gif option will remain), and CTF correction will (hopefully) be put into a future release pending testing. David Mastronarde has given me the go-ahead for using ctfplotter and ctfphaseflip in my workflow and he has allowed me to include it in Appion releases if desired.

Here's what has changed since the last update:

Added: Parallelized batch processing! Each processing step —Prep, Coarse Alignment, Refinement, Reconstruction— can be called separately or together. Any combination of ranges of tiltseries is accepted (eg. 3,7-22,40-53,100,102). All alterable parameters are imported from previous pages in the workflow and are changeable.
Added: An option to reset the shifts imported from leginondb to zero shift. Sometimes Leginon records the relative shifts of tilt series images as being greater than the width of the image, which breaks Protomo. Adding this option should increase the reliability of batch alignments.
Fixed: Various help docs.
Fixed: Made the multiprocessing handler more robust so it properly counts the running processes.
Fixed: Python scripts weren't changing parameters for Round 1, now they are.
Fixed: Tilt series gifs weren't shifted properly. Shifting up and down had to be reversed and rotations were in the wrong direction.
Updated: Moved more multi-use code to apProTomo2Aligner.py library.
Solved: The ram-consumming .tlt file preparation step by relegating it to its own parallel processing step where the number of processors used is set to max=5.

To do: Add a parameter output log.
To do: Add a runtime output log.
To do: Find a way to not load content beneath a button (Summary pages).
To do: Add 'upload to Appion' functionality after the reconstruction webpage to save parameters for later use.
To do: Remove the 'Just Show Command' button from Summary webpages.
To do: Add docpops to iteration summary pages (where are they imported from?)
To do (longer-term): Add dual-tilt functionality (I've never even tried using Protomo on a dual tilt series, so I don't know what will be required).

There are still too many things in flux right now for a release, but I'm aiming for Friday April 25 so that Gabe can start trying to use it. There are still some batch processing things to iron out before then.

Here is what has changed since the last update:

Added: Functionality to remove tilt series images completely from a workflow if their shifts exceed a percentage of the image size. A small percentage of tilt series (~5% from my experience; depends on leginon shift estimation when tilting) have one or more images with shift records in leginondb that are shifted too far for Protomo to deal with, so it gives up. There is a new option to automatically detect and remove these images from the .tlt file so that alignment can continue. Almost all of the offending images, from my experience, are at the highest tilts and don't appear to be alignable under any conditions, so I anticipate that this feature will increase the reliability of tilt series alignment without losing usable data.
Added: HTML5 video output (mp4, webm, and ogv) for all video depictions in addition to gif output. HTML5 video is now default, gifs are the backup option. HTML5 video uses better compression with less quality loss, generally loads faster than gifs, is portable, is supported by all modern browsers, and it's 25 years newer. You can also scroll through videos. (Requires ffmpeg)
Added: How many frames each video is out of to the bottom of the videos.
Added: Tilt degrees in tilt series videos (can't get degrees symbol to work...).
Added: 'Download video' links for mp4 depiction videos.

Fixed: If the number of iterations for round 2 is set to zero, the script used to change parameters incorrectly. This is now fixed.
Fixed: It was unclear whether the tilt information used at the beginning of a refinement carried over from the coarse alignment (this information is hidden in protomo's binary .i3t files). This has been fixed by forcing Protomo to create a new .i3t file at this stage explicitly using the post-coarse alignment .tlt file to make a new .i3t file instead of relying on the .i3t file.

Updated: Moved the video creation step for the unaligned tilt series to begin before coarse alignment and run in the background. This allows for the user to see what the unshifted tilt series looks like even if coarse alignment fails, and it's a time-saver.

Removed: The option to unshift images. This option didn't accomplish what I'd hoped for and has been replaced by the first new Addition above.
Removed: The option to optimize gifs. They are only optimized now that html5 video is default.

To do: Add a screening option that can be run while collecting. This option should continually wait for new full tilt series to appear in leginondb, then run them through a coarse alignment. This will make it easier to determine collection parameters.
To do: Add background functionality that guesses which iteration is the best. This is actually almost done I just need to test it.
To do: Add x and y slice-throughs for tomograms like in IMOD.
To do: Get the degrees symbol in the videos. This is surprisingly difficult.
To do: Add 'upload to Appion' functionality after the reconstruction and batch reconstruction webpages.
To do: Remove the 'Just Show Command' button from Summary webpages.
To do: Add docpops to iteration summary pages (where are they imported from?)
To do: General code cleanup. But who has time for that?
To do? (longer-term): Add dual-tilt functionality (I've never even tried using Protomo on a dual tilt series, so I don't know what will be required).

Apologies for not committing yet. We had a tilt series session, so most of my effort has been towards tilt series and tomogram processing (and it works great!!). The main additions below are a result of my desire to align as many tilt series as possible. We collected ~80 tilt series, where ~40 of them were at a reasonable dose (we had brightness issues with the tip). Of those 40, ~20 were reasonably aligned or well aligned and CTF corrected after about 3 days of processing on ~50 cpus using the python backend of this Appion-Protomo implementation. I have a workflow on paper that will automatically align tilt series in batch until convergence, and CTF correct (see 1st 'To do' below). This, writing a manuscript for JSB, and finishing the front-end PHP are my top priorities.

Added: A quality assessment plot (and underlying statistics) is used to estimate both which iteration is the best and whether an iteration is well-aligned. A candidate for a well aligned iteration should have the sum of the average residual x and y correction factors and their standard deviations below a value of 0.025. Once a candidate tilt series is found, the user should determine if each of the average residual correction factors are between 0.99 and 1.01, if the standard deviations are below 0.005, and the rotation factor is below 1 degree and is more or less smoothly monotonic. The addition of this metric allows for the user to immediately determine which iteration is the best (and how good it is) in the single tilt series alignment webpages, and in batch mode which tilt series is most alignable for a given set of parameters. This metric will be used to present the user with the highest quality refinement iteration for each tilt series in a batch of run tilt series alignments.
Added: The ability to restart a batch refinement using the best iterations from the previous refinements. Often the user will find when batch refining that various iterations were aligned better than others for many tilt series in a session and the user would like to restart refinement on those iterations. Simply adding --restart_cycle=best --refresh_i3t=True to the protomo2batch.py command does this. Unfortunately Protomo then begins overwriting some files so that the depiction videos are then jumbled up, so instead of restarting from an iteration with the same i3t file, the previous run is now saved and the current run starts essentially from scratch, just with a new tlt file. Useful information from previous runs are now kept in media/quality_assessment/run## directories. Users can now essentially align tilt series blindly!
Added: Automatic CTF correction by phase flipping using defocus estimations from leginondb. Defocus estimates are queried per image from leginon as estimated by any package through Appion. Accurate defocus is determined using confidence values in the database and by filtering using 1 stdev from the mean as a threshold. Defocus values that could not be accurately determined are interpolated and phase flipping is done using Imod's ctfphaseflip (strip method). This is available in batch mode only and is all done in parallel.
Added: Round information on the top of quality assessment plots so the user can quickly determine what binning was used for which range of iterations.
Added: Mean and std, x and y labels, and legends to correlation plots.
Added: An option to refresh i3t files in batch mode. If a Protomo run is interrupted the i3t file may be unusable. This option removes the i3t file so that a new one can be created. Unfortunately if this is done Protomo will forget all of the previous iterations that have been run and so will start overwriting files beginning at iteration 1.
Added: mp4 Download button for videos when using gif depiction on webpages.

Updated: Correlation peak .img files are now removed after converting to mrcs. This slightly reduces disk space use.
Updated: Some cache files from coarse alignment are now removed after coarse alignment. These took up at least as much space as the raw images, but not any more!

To do: Fully Automated algorithm: Run for [85] iterations (20 at b8, 20 at b6, 20 at b4, 15 at b2, 10 at b1) or until convergence [<0.015> at b2] (if it hits convergence then run for [5] more iterations at b1, then take the best and reconstruct at [b8 for picking and b2 for processing] using all images except those at [highest 2 tilt angles if cofx or cofy > 1%]).
To do: Try to automate the already existent tomogram template picking that is in i3.
To do: Debug already existent screening option that can be run while collecting.
To do: Add already functional x- and y-tomogram slicethroughs to alignment workflow. This already works, I just haven't made options for it in the python scripts.
To do: Add mpi support so that the batch script can be run on hpc cpu clusters.
To do: Get the degrees symbol in the videos. This is surprisingly difficult.
To do: Add 'upload to Appion' functionality after the reconstruction and batch reconstruction webpages.
To do: Remove the 'Just Show Command' button from Summary webpages.
To do: Add docpops to iteration summary pages (where are they imported from?)

Instructions for batch alignment (use until next release).

Go to imageviewer. Choose your session. Click 'processing' up top. On the next page go to 'Align Tilt Series' and click Protomo 2.4.1.

(This next set of steps is to get parameters that align one of your tilt series from a session reasonably well)
On the next page choose a tilt series to align. These are numbered as they are in the Tomography viewer. Your output directory and runname are probably fine. Scroll through the options, changing any that you want (you don't have to change anything). Click 'Just Show Command' and run that command in a terminal that has access to the output directory and leginondb. Click 'View Coarse Alignment Results' and wait for the processing to finish. When it finishes, refresh the page. Read and follow the 'Image Info' help docs under each of the diagnostic videos. Click on the Refinement link when you're happy. On the next page with about 100 parameters, change whatever you want (most important values are thickness and lowpass diameters) or click 'Just Show Command' and run the next command. Click on the 'View Refinement Results' link while it's running and refresh the page after each subsequent iteration completes. You can look at summary pages by clicking on correlation peak videos. When you're happy, click the go to Reconstruction page link. Choose one of the iterations (doesn't matter which, we just want to create a parameter file here for use in batch processing). Choose binned by 4 or maybe 2. Run the command. Now you should have three .param in <output directory>/<runname>/ready_for_batch/. These files will be used as inputs for batch processing (the step you've been waiting for!).

On a dedicated workstation with ~4GB of ram per cpu, run this command (edit every option that has <these brackets> then remove the <brackets> - the values should be the same as on your Refinement page) to create all required directories, to go through a coarse alignment, and to go through the first 60 refinement iterations:

protomo2batch.py --sessionname=<15may04a> --rundir=</lustre/cryo/lustre/appiondata/15may04a/protomo_alignments/> --tiltseriesranges=<1-87> --link=False --procs=all --coarse_param_file=</lustre/cryo/lustre/appiondata/15apr12b/protomo_alignments/protomorun4/coarse_series0008.param> --refine_param_file=</lustre/cryo/lustre/appiondata/15apr12b/protomo_alignments/protomorun4/series0008.param> --recon_param_file=</lustre/cryo/lustre/appiondata/15apr12b/protomo_alignments/protomorun4/series0008.param> --pixelsize=<2.02698> --sampling=8 --map_sampling=8 --image_file_type=mrc --all_tilt_videos=False --all_recon_videos=False --prep_files=True --coarse_align=True --refine=True --reconstruct=False --screening_mode=False --coarse_retry_align=10 --coarse_retry_shrink=0.9 --refine_retry_align=12 --refine_retry_shrink=0.9 --region_x=<2304> --region_y=<3072> --thickness=<1000> --r1_region_x=<1920> --r2_region_x=<1920> --r3_region_x=<1920> --r1_region_y=<2560> --r2_region_y=<2560> --r3_region_y=<2560> --r1_sampling=8 --r2_sampling=6 --r3_sampling=4 --r1_mask_width_x=<360> --r2_mask_width_x=<360> --r3_mask_width_x=<360> --r1_mask_width_y=<480> --r2_mask_width_y=<480> --r3_mask_width_y=<480> --r1_lowpass_diameter_x=<15> --r2_lowpass_diameter_x=<15> --r3_lowpass_diameter_x=<15> --r1_lowpass_diameter_y=<15> --r2_lowpass_diameter_y=<15> --r3_lowpass_diameter_y=<15> --r1_highpass_diameter_x=<250> --r2_highpass_diameter_x=<350> --r3_highpass_diameter_x=<500> --r1_highpass_diameter_y=<250> --r2_highpass_diameter_y=<350> --r3_highpass_diameter_y=<500> --r1_mask_apod_x=<360> --r2_mask_apod_x=<360> --r3_mask_apod_x=<360> --r1_mask_apod_y=<480> --r2_mask_apod_y=<480> --r3_mask_apod_y=<480> --r1_lowpass_apod_x=<25> --r2_lowpass_apod_x=<15> --r3_lowpass_apod_x=<10> --r1_lowpass_apod_y=<25> --r2_lowpass_apod_y=<15> --r3_lowpass_apod_y=<10> --r1_highpass_apod_x=<75> --r2_highpass_apod_x=<100> --r3_highpass_apod_x=<150> --r1_highpass_apod_y=<75> --r2_highpass_apod_y=<100> --r3_highpass_apod_y=<150> --r1_iters=20 --r2_iters=20 --r3_iters=20 --r1_kernel_x=<360> --r2_kernel_x=<360> --r3_kernel_x=<360> --r1_kernel_y=<480> --r2_kernel_y=<480> --r3_kernel_y=<480> --r1_peak_search_radius_x=<300> --r2_peak_search_radius_x=<300> --r3_peak_search_radius_x=<300> --r1_peak_search_radius_y=<300> --r2_peak_search_radius_y=<300> --r3_peak_search_radius_y=<300> --shift_limit=30 --angle_limit=40 --dimx=<3840> --dimy=<5120>

(If you edited your raw images after leginon, add --fix_frames=True to the command...Protomo is really picky about the mrc format.)
After that finishes, run the same command to go through the next 30 iterations of refinement, but change the following options:

--prep_files=False --coarse_align=False --r1_sampling=4 --r2_sampling=2 --r3_sampling=1 --r1_iters=10 --r2_iters=20 --r3_iters=10 --fix_frames=False

and add these two options:

--restart_cycle=best --refresh_i3t=True

After all of the iterations finish, manually look at the plots in <outdir>/tiltseries####/media/quality_assessment/quality_assessment.gif to see which tilt series have iterations that dip below 0.015. Your goal is to have an iteration at binned by 2 or 1 dip below 0.015 (one of mine reached 0.005!). When this happens, that tilt series is well aligned. For tilt series that still need processing, re-run the previous command with the appropriate binning and number of iterations (note: --r1_sampling=8 --r1_iters=20 --r2_sampling=4 --r2_iters=15 means 'refine for 20 iterations at binned by 8, then 15 iterations at binned by 4; the initial 'r' means 'round'). Each time you re-run the refinement command with the two additional options above, the best iteration from the previous run will be placed in media/quality_assessment/run##/ .

(These two commands, the checking, and the restarting you have to do will optionally be included in a single, no-fuss command in the next release.)

If you want to CTF correct, make sure that you've run all of your images through a defocus estimator in Appion and that at least half of your images had confidently estimated defoci (yellow or green rings in imageviewer), and make sure you have Imod in your path. Then run this command:

protomo2batch.py --sessionname=<15may04a> --projectid=<ap20> --rundir=</lustre/cryo/lustre/appiondata/15may04a/protomo_alignments/> --tiltseriesranges=<1-87> --procs=all --pixelsize=<2.02698> --voltage=<300> --cs=<2.7> --amp_contrast=<0.07> --ctf_correct=True

Make sure you don't accidentally CTF correct twice (you can check by looking for edge ripples in a raw image)!

Post bugs here.
Contact me: ajn10d@fsu.edu

The Appion-Protomo paper has been submitted to the JSB special issue! I will commit to the svn this evening. A highlight from the paper that is in the upcoming commit is the Fully Automated Batch option. I show in the paper that using only the default values (and an alignable set of tilt-series), a completely novice user can blindly align a tilt-series session over the course of ~3 days (provided you have enough processors) such that at least half reach convergence without any user intervention.

Here are the changes since the last commit:

Added: Full Automation mode that exhaustively searches tilt geometry phase space for up to 160 iterations over 5 sampling rates (default values) or until convergence is met at a binning of 2 or 1. All the user needs to have are three parameter files from the individual tilt series processing webpages. Everything else - from microscope to reconstruction - is done automatically with a single command.
Added: Added a 5th optional round to refinements.
Updated: CTF correction now properly checks for previous CTF correction run so that you don't CTF correct more than once.
Updated: Reconstruction webpage and python script now completely works.
Fixed: CTF correction wasn't taking the tilt azimuth into account correctly.
Fixed: An off by one or more for round changing in refinement.
Fixed: The pixelsize option as inputted into ctfphaseflip was off by a factor of 10 (ctfphaseflip requires pixelsize to be in nm). Fixed this.
Fixed: Made the CCMS plot's y-axis start at 0 so it's easier to see which tilt series have the most potential when looking at multiple plots at once.
Fixed: Lots of bug fixes and code cleanup.
Updated: Made the convergence criteria 0.02 instead of 0.025, with a recommended value of 0.015 or below.
Updated: Default parameter values have been updated.
Updated: Citations now include IMOD for batch CTF correction.
Updated: protomo2aligner.log now includes timestamp in filename to avoid overwriting.
Removed: Option to link files for three reasons: 1) At some point everything needs to be y-flipped because Protomo reads bottom to top. Might as well do it from the beginning. 2) Protomo is very picky about mrc format. Mrc files made with Eman fail in Protomo. Mrc files made with Pyami work fine. Y-flipping is done in Pyami so this solves multiple issues at once. 3) It's dangerous to link what is potentially the only copy of the file. Users might inadvertently edit the images, destroying the original copy.
To do: Fully test Screening Mode. I've only had a chance to test this part of the script once and it's not looping properly. Whenever someone collects tilt series next on our microscope, I will test it again and (hopefully) get it working.
Known issue: Full automation CCMS plots sometimes get temporarily stuck. This needs to be investigated.

Edit: svn is not behaving properly...hold off on the update for now. (7-6-15 6pm ET)
Edit2: It wasn't svn, it was a problem on our end. The files from the July 6 revisions are fine I think. (7-7-15 12pm ET)

You should get the latest revision. The individual tilt-series processing workflow is now well tested and should be ready for anybody to use. A similar commit for batch processing is upcoming asap. All the Python scripts are there, I just need to make sure all of the user interface works properly. If you'd like to use the batch processing, the webpages might work. If not, refer to Update #17 above.

The release notes below allow for much easier tilt-series alignment analysis than in previous releases.

Added: Round and Switch parameter (in Protomo units) now outputs to protomo2aligner log and to screen.
Added: More critical information has been put into the CCMS plot titles for easier analysis.
Updated: For individual tilt-series processing, the image processing for the reconstruction video and tilt series video is now done in parallel.
Fixed: The last z-slice of reconstruction and correlation videos was missing (off by one).

I found a couple of bugs. 1) when trying to restart from a given iteration, I get this error.

[sstagg@krios-0-2 ~]$ protomo2aligner.py --r1_region_x=1920 --r2_region_x=1920 --r3_region_x=1920 --r4_region_x=1920 --r1_region_y=2560 --r2_region_y=2560 --r3_region_y=2560 --r4_region_y=2560 --r1_sampling=8 --r2_sampling=4 --r3_sampling=2 --r4_sampling=1 --map_sampling=8 --thickness=1000 --r1_mask_width_x=240 --r2_mask_width_x=240 --r3_mask_width_x=240 --r4_mask_width_x=240 --r1_mask_width_y=320 --r2_mask_width_y=320 --r3_mask_width_y=320 --r4_mask_width_y=320 --r1_lowpass_diameter_x=40 --r2_lowpass_diameter_x=40 --r3_lowpass_diameter_x=40 --r4_lowpass_diameter_x=40 --r1_lowpass_diameter_y=40 --r2_lowpass_diameter_y=40 --r3_lowpass_diameter_y=40 --r4_lowpass_diameter_y=40 --r1_highpass_diameter_x=560 --r2_highpass_diameter_x=560 --r3_highpass_diameter_x=560 --r4_highpass_diameter_x=560 --r1_highpass_diameter_y=560 --r2_highpass_diameter_y=560 --r3_highpass_diameter_y=560 --r4_highpass_diameter_y=560 --r1_mask_apod_x=240 --r2_mask_apod_x=240 --r3_mask_apod_x=240 --r4_mask_apod_x=240 --r1_mask_apod_y=320 --r2_mask_apod_y=320 --r3_mask_apod_y=320 --r4_mask_apod_y=320 --r1_lowpass_apod_x=16 --r2_lowpass_apod_x=16 --r3_lowpass_apod_x=16 --r4_lowpass_apod_x=16 --r1_lowpass_apod_y=16 --r2_lowpass_apod_y=16 --r3_lowpass_apod_y=16 --r4_lowpass_apod_y=16 --r1_highpass_apod_x=75 --r2_highpass_apod_x=100 --r3_highpass_apod_x=150 --r4_highpass_apod_x=200 --r1_highpass_apod_y=75 --r2_highpass_apod_y=100 --r3_highpass_apod_y=150 --r4_highpass_apod_y=200 --corr_mode=mcf --r1_iters=15 --r2_iters=10 --r3_iters=5 --r4_iters=5 --binning=true --preprocessing=true --border=100 --clip_low=3.5 --clip_high=3.5 --gradient=true --iter_gradient=false --filter=median --r1_kernel_x=5 --r2_kernel_x=5 --r3_kernel_x=5 --r4_kernel_x=5 --r1_kernel_y=5 --r2_kernel_y=5 --r3_kernel_y=5 --r4_kernel_y=5 --grow=0 --window_area=0.95 --do_estimation=false --norotations=false --correlation_size_x=160 --correlation_size_y=160 --r1_peak_search_radius_x=300 --r2_peak_search_radius_x=300 --r3_peak_search_radius_x=300 --r4_peak_search_radius_x=300 --r1_peak_search_radius_y=300 --r2_peak_search_radius_y=300 --r3_peak_search_radius_y=300 --r4_peak_search_radius_y=300 --orientation=true --azimuth=true --elevation=false --rotation=true --scale=false --logging=true --loglevel=2 --retry=20 --retry_shrink=0.9 --create_tilt_video=true --create_reconstruction=false --keep_recons=false --map_size_x=480 --map_size_y=640 --map_size_z=200 --video_type=html5vid --restart_cycle=2 --runname=protomorun6 --rundir=/lustre/cryo/lustre/appiondata/15apr12b/protomo_alignments/protomorun6 --tiltseries=8 --sessionname=15apr12b --pixelsize=2.02698 --dimx=3840 --dimy=5120 --image_file_type=mrc --coarse=False --projectid=51 --expid=8021 --jobtype=protomo
/usr/lib64/python2.6/site-packages/numpy/oldnumeric/__init__.py:11: ModuleDeprecationWarning: The oldnumeric module will be dropped in Numpy 1.9
warnings.warn(_msg, ModuleDeprecationWarning)
... Time stamp: 15jul08j52
... Function name: protomo2aligner
... Writing function log to: protomo2aligner.log
... Running on host: krios-0-2.local
... Looking up session, 15apr12b
... Looking up session first, 8021
... getting imagelist
... highest tilt angle is 59.989201
... Starting Protomo alignment
protomo: opening tilt series "series0008"
protomo: cycle 08
... Beginning Refinements
... No Round parameters changed for Iteration #1

Traceback (most recent call last):
File "/panfs/storage.local/imb/stagg/software/myami_dev/appion/bin/protomo2aligner.py", line 1321, in <module>
protomo2aligner.start()
File "/panfs/storage.local/imb/stagg/software/myami_dev/appion/bin/protomo2aligner.py", line 1165, in start
apDisplay.printMsg("Beginning Refinement Iteration #%s, Round #%s\n" % (n+1,r))
UnboundLocalError: local variable 'r' referenced before assignment

2) When starting a refinement from the beginning, you need to add a line to the report webpage where it gets you to erase the current .i3t file.

Another issue: In the refinement, when depiction movies are made of the reconstructions, the x,y, and z dimensions should be determined from the window size and thickness parameters in the refinement. The purpose of those videos is to help monitor the refinement, and diagnose problems. Many of those problems can be determined by looking at the videos, but this is not possible if the videos are not reflecting what is going on in the refinement.

This release has useful new features. Get it! Also, bug 2) from Update #20 is fixed. Feature request in Update #21 is listed as the first addition.

Added: The z-height of the preliminary reconstructions is now determined to be twice the thickness. The window size is now shown in the reconstruction videos, allowing for much easier identification of alignment issues.
Added: Round and Switch parameter (in Protomo units) now outputs to protomo2aligner log and to screen.
Added: More critical information has been put into the CCMS plot titles for easier analysis.
Updated: To accommodate for Protomo's pickiness in image datatype, after y-flipping during File Preparation, images are now normalized and their datatype is now converted to float32.
Updated: For individual tilt-series processing, the image processing for the reconstruction video and tilt-series video is now done in parallel.
Updated: The list of files to be removed if restarting a refinement.
Fixed: All of the math done in the param files is now done explicitly in Python to ensure proper variables for each iteration. Before this release the bandpass and body size values weren't being calculated properly.
Fixed: The last z-slice of reconstruction and correlation videos was missing (off by one).

This written update encompasses the updates since the last written update. The most exciting update is the addition of modular presets to the Refinement webpage. This lowers the required knowledge for running a customized Protomo run to nearly zero!

Added: Presets to Refinement. Now things can be super easy. There are now three modular preset dropdown menus that allow you to easily change all critical values in the ~100 parameter form to conform to the sample in your tilt-series, the robustness of alignments, and the accuracy of the alignments.
Added: Correlation type is now included as a set of Round parameters.
Added: Tilt azimuth and tilt angle refinement plots are now made and included on the Refinement summary page so the user can track their convergence.
Updated: Window shrinking for restarting failed alignments is now done without any user input. Before this update the new window size was determined by shrinking a user-given percentage up to a user-given number of times. Now the shrunken values are determined from a list of numbers that have prime factors no greater than 7. This update eliminates unnecessary user intervention and chooses dimensions that expedite FFTing.
Updated: Help files have been significantly improved.
Fixed: Log files used to conflict if multiple protomo2aligner.py commands were run in the same directory at the same time. This should be fixed.
Fixed: Filter values are now (hopefully) correct and predictive.
Fixed?: Restarting Refinements (ref. Update 20, 1) ) might be working now, but I haven't had time to check it. I wouldn't be surprised if something is still awry.

There have been so many updates, I probably will miss some. I have the python side working for non-Leginon collected tilt-series, but there's one amazingly convoluted hurdle I haven't yet gotten over: PHP.

Added: Dose compensation using equation (3) from Grant & Grigorieff, 2015. The user can either use one of three presets or input their own values for a, b, and c.
Added: Two plots for dose compensation: One showing the accumulated dose as a function of tilt angle, and the other showing the lowpass filter applied as a function of dose.
Added: Two plots for CTF correction: One showing the measured and interpolated defocus values over all tilt angles, and the other showing a plot of the CTF^2 using the average defocus value and attenuated by estimated envelope functions.
Added: You can now limit your final reconstructions to exclude high tilts.
Added: Reconstruction webpages assist you in determining which high tilt images you may wish to exclude by showing the correction factor plots for the chosen reconstruction iteration.
Added: Plots for angular refinements (tilt elevation, orientation angles, tilt azimuth) for all iterations are now included on the refinement summary page.
Added: Settings verbosity. The user can choose Express (just presets), Basic (Express + parameters that presets change), or Advanced (Everything).
Added: Presets to Coarse Alignment.
Added: Now CCMS values are calculated for shifts, rotations, and scalings. A scaled sum of CCMS values is now outputted for each iteration along with each of the new CCMS values.
Added: Option to turn off parallelization in individual tilt-series workflow. This otherwise breaks systems that run out of memory:(
Added: Binning, lowpass, and thickness are now written to the bottom of the reconstruction videos.
Added: Protomo's 'slab' option to Coarse Alignment and Refinement. I've included documentation from Hanspeter and my private discussions because the Protomo user guide doesn't explain anything.
Added: Grant & Grigorieff dose compensation paper to the citations list.
Added: Appion-Protomo paper to the citation lists.

Updated: CTF correction and dose compensation can be performed before Coarse Alignment or before Reconstruction from the respective individual tilt-series processing webpages.
Updated: CCMS plot now includes all of the CCMS values and their scaled sum. The scaled sum should be used along with convergence of the angular refinements to evaluate tilt-series alignment quality.
Updated: Plots for individual refinement iterations now include both image number and image tilt angle on the x-axes.
Updated: Reconstruction in individual tilt-series alignment workflow now leaves all of the alignment files alone and reconstructs in a subdirectory.
Updated: PHP webpages are now confiigured (or so I think) to time the session out after one week, instead of whenever garbage collection randomly occurs.
Updated: The size of the reconstructions is now only dependent on the requested sampling. Before, if the user wanted to change the sampling of the reconstruction they also had to change the x and y dimensions.
Updated: Removed the useless 'Relative angle between correction factors x and y' plot and replaced it with the useful 'Correction Factors for Scaling' plot.
Updated: The Protomo option to select certain images has been removed because it is redundant. The exclude images option remains.
Updated: The Protomo exclude images option has been replaced by an option of the same name that instead removes the excluded images from the .tlt file. I don't trust Protomo.
Updated: Removed the Voltage and Spherical Aberration parameters from CTF correction on the user's side. These are now queried from the database in the python.
Updated: Removed 'Alternate Raw Image Path' option in Coarse Alignment and Refinement. This will effectively replaced by the manual tilt-series option.
Updated: Going back to the 'Align Tilt-Series' page clears all variables.
Updated: Made import errors into apDisplay Warnings or Errors.
Updated: Some code cleanup.
Updated: Lots of help doc updates.

Fixed: Added os.environ["MAGICK_THREAD_LIMIT"] = "1" to before all Imagemagick 'convert' calls. 'convert' by default uses as many threads as possible. Some installations of Imagemagick throw an error if convert determines that there are insufficient resources. This causes some images not to be produced properly. Setting this environmental variable tells convert to only use one thread, which solves the issue. This doesn't hurt performance because the steps that use convert are already parallelized.

Batch mode is now fully implemented. Batch mode is essentially modular. Batch functions include:

File Preparation
Coarse Alignment
Refinement
Reconstruction
CTF Correction
Dose Compensation
Screening Mode
Fully Automated

I'll be making a Wiki page and a 10+ part video tutorial next.