Appion

I am already starting the web part : runTomoAligner.php for user input the values. I will also do the result page.
Amber, Please update the python part. The file need to update is tomoaligner.py

We need to finish this before Friday, because Hanspeter only be here this week.

Thanks.

Eric

There is a new project for us to do development. The project name is protomo tutorial data.
All the example images has been upload to session : 10nov10z

We've added a few parameters that need to be sent from the PHP to the python file:

self.parser.add_option("--windowsize_x", dest="windowsize_x",  type="int",
            help="Protomo2 only: Region of interest in the x dimension used for alignmnet (in pixels), e.g. --windowsize_x=1024", metavar="int")

self.parser.add_option("--windowsize_y", dest="windowsize_y",  type="int",
            help="Protomo2 only: Region of interest in the y dimension used for alignmnet (in pixels), e.g. --windowsize_y=1024", metavar="int")

self.parser.add_option("--lowpass_diameter_x", dest="lowpass_diameter_x",  type="float",
            help="Protomo2 only: TODO: To be determined (in reciprical pixels), e.g. --lowpass_diameter_x=0.4", metavar="float")

self.parser.add_option("--lowpass_diameter_y", dest="lowpass_diameter_y",  type="float",
            help="Protomo2 only: TODO: To be determined (in reciprical pixels), e.g. --lowpass_diameter_y=0.4", metavar="float")

self.parser.add_option("--highpass_diameter_x", dest="highpass_diameter_x",  type="float",
            help="Protomo2 only: TODO: To be determined (in reciprical pixels), e.g. --highpass_diameter_x=0.02", metavar="float")

self.parser.add_option("--highpass_diameter_y", dest="highpass_diameter_y",  type="float",
            help="Protomo2 only: TODO: To be determined (in reciprical pixels), e.g. --highpass_diameter_y=0.02", metavar="float")

self.parser.add_option("--backprojection_bodysize", dest="backprojection_bodysize",  type="float",
            help="Protomo2 only: Back projection body size for the reconstruction (in pixels), e.g. --backprojection_bodysize=80.0", metavar="float")

self.parser.add_option("--protomo2paramfile", dest="protomo2paramfile",  type="FILE",
            help="Protomo2 only: Path and filename of the protomo2 parameter file, e.g. --protomo2paramfile=/path/to/max.param", metavar="FILE")

self.parser.add_option("--max_iterations", dest="max_iterations",  type="int",
            help="Protomo2 only: Number of alignment and geometry refinement iterations, e.g. --max_iterations=4", metavar="int")

The basic requirement variables for basic setting for Protomo 2 is nwo in svn, but the validation is not done yet (only check empty now). For now, make sure you enter the valid values for them.

Couple things need to do:
1. inputs validation
2. advance option (still have some more input boxes need to created).
3. user upload the config file and website will automatic populate all the values in the boxes.

Thanks.

Eric

Here is the example of the command look like with basic setup:
tomoaligner.py --session=10nov10z --tiltseriesnumber=1 --projectid=291 --runname=align1 --alignmethod=protomo2 --sample=2 --windowsize_x=500 --windowsize_y=500 --lowpass_diameter_x=0.40 --lowpass_diameter_y=0.40 --highpass_diameter_x=0.40 --highpass_diameter_y=0.40 --backprojection_bodysize=82.50 --max_iterations= --description="testing" --commit

1. protomo needs python 2.6, we are running 2.4. Christopher installed 2.6 at /ami/sw/packages/Python2.6/bin/python along with numpy,scipy and PIL to us going.
2. Hanspeter needs to make some modifications to accept string input for paths. Right now they are hard coded in the config file.
3. We need to write out a config file template in the python wrapper, rather than relying on the file being present.
4. The images can't start with a number. We prepend numbers to uploaded images. We copied his images to a new directory (/ami/data00/leginon/10nov10z/rawdatatest) and changed the file names to be compliant. Hanspeter will need to modify his code to allow the numbers.
5. We have hard coded a path to a geometry file (home/amber/max.tlt). This needs to be built in the python script using values from the database.
6. Hanspeter needs to implement an automatically generated filename for the correction factors data file. It is currently hard coded.

• ssh to amibox02
• set the Path variables:
  (use my workspace for appion files)
  setenv PYTHONPATH /home/amber/amiworkspace/myami:/home/amber/amiworkspace/myami/appion:/usr/local/EMAN/lib
  setenv APPTIONDIR /home/amber/amiworkspace/myami/appion
  setenv PATH /home/amber/amiworkspace/myami/appion/bin:${PATH}
  (use python 2.6)
  setenv PATH /ami/sw/packages/Python2.6/bin:${PATH}
  (use the new protomo files)
  setenv I3ROOT /ami/sw/packages/PROTOMO
  setenv PATH ${I3ROOT}/bin:${I3ROOT}/bin/linux/x86-64:${PATH}
  setenv LD_LIBRARY_PATH ${I3ROOT}/lib/linux/x86-64
  setenv PYTHONPATH ${I3ROOT}/lib/linux/x86-64:${PYTHONPATH}
• run the command
  

  /home/amber/amiworkspace/myami/appion/bin/tomoaligner.py --session=10nov10z --tiltseriesnumber=1 --projectid=291 --runname=align1 --alignmethod=protomo2 --sample=2 --windowsize_x=500 --windowsize_y=500 --lowpass_diameter_x=0.40 --lowpass_diameter_y=0.40 --highpass_diameter_x=0.40 --highpass_diameter_y=0.40 --backprojection_bodysize=82.50 --max_iterations=2 --description="testing protomo2" --commit --protomo2paramfile=/home/amber/max.param --rundir=/ami/data17/appion/10nov10z/rawdata/tiltseries1/align/align1
  

• check the results in /ami/data00/appion/10nov10z/tomo/tiltseries1/align/align1/out/

Hanspeter,
I'd like to update our version of Protomo so I can make the points 2 and 6 changes, but the link you provided is no longer valid. Could you repost it please?
Thanks,
Amber

The updated versions are at
http://www.sb.fsu.edu/~winkler/protomo/protomo-2.0.10.tar.bz2
http://www.sb.fsu.edu/~winkler/protomo/protomo-tutorial-2.0.10.pdf

In section 4.5 of the PDF file you can find the commands to write out the *.tlt

The file correctionFactors.dat is only needed if you want to generate a plot and it should be written out prior to processing. It is a text file and the columns contain the following data:
(1) image number
(2) rotation
(3) and (4) stretch/compression factors in two orthogonal directions
(5) angle of stretch/compression
(6) isotropic scale factor

I usually only look at the plot of (3) and (4)

Hanspeter, I tried to run without the .dat file and it would not run. Do I need to set a parameter to allow it to run without it?

The procedure should run without the *.dat file. It is not needed by any of my python functions, the problem is most likely in the appion code.

Hanspeter, did the format of the max.param file change since 2.0.5?
I'm getting the following error when using 2.0.10:
protomo.error: undefined parameter

from the following line of code:
protomo2param.set("lowpass.diameter", "{ %d %d }" % (self.params['lowpass_diameter_x'], self.params['lowpass_diameter_y']))

The format hasn't changed, but I think the parameter name is incomplete, it should be
window.lowpass.diameter

Hanspeter, you were right, adding "window" fixed that one. Now I'm getting an error in the align() function. It doesn't seem to like the apodization parameter, however I have not changed the parameter file that worked with 2.0.5. The param file that I'm using is already attached to this issue.

File "/home/amber/amiworkspace/myami/appion/appionlib/apProTomo2.py", line 97, in run
    protomo2series.align()
protomo.error: invalid parameter, apodization, section tiltseries.window.lowpass
  invalid parameter, apodization, section tiltseries.window.highpass

Any thoughts?

I have never seen this error, but it could occur if window.lowpass.diameter is undefined, or if window.lowpass.sigma is set, which is incompatible with apodization. Same for window.highpass.

Hanspeter,
I'm trying to set string parameters which was not supported in the first version, but I believe is in the version I'm using now. I'm able to set directories with string values, but the following parameters are not working for me. Should these be set with something other than a string? I tried "binning" with literal "true" and "True" and 1, with no luck.

protomo2param.set("binning", '"%s"' % (self.params['do_binning'],) ) # set to "True"
protomo2param.set("preprocessing", '"%s"' % (self.params['do_preprocessing'],) )
protomo2param.set("align.estimate", '"%s"' % (self.params['do_estimation'],) )
protomo2param.set("correlation.mode", '"%s"' % (self.params['correlation_mode'],) )

For boolean variables, use the strings "true" or "false" (all lowercase).
The parameter "correlation.mode" should probably be "align.correlation.mode", valid string values are "xcf", "mcf", "pcf", and "dbl". The value is not checked at this point, but you get an error later when you run the alignment if it is anything else.

If it still doesn't work, what are the error messages?

Checked in progress so far with r15609. Able to set all parameters without error. Problem was that strings need to be wrapped in quotes when being passed to protomo2, but the strings that are representing boolean true and false can't have quotes.
Now getting hung up in the protomo2series.align() function with error message:

protomo: creating tilt series "max" 
Traceback (most recent call last):
  File "/home/amber/amiworkspace/myami/appion/bin/tomoaligner.py", line 484, in <module>
    app.start()
  File "/home/amber/amiworkspace/myami/appion/bin/tomoaligner.py", line 356, in start
    self.runProtomo2(sessiondata, processdir, seriesname, ordered_imagelist, refimg, center, corr_bin, commit, tilts)
  File "/home/amber/amiworkspace/myami/appion/bin/tomoaligner.py", line 299, in runProtomo2
    apProtomo2.run()
  File "/home/amber/amiworkspace/myami/appion/appionlib/apProTomo2.py", line 98, in run
    protomo2series.align()
protomo.error: invalid value, mode, section tiltseries.align.correlation

The tiltseries.align.correlation value is being set to "mcf".

I suspect that this has something to do with the parameter names. In issue #21, did you specify "tiltseries" in front of the names? When I run this from the python command line, it is not needed. Also, the actual name should be "align.correlation.mode". You can construct the names by looking at the parameter file and concatenating the identifiers, for example

align { correlation { mode: "mcf" } }

would result in "align.correlation.mode"

AHA! that is very useful information on how to structure the set() function call for parameters. That should definatly go in your user guide.

Yes, I changed correlation.mode to align.correlation.mode. I have not specified tiltseries in front of any names. Just tried adding it and I got an undefined parameter error.

You can see the code that I am using by clicking on the revision link r15609. Then click on the file apProtomo2.py. Parameters are set in setUserModifiedParams().

I have added "align." in front of correlation.size and peaksearch.radius in my sandbox to comply with your naming convention. After changing these, I am still getting the error specified above.

I am not so familiar with python, are these two statements equivalent
(assuming that the stored values are 128 128):

protomo2param.set("align.correlation.size", "{ %d %d }" % (self.params['correlation_size_x'], self.params['correlation_size_y']))

protomo2param.set( "align.correlation.size", "{128 128}" )

Something else came to my mind: Do you create the series object before or after running setUserModifiedParams? If you do it before, then you have to replace the parameters in the series object, because it stores a copy of the parameter object before the modifications:
seriesobject.param( protomo2param )

Yes, those are equivilant statements, provided the values in self.params['correlation_size_x'] and self.params['correlation_size_y'] are 128. Here is my order of execution:

        # use the tilt angles to set the reference image, which is the image with tilt values closest to zero.        

        # Write a default parameter file if it does not already exist

        # Create the parameter object

        # Write a default tilt geometry file if it does not already exist 

        # Create the protomo2 geometry object

        # Modify the parameter object with the user input values
        self.setUserModifiedParams( protomo2param )

        # Create a tilt series object (contains input geometry, creates updated geometry, info to do the alignment)
        protomo2series = protomo.series(protomo2param, protomo2geom)

        # Run the specified number of alignment iterations
        for i in range(0, self.params['max_iterations']):
            protomo2series.align()
            # correction factors, automatically generates a file name using the iteration number(? TODO: Hanspeter to implement)
            protomo2series.corr() #( self.correctionFactorFile )
            protomo2series.fit()
            protomo2series.update()

The code is in the run() function of apProtomo2.py. Looks like I set the params then create the series object.

Before you call align(), can you print the parameter that produces an error later to check if it is set properly:
protomo2series.setparam("align.correlation.size")
This prints it to stderr