Appion

1.2 I think the polygon should be loaded into the gui when it pops up with the particular image

2.1.1 and 2.1.2 - I think these should be lower priority, and maybe not a priority at all. It would be great to have fancy tools to display and view the masks, but the important thing is to have a way of automating the way that the gui viewer reads and displays the polygon to the user, such that the user can then accept / reject / or adjust the polygon.

2.3 - I don't know how this is currently done with, for example, the autotiltpairaligner and manualtiltpairaligner, but I would imagine that it should be modeled on that

2.3-3.2 - the only problem that I can gather from this strategy is that you would have to go through the "bad" images twice - once for marking them as "redo", and again for actually redoing. Perhaps a better way to do it would be to simply adjust the mask in place instead of marking it as rejected and then adjusting the mask in a separate session. This is how its done in autotiltpairaligner / manualtiltpairaligner.

Thanks for the feedback Dmitry.
Bridget, Anchi, Arne and I met on Monday to discuss the implementation, and one of the issues that Arne stressed about using the manual masker is that it takes a very long time for each image to load. So we thought it would be faster to quickly run through the images in the image viewer with the mask regions displayed and allow the user to mark for redo the images that did not get properly masked. Bridget estimated that about 5% of the images would need to be redone. Then the user could just use the manual masker on 5% of the images. Bridget also said that when the auto masker fails, it fails completely. So she thinks that it would be easier to make a new mask from scratch rather than fiddle around editing an entirely wrong mask. This is why we thought we could just discard masks that the user marked for redo and start from scratch in the manual masker.

After this meeting Anchi and I continued the discussion regarding how to display the mask regions in the image viewer, and it seemed that having the polygons may be easier, since we could use the same framework that is in place for showing particle picks. However, mask regions from crud finding and manual masking are not in polygons, so it could be better in the long run to add the ability to display regions in image viewer based on mask images rather than region polygons. We have not determined the best solution for this yet, which is why I have listed all the options.

Let me know if you have any questions on this, or further thoughts on the workflow based on this...

Sargis,
Could you take a quick look at what it will take to install this on Guppy to see if you could do it this morning?
Thanks!

Downloaded and unpacked em_hole_finder code from https://github.com/hbradlow/em_hole_finder to guppy head node under /opt. /opt is cross-mounted on all guppy work nodes as well, so once I have em_hole_finder installed and working on guppy head node, I'll repeat these steps on guppy work nodes as well:

Here are the steps I did to install em_hole_finder in guppy head node:

[root@guppy em_hole_finder]# easy_install pip
[root@guppy em_hole_finder]# pip install -r requirements.txt

...
    C compiler: gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC

    compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.6 -c'
    gcc: _configtest.c
    gcc -pthread _configtest.o -o _configtest
    _configtest
    failure.
    removing: _configtest.c _configtest.o _configtest
    building data_files sources
    build_src: building npy-pkg config files
Downloading/unpacking scikit-image==0.8.2 (from -r requirements.txt (line 5))
  Running setup.py egg_info for package scikit-image
    Traceback (most recent call last):
      File "<string>", line 16, in <module>
      File "/tmp/pip-build-root/scikit-image/setup.py", line 108, in <module>
        check_requirements()
      File "/tmp/pip-build-root/scikit-image/setup.py", line 103, in check_requirements
        % ((package_name, ) + min_version))
    ImportError: You need `Cython` version 0.15 or later.
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 16, in <module>

  File "/tmp/pip-build-root/scikit-image/setup.py", line 108, in <module>

    check_requirements()

  File "/tmp/pip-build-root/scikit-image/setup.py", line 103, in check_requirements

    % ((package_name, ) + min_version))

ImportError: You need `Cython` version 0.15 or later.

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-root/scikit-image
Storing complete log in /root/.pip/pip.log

cd /tmp/pip-build-root/Cython
python setup.py install

ImportError: You need `numpy` version 1.6 or later.

[root@guppy em_hole_finder]# rpm -qa|grep numpy
numpy-f2py-1.4.1-9.el6.x86_64
numpy-1.4.1-9.el6.x86_64

Installed em_hole_finder on guppy head and work nodes. Install note reads pip install -r requirements.txt, but this wasn't working on guppy. I had to run the following commands to install em_hole_finder:

  yum install python-virtualenv
  cd /opt/em_hole_finder
  virtualenv env
  source env/bin/activate
  yum install lapack-devel.x86_64
  ~/safadmin yum install lapack-devel.x86_64 -y
  pip install numpy
  pip install Cython
  pip install PIL
  pip install scipy
  pip install scikit-image
  pip install ipython
  pip install wsgiref

Run the following command to verify that installation was successful:

(env)[root@guppy em_hole_finder]# pwd
/opt/em_hole_finder
(env)[root@guppy em_hole_finder]# python test.py
Took 10.0 seconds to complete

source /opt/em_hole_finder/env/bin/activate.csh #csh
or
source /opt/em_hole_finder/env/bin/activate #bash

Hey Sargis,
I'm trying to test the run step of hole finder, but get this:

amber@guppy ~] /ami/data00/dev/amber/appion runJob.py /ami/data00/dev/amber/appion automasker.py --projectid=303 --preset=en --session=zz07jul25b --runname=maskrun8 --rundir=/ami/data00/appion/zz07jul25b/mask/maskrun8 --no-rejects --no-wait --commit --limit=10 --continue --expid=8556 --jobtype=maskmaker --ppn=1 --nodes=1 --walltime=240 --jobid=1687 
Traceback (most recent call last):
  File "/ami/data00/dev/amber/myami/appion/bin/runJob.py", line 5, in <module>
    from appionlib import apAgent
  File "/ami/data00/dev/amber/myami/appion/appionlib/apAgent.py", line 4, in <module>
    from appionlib import apRefineJobXmipp
  File "/ami/data00/dev/amber/myami/appion/appionlib/apRefineJobXmipp.py", line 10, in <module>
    from appionlib import apSymmetry
  File "/ami/data00/dev/amber/myami/appion/appionlib/apSymmetry.py", line 8, in <module>
    from appionlib import appiondata
  File "/ami/data00/dev/amber/myami/appion/appionlib/appiondata.py", line 7, in <module>
    import sinedon.data
  File "/opt/myamisnap/lib/sinedon/__init__.py", line 23, in <module>
    from data import Data
  File "/opt/myamisnap/lib/sinedon/data.py", line 12, in <module>
    import dbdatakeeper
  File "/opt/myamisnap/lib/sinedon/dbdatakeeper.py", line 10, in <module>
    import sqldict
  File "/opt/myamisnap/lib/sinedon/sqldict.py", line 151, in <module>
    import sqlexpr
  File "/opt/myamisnap/lib/sinedon/sqlexpr.py", line 56, in <module>
    import MySQLdb
ImportError: No module named MySQLdb

I get the same error just running dog picker. Looks like sinedon needs to be included in the environment, but it's not clear to me how I should modify the virtual environment. Can you help out with getting this to run?

Hey Amber,

I'm getting:

[sargis@guppy ~]$ /ami/data00/dev/amber/appion runJob.py /ami/data00/dev/amber/appion automasker.py --projectid=303 --preset=en --session=zz07jul25b --runname=maskrun8 --rundir=/ami/data00/appion/zz07jul25b/mask/maskrun8 --no-rejects --no-wait --commit --limit=10 --continue --expid=8556 --jobtype=maskmaker --ppn=1 --nodes=1 --walltime=240 --jobid=1687 
Appion config file : /ami/data00/dev/amber/myami/.appion.cfg doesn't exist.  Can't setup processing host

I sourced the environment in my .cshrc file

source /opt/em_hole_finder/env/bin/activate.csh

My .appion.cfg file is located in my home directory. Since you were not running as me, neither of those were in place when you tried the command.
Should I just be activating the environment right before calling the em_hole_finder command in appion loop? Would that work? I'm using a subprocess to call find_mask.py from the Appion script for each image. I thought the subprocess environment is inheriting from the parent, but I can check to see if there is a way to set up a new environment.

r17638 adds the bulk of the masking functionality that we have discussed.

This should be available from http://longboard.scripps.edu/betamyamiweb starting tomorrow. Redux will need to be updated and restarted on cronus3 before it will work there.

1. Go to "Region Mask Creation" in the Appion Pipeline.
2. Select "Run Automated Masking"
3. Select "Auto Masking"
4. Press the Run Auto Masker button to begin the run. Note that this will only run on Guppy at the moment.
5. When the run is complete, you can view the results by clicking on the "# complete" link below the Automated Masking menu item.
6. Click through the run name link on a couple screens and you will get to the multi-image assessor, which should automatically load the masking run you selected. Auto-masked regions appear darker.
7. Change the "Mask Assessment Status" to "reject" for any images where the auto masking tool failed. Rejected masks should not be used during stack creation.
8. When creating a stack, select the completed auto masking run. Particles within non-rejected mask regions should not be included in the stack.

1. The results link on the pipeline menu puts all masking run results (Automated and Manual) under the Automated section.
2. The "Run Automated Masking selection page does not have appropriate descriptions for the various masking methods.
3. The auto masker is only installed on Guppy and setting up the environment and command is hard coded for Guppy.
4. The environment setup is targeted to csh and not bash. Still need to do some shell detection to choose the right syntax.
5. Need to complete changes to the manual masker, which will allow a user to load only images that do not have accepted mask regions and manually mask the images that the auto masker failed on.
6. Need to make modifications to the image viewer to display mask regions and allow the user to reject them from there.

1. loadimg.php included a line that did not seem to be working with redux, particular to images needing resizing. I'm bypassing that, but I may have broken other things...need to do some searches and testing.
2. I made many changes to apMask.py and apCrud.py to functions that are also used for manual masking, so I may have introduced bugs to manual masking.
3. I made many changes to muli-image assessor, so if anyone was using it before, it may not be exactly the same.

Let me know of any issues or concerns. There is still a good deal of work and cleanup to do on this.

r17648 adds the ability to run the manual masker on images with rejected auto masks.

1. run the auto masker as stated in the previous update.
2. in the appion pipeline, go to Region Mask Creation->Run Manual Masking
3. under "Accept the mask region as" section of the parameters, select the 2nd radio button, "Combined Run with Existing Assessment Run".
4. using the drop down list adjacent to this radio button, select the name of the original automated masking run that you have already assessed (selected Keep or Reject status) with the Multi Image Assessor.
5. copy and paste the command to run. You should only see images that do not have an accepted mask from the previous auto mask run. This includes images where no mask was created in the auto run, and images where a mask was created, but you rejected it with the multi image assessor.
6. When you complete the manual run, you can create a stack. When creating the stack, at the "Mask Assessment" drop down list, select the name of the ORIGINAL automated masking run. (The manual run assessment was added to the original auto run, rather creating a new mask assessment run in the database.)

1. Try adding mask assessment capabilities to the image viewer.
2. Clean up the code.

This is a really great addition to the pipeline! There are already 3 people that have successfully used it on their data, including myself, and I'm sure this will be one of the most popular tools in the pipeline. Based on the experience that I had, here are my suggestions:

1) the manual run after the auto-run can be made faster. Currently, it still goes through all the images, although you ONLY have to mask the ones that were rejected. As it stands, it is often the case that it will go through, e.g. 5 images (which would take ~30 seconds or so), and then ask you to mask the one that hadn't been done. In practice, this means that you still have to wait for the script to go through all of the images, and only once in a while make your mask. It would be great if either (1) the script could go through the non-rejected images in the beginning or (2) not go through them at all, and just have the information duplicated from the database, or something of that matter (better, I think).

2) The way in which the masks are displayed are different between the manual and auto runs. In the manual run, the mask is white on black, but in the auto-run the mask is black on white. To me it makes sense that the mask is black on white, because the black is 0, but the white is 1 (i.e. as it is in the auto-run).

3) I don't quite understand how the masks are displayed in the 2nd run (the manual run). In it, I see masks from the manual run, combined with masks from the auto-run, as well as masks that shouldn't be there (i.e. where the auto run result was rejected, and NO mask was applied in the manual run).

Dmitry

There are several input parameters that could be added (and made available in an advanced GUI) that might help reduce failures. Or it could be run several times using different parameters and see if some of the failures succeed using a different set of parameters. It really needs to be "testable" just like the particle finders.
I am putting the parameter descriptions here for the record:

Downsampling rate: https://github.com/hbradlow/em_hole_finder/blob/master/find_mask.py#L22
Component size thresholding: https://github.com/hbradlow/em_hole_finder/blob/master/pipeline.py#L328
Adaptive thresholding factor: https://github.com/hbradlow/em_hole_finder/blob/master/pipeline.py#L272
Blur factor: https://github.com/hbradlow/em_hole_finder/blob/master/pipeline.py#L250
Dilation factor: https://github.com/hbradlow/em_hole_finder/blob/master/pipeline.py#L21
Erosion factor: https://github.com/hbradlow/em_hole_finder/blob/master/pipeline.py#L22

Dmitry,
Regarding your point number 1, I just checked in changes to the manual masker to put the mask checking on the front end, reducing time between images while actively masking.
Give it a try and let me know if it needs adjustments.

Point 2 is part of the code cleanup that I need to take care of still.

1. Run the Auto Masker
2. Assess the auto masker results
3. Run the Manual Masker using the auto mask run as the assess name (one time only!)
4. Make a stack with the Auto mask run name

In other words, running the manual masker with the same auto assess name more than once is bad!

I'll start work on exposing the adjustable parameters that Bridget has listed now.

r17673 exposes the em_hole_finder's input parameters.
Still need to implement the "Test on a single image" function.

r17674 adds ability to test on a single image.

TODO: as stated in #2437, we need to set a default downsample size in the GUI that makes sense for the image shape.

13apr01w is a good test session.