Appion

this should be fixed now in r17515

I'll put in a request on Monday to update the myami package on Garibaldi.
What is added to the .cshrc file?

this also doesn't work on guppy...

could someone walk me through source gabes .cshrc on garibaldi, please? I tried creating the command through appion through 'just show command', then starting an interactive session on garibaldi, then sourcing /home/glander/.cshrc and then submitting and it doesn't seem to work as it says " $IMAGIC_ROOT directory is not specified."

thanks!!

Melody - if you log in to garibaldi & just load the imagic module, and then echo $IMAGIC_ROOT, does that show you "/opt/applications/imagic/100312"
My .cshrc is just setting the environment variables to use my local install of myami.

awesome, this works, thanks so much gabe!! have a good weekend!!

Hey Gabe,
Do have any time today to come chat with me about this? Guppy is not working, and the imagic version has not changed.

Hi,

On April 16th some change occurred between 13:31 and 17:45.

Here's the last imagic msa that worked for me: /ami/data15/appion/13jan26d/align/imagicmsa17
and here's the first one that DIDN'T work: /ami/data15/appion/13jan26d/align/imagicmsa18

Cheers,
Melody

Here is one that I ran on garibaldi yesterday, generating the command through longboard/beta, then pasting into garibaldi after starting an interactive session on a myami node, loading the imagic module, and sourcing gabe's cshrc.

/ami/data15/appion/13mar22c/align/imagicmsa-bin4

Ok I think we got to the bottom of this,
I'm incorporating the changes that will be necessary if we ever update our version of imagic.
r17527

Well, according to: http://www.open-mpi.org/faq/?category=building#build-rte-tm
I should be able to test if the torque enabled version of mpi is used by looking for the following components:

shell$ ompi_info | grep tm
                 MCA pls: tm (MCA v1.0, API v1.0, Component v1.0)
                 MCA ras: tm (MCA v1.0, API v1.0, Component v1.0)

We came to the conclusion that garibaldi works and guppy does not because they have different versions of open mpi, however,

garibaldi00 home/amber> ompi_info | grep tm
              MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.3)
                 MCA ras: tm (MCA v2.0, API v2.0, Component v1.4.3)
                 MCA plm: tm (MCA v2.0, API v2.0, Component v1.4.3)

amber@guppy ~] ompi_info | grep tm
                 MCA ras: tm (MCA v2.0, API v2.0, Component v1.6.4)
                 MCA plm: tm (MCA v2.0, API v2.0, Component v1.6.4)
                 MCA ess: tm (MCA v2.0, API v2.0, Component v1.6.4)

it looks like they are both using torque enabled versions. Or perhaps they are different versions but I'm not sure how to find out?

Sargis, do you have any ideas on how to tells between the versions, if they are different?

Component v seems to indicate the version of ompi. I have checked IMAGIC on guppy and it includes openmpi version 1.4.1:

[root@guppy bin]# pwd
/opt/Imagic/openmpi/bin
[root@guppy bin]# ./ompi_info|grep tm
              MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4.1)

OK. Is imagic using the openmpi at /opt/Imagic/openmpi/bin? It looks like this version of openmpi has NOT been compiled for use with Torque since it does not list the tm components.

1. The version of Open MPI that Imagic uses on Guppy, is not Torque enabled and is version 1.4.1
   

   amber@guppy /opt/Imagic/openmpi/bin] ./ompi_info
                    Package: Open MPI imagic@Obelix Distribution
                   Open MPI: 1.4.1
      Open MPI SVN revision: r22421
      Open MPI release date: Jan 14, 2010
   

2. Other software running on Guppy is using the Torque enabled version:
   

   [amber@guppy ~]$ ompi_info
                    Package: Open MPI root@guppy Distribution
                   Open MPI: 1.6.4
      Open MPI SVN revision: r28081
      Open MPI release date: Feb 19, 2013
   

3. On Garibaldi if I only load module imagic, openmpi is configured for torque and ompi_info gives:
   

   garibaldi00 home/amber> ompi_info
                    Package: Open MPI jcducom@garibaldi00 Distribution
                   Open MPI: 1.4.3
      Open MPI SVN revision: r23834
      Open MPI release date: Oct 05, 2010
   

4. Running imagic on garibaldi works fine with torque enabled v1.4.3
5. Running on Guppy does not work.

Should we change the version of OpenMPI that Imagic uses on guppy to one that is Torque enabled?

I really don't like running openMPI out of the IMAGIC directory, and really think that all programs should use the same openMPI. IMAGIC should be compiled to work with the global torque-enabled install of openmpi.

Sounds good. As far as I know, the way it's currently setup in appion, IMAGIC is using the default openMPI on the system. I don't know how to compile IMAGIC with the global torque-enabled install of openmpi, but I have found some options that we can change at /opt/Imagic/imagic.drv:

#
# IMAGIC LICENSE (template)
#
# Be careful with changing the lines below
#
RELEASE = 5.7                  # IMAGIC major.minor release
CHECK   = ON                   # automatic version checking (ON/OFF)
CPU     = 1                    # number of CPUs (processes)
MPI     = OPENMPI              # MPI implementation (see: MPI_EXIST in mpi_lib.f)
PBS     = NO                   # PBS support (YES/NO)
FFT     = SINGLETON            # Fourier transform (CUDA/FFTW/SINGLETON)
DISP1   = 1,X_WINDOWS,MYUNX    # XWindows display device
PLOT1   = 1,X_WINDOWS,MYUNX    # plot device XWindows
PLOT2   = 2,PS,MYUNX           # plot device PostScript
PLOT3   = 3,FILE,MYUNX         # plot device IMAGIC image file
CAMERA1 = 1,HEEL-O-MAT,MYUNX   # scanner device
EOF

Here is a command to test with

/opt/myamisnap/bin/appion imagicMSA.py --projectid=303 --alignid=31 --lpfilt=5 --hpfilt=600 --mask_radius=0.9 --mask_dropoff=0.1 --bin=1 --numiters=5 --MSAdistance=modulation --overcorrection=0.8 --description="test for guppy" --nproc=1 --commit --rundir=/ami/data15/appion/zz07jul25b/align/imagicmsa5 --runname=imagicmsa5 --expid=8556 --jobtype=alignanalysis

Thanks Amber. It runs fine for me:

[sargis@guppy ~]$ /opt/myamisnap/bin/appion imagicMSA.py --projectid=303 --alignid=31 --lpfilt=5 --hpfilt=600 --mask_radius=0.9 --mask_dropoff=0.1 --bin=1 --numiters=5 --MSAdistance=modulation --overcorrection=0.8 --description="test for guppy" --nproc=1 --commit --rundir=/ami/data15/appion/zz07jul25b/align/imagicmsa5 --runname=imagicmsa5 --expid=8556 --jobtype=alignanalysis

 ... Time stamp: 13apr24l51
 ... Function name: imagicMSA
 ... Appion directory: /opt/myamisnap/lib
 ... Using split database
Connected to database: 'ap303'
 ... Committing data to database
 ... Run directory: /ami/data15/appion/zz07jul25b/align/imagicmsa5
 ... Writing function log to: imagicMSA.log
 ... Running on host: guppy
 ... Uploading ScriptData....
 ... Running on host: guppy
 ... Found 8 processors on this machine
 ... Running on host: guppy
 ... Running Appion version 'trunk'
 ... copying aligned stack into working directory for operations with IMAGIC
{'bin': 1, 'lpfilt_imagic': 0.65199999999999991, 'hpfilt_imagic': 0.0054333333333333326, 'expid': 8556, 'overcorrection': 0.80000000000000004, 'description': 'test for guppy', 'projectid': 303, 'boxsize': 120, 'mask_dropoff': 0.10000000000000001, 'apix': 1.6299999999999999, 'hpfilt': 600, 'alignid': '31', 'runname': 'imagicmsa5', 'nproc': 1, 'commit': True, 'rundir': '/ami/data15/appion/zz07jul25b/align/imagicmsa5', 'MSAdistance': 'modulation', 'lpfilt': 5, 'mask_radius': 0.90000000000000002, 'jobtype': 'alignanalysis', 'numiters': 5}
... aligned stack pixel size: 1.63
... aligned stack box size: 120
Running IMAGIC .batch file: See imagicMultivariateStatisticalAnalysis.log for details
IMAGIC: /ami/data15/appion/zz07jul25b/align/imagicmsa5/imagicMultivariateStatisticalAnalysis.batch
 ... completed in 25.27 sec
EMAN: proc2d /ami/data15/appion/zz07jul25b/align/imagicmsa5/eigenimages.img /ami/data15/appion/zz07jul25b/align/imagicmsa5/eigenimages.img inplace
 ... inserting Align Analysis Run parameters into database
 ... Alignment time: 25.28 sec
 ... Database Insertion time: 31.48 msec
 ... Closing out function log: imagicMSA.log
 ... Ended at Wed, 24 Apr 2013 11:52:19
 ... Memory increase during run: 2.918 MB
Total run time:    28.49 sec
 ... Run directory:
 /ami/data15/appion/zz07jul25b/align/imagicmsa5
[sargis@guppy ~]$ which mpirun
/usr/bin/mpirun
[sargis@guppy ~]$ more /ami/data15/appion/zz07jul25b/align/imagicmsa5/
eigenimages.hed                              imagicMSA.log                                my_msa.lis                                   start.img 
eigenimages.img                              imagicMultivariateStatisticalAnalysis.batch* my_msa.plt                                   thread000.log 
eigenpixels.hed                              imagicMultivariateStatisticalAnalysis.log    pixcoos.hed                                  thread001.log 
eigenpixels.img                              msamask.hed                                  pixcoos.img                                  
_imagic.dff                                  msamask.img                                  start.hed                                    
[sargis@guppy ~]$ more /ami/data15/appion/zz07jul25b/align/imagicmsa5/
eigenimages.hed                              imagicMSA.log                                my_msa.lis                                   start.img 
eigenimages.img                              imagicMultivariateStatisticalAnalysis.batch* my_msa.plt                                   thread000.log 
eigenpixels.hed                              imagicMultivariateStatisticalAnalysis.log    pixcoos.hed                                  thread001.log 
eigenpixels.img                              msamask.hed                                  pixcoos.img                                  
_imagic.dff                                  msamask.img                                  start.hed                                    
[sargis@guppy ~]$ more /ami/data15/appion/zz07jul25b/align/imagicmsa5/imagicMSA.log
[ sargis@guppy: Wed Apr 24 11:51:51 2013 ]
/opt/myamisnap/bin/imagicMSA.py \
  --projectid=303 --alignid=31 --lpfilt=5 --hpfilt=600 --mask_radius=0.9 \
  --mask_dropoff=0.1 --bin=1 --numiters=5 --MSAdistance=modulation \
  --overcorrection=0.8 --description='test for guppy' --nproc=1 \
  --commit --rundir=/ami/data15/appion/zz07jul25b/align/imagicmsa5 \
  --runname=imagicmsa5 --expid=8556 --jobtype=alignanalysis 
[Wed Apr 24 11:52:19 2013]
finished run of imagicMSA

the example Amber sent only runs on 1 processor, and doesn't use the MPI version.
Run with more than 1 processor by changing the "nproc" parameter in the command.
Also, check the batch file created by appion to make sure that it is using the global mpirun install, and not IMAGIC's.
you should see a file "imagicMultivariateStatisticalAnalysis.batch" in the run directory, and it should contain a line that looks like:
mpirun -np 8 -x IMAGIC_BATCH /opt/Imagic/align/msa.e_mpi << EOF

mpirun isn't working for me on guppy. Using nodes=2:ppn=4, executing the command:

mpirun -np 8 -x IMAGIC_BATCH /opt/Imagic/msa/msa.e_mpi

I get this error:
--------------------------------------------------------------------------
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.

Host: guppy-16
Framework: ess
Component: tm
--------------------------------------------------------------------------
[guppy-16:07162] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file runtime/orte_init.c at line 120
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

orte_ess_base_open failed
  --> Returned value Error (1) instead of ORTE_SUCCESS
-------------------------------------------------------------------------
[guppy-16:07162] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file orted/orted_main.c at line 325

Sorry sargis, on the test command you need to use nproc > 1 to use mpi.

Gabe, looks like it could not find the torque component of openmpi on guppy-16.

Which version of IMAGIC was at Berkeley and did you have ABSOFT or INTEL Fortran compiler? We have IMAGIC 5.7 on guppy that includes Open MPI: 1.4.1. I have compiled openmpi-1.4.5-2.src.rpm with Torque support and can replace /opt/Imagic/openmpi so that it can run parallel (MPI) jobs across nodes, if needed.