Appion

Melody is reporting this issue as well on both guppy and garibaldi.

Sargis, I just realized that this morning when you asked if I was working on the bug that David reported you probably thought I meant this one. I was working on #2744 this morning.

Sorry David and Melody, looks like our communication snafu prevented work on this urgent issue today...

I just tried again and cl2d jobs don't work either via the webinterface or in submitting the jobfile.

Sargis, here is an example /ami/data15/appion/14aug02a/align/cl2d7

Thanks
David

CL2D appears to be running fine right now. My clustering results are currently being uploaded to the database. This is the folder: /ami/data15/appion/14aug05b/align/cl2d2/

This still doesn't work for me.
When submitting jobs from the webinterface, I still have the same error message.
When deleting all the files but the jobfile and submitting it manually to guppy, the job crashes with an mpi error (/ami/data15/appion/14aug02a/align/cl2d8).

I have also been trying to run cl2d jobs for other projects on garibaldi and although they go to completion, they yield only 2 class averages out of 64 or 128 required (depending on the job).

Thanks for the update Reggie. Sorry this still doesn't work for you David. I've run /ami/data15/appion/14aug02a/align/cl2d7 as David and it worked for me too. Things to try for David: use betamyamiweb instead of myamiweb or replace /opt/myami-3.0 with /opt/myamisnap in the job file.

By doing diff for the log when it works and when it fails I see that in both cases xmipp_mpi_class_averages works fine. This seems to be a problem with proc2d as shown in the snippet of diff below.

96,99c77
< EMAN: proc2d /ami/data15/appion/14aug02a/align/cl2d7/part14aug08j36_lev
el_05_.hed /ami/data15/appion/14aug02a/align/cl2d7/average.mrc average
< EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $).
< Run 'proc2d help' for detailed help.
1 images60 complete
---
> !!! WARNING: could not create stack average, average.mrc

Hi All,

I'm still also having this problem about half the time I run. I ran from cronus3/betamyami and it has /opt/myamisnap in the job file.

Here is part of the job file

webcaller.py '/opt/myamisnap/bin/appion runXmippCL2D.py --stack=27 --lowpass=15 --highpass=2000 --num-part=22638 --num-ref=128 --bin=4 --max-iter=15 --nproc=24 --fast --classical_multiref --commit --description=correntropy --runname=cl2d11 --rundir=/ami/data15/appion/14jul24d/align/cl2d11 --projectid=450 --expid=13722 ' /ami/data15/appion/14jul24d/align/cl2d11/cl2d11.appionsub.log

And the error is:

Getting stack data for stackid=27
Old stack info: 'small ones ... 22638 particle substack with 0,1,2,3,4,5,7,8,10,13,15,16,18,19,29,25,23,22,21,20,30,31,33,38,39,45,50,68,66,70,71,78,80,90,93,98,106,103,100,114,127,126,123,125 classes included'
... Stack 27 pixel size: 1.364
... averaging stack for summary web page
!!! WARNING: could not create stack average, average.mrc
... Inserting CL2D Run into DB

lines= ['\tlibmpi.so.1 => /usr/lib64/libmpi.so.1 (0x000000372fa00000)\n', '\tlibmpi_cxx.so.1 => /usr/lib64/libmpi_cxx.so.1 (0x00007fdfffc0f000)\n']
/ami/data15/appion/14jul24d/align/cl2d11/alignedStack.hed
Traceback (most recent call last):
File "/opt/myamisnap/bin/runXmippCL2D.py", line 624, in <module>
cl2d.start()
File "/opt/myamisnap/bin/runXmippCL2D.py", line 605, in start
self.insertAlignStackRunIntoDatabase("alignedStack.hed")
File "/opt/myamisnap/bin/runXmippCL2D.py", line 386, in insertAlignStackRunIntoDatabase
apDisplay.printError("could not find average mrc file: "+avgmrcfile)
File "/opt/myamisnap/lib/appionlib/apDisplay.py", line 65, in printError
raise Exception, colorString("\n * FATAL ERROR *\n"+text+"\n\a","red")
Exception: * FATAL ERROR *
could not find average mrc file: /ami/data15/appion/14jul24d/align/cl2d11/average.mrc

This was running on guppy

I've checked /var/log/messages on guppy-14 and saw this there:

Aug 12 11:54:07 guppy-14 abrt: detected unhandled Python exception in '/opt/myamisnap/bin/runXmippCL2D.py'
Aug 12 11:54:07 guppy-14 abrt: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory
Aug 12 11:54:15 guppy-14 pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/torque/spool/670735.guppy.emg.nysbc.org.OU melody@guppy.emg.nysbc.org:/ami/data15/appion/14jul24d/align/cl2d11/cl2d11.appionsub.job.o670735' failed with status=1, giving up after 4 attempts

The error output in guppy-14:/var/spool/torque/undelivered/670741.guppy.emg.nysbc.org.OU reads:

Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
PBS: chdir to /home/melody failed: No such file or directory
[guppy-18:28212] [[24343,0],4] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-18:28212] [[24343,0],4] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-18:28212] [[24343,0],4] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-18:28212] [[24343,0],4] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-16:19614] [[24343,0],2] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-16:19614] [[24343,0],2] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-16:19614] [[24343,0],2] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-16:19614] [[24343,0],2] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-15:18344] [[24343,0],1] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-15:18344] [[24343,0],1] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-15:18344] [[24343,0],1] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-15:18344] [[24343,0],1] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-19:08589] [[24343,0],5] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-19:08589] [[24343,0],5] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer
[guppy-19:08589] [[24343,0],5] ORTE_ERROR_LOG: Not found in file ess_tm_module.c at line 241
[guppy-19:08589] [[24343,0],5] -> [[24343,0],0] (node: NULL) oob-tcp: Number of attempts to create TCP connection has been exceeded.  Can not communicate with peer

Hi,

I moved the old directory to
/ami/data15/appion/14jul24d/align/OLD_cl2d11_OLD/

I re-created the directory and ran the same job using the exact same job file and got the same error as before:
... averaging stack for summary web page
!!! WARNING: could not create stack average, average.mrc
... Inserting CL2D Run into DB

lines= ['\tlibmpi.so.1 => /usr/lib64/libmpi.so.1 (0x0000003913600000)\n', '\tlibmpi_cxx.so.1 => /usr/lib64/libmpi_cxx.so.1 (0x00007f4abc346000)\n']
/ami/data15/appion/14jul24d/align/cl2d11/alignedStack.hed
Traceback (most recent call last):
File "/opt/myamisnap/bin/runXmippCL2D.py", line 624, in <module>
cl2d.start()
File "/opt/myamisnap/bin/runXmippCL2D.py", line 605, in start
self.insertAlignStackRunIntoDatabase("alignedStack.hed")
File "/opt/myamisnap/bin/runXmippCL2D.py", line 386, in insertAlignStackRunIntoDatabase
apDisplay.printError("could not find average mrc file: "+avgmrcfile)
File "/opt/myamisnap/lib/appionlib/apDisplay.py", line 65, in printError
raise Exception, colorString("\n * FATAL ERROR *\n"+text+"\n\a","red")
Exception: * FATAL ERROR *
could not find average mrc file: /ami/data15/appion/14jul24d/align/cl2d11/average.mrc

If the stack average is really just for the summary page could we just comment out this part? It's not needed for the actual alignment or processing and we would be very happy to be able to proceed using just cl2d without this average image for the appion webpage.

I've researched the mpi error message and it seems that this can be fixed by setting mpi_preconnect_all to 1:
https://groups.google.com/forum/#!msg/mom-users/G4Nknz-Vp5I/1DIl7czOP1UJ

I've added mpi_preconnect_all = 1 to /etc/openmpi-mca-params.conf on guppy head and work nodes.

Please try again. If it fails, try running the same job and request one node instead of 4 nodes to see if that helps.

I just tested a small stack (~1200 particles) after Sargis modified: didn't run

mpirun: killing job...

mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

/ami/data15/appion/14aug12b/align/cl2d4

Thanks Jana. This job was terminated because it exceeded walltime limit:

[root@guppy cl2d4]# more cl2d4.appionsub.job.o670882
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
=>> PBS: job killed: walltime 7240 exceeded limit 7200 = (2*60*60)
Terminated

It still doesn't work but now I am having a new error:

lines= ['\tlibmpi.so.1 => /usr/lib64/libmpi.so.1 (0x000000308f800000)\n', '\tlibmpi_cxx.so.1 => /usr/lib64/libmpi_cxx.so.1 (0x00007f59fe17a000)\n']
/ami/data15/appion/14aug07f/align/cl2d3/alignedStack.hed
Traceback (most recent call last):
File "/opt/myamisnap/bin/runXmippCL2D.py", line 624, in <module>
cl2d.start()
File "/opt/myamisnap/bin/runXmippCL2D.py", line 605, in start
self.insertAlignStackRunIntoDatabase("alignedStack.hed")
File "/opt/myamisnap/bin/runXmippCL2D.py", line 389, in insertAlignStackRunIntoDatabase
apDisplay.printError("could not find reference stack file: "+refstackfile)
File "/opt/myamisnap/lib/appionlib/apDisplay.py", line 65, in printError
raise Exception, colorString("\n * FATAL ERROR *\n"+text+"\n\a","red")
Exception: * FATAL ERROR *
could not find reference stack file: /ami/data15/appion/14aug07f/align/cl2d3/part14aug17k12_level_-1_.hed

This bug seems reproducible as it happens several times for me.

I'm now running /ami/data15/appion/14aug07f/align/cl2d3/cl2d3.appionsub.job as David. I'm using /ami/data00/dev/sargis/appion that should keep intermediate results and print more debug messages.

Since this worked for me last time I run a similar job as David, please run env|grep PATH on guppy so we can compare our env paths. Here is what I get:

[sargis@guppy cl2d2]$ env|grep PATH
PATH=.:/opt/Xmipp/bin:/opt/protomo/bin:/opt/protomo/x86_64/bin:/opt/protomo2/bin/linux/x86-64:/opt/phoelix/bin64:/usr/kerberos/bin:/opt/IMOD/bin:/usr/local/EMAN/bin:/opt/EMAN2/bin:/opt/myamisnap/bin:/opt/em_hole_finder:/usr/local/bin:/bin:/usr/bin:/usr/local/ccp4-6.3.0/share/xia2/xia2core//Test:/usr/local/ccp4-6.3.0/share/xia2/xia2//Applications:/usr/local/ccp4-6.3.0/etc:/usr/local/ccp4-6.3.0/bin:/usr/local/ccp4-6.3.0/share/ccp4i/bin:/usr/local/ccp4-6.3.0/share/dbccp4i/bin:/usr/local/relion/bin:/usr/local/SIMPLE/simple_linux_120521/apps:/usr/local/SIMPLE/simple_linux_120521/bin:/opt/suprim/bin64
MATLABPATH=/opt/myamisnap/ace
LD_LIBRARY_PATH=/opt/Xmipp/lib:/opt/suprim/lib64:/opt/suprim/lib64:/opt/phoelix/lib64:/opt/IMOD/lib:/opt/EMAN2/lib:/usr/lib64:/usr/local/matlab/bin/glnxa64:/usr/local/ccp4-6.3.0/lib:/usr/local/EMAN/lib:/opt/Imagic/lib:/opt/Imagic/fftw/lib:/opt/protomo2/lib/linux/x86-64
PYTHONPATH=/opt/EMAN2/lib:/opt/EMAN2/bin:/opt/myamisnap/lib::/usr/local/ccp4-6.3.0/share/python:/usr/local/EMAN/lib:/opt/protomo2/lib/linux/x86-64
MANPATH=/opt/IMOD/man:/usr/share/man:/usr/local/ccp4-6.3.0/man
CLASSPATH=.:/usr/local/ccp4-6.3.0/bin
DYLD_LIBRARY_PATH=/usr/local/ccp4-6.3.0/lib
XUSERFILESEARCHPATH=/usr/local/ccp4-6.3.0/lib/X11/app-defaults/%N:/usr/lib/X11/app-defaults
RASMOLPATH=/usr/local/ccp4-6.3.0/x-windows/RasMol/src
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles
SUPRIM_PATH=/opt/suprim

Also, has anyone tried running this on a single node instead of 4 nodes to see if that works?

Update: The job I run as David is now running the mpi part and it seems to be working fine so far. I'll need to see David or Molody so we can lunch a modified job file together to debug this.

Hi Sargis,

I tried running cl2d using the version 3 of Xmipp as you suggested and got the following error.

/ami/data15/appion/14aug14a/align/cl2d6

Traceback (most recent call last):
File "/opt/myamisnap/bin/runXmipp3CL2D.py", line 647, in <module>
cl2d.start()
File "/opt/myamisnap/bin/runXmipp3CL2D.py", line 599, in start
star.read()
File "/opt/myamisnap/lib/appionlib/starFile.py", line 132, in read
raise Exception("Trying to read a star format file that does not exist: %s" % (self.location))
Exception: Trying to read a star format file that does not exist: images.xmd

Hi,

I ran two more CL2D jobs unsuccessfully on guppy.

/ami/data15/appion/14aug18a/align/cl2d1
/ami/data15/appion/14aug18a/align/cl2d3

They both had the same error:
=>> PBS: job killed: mem job total 125840520 kb exceeded limit 125829120 kb
Terminated

This was a very small job and I'm very surprised that it would need 125 gb of memory.... is there 16 gb of memory per node on guppy?

Thanks for the update. 14aug07f/align/cl2d3 finished succesfully. I've started a new Xmipp 3 cl2d job using http://longboard/betamyamiweb using the same parameters as in /ami/data15/appion/14aug14a/align/cl2d6.

Dmitry reported that he was getting PBS: chdir to '/home/dlyumkis' failed on guppy-17 yesterday. There were similar error message in the David's job file. /home folder was somehow unmounted from guppy-17. I have remounted /home on guppy-17.

For Melody's CL2D jobs there was no PBS: chdir errors, which is a good news. I'm now working on the memory error.

Yes, there is 16 gb of memory per node on guppy. pbsnodes gives me the following info: physmem=16331812kb,availmem=24125456kb,totmem=24564764kb. This is about the same for all guppy nodes.

[root@guppy align]# more */*.appionsub.job.o*
::::::::::::::
cl2d1/cl2d1.appionsub.job.o670982
::::::::::::::
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
=>> PBS: job killed: mem job total 18367888 kb exceeded limit 16777216 kb
Terminated
::::::::::::::
cl2d3/cl2d3.appionsub.job.o670989
::::::::::::::
Warning: no access to tty (Bad file descriptor).
Thus no job control in this shell.
=>> PBS: job killed: mem job total 125840520 kb exceeded limit 125829120 kb
Terminated
::::::::::::::

I am able to run CL2D on guppy without any error messages. I submitted the job via Appion and longboard/betamyamiweb.

The command and folder are below:

/opt/myamisnap/bin/runXmippCL2D.py \
--stack=190 --lowpass=10 --highpass=2000 --num-part=1349 --num-ref=20 \
--bin=7 --max-iter=15 --nproc=16 --fast --classical_multiref \
--correlation --commit --description='CL2D 26minus minus' --runname=cl2d3 \
--rundir=/ami/data15/appion/14jan24a/align/cl2d3 --projectid=367 \
--expid=12853

Sargis,
The run /ami/data15/appion/14aug14a/align/cl2d7 using xmipp 3 successfully completed.
However, the classes aren't properly uploaded as nothing shows up when clicking the "show composite page" link.

Hi,

This is just a note, on a cl2d run I ran last August 19th the particles were still correctly correlated with the 2D class averages. Therefore, my guess is the problem has occurred to changes made within the last month. This was with Xmipp2.

Please let us know if there is anything more we can do to test this problem, we would really like to get it fixed ASAP. Jana/Yong ZI are currently running a small proteasome test with 3k particles as their preferred orientation and high contrast make is easy to tell if the particles are correlated with the 2D average or not.

I made a stack with 3 really clear views (50S ribosome side view, 50S ribosome crown view, and 30S ribosome) and then classified them (/ami/data15/appion/14sep14a/align/cl2d9/). Level 2 uploaded properly, but 1 and 0 didn't. You can see the number of particles in the level 1 classes is the exact same number as the first four level 2 classes, and the particles no longer correspond.

I've looked at the code (php and python) and can't figure out what might be causing this. There are no recent changes there or in Xmipp2, which makes it a bit hard to debug. In /ami/data15/appion/14sep14a/align/cl2d9/ there are .sel files that point to the aligned particles and that part (View Alighned Partices) seems to be working OK. I'm now going through View Raw Partices code (both php and upload part in runXmippCL2D.py) to see what can be done there. A smaller test case with fewer particles would make it eaiser to debug this.

Hey Emily,

Bridget and I just looked through your 14sep14a cl2d9 on data15 for all the levels and it seems like they all work... could you go back and double check and see if they are still wrong for you? (we're not ruling out the possibility that something has changed since when you looked at them this morning...)

Thanks

Hi Sargis,

I've made a small stack with ~3k particles and ran cl2d on guppy. The directory can be found here:
/ami/data22/appion/14aug07e/align/cl2d10-10im_guppy

If you look in appion at the 64 classes (/ami/data22/appion/14aug07e/align/cl2d10-10im_guppy/part14sep24n47_level_05_.hed) you can see that for class #13 or #23 or #41 looking at the aligned particles, they are clearly not matched up correctly. The particles associated with class #57 might be the correct ones for class #13, #23 or #41.

Thanks Sargis. We are currently baffled. Emily did a small run - see above /ami/data15/appion/14sep14a/align/cl2d9/
and reported problems. But then Melody and I looked at it and it looks prefect. 14aug07e - latest run.
Melody did a small run and it looks wrong.
Emily ran on garibaldi and Melody ran on both but so far we'ver only looked at guppy and that is wrong.

Maybe the classes can be displayed before the database is done updating?
Did the classes seem god to you in spite of the incorrect total particle counts? i.e. did the raw particles seem to belong to the right class. Right now they all look very good.
Did you always run on garibaldi or sometimes on guppy beofre?

My test case was run on guppy, as Anchi requested since we have better control over it, and I am guessing that you are right and it wasn't finished uploading-sorry for jumping the gun. That's a mild problem since Appion told me the run was done, but a different problem. At least one of my experimental runs is still incorrect and was run on garibaldi. The error persists through the different levels and affects both raw and aligned particles. I will try submitting the same 600 particle test job to garibaldi and see if it uploads correctly. I'll also try to make a 100 particle one so that Sargis has less to search through.

Same problem on garibaldi:

/ami/data22/appion/14aug07e/align/cl2d10-10im_garibaldi

I was waiting until Melody's test finished but now it is. I think we've figured out the problem-it started when we switched to using data21 and 22 after the data15 crash. Here are 2 runs with only 127 particles:
/ami/data15/appion/14sep14a/align/cl2d11/
/ami/data21/appion/14sep14a/align/cl2d12/

cl2d12 was run hours ago but hasn't fully uploaded in addition to having some incorrect particles. Usually the first class average is off and after that it's sporadic.

I tried it also with a 650 particle stack with the same problem (cl2d10 and cl2d10_data21) and Melody tried it with the proteasome. It seems to be consistent. The particles are way more clear for the proteasome, but that stack is a lot larger.

Hope this helps.

it's a miracle!

last night emily emailed me pointing out that it seemed like her alignments worked but only on data15.
i re-ran this with the proteaosme and i found at least for guppy on data15 that the particles are properly correlated (for this one run)

/ami/data15/appion/14aug07e/align/cl2d10-10im_guppy_data15

this seems like more than just a coincidence as emily has also seen it work properly on data15 and that the 4 other runs i ran on data22 in the same session wiht the same parameters/stack have incorrect class average/particle correlation

anyway, could it be something gets crossed when reading/writing to another disk?

I have rerun last two jobs without clearing intermediate results and the raw results produced my Xmipp2 (.spi, .xmp, .sel) look identical.

/ami/data15/appion/14aug07e/align/cl2d10-10im_guppy_data15
/ami/data22/appion/14aug07e/align/cl2d10-10im_guppy

I'll now go through the steps in createReferenceStack (https://emg.nysbc.org/projects/appion/repository/entry/trunk/appion/bin/runXmippCL2D.py#L192). I think somthing is happending when it creates the class averages and alignedStack that makes the result look different.

I found what's causing this. runXmippCL2D.py is using glob function and this function is returned ordered list of files on data15. On data22, however, it returns this list in arbitrary order. I have made revision r18592 that sorts this list before making imagic files. Please try running another cl2d job on data21 or data22 tomorrow after a nightly cron jobs updates appion on guppy.

There is more info on how pythons glob.glob ordered at http://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered.

I have found why redmine was not sending notification emails after reading http://www.redmine.org/issues/8157 and going through the logs (Email delivery error: 550 5.1.1 <craigyk@scripps.edu>... User unknown).

Hey everyone,

I finished my identical data15 and data21 runs on an experimental dataset, and things seem to have uploaded properly on both! I didn't check every single class, but I randomly opened classes of distinct particles in different levels and they were all correct. Runs are on 14sep17c (cl2d5 and cl2d5_data21) if anyone wants to take a look. I ran on garibaldi but, as Sargis made the same updates on both, I think it should be fine.

Happy classifying!

Thanks Emily and everyone else for the careful testing and bug reporting. And thanks Sargis for hunting this down and fixing it.