Project

General

Profile

Actions

Bug #5048

closed

Generating missing stack file error

Added by Ashleigh Raczkowski almost 7 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Carl Negro
Category:
-
Target version:
Start date:
06/24/2017
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 3.2
Show in known bugs:
No
Workaround:

Description

Unable to generate missing stack file or create stack file when making a substack from 2D classes. Working as user mfalzone on SEMC cluster.

generateMissingStack.py --projectid=197 --expid=4260 --stackid=35
Connected to database: 'nyap_197'
 ... Getting stack data for stackid=35
Old stack info: 'sub ... 15881 particle substack with 4,7,9,11,13,15,16,19,21,43,40,59,58,36,57,81,99,50,29,25,24,66,47,48,69,93 classes included'
 ... Querying original particles from stackid=35
 ... original stackid: 33
connecting
Original stack: /gpfs/appion/mfalzone/17jun23b/stacks/centered31/ali.hed
 ... generating stack: '/gpfs/appion/mfalzone/17jun23b/stacks/alignsub18/ali.hed' with 15881 particles
 ... creating a new stack
        /gpfs/appion/mfalzone/17jun23b/stacks/alignsub18/ali.hed
from the oldstack
        /gpfs/appion/mfalzone/17jun23b/stacks/centered31/ali.hed

Traceback (most recent call last):
  File "/opt/myamisnap/bin/generateMissingStack.py", line 35, in <module>
    apVirtualStack.generateMissingStack(params['stackid'])
  File "/opt/myamisnap/lib/appionlib/apVirtualStack.py", line 25, in generateMissingStack
    apStack.makeNewStack(vstackdata['filename'], fname, plist)
  File "/opt/myamisnap/lib/appionlib/apStack.py", line 50, in makeNewStack
    partlist = emanLstFileToPartList(listfile)
  File "/opt/myamisnap/lib/appionlib/apStack.py", line 70, in emanLstFileToPartList
    f = open(listfile, "r")
TypeError: coercing to Unicode: need string or buffer, list found
Actions #1

Updated by Carl Negro almost 7 years ago

I merged changes from trunk to myami-beta that should fix this; this will be pushed to our production at midnight tonight.

Actions #2

Updated by Ashleigh Raczkowski almost 7 years ago

Still seeing the same error message today.

Actions #3

Updated by Neil Voss almost 7 years ago

This was fixed in commit commit:bf5914f6.

Actions #4

Updated by Ashleigh Raczkowski almost 7 years ago

This seems to work for other sessions but not for 17jun23b. I a re-running maxlike in case something was wrong with the first two alignments.

Update: no change.

Actions #5

Updated by Anchi Cheng almost 7 years ago

  • Status changed from New to Assigned
  • Assignee set to Gabriel Lander
  • Priority changed from Normal to Urgent
  • Target version set to Appion/Leginon 3.3

This is a real bug also in the trunk.

This particular missing stack is a substack of the existing stack (stackid=35) which selects 15881 particles from 51760. This size of substack need to use particlelist file to pass the selections.

plist sent from apVirtualStack.py, line 25 is a list
apStack.makeNewStack function did not make a list file with it.
Therefore, when apStack.emanLstFileToPartList(listfile) is called, it failed since listfile is still plist.

Actions #6

Updated by Neil Voss almost 7 years ago

Okay, I am sure there are more of these, so I made a change to take either a list or a string for the makeNewStack function.

see commits commit:c519afed and commit:3cd178915

Actions #7

Updated by Neil Voss almost 7 years ago

  • Status changed from Assigned to In Test
  • Assignee changed from Gabriel Lander to Carl Negro

put Carl in charge to make sure commit gets moved over.

Actions #8

Updated by Carl Negro almost 7 years ago

Merged, this will update at NYSBC tonight. Thanks

Actions #9

Updated by Ashleigh Raczkowski almost 7 years ago

Tried to generate the stack again today:

generateMissingStack.py --projectid=197 --expid=4260 --stackid=35
Connected to database: 'nyap_197'
 ... Getting stack data for stackid=35
Old stack info: 'sub ... 15881 particle substack with 4,7,9,11,13,15,16,19,21,43,40,59,58,36,57,81,99,50,29,25,24,66,47,48,69,93 classes included'
 ... Querying original particles from stackid=35
 ... original stackid: 33
connecting

Original stack: /gpfs/appion/mfalzone/17jun23b/stacks/centered31/ali.hed
 ... generating stack: '/gpfs/appion/mfalzone/17jun23b/stacks/alignsub18/ali.hed' with 15881 particles
 ... creating a new stack
    /gpfs/appion/mfalzone/17jun23b/stacks/alignsub18/ali.hed
from the oldstack
    /gpfs/appion/mfalzone/17jun23b/stacks/centered31/ali.hed

 ... creating sub-stack from large stack
 ... ProcessStack2: Free memory: 6.5 GB
 ... ProcessStack2: Box size: 56
 ... ProcessStack2: Memory used per part: 12.2 kB
 ... ProcessStack2: Max particles in memory: 557831
 ... ProcessStack2: Particles allowed in memory: 1000
 ... ProcessStack2: Number of particles in stack: 15881
 ... ProcessStack2: Particle loop num chunks: 16
 ... ProcessStack2: Particle loop step size: 992
 ... ProcessStack2: partnum 1 to 992 of 15881
 ... going to read 15881 particles from file
 ... readParticlesListFromFile 0 to 46288
Traceback (most recent call last):
  File "/opt/myamisnap/bin/generateMissingStack.py", line 35, in <module>
    apVirtualStack.generateMissingStack(params['stackid'])
  File "/opt/myamisnap/lib/appionlib/apVirtualStack.py", line 25, in generateMissingStack
    apStack.makeNewStack(vstackdata['filename'], fname, plist)
  File "/opt/myamisnap/lib/appionlib/apStack.py", line 54, in makeNewStack
    stackTools.createSubStack(oldstack, newstack, partlist, msg=True)
  File "/opt/myamisnap/lib/appionlib/StackClass/stackTools.py", line 81, in createSubStack
    subStack.start(instack, partlist)
  File "/opt/myamisnap/lib/appionlib/StackClass/ProcessStack.py", line 143, in start
    stackarray = self.stackClass.readParticlesFromFile(partlist)
  File "/opt/myamisnap/lib/appionlib/StackClass/baseClass.py", line 143, in readParticlesFromFile
    partdatalist = self._readParticleListFromFile(particleNumbers)
  File "/opt/myamisnap/lib/appionlib/StackClass/imagicClass.py", line 52, in _readParticleListFromFile
    a = apImagicFile.readSingleParticleFromStack(self.filename, partnum=partnum, msg=self.debug)
  File "/opt/myamisnap/lib/appionlib/apImagicFile.py", line 492, in readSingleParticleFromStack
    apDisplay.printError("particle numbering starts at 1")
  File "/opt/myamisnap/lib/appionlib/apDisplay.py", line 65, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception: 
 *** FATAL ERROR ***
particle numbering starts at 1

Actions #10

Updated by Neil Voss almost 7 years ago

Hi Ashleigh, I just pushed a fix to the trunk, so give it a try tomorrow.

Actions #11

Updated by Neil Voss almost 7 years ago

Oh, I guess I need Carl to merge this as well.

Actions #12

Updated by Carl Negro almost 7 years ago

I see one of your commits in trunk and one in myami-beta? Is the trunk up to date where it should be?

Actions #13

Updated by Neil Voss almost 7 years ago

Hmm, I did a git rebase command to merge my last two commits and maybe something went wrong.

It is commit commit:a5af8b0c8b7adb2e8eb19ee036299c1d236be594

commit commit:4470bc481af80f087660e3a5491a2c917103336a should not exist, so I reverted it

Actions #14

Updated by Neil Voss almost 7 years ago

I think the myami-beta commit is not really there, redmine just thinks it is there.

Actions #15

Updated by Carl Negro almost 7 years ago

Merged trunk to myami-beta, thanks.

Actions #16

Updated by Ashleigh Raczkowski almost 7 years ago

Thank you all. It is working!

Actions #17

Updated by Anchi Cheng over 6 years ago

  • Status changed from In Test to Closed
Actions

Also available in: Atom PDF