Appion

This error is reported by a local user on semc-cluster. It can be reproduced with the stack of 154635 particles of 160 boxsize. Stack average part failed due to the memory needed by the chunk based on self.freememory is larger than the hard-coded memorylimit value defined at the start of apImageFile module.

>>> from appionlib import apStack
>>> a = apStack.AverageStack(True)
>>> a.start('start.hed',None)

 ... processStack: Free memory: 206.4 GB
 ... processStack: Box size: 160
 ... processStack: Memory used per part: 100.0 kB
 ... processStack: Max particles in memory: 2164206
 ... processStack: Particles allowed in memory: 108210
 ... processStack: Number of particles in stack: 154635
 ... processStack: Particle loop num chunks: 2
 ... processStack: Particle loop step size: 77317
 ... processStack: partnum 1 to 77317 of 154635
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/acheng/myami/appion/appionlib/apImagicFile.py", line 974, in start
    stackdata = readImagic(stackfile, first=first, last=last, msg=False)
  File "/home/acheng/myami/appion/appionlib/apImagicFile.py", line 168, in readImagic
    %(numpart, apDisplay.bytes(partbytes*numpart)))
  File "/home/acheng/myami/appion/appionlib/apDisplay.py", line 65, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception: 
 *** FATAL ERROR ***
Stack is too large to read 77317 particles, requesting 7.4 GB

Neil, how do you want this to be handled ? Many cluster nodes have large memory but shared by many processors. Unless the user always blocks everyone else from accessing a node, the shared memory is not really all available. I hope the rest of you weigh in as well since if we assign it too small it will slow things down.