Bug #3529
openmemmap error
0%
Description
I got the following error when I tried to make a substack:
... averaging stack for summary web page
Traceback (most recent call last):
File "/panfs/storage.local/imb/stagg/software/myami_dev/appion/bin/alignSubStack.py", line 293, in <module>
subStack.start()
File "/panfs/storage.local/imb/stagg/software/myami_dev/appion/bin/alignSubStack.py", line 279, in start
apStack.averageStack(stack=oldstack,outfile=outavg,partlist=includeParticle)
File "/panfs/storage.local/imb/stagg/software/myami_dev/appion/appionlib/apStack.py", line 328, in averageStack
particles = imagic.read(stack)
File "/panfs/storage.local/imb/stagg/software/myami_dev/pyami/imagic.py", line 73, in read
a = readImagicData(pair['img'], header_dict, frame)
File "/panfs/storage.local/imb/stagg/software/myami_dev/pyami/imagic.py", line 43, in readImagicData
a = numpy.memmap(filename, dtype=dtype, mode='r', offset=start, shape=shape, order='C')
File "/usr/lib64/python2.7/site-packages/numpy/core/memmap.py", line 257, in new
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
mmap.error: [Errno 12] Cannot allocate memory
Updated by Neil Voss almost 10 years ago
I think I am using memmap incorrectly.
http://stackoverflow.com/questions/1661986/why-doesnt-pythons-mmap-work-with-large-files
We need to scrap memmap, I do not think 1 particle per iteration is ideal, but we can check system memory and do a chunk of particles at a time.
I already have the tools do implement the functionality, but I do not have the time. Probably a good idea to create a class for handling stack files.
Neil
P.S. Scott you should add yourself as a watcher on these bugs as well.
Updated by Scott Stagg almost 10 years ago
Well what should we do? Appion trunk and branch3.2 is broken almost entirely with the current setup.
Updated by Neil Voss almost 10 years ago
So what is using the new approc2d right now? Makestack should be okay. It is only crashing on the average stack step correct?
Updated by Neil Voss almost 10 years ago
I changed averageStack back to EMAN1 in r19126
Updated by Gabriel Lander over 9 years ago
since we're back to using eman1 I added code to write an LST file so this will work with virtual substacks