Project

General

Profile

Actions
Copy link

Bug #1435

closed

mrc can not read large stack even with memory mapping

Added by Anchi Cheng about 13 years ago. Updated over 11 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Jim Pulokas
Category:
Python scripting
Target version:
Appion/Leginon 3.0.0
Start date:
09/28/2011
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 2.1.0
Show in known bugs:
No
Workaround:

Description

try:/ami/data00/ami/data00/appion/11sep14d/ddstack/ddraw1/11sep14d_00010sq_v01_00003hl_v01_00002ed_st.mrc
The stack is 4096x3072x60

got
Warning: overflow encountered in long_scalars
Traceback (most recent call last):
File "updateStackHeader.py", line 27, in ?
updateHeaderStats(stackpath,-1)
File "updateStackHeader.py", line 14, in updateHeaderStats
a = mrc.mmap(filepath)
File "/home/acheng/myami/pyami/mrc.py", line 695, in mmap
mrcdata = numpy.memmap(filename, dtype=headerdict['dtype'], mode='r', offset=1024, shape=headerdict['shape'], order='C')
File "/usr/lib/python2.4/site-packages/numpy/core/memmap.py", line 67, in new
mm = mmap.mmap(fid.fileno(), bytes, access=acc)
OverflowError: memory mapped size must be positive

Also tried mrc.read() and got
754974720 items requested but only 0 read
Segmentation fault

Actions
Copy link
 #1

Updated by Jim Pulokas about 13 years ago

determined that it works fine on my ubuntu 11 system, but not on defcon1. continuing to investigate...

Actions
Copy link
 #2

Updated by Jim Pulokas about 13 years ago

Looks like it may require python 2.5 or greater to use mmap on a very large file:
http://mail.scipy.org/pipermail/numpy-discussion/2007-February/025950.html

Instead, I just made a different way to read a single slice from a large stack. All you have to do is give the Z slice you want to the read function:

a = read('test.mrc', zslice=4)

remember zslice start from 0. For now, zslice must be a single integer, but if it becomes necessary, we can expand this feature to allow a range of z.

feature added in r16084

Actions
Copy link
 #3

Updated by Jim Pulokas about 13 years ago

  • Status changed from Assigned to In Code Review
  • Assignee changed from Jim Pulokas to Anchi Cheng
Actions
Copy link
 #4

Updated by Anchi Cheng almost 13 years ago

  • Category set to Python scripting
  • Priority changed from Urgent to Normal
  • Target version set to Appion/Leginon 2.2.0

The current solution works at python 2.4 Should reinvestigate optimization when we move to python 2.6

Actions
Copy link
 #5

Updated by Anchi Cheng almost 13 years ago

  • Assignee changed from Anchi Cheng to Jim Pulokas
Actions
Copy link
 #6

Updated by Anchi Cheng almost 13 years ago

  • Status changed from In Code Review to Assigned
Actions
Copy link
 #7

Updated by Amber Herold over 12 years ago

  • Target version changed from Appion/Leginon 2.2.0 to Appion/Leginon 3.0.0

I'm moving the optimized version of this to 3.0 target version.

Actions
Copy link
 #8

Updated by Amber Herold over 11 years ago

  • Target version changed from Appion/Leginon 3.0.0 to Appion/Leginon Future Version
Actions
Copy link
 #9

Updated by Anchi Cheng over 11 years ago

  • Status changed from Assigned to Closed
  • Target version changed from Appion/Leginon Future Version to Appion/Leginon 3.0.0

Tested the methods used in python 2.4 on python 2.6 machine, and it works equally well. Both mmap and zslice in read functions worked fine. Reading the whole stack also did not crash.

This bug can therefore be closed.

Actions
Copy link

Also available in: Atom PDF