Project

General

Profile

Actions
Copy link

Bug #4513

open

runLearnStackCleaner.py uses too much memory

Added by Scott Stagg almost 8 years ago. Updated almost 8 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
Neil Voss
Category:
-
Target version:
-
Start date:
10/07/2016
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 3.2
Show in known bugs:
No
Workaround:

Description

Neil, is runLearnStackCleaner.py supposed to load the entire stack into memory? If so, it makes this script unusable for large stacks. Seems like it wouldn't have to load the whole stack if it's only choosing random particles to view.

Actions
Copy link
 #1

Updated by Neil Voss almost 8 years ago

Hi Scott, no it only loads images to fill the viewer. I have tried this on 10GB stacks.

I guess it would have to load all the classified particles to do PCA. But this was a big concern of mine.

Can you provide more info, does it crash on load?

Actions
Copy link
 #2

Updated by Neil Voss almost 8 years ago

  • Status changed from New to Assigned
  • Assignee set to Scott Stagg
Actions
Copy link
 #3

Updated by Scott Stagg almost 8 years ago

I get a memory error on one machine

[sstagg@hpc-login-34 ~]$ runLearnStackCleaner.py --stack-id=291 --commit --rundir=/lustre/cryo/lustre/appiondata/16sep07b/stacks/learn291 --runname=learn291 --projectid=1 --expid=11581 --jobtype=makestack

 ... Time stamp: 16oct07n28
 ... Function name: runLearnStackCleaner
 ... Appion directory: /panfs/storage.local/imb/stagg/software/myamiss/appion
 ... Processing hostname: hpc-login-34.rcc.fsu.edu
 ... Using split database
Connected to database: 'ap1'
 ... Committing data to database
 ... Run directory: /lustre/cryo/lustre/appiondata/16sep07b/stacks/learn291
!!! WARNING: directory '/lustre/cryo/lustre/appiondata/16sep07b/stacks/learn291' already exists.
 ... Writing function log to: runLearnStackCleaner.log
 ... Uploading ScriptData....
 ... Found 24 processors on this machine
 ... Running Appion version 'trunk'
!!! WARNING: Assuming apix is 1.0 A/pixel
Traceback (most recent call last):
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/runLearnStackCleaner.py", line 135, in <module>
    learnstack.start()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/bin/runLearnStackCleaner.py", line 68, in start
    a.run()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/proc2dLib.py", line 78, in run
    self.approc2d.start()
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/proc2dLib.py", line 405, in start
    indata = self.readFileData(self.params['infile'])
  File "/panfs/storage.local/imb/stagg/software/myamiss/appion/appionlib/proc2dLib.py", line 228, in readFileData
    data = imagic.read(filename)
  File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/imagic.py", line 73, in read
    a = readImagicData(pair['img'], header_dict, frame)
  File "/panfs/storage.local/imb/stagg/software/myamiss/pyami/imagic.py", line 43, in readImagicData
    a = numpy.memmap(filename, dtype=dtype, mode='r', offset=start, shape=shape, order='C')
  File "/usr/lib64/python2.7/site-packages/numpy/core/memmap.py", line 257, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
mmap.error: [Errno 12] Cannot allocate memory

I can launch it on a high memory machine, but it take up a big chunk when it is starting up but then it goes down.

Actions
Copy link
 #4

Updated by Scott Stagg almost 8 years ago

  • Assignee changed from Scott Stagg to Neil Voss
Actions
Copy link

Also available in: Atom PDF