Feature #4704
open
Create a quicker make stack
Added by Neil Voss almost 8 years ago.
Updated about 6 years ago.
Description
I was experimenting with a faster make stack, out of necessity it will do less and it be multiprocess by design.
The biggest problem is that when you launch a multiprocess from python, you cannot use the database, because the MySQL connection does not transfer, so that is a major drawback. I am thinking of using star files for the transfer of information between the individual processes, rather than trying to make a new connection to the database.
I wanted to note that I am starting from scratch and will not modify the existing makestack2.py, perhaps this could eventually become appionLoop3.py
- Description updated (diff)
Made a 218,000 particle stack in
Total run time: 3 min 3 sec
Wow! What is the timing using the old method?
Big stack timing:
Timing stats
Batch Boxer: 2.67 +/- 0.68 sec (19.07 percent)
Stack Merging: 401.44 +/- 109.19 msec (2.86 percent)
Mean/Std Read: 5.71 +/- 1.85 sec (40.72 percent)
COMPLETE LOOP: 3 hr 21 min for 603 images
... Closing out function log: makestack2.log
... Ended at Fri, 13 Jan 2017 14:32:32
Total run time: 3 hr 21 min
SUMMARY: makestack2
------------------------------------------
PEAKS: 339 peaks of 193394
AVG PEAKS: 319.7 +/- 86.4 peaks
(- ESTIMATE: 193670 to 193756 total peaks -)
------------------------------------------
TIME: 18.03 sec
AVG TIME: 14.0 +/- 3.71 sec
------------------------------------------
on my docker install:
Timing stats
Batch Boxer: 1.22 +/- 0.09 sec (41.38 percent)
Stack Merging: 35.72 +/- 9.31 msec (1.21 percent)
Mean/Std Read: 0.94 +/- 0.15 sec (31.70 percent)
DB Insertion: 61.72 +/- 9.57 msec (2.09 percent)
COMPLETE LOOP: 47.03 sec for 11 images
Quick Stack:
Total run time: 2.32 sec
Sorry for the redmine spam, just taking notes, Another quick test:
normal makestack2.py for 10,000 particles:
Timing stats
Batch Boxer: 2.35 +/- 0.6 sec (22.65 percent)
Stack Merging: 313.5 +/- 86.84 msec (3.02 percent)
Mean/Std Read: 5.07 +/- 1.54 sec (48.78 percent)
DB Insertion: 450.03 +/- 123.28 msec (4.33 percent)
COMPLETE LOOP: 7 min 11 sec for 35 images
Total run time: 7 min 31 sec
remove particle stats for 10,000 particles:
Timing stats
Batch Boxer: 2.35 +/- 0.61 sec (42.78 percent)
Stack Merging: 318.85 +/- 89.92 msec (5.80 percent)
Mean/Std Read: 267.55 +/- 68.74 msec (4.87 percent)
DB Insertion: 417.97 +/- 115.67 msec (7.60 percent)
COMPLETE LOOP: 4 min 14 sec for 35 images
Total run time: 4 min 37 sec
This is a pretty big speed-up for removing some functions, so I am going to commit.
Just committed quickStack to trunk, please try it out.
Excellent work Neil. I've made the following changes to web ui.
- It now auto-fills Initial Micrograph Boxsize similar to regular Make Stack. It also auto-fills Final Reduced Particle Boxsize (binned Initial Micrograph Boxsize by 2).
- Added Number of CPU Cores (nproc) option. This is important since otherwise it was using nproc=1.
Also available in: Atom
PDF