Project

General

Profile

Actions
Copy link

Feature #4704

open

Create a quicker make stack

Added by Neil Voss almost 8 years ago. Updated about 6 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
Neil Voss
Category:
-
Target version:
-
Start date:
01/13/2017
Due date:
% Done:

0%

Estimated time:

Description

I was experimenting with a faster make stack, out of necessity it will do less and it be multiprocess by design.

The biggest problem is that when you launch a multiprocess from python, you cannot use the database, because the MySQL connection does not transfer, so that is a major drawback. I am thinking of using star files for the transfer of information between the individual processes, rather than trying to make a new connection to the database.

I wanted to note that I am starting from scratch and will not modify the existing makestack2.py, perhaps this could eventually become appionLoop3.py

Actions
Copy link
 #1

Updated by Neil Voss almost 8 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Neil Voss almost 8 years ago

Made a 218,000 particle stack in 

Total run time:    3 min 3 sec

Actions
Copy link
 #3

Updated by Bridget Carragher almost 8 years ago

Wow! What is the timing using the old method?

Actions
Copy link
 #4

Updated by Neil Voss almost 8 years ago

Big stack timing:

Timing stats
Batch Boxer: 2.67 +/- 0.68 sec (19.07 percent)
Stack Merging: 401.44 +/- 109.19 msec (2.86 percent)
Mean/Std Read: 5.71 +/- 1.85 sec (40.72 percent)
COMPLETE LOOP:    3 hr 21 min for 603 images
 ... Closing out function log: makestack2.log
 ... Ended at Fri, 13 Jan 2017 14:32:32
Total run time:    3 hr 21 min

    SUMMARY: makestack2
    ------------------------------------------
    PEAKS:        339 peaks of 193394
    AVG PEAKS:    319.7 +/- 86.4 peaks
    (- ESTIMATE: 193670 to 193756 total peaks -)
    ------------------------------------------
    TIME:         18.03 sec
    AVG TIME:     14.0 +/- 3.71 sec
    ------------------------------------------
Actions
Copy link
 #5

Updated by Neil Voss almost 8 years ago

on my docker install:

Timing stats
Batch Boxer: 1.22 +/- 0.09 sec (41.38 percent)
Stack Merging: 35.72 +/- 9.31 msec (1.21 percent)
Mean/Std Read: 0.94 +/- 0.15 sec (31.70 percent)
DB Insertion: 61.72 +/- 9.57 msec (2.09 percent)
COMPLETE LOOP:    47.03 sec for 11 images

Quick Stack:

Total run time:    2.32 sec

Actions
Copy link
 #6

Updated by Neil Voss almost 8 years ago

Sorry for the redmine spam, just taking notes, Another quick test:

normal makestack2.py for 10,000 particles:

Timing stats
Batch Boxer: 2.35 +/- 0.6 sec (22.65 percent)
Stack Merging: 313.5 +/- 86.84 msec (3.02 percent)
Mean/Std Read: 5.07 +/- 1.54 sec (48.78 percent)
DB Insertion: 450.03 +/- 123.28 msec (4.33 percent)
COMPLETE LOOP:    7 min 11 sec for 35 images
Total run time:    7 min 31 sec

remove particle stats for 10,000 particles:

Timing stats
Batch Boxer: 2.35 +/- 0.61 sec (42.78 percent)
Stack Merging: 318.85 +/- 89.92 msec (5.80 percent)
Mean/Std Read: 267.55 +/- 68.74 msec (4.87 percent)
DB Insertion: 417.97 +/- 115.67 msec (7.60 percent)
COMPLETE LOOP:    4 min 14 sec for 35 images
Total run time:    4 min 37 sec

This is a pretty big speed-up for removing some functions, so I am going to commit.

Actions
Copy link
 #7

Updated by Neil Voss over 7 years ago

Just committed quickStack to trunk, please try it out.

Actions
Copy link
 #8

Updated by Sargis Dallakyan about 6 years ago

Excellent work Neil. I've made the following changes to web ui.

- It now auto-fills Initial Micrograph Boxsize similar to regular Make Stack. It also auto-fills Final Reduced Particle Boxsize (binned Initial Micrograph Boxsize by 2).
- Added Number of CPU Cores (nproc) option. This is important since otherwise it was using nproc=1.

Actions
Copy link

Also available in: Atom PDF