Project

General

Profile

Actions

Feature #2337

closed

create fftw wisdom plan before starting redux

Added by Anchi Cheng over 11 years ago. Updated almost 11 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Category:
Web Image Viewer
Start date:
04/23/2013
Due date:
% Done:

0%

Estimated time:
Deliverable:

Description

fftw runs much faster with saved wisdom plan. However, there is still memory leak problem in creating the plan dynamically. This feature and script benchmark the fftw while saving the plan

The wisdom file is saved in the home directory of the user that runs the script and is named fftw3-wisom-_computer_hostname_


Related issues 1 (0 open1 closed)

Related to Leginon - Bug #2251: python fftw3 wrapper (pyfftw) has memory leaksClosedJim Pulokas02/15/2013

Actions
Actions #1

Updated by Anchi Cheng over 11 years ago

  • Category set to Web Image Viewer
  • Status changed from New to Assigned
  • Priority changed from Normal to Urgent
  • Target version set to myami-3.0

r17533 r17534 are changes for this feature.

To run this on K2 counted image (dimension y=3838, x=3710) and to make it take effect in the redux server:

  1. python fftwsetup.py 2 3838 3710
    **This takes a while, but re-run it will be fast because the plan is now saved
  2. restart redux server for the plan to be used

Jim. please add instruction

Actions #2

Updated by Anchi Cheng over 11 years ago

Here is Jim's e-mail for more details:

It is best to create the wisdom ahead of time using the stand alone script
pyami/fft/fftwsetup.py. It will store the wisdom you create in your home
directory, but you can then copy it to any other home directory that wants
to use it, or copy it as root to a file called /etc/fftw/wisdom which any
user can access.

Using fftwsetup.py:

I just committed a fix to svn so it prints the usage message properly.
usage: ./fftwsetup.py N shape0 shape1

Make sure to use shape0 and shape1 as in numpy dimensions (row,col) not (X,Y)

N is the number of iterations. You only have to run 1 iteration to
generate wisdom. The only reason to run 2 or more is just to have it
verify that it actually runs faster after the first iteration, because only
the first iteration will have to generate the wisdom. Other iterations
will use that wisdom and be faster.

After you run fftwsetup.py, it will print out a timing report showing the
time it took to generate wisdom and run the fft for each iteration. It
will also save the new wisdom to your home directory as:
fftw3-wisdom-hostname

Every time you run fftwsetup.py, it will first read the home directory
wisdom if it exists, and also the /etc/fftw/wisdom if it exists. It will
combine those, plus the new wisdom it creates, and store it back to the
home directory wisdom. If you want /etc/fftw/wisdom to contain the new
wisdom, be sure to copy the new wisdom file from the home directory to
there (has to be called "wisdom" in /etc/fftw). Or if you are always sure
to run as root, then no need to copy to /etc/fftw and always just use the
home directory wisdom.

Power of two sized images will be faster to create wisdom, but odd sized
images will take longer. For example on defcon1 (4 core Xeon 2.5 GHz):

./fftwsetup.py 2 8192 8192
-> takes about 15 seconds to create wisdom the first time
-> if wisdom is already created, takes about 2 seconds to do fft

./fftwsetup.py 2 8191 8191
-> takes about 40 seconds to create wisdom the first time
-> if wisdom is already created, takes about 5 seconds to do fft

Actions #3

Updated by Anchi Cheng over 11 years ago

I added the instruction at Using_Redux_to_serve_images_on_myamiweb

Actions #4

Updated by Anchi Cheng over 11 years ago

Jim,

Please check the instruction and decide if you want to do more work on this or to close it.

Actions #5

Updated by Jim Pulokas over 11 years ago

  • Status changed from Assigned to Closed

looks good

Actions #6

Updated by Anchi Cheng almost 11 years ago

  • Target version changed from myami-3.0 to Appion/Leginon 3.0.0
Actions

Also available in: Atom PDF