Feature #2192
closedmyamiweb interface for DD frame alignment
0%
Description
makeDDRawFrame.py gui needs:
1. --align (boolean flag, default True)
2. --defergpu (boolean flag) If launched on computer without gpu, this should be default True if --align is True.
If --defergpu is set, the program will first create the ddstack run in the database, and loop thru images to make dd frame stack but the user can start catchUpDDAlign.py to run the alignment on gpu equipped computer like this:
catchUpDDAlign.py --projectid=285 --ddstack=11 --expid=10806 --jobtype=catchupddalign
--ddstack is the DEF_id of ApDDStackRunData.
In the long term, myamiweb should be configured to specify gpu availability and handle gpu usage on gpu cluster settings. At moment people will copy and paste the comment. Since we have not been able to have more than one job running on gpu. It would kick out one of them. Feel free to add defergpu to ApDDStackParamsData if --defergpu need to be logged in the ddstack run.
Session 12dec10anchik2 is a good test session.
Example scripts:
makeDDRawFrameStack.py --description="test amibox04" --projectid=285 --ddstack=11 --session=12dec10anchik2 --expid=10806 --jobtype=catchupalign --align --defergpu
catchUpDDAlign.py --projectid=285 --ddstack=11 --expid=10806 --jobtype=catchupddalign
Updated by Amber Herold almost 12 years ago
So is this sometimes a two step process, when defergpu is set?
Would I need to set up the interface similar to a preprefine job such that after the first step runs, then the user must start the second? We did add gpu info to the config file already, so we should be able to look it up to know if there is a gpu or not. Is there a way to query the gpu to see if it is busy?
Perhaps we should have a face to face before I start coding this...
Updated by Anchi Cheng over 11 years ago
- Priority changed from High to Immediate
gui can check ApDDStackRunData for ddstack.
If the ddstack is meant to be aligned, ApDDStackParamsData.align should be true.
Updated by Amber Herold over 11 years ago
Anchi, For catchupDDAlign, should the run name and output directory be the same as the original make DD stack, or something else?
Updated by Anchi Cheng over 11 years ago
At script level it does not need runname and rundir because it gets it from ddstackid. If you must put it in, then, yes, it is the same.
Updated by Amber Herold over 11 years ago
- Status changed from Assigned to In Test
- Assignee changed from Amber Herold to David Veesler
David, Could you test this out?
It is under direct detector tools. You can select to defer alignment when creating a new dd stack and later launch the Catchup script after the first image is in the stack.
Updated by David Veesler over 11 years ago
I tried a couple of times and the web interface didn't work to submit the job but when pasting the command in a terminal it does work.
Here is the error message I got:
Appion Directory /ami/data15/appion/12nov30d/ddstack/ddstack25
Job File Name ddstack25.appionsub.job
Hostname guppy.scripps.edu
Username dveesler
Local server ip 137.131.204.71
Project Id
Command /opt/myamisnap/bin/appion runJob.py /opt/myamisnap/bin/appion makeDDRawFrameStack.py --description="test" --projectid=373 --preset=es --session=12nov30d --runname=ddstack25 --rundir=/ami/data15/appion/12nov30d/ddstack/ddstack25 --no-rejects --no-wait --commit --limit=1 --continue --align --defergpu --expid=10730 --jobtype=makeddrawframestack --ppn=1 --nodes=1 --walltime=240 --jobid=334
ERROR in job submission. Check the cluster. makeddrawframestack ['/opt/myamisnap/bin/appion', 'makeDDRawFrameStack.py', '--description=test', '--projectid=373', '--preset=es', '--session=12nov30d', '--runname=ddstack25', '--rundir=/ami/data15/appion/12nov30d/ddstack/ddstack25', '--no-rejects', '--no-wait', '--commit', '--limit=1', '--continue', '--align', '--defergpu', '--expid=10730', '--jobtype=makeddrawframestack', '--ppn=1', '--nodes=1', '--walltime=240', '--jobid=334']
I submitted the following command into a terminal logged on guppy and it worked smoothly:
[dveesler@guppy ~]$ /opt/myamisnap/bin/appion makeDDRawFrameStack.py --description="test" --projectid=373 --preset=es --session=12nov30d --runname=ddstack25 --rundir=/ami/data15/appion/12nov30d/ddstack/ddstack25 --no-rejects --no-wait --commit --limit=1 --continue --align --defergpu --expid=10730 --jobtype=makeddrawframestack
Updated by Amber Herold over 11 years ago
David, there was a bug with all web submitted jobs last week that should be sorted out now. Could you please try launching this again to confirm functionality?
Updated by David Veesler over 11 years ago
The web submission problem is persisting for the DD frames alignment. I just tried it again (cf. below).
However, I think we should have a binning option in the web interface because we have to add bin by 2 in the command line submission to decime super-resolution images from ~8Kx8K to ~4Kx4K in order to get Yifan's soft to work.
Appion Directory /ami/data15/appion/12nov30d/ddstack/ddstack26
Job File Name ddstack26.appionsub.job
Hostname guppy.scripps.edu
Username dveesler
Local server ip 137.131.204.44
Project Id
Command /opt/myamisnap/bin/appion runJob.py /opt/myamisnap/bin/appion makeDDRawFrameStack.py --description="test web interface" --projectid=373 --preset=es --session=12nov30d --runname=ddstack26 --rundir=/ami/data15/appion/12nov30d/ddstack/ddstack26 --no-rejects --no-wait --commit --limit=1 --continue --align --defergpu --expid=10730 --jobtype=makeddrawframestack --ppn=1 --nodes=1 --walltime=240 --jobid=339
ERROR in job submission. Check the cluster. makeddrawframestack ['/opt/myamisnap/bin/appion', 'makeDDRawFrameStack.py', '--description=test web interface', '--projectid=373', '--preset=es', '--session=12nov30d', '--runname=ddstack26', '--rundir=/ami/data15/appion/12nov30d/ddstack/ddstack26', '--no-rejects', '--no-wait', '--commit', '--limit=1', '--continue', '--align', '--defergpu', '--expid=10730', '--jobtype=makeddrawframestack', '--ppn=1', '--nodes=1', '--walltime=240', '--jobid=339']
Updated by Amber Herold over 11 years ago
I added the binning option to the web gui for makeDDrawFrameStack in r17498.
Updated by Anchi Cheng over 11 years ago
I changed the default binning to 1 in r17536
Updated by Anchi Cheng over 11 years ago
- Status changed from In Test to Assigned
- Assignee changed from David Veesler to Amber Herold
Amber, please make this first priority:
the command created by runCatchUpDDAlign.php is incorrect.
rundir should be, for example, --rundir=/ami/data15/appion/13apr29anchitestT1/ddstack/ddstack2
not --rundir=/ami/data15/appion/13apr29anchitestT1/ddstack/ddstack2/ddstack2
Updated by Amber Herold over 11 years ago
- Status changed from Assigned to In Test
- Assignee changed from Amber Herold to Anchi Cheng
- Priority changed from Immediate to Normal
Forgot to change the status of this when I checked in the last fix...
Updated by Amber Herold almost 11 years ago
Hey Anchi,
I took at look at the code in r18022.
I processing.inc, I would prefer not to create the new function
function gpuDriftCorrRef() { return showReference("gpudriftcorr"); }
Those function are only there for backward compatibility, and eventually I would like to remove them all and instead use
function showReference( $method )
So to add a reference you would use
showReference( "gpudriftcorr" )directly.
I know I had a note that I want to revisit that function, but it will be easier to maintain in the long run if we don't use a separate function for each type of reference.
Thoughts?
Updated by Anchi Cheng almost 11 years ago
Sounds fine, except that now you need to input publication.inc everywhere. It is up to you. Fine with me either way.
Updated by Amber Herold almost 11 years ago
showReference() is in processing.inc, so you should not need to add publication.inc. Did I miss something?
Updated by Anchi Cheng almost 11 years ago
- Assignee changed from Anchi Cheng to Amber Herold
You are right. Changed it in r18029 and merged to 3.0 in r18030
Updated by Anchi Cheng almost 11 years ago
- Status changed from In Test to In Code Review
r18053 changes runCatchUpDDAlign.php to use inc/forms/ddstackForm.inc and give the correct help and label to selection.
Could not create DDStackForm instance in createCommand function since it kept complaining that runname was undefined. This is not a general case, any way, since its rundir and runname is dependent on the selection of ddstack made.
Updated by Amber Herold over 10 years ago
- Status changed from In Code Review to In Test
- Assignee changed from Amber Herold to Anchi Cheng
reviewed code...can we assume this has been tested?
Updated by Anchi Cheng almost 7 years ago
- Status changed from In Test to Closed