Feature #1098
closed
use configuration file to get default appion base directory
Added by Anchi Cheng almost 14 years ago.
Updated over 12 years ago.
Description
With completed appion and leginon retults moved to disks that potentially are full, new runs can not use the existing leginon or appion run path to set default. The proposal here is to add a constant in config.php and creates the new appion path default from that. On the python side, image path in leginon.cfg can be used for the same purpose.
Potential question is what to do with cases that the outdir is default not by leginon session but by, for example, ClusteringStackData. In such a case and with combined stack, the clusteringstack may be originated from a run of a different session and therefore assign the default outdir as such.
Question for Neil, Scott, Gabe,
Would it be bad if I use current session to generate the default which we now pass into all pages in the url?
- Status changed from Assigned to In Code Review
- Assignee changed from Anchi Cheng to Amber Herold
r15192 committed almost all files needing the change. Exception are uploadtomo.php and a few that get the default path not from the leginon session but from its parent tasks. I will wait until responses from Scott and Neil are back to do these one way or the other.
To use this, myamiweb/config.php need to define DEFAULT_APPION_PATH like this:
define('DEFAULT_APPION_PATH',"/ami/data00/appion");
To test, click on the run pages to see if they give what you put as default path on an old session. and use Just Show Commands to confirm that the rundir is set right with all '/' properly inserted. Some are more strict than other as it turn out.
I will do the python side later. This part is more critical for use. Please review these now.
I'm not sure I understand all of the implications of this, but it seems like it would be a way to easily migrate users from one disk to another, which would be good. I'm up for giving it a shot.
IIRC, Because SPIDER cannot take strings longer than like 64 characters, the clustering needs to be run in the same directory as the feature analysis, because I cannot pass the full directory path to SPIDER without getting an error.
Neil,
We will leave the "run particle clustering" pages alone, then. How about createmodel.php which use Clustered Classes to run eman common-line program to get a 3D-model. This one uses the ClusteringStackData to get the default path. I think it is o.k. to default to the current appion processing session path even though with combined stack etc it might be saved in the path of another session. Am I right?
Scott,
My concern was for two related processing to end up under paths of different session, whether that is too confusing.
This is already happening for GroEL data when we combined two sessions and proceed with reconstruction in either one. The report did help us trace back o.k. I just want to see if you remember cases when such switch is more damaging. Neil's example on SPIDER clustering is one of these examples.
We have tried in the past to keep all data and processing of the same session or project on the same disk and migrate them to a disk with space if processing is to be continue on it. It turns out to be unpredictable since people may keep processing the same session on one project but very little on the other. The whole raw and processing data ended up having to be moved very often to make room for the faster-grower and not to waste space on the slow-grower. Our current system is to save data on the same disk you may call it a buffer and Christopher has an automated script that migrates the data to other more permanent storage disks regularly. as long as they don't grow on the permanent disks by themselves, we can increase the disk usage efficiency by packing closer to full on the permanent disks.
Thanks Anchi, I think create model can work in any directory. And I am unaware of any other scripts that require a specific directory, just the SPIDER coran stuff.
r15208 convert python scripts start with upload.
Some 'upload*.py' scripts are not changed
- uploadFrealign.py - use jobdata
- uploadImages.py - use leginon image_path already
- uploadRecon.py - use jobdata
- uploadXmippRefine.py - not sure if we can rely on stack coming from a session.
r15217 does the similar thing for uploadtomo.php but as a basedir because tomography has more hierarchy in its data structure and would not know what the outdir should be until all parameters are input in the form.
The only remaining webpage that might need change now is
The python-side is pending appionwrapper conversion because I can not make a universal solution with mixed methods of getting SessionData information (some by session name, some by stackid). The new SubmitOrShowCommand function will output --expid from php pages and we will be able to use that. see also #1114
This path is defined in the config file, and it is where appion data will be stored during processing, not where appion has been installed. Need to test what happens when the path is not defined.
- Status changed from In Code Review to In Test
Modified the comment for DEFAULT_APPION_PATH. CAn go to test now.
Testing:
The new function to get the path has already been used extensively at AMI, however, it is likely that not all processing modules have been executed.
Anchi did visually inspect all the paths that result from her changes.
Need to test the case that the path is not set.
- Status changed from In Test to Closed
I tested the path variable not existing in the config file and the path being empty. Looks good.
- Status changed from Closed to In Code Review
r15703 and r15704 add the python-side of using expId (now available for all appionwrapper-allowed php codes) to get default appion path. It is written as an option when sessionname is not available so that it does not break old codes.
Testing:
1. copy any command generated by appion web to a processing computer text terminal.
2. remove the --rundir assignment and make sure that it has --expId assginment but not --sessionname and try to run it.
- Status changed from In Code Review to In Test
there was a typo - line 130 sessionid - but has been fixed already.
- Status changed from In Test to Closed
Also available in: Atom
PDF