Adding refine runjob » History » Revision 14
Revision 13 (Amber Herold, 07/21/2014 12:48 PM) → Revision 14/17 (Amber Herold, 07/21/2014 12:53 PM)
h1. Adding refine job h3. How a cluster job file is built and launched * To run a cluster job, the user runs a command on the head node of the cluster that looks like this: <pre>runJob.py --jobtype=frealignrecon --outerMaskRadius=65 --innerMaskRadius=0 --symmetry="D7 (z)" --endIter=10 --percentDiscard=15 --wgh=0.07 --hp=300 --lp=10 --ffilt --psi --theta --phi --x --y --modelnames=initmodel0001.mrc --stackname=start.mrc --apix=1.63 --boxsize=120 --totalpart=300 --cs=2 --kv=120 --description="test frealign get preset" --runname=frealign_recon69 --rundir=/ami/data17/appion/zz07jul25b/recon/frealign_recon69 --nodes=2 --ppn=4 --rpn=8 --walltime=240 --cput=2400 --localhost=guppy.scripps.edu --remoterundir=/ami/data15/appion/zz07jul25b/recon/frealign_recon69 --projectid=303 --expid=8556</pre> * *runJob.py* (source:trunk/appion/bin/runJob.py) is a python script that passes all the command parameters to a python class called *apAgent* (source:trunk/appion/appionlib/apAgent.py). The *apAgent* object instantiates 2 more classes: # _Job object_: ** this can be a *genericJob* (source:trunk/appion/appionlib/apGenericJob.py) which does not require a specialized job file, or a job class based on *apRefineJob*, or *apRemoteJob* ** See source:trunk/appion/appionlib/apSparxISAC.py for an example based on remoteJob ** The job object is passed the cluster processing parameters and the name of the job ** Most importantly, this object is responsible for knowing what the guts of the job file should be. These guts are maintained in a list called *command_list* # _Processing Host object_: ** The base processing host will work for most resource managers, variations can create a new class based on processingHost.py (source:trunk/appion/appionlib/processingHost.py) apProcessingHost such as source:trunk/appion/appionlib/torqueHost.py ** The processing host class knows how to format the job file so that the resource manager running on the current cluster can read it ** It also keeps track of the correct commands to use to launch a job and check on the job status * So, when a job is run with a command starting with runJob.py: ## an apAgent object will be created ## the apAgent object will create a processingHost object whose type (torque, moab, etc) depends on the specific cluster that is being run on ## the apAgent object will create a job object based on the --jobtype parameter that was passed to runJob.py ## the apAgent object will then call the launchJob() function defined in the processingHost class ## inside the launchJob() function, the details of the job cluster parameters (number of processors, memory, etc) and the guts of the job file are requested from the Job Object ## the information from the job object is used by the processingHost object to build the job file ## the processingHost object submits the newly created job file to the cluster h3. To add a new job type To add a new job type: # Write a new class in myami/appion/appionLib that is based on apRefineJob or apRemoteJob # Add the new job type to the Agent class Agent. After you have added a new refinement job class it needs to be added to the job running agent by editing the file apAgent.py in appionlib. # Add the name of the module you created to the import statements at the top of the file. # In the method _createJobInst_ add the new refinment job type to the condition statements. <pre> Ex. elif "newJobType" == jobType: jobInstance = newModuleName.NewRefinementClass(command) </pre> !Agent_class_diag.png!