Adding refine runjob » History » Revision 15
Revision 14 (Amber Herold, 07/21/2014 12:53 PM) → Revision 15/17 (Amber Herold, 07/21/2014 12:53 PM)
h1. Adding refine job
h3. How a cluster job file is built and launched
* To run a cluster job, the user runs a command on the head node of the cluster that looks like this:
<pre>runJob.py --jobtype=frealignrecon --outerMaskRadius=65 --innerMaskRadius=0 --symmetry="D7 (z)" --endIter=10 --percentDiscard=15 --wgh=0.07 --hp=300 --lp=10 --ffilt --psi --theta --phi --x --y --modelnames=initmodel0001.mrc --stackname=start.mrc --apix=1.63 --boxsize=120 --totalpart=300 --cs=2 --kv=120 --description="test frealign get preset" --runname=frealign_recon69 --rundir=/ami/data17/appion/zz07jul25b/recon/frealign_recon69 --nodes=2 --ppn=4 --rpn=8 --walltime=240 --cput=2400 --localhost=guppy.scripps.edu --remoterundir=/ami/data15/appion/zz07jul25b/recon/frealign_recon69 --projectid=303 --expid=8556</pre>
* *runJob.py* (source:trunk/appion/bin/runJob.py) is a python script that passes all the command parameters to a python class called *apAgent* (source:trunk/appion/appionlib/apAgent.py).
The *apAgent* object instantiates 2 more classes:
# _Job object_:
** this can be a *genericJob* (source:trunk/appion/appionlib/apGenericJob.py) which does not require a specialized job file, or a job class based on *apRefineJob*, or *apRemoteJob*
** See source:trunk/appion/appionlib/apSparxISAC.py for an example based on remoteJob
** The job object is passed the cluster processing parameters and the name of the job
** Most importantly, this object is responsible for knowing what the guts of the job file should be. These guts are maintained in a list called *command_list*
# _Processing Host object_:
** The base processing host will work for most resource managers, variations can create a new class based on processingHost.py (source:trunk/appion/appionlib/processingHost.py) such as source:trunk/appion/appionlib/torqueHost.py
** The processing host class knows how to format the job file so that the resource manager running on the current cluster can read it
** It also keeps track of the correct commands to use to launch a job and check on the job status
* So, when a job is run with a command starting with runJob.py:
## an apAgent object will be created
## the apAgent object will create a processingHost object whose type (torque, moab, etc) depends on the specific cluster that is being run on
## the apAgent object will create a job object based on the --jobtype parameter that was passed to runJob.py
## the apAgent object will then call the launchJob() function defined in the processingHost class
## inside the launchJob() function, the details of the job cluster parameters (number of processors, memory, etc) and the guts of the job file are requested from the Job Object
## the information from the job object is used by the processingHost object to build the job file
## the processingHost object submits the newly created job file to the cluster
h3. To add a new job type
To add a new job type:
# Write a new class in myami/appion/appionLib that is based on apRefineJob or apRemoteJob
# Add the new job type to the Agent class
After you have added a new refinement job class it needs to be added to the job running agent by editing the file apAgent.py in appionlib.
# Add the name of the module you created to the import statements at the top of the file.
# In the method _createJobInst_ add the new refinment job type to the condition statements.
<pre>
Ex.
elif "newJobType" == jobType:
jobInstance = newModuleName.NewRefinementClass(command)
</pre>
!Agent_class_diag.png!