Notes from Recon meeting » History » Version 6
Amber Herold, 06/22/2011 05:42 PM
1 | 1 | Amber Herold | h1. Notes from Recon meeting |
---|---|---|---|
2 | |||
3 | Moving forward, refinements will all be split into 2 steps, prep and run. |
||
4 | |||
5 | h2. Prepare refine |
||
6 | |||
7 | When the user selects to prep a refinement, a web form is provided to select the: |
||
8 | # refinement method - eman, xmipp, frealign, etc... |
||
9 | # stack |
||
10 | # model |
||
11 | # run parameters - runname, rundir, description |
||
12 | # stack prep params - lp, hp, last particle, binning |
||
13 | |||
14 | 4 | Amber Herold | The web server then calls prepRefine.py located on the local cluster to prepare the refinement. |
15 | 1 | Amber Herold | |
16 | h2. Run Refine |
||
17 | |||
18 | When the user selects to run a prepared refinement, a web form is provided to select the: |
||
19 | # prepped refine |
||
20 | # cluster parameters - ppn, nodes, walltime, cputime, memory, mempernode |
||
21 | # refine params, both general and method specific |
||
22 | |||
23 | The web server will then: |
||
24 | # verify the cluster params by checking default_cluster.php |
||
25 | # if needed, copy the stack and model to a location that can be accessed by the selected cluster |
||
26 | # verify the user is logged into the cluster |
||
27 | 6 | Amber Herold | # pass the refine command to runJob.py (extended from the Agent class), located on the remote cluster |
28 | 1 | Amber Herold | |
29 | runJob.py will: |
||
30 | # set the job type which was passed in the command |
||
31 | # create an instance of the job class based on the job type |
||
32 | # create an instance of the processing host class |
||
33 | # launch the job based via the processing host |
||
34 | # update the job status in the appion database (do we have db access from the remote cluster?) |
||
35 | |||
36 | h2. Object Model |
||
37 | |||
38 | h3. Processing Host |
||
39 | |||
40 | Each processing host (eg. Garibaldi, Guppy, Trestles) will define a class extended from a base ProcessingHost class. |
||
41 | The extended classes know what headers need to be placed at the top of job files and they know how to execute a command based on the specific clusters requirements. |
||
42 | The base ProcessingHost class could be defined as follows: |
||
43 | <pre> |
||
44 | 2 | Amber Herold | abstract class ProcessingHost(): |
45 | def generateHeader(jobObject) # abstract, extended classes should define this, returns a string |
||
46 | def executeCommand(command) # abstract, extending classes define this |
||
47 | def createJobFile(header, commandList) # defined in base class, commandList is a 2D array, each row is a line in the job file. |
||
48 | def launchJob(jobObject) # defined in base class, jobObject is an instance of the job class specific to the jobtype we are running |
||
49 | 1 | Amber Herold | header = generateHeader(jobObject) |
50 | jobFile = createJobFile(header, jobObject.getCommandList()) |
||
51 | executeCommand(jobFile) |
||
52 | </pre> |
||
53 | |||
54 | 2 | Amber Herold | h3. Job |
55 | |||
56 | Each type of appion job (eg Emanrefine, xmipprefine) will define a class that is extended from a base Job class. |
||
57 | The extending classes know parameters that are specific to the job type and how to farmat the parameters for the job file. |
||
58 | The base Job class could be defined as follows: |
||
59 | <pre> |
||
60 | class Job(): |
||
61 | self.commandList |
||
62 | self.name |
||
63 | self.rundir |
||
64 | self.ppn |
||
65 | self.nodes |
||
66 | self.walltime |
||
67 | self.cputime |
||
68 | self.memory |
||
69 | self.mempernode |
||
70 | 3 | Amber Herold | def __init__(command) # constructor takes the command (runJob.py --runname --rundir ....) |
71 | 1 | Amber Herold | self.commandList = self.createCommandList(paramDictionary) |
72 | 3 | Amber Herold | def createCommandList(command) # defined by sub classes, returns a commandList which is a 2D array where each row corresponds to a line in a job file |
73 | </pre> |
||
74 | |||
75 | h3. Agent |
||
76 | |||
77 | There will be an Agent class that is responsible for creating an instance of the appropriate job class and launching the job. |
||
78 | It will be implemented as a base class, where sub classes may override the createJobInst() function. For now, there will be only one sub class defined |
||
79 | called RunJob. The same runJob.py will be installed on all clusters. This implementation will allow flexibility for the future. |
||
80 | The base Agent class may be defined as follows: |
||
81 | <pre> |
||
82 | class Agent(): |
||
83 | def main(command): |
||
84 | jobType = self.getJobType(command) |
||
85 | job = self.createJobInst(jobType, command) |
||
86 | processHost = new ProcessingHost() |
||
87 | jobId = processHost.launchJob(job) |
||
88 | self.updateJobStatus() |
||
89 | def getJobType(command) # parses the command to find and return the jobtype |
||
90 | def createJobInst(jobType, command) # sub classes must override this the create the appropriate job class instance |
||
91 | def updateJobStatus() # not sure about how this will be defined yet |
||
92 | </pre> |
||
93 | |||
94 | Sub classes of Agent will define the createJobInst() function. |
||
95 | We could create a single subclass that creates a job class for every possible appion job type. |
||
96 | (we could make a rule that job sub classes are named after the jobtype with the word job appended. then this function would never need to be modified) |
||
97 | A sample implementation is: |
||
98 | <pre> |
||
99 | class RunJob(Agent): |
||
100 | def createJobInst(jobType, command) |
||
101 | switch (jobType): |
||
102 | case "emanrefine": |
||
103 | job = new EmanJob(command) |
||
104 | break |
||
105 | case "xmipprefine": |
||
106 | job = new XmippJob(command) |
||
107 | break |
||
108 | return job |
||
109 | 2 | Amber Herold | </pre> |