Project

General

Profile

Notes from Recon meeting » History » Version 6

Amber Herold, 06/22/2011 05:42 PM

1 1 Amber Herold
h1. Notes from Recon meeting
2
3
Moving forward, refinements will all be split into 2 steps, prep and run.
4
5
h2. Prepare refine
6
7
When the user selects to prep a refinement, a web form is provided to select the:
8
# refinement method - eman, xmipp, frealign, etc... 
9
# stack
10
# model
11
# run parameters - runname, rundir, description
12
# stack prep params - lp, hp, last particle, binning
13
14 4 Amber Herold
The web server then calls prepRefine.py located on the local cluster to prepare the refinement.
15 1 Amber Herold
16
h2. Run Refine
17
18
When the user selects to run a prepared refinement, a web form is provided to select the:
19
# prepped refine 
20
# cluster parameters - ppn, nodes, walltime, cputime, memory, mempernode
21
# refine params, both general and method specific
22
23
The web server will then:
24
# verify the cluster params by checking default_cluster.php
25
# if needed, copy the stack and model to a location that can be accessed by the selected cluster
26
# verify the user is logged into the cluster
27 6 Amber Herold
# pass the refine command to runJob.py (extended from the Agent class), located on the remote cluster
28 1 Amber Herold
29
runJob.py will:
30
# set the job type which was passed in the command
31
# create an instance of the job class based on the job type
32
# create an instance of the processing host class
33
# launch the job based via the processing host
34
# update the job status in the appion database (do we have db access from the remote cluster?)
35
36
h2. Object Model
37
38
h3. Processing Host
39
40
Each processing host (eg. Garibaldi, Guppy, Trestles) will define a class extended from a base ProcessingHost class. 
41
The extended classes know what headers need to be placed at the top of job files and they know how to execute a command based on the specific clusters requirements.
42
The base ProcessingHost class could be defined as follows:
43
<pre>
44 2 Amber Herold
abstract class ProcessingHost():
45
    def generateHeader(jobObject) # abstract, extended classes should define this, returns a string
46
    def executeCommand(command) # abstract, extending classes define this
47
    def createJobFile(header, commandList) # defined in base class, commandList is a 2D array, each row is a line in the job file.
48
    def launchJob(jobObject) # defined in base class, jobObject is an instance of the job class specific to the jobtype we are running
49 1 Amber Herold
        header = generateHeader(jobObject)
50
        jobFile = createJobFile(header, jobObject.getCommandList())
51
        executeCommand(jobFile)
52
</pre>
53
54 2 Amber Herold
h3. Job
55
56
Each type of appion job (eg Emanrefine, xmipprefine) will define a class that is extended from a base Job class.
57
The extending classes know parameters that are specific to the job type and how to farmat the parameters for the job file.
58
The base Job class could be defined as follows:
59
<pre>
60
class Job():
61
    self.commandList
62
    self.name
63
    self.rundir
64
    self.ppn
65
    self.nodes
66
    self.walltime
67
    self.cputime
68
    self.memory
69
    self.mempernode
70 3 Amber Herold
    def __init__(command) # constructor takes the command (runJob.py --runname --rundir ....)
71 1 Amber Herold
        self.commandList = self.createCommandList(paramDictionary)   
72 3 Amber Herold
    def createCommandList(command) # defined by sub classes, returns a commandList which is a 2D array where each row corresponds to a line in a job file
73
</pre>
74
75
h3. Agent
76
77
There will be an Agent class that is responsible for creating an instance of the appropriate job class and launching the job.
78
It will be implemented as a base class, where sub classes may override the createJobInst() function. For now, there will be only one sub class defined
79
called RunJob. The same runJob.py will be installed on all clusters. This implementation will allow flexibility for the future.
80
The base Agent class may be defined as follows:
81
<pre>
82
class Agent():
83
    def main(command):
84
        jobType = self.getJobType(command)
85
        job = self.createJobInst(jobType, command)
86
        processHost = new ProcessingHost()
87
        jobId = processHost.launchJob(job)
88
        self.updateJobStatus()
89
    def getJobType(command) # parses the command to find and return the jobtype
90
    def createJobInst(jobType, command) # sub classes must override this the create the appropriate job class instance
91
    def updateJobStatus() # not sure about how this will be defined yet
92
</pre>
93
94
Sub classes of Agent will define the createJobInst() function.
95
We could create a single subclass that creates a job class for every possible appion job type.
96
(we could make a rule that job sub classes are named after the jobtype with the word job appended. then this function would never need to be modified)
97
A sample implementation is:
98
<pre>
99
class RunJob(Agent):
100
    def createJobInst(jobType, command)
101
        switch (jobType):
102
            case "emanrefine":
103
                job = new EmanJob(command)
104
                break
105
            case "xmipprefine":
106
                job = new XmippJob(command)
107
                break
108
         return job
109 2 Amber Herold
</pre>