Setup job submission server » History » Version 27
Amber Herold, 06/08/2010 02:40 PM
1 | 1 | Neil Voss | h1. Setup job submission server |
---|---|---|---|
2 | |||
3 | 23 | Amber Herold | In this case, we are setting up a job submission server that will have all of the data directories mounted and external packages installed (EMAN, Xmipp, etc.) on the compute nodes. Most institutions have a job submission server already, but the data is not accessible. Appion is not set up for this scenario except for large reconstruction jobs. |
4 | 1 | Neil Voss | |
5 | 7 | Neil Voss | ______ |
6 | |||
7 | 6 | Neil Voss | h2. PBS and the Torque Resource Manager |
8 | 1 | Neil Voss | |
9 | 6 | Neil Voss | PBS stands for a "Portable Batch System":http://en.wikipedia.org/wiki/Portable_Batch_System. It is a job submission system meaning that users submit many jobs and the server prioritizes and executes each job as resources permit. Below we show how to install the popular open source PBS system called "TORQUE":http://en.wikipedia.org/wiki/TORQUE_Resource_Manager. |
10 | 1 | Neil Voss | |
11 | 6 | Neil Voss | A TORQUE cluster consists of one head node and many compute nodes. The head node runs the *pbs_server daemon* and the compute nodes run the *pbs_mom daemon*. Client commands for submitting and managing jobs can be installed on any host (including hosts not running pbs_server or pbs_mom). More documentation about Torque is "available here.":http://www.clusterresources.com/products/torque/docs/ |
12 | |||
13 | 7 | Neil Voss | ______ |
14 | |||
15 | 6 | Neil Voss | h2. Head node installation |
16 | |||
17 | 14 | Neil Voss | h3. Install Torque-server |
18 | 6 | Neil Voss | |
19 | 24 | Amber Herold | Torque available with Fedora and CentOS 5.4 (through the EPEL). For YUM based systems type: |
20 | 6 | Neil Voss | |
21 | 1 | Neil Voss | <pre> |
22 | 25 | Amber Herold | sudo yum -y install torque-server torque-scheduler torque-client |
23 | 1 | Neil Voss | </pre> |
24 | 7 | Neil Voss | |
25 | 1 | Neil Voss | h3. Initialize Torque-server, because PATH setting you will need to become root |
26 | 25 | Amber Herold | |
27 | Make sure the directory containing the _pbs_server_ executable is in your PATH. For CentOS this is usually /usr/sbin. |
||
28 | 8 | Neil Voss | |
29 | 10 | Neil Voss | <pre> |
30 | 12 | Neil Voss | sudo su |
31 | /usr/share/doc/torque-2.3.10/torque.setup root |
||
32 | exit |
||
33 | 10 | Neil Voss | </pre> |
34 | 9 | Neil Voss | |
35 | 8 | Neil Voss | h3. Activate Torque-server |
36 | |||
37 | Enable the torque pbs_mom daemon on reboot: |
||
38 | 1 | Neil Voss | |
39 | <pre> |
||
40 | 9 | Neil Voss | sudo /sbin/chkconfig pbs_server on |
41 | 15 | Neil Voss | sudo /sbin/service pbs_server restart |
42 | 22 | Neil Voss | sudo /sbin/chkconfig pbs_sched on |
43 | sudo /sbin/service pbs_sched start |
||
44 | 8 | Neil Voss | </pre> |
45 | |||
46 | 16 | Neil Voss | h3. Add nodes to Torque-server nodes file: /var/torque/server_priv/nodes |
47 | 8 | Neil Voss | |
48 | 17 | Neil Voss | The format is: |
49 | <pre> |
||
50 | node-name[:ts] [np=] [properties] |
||
51 | </pre> |
||
52 | |||
53 | To add the localhost with two processors as a node, you would add: |
||
54 | |||
55 | <pre> |
||
56 | localhost np=2 |
||
57 | </pre> |
||
58 | |||
59 | You should add every *compute node* to this file, e.g., |
||
60 | |||
61 | <pre> |
||
62 | node01.INSTITUTE.EDU np=2 |
||
63 | node02.INSTITUTE.EDU np=4 |
||
64 | node03.INSTITUTE.EDU np=2 |
||
65 | </pre> |
||
66 | |||
67 | 7 | Neil Voss | ______ |
68 | 1 | Neil Voss | |
69 | 6 | Neil Voss | h2. Compute node installation |
70 | |||
71 | h3. Install Torque-mom |
||
72 | |||
73 | Torque available in with Fedora and CentOS 5.4 (through the EPEL). For YUM based systems type: |
||
74 | |||
75 | <pre> |
||
76 | sudo yum -y install torque-mom torque-client |
||
77 | </pre> |
||
78 | |||
79 | 18 | Neil Voss | h3. Configure node to receive jobs from headnode: |
80 | |||
81 | bq. see http://www.clusterresources.com/products/torque/docs/1.2basicconfig.shtml#initializenode for more details |
||
82 | |||
83 | Edit the /var/torque/mom_priv/config file: |
||
84 | |||
85 | <pre> |
||
86 | 21 | Neil Voss | $pbsserver headnode.INSTITUTE.EDU # hostname running pbs_server |
87 | 18 | Neil Voss | </pre> |
88 | 1 | Neil Voss | |
89 | For the localhost add: |
||
90 | |||
91 | <pre> |
||
92 | 21 | Neil Voss | $pbsserver localhost # hostname running pbs_server |
93 | 1 | Neil Voss | </pre> |
94 | 18 | Neil Voss | |
95 | 19 | Neil Voss | h3. Activate Torque-mom |
96 | 18 | Neil Voss | |
97 | 19 | Neil Voss | Enable the torque pbs_mom daemon on reboot: |
98 | |||
99 | <pre> |
||
100 | sudo /sbin/chkconfig pbs_mom on |
||
101 | sudo /sbin/service pbs_mom start |
||
102 | </pre> |
||
103 | 20 | Neil Voss | |
104 | _________ |
||
105 | |||
106 | h2. Test Torque Setup |
||
107 | |||
108 | On the head node, see if you can run a @qstat@:<pre>qstat</pre> |
||
109 | |||
110 | 26 | Amber Herold | You can type pbsnodes to check the state of the compute clusters. |
111 | |||
112 | 20 | Neil Voss | On the head node, create a job and submit it: |
113 | <pre> |
||
114 | echo "sleep 60" > test.job |
||
115 | echo "echo hello" >> test.job |
||
116 | qsub test.job |
||
117 | qstat |
||
118 | </pre> |
||
119 | |||
120 | |||
121 | |||
122 | |||
123 | |||
124 | |||
125 | 2 | Neil Voss | _________ |
126 | |||
127 | 27 | Amber Herold | [[Setup Remote Processing|^ Setup Remote Processing]] | [[Install SSH module for PHP|Install SSH module for PHP >]] |
128 | 2 | Neil Voss | |
129 | ______ |