Project

General

Profile

Start with existing ANY Linux flavor OS » History » Version 14

Amber Herold, 01/27/2010 09:08 AM

1 1 Amber Herold
h1. Start with existing ANY Linux flavor OS
2
3 3 Amber Herold
{{>toc}}
4
5 1 Amber Herold
*Appion Install Manual Referenced to Leginion Installation*
6
7
=====================
8
9
_The Appion Team_
10
11
email: *appion@scripps.edu* for any help you need.
12
13 2 Amber Herold
h2. 1. Introduction 
14 1 Amber Herold
15 2 Amber Herold
This document describes a general installation of Appion, concentrating on the installation and setup on the database and web servers.  Most sections refer to the Leginon installation documentation since the two share the same general architecture.   If you want to run real Leginon on the microscope, you just need to follow the additional installation starting from "Processing Server Window Installation Chapter of Leginon installation manual":http://emg.nysbc.org/documentation/leginon/bk02ch04s08.php.
16 1 Amber Herold
17 2 Amber Herold
h2. 2. Setup MySQL databases 
18
19 5 Amber Herold
h3. see "Leginon database server-side installation manual":http://emg.nysbc.org/documentation/leginon/bk02ch04s06.php  for details, including how to increasing cache size. 
20 2 Amber Herold
21
h3. Preliminary steps if you have an existing project database from a previous installation 
22
23 1 Amber Herold
We need to remove a table in the project database called "install".  This will allow the new default tables to be defined when we set up on the web-server side.
24 2 Amber Herold
25
26
<pre>
27 1 Amber Herold
$ mysql projectdata -u usr_object
28
mysql> drop table install;
29
mysql> exit
30 2 Amber Herold
</pre>
31
32
33
h3. Additional work 
34
35 1 Amber Herold
You need to decide what prefix you will use for the processing databases.  We will be creating them on the fly later.  Our default is *ap* followed by a project id number.  More about this later.
36
At this point, you need to do the following to grant privileges to users for any database whose name starts with *ap*
37 2 Amber Herold
38
39
<pre>
40 1 Amber Herold
$ mysql -u root -p
41 2 Amber Herold
</pre>
42
43
44 1 Amber Herold
Note: If you didn't set a mysql root password, don't use -p option.
45 2 Amber Herold
46
47
<pre>
48 1 Amber Herold
mysql> GRANT ALL PRIVILEGES ON `ap%`.* TO usr_object@"%";
49
mysql> exit
50 2 Amber Herold
</pre>
51
52
53
h2. 3. Web server-side installation 
54 3 Amber Herold
55 2 Amber Herold
h3. Follow instructions in "Leginon Database server-side installation manual":http://emg.nysbc.org/documentation/leginon/bk02ch04s07.php. 
56 3 Amber Herold
57 2 Amber Herold
h3. Continue with the instructions in "Leginon Web server set up and Installation":http://emg.nysbc.org/documentation/leginon/bk02ch04s08.php. 
58 3 Amber Herold
59 2 Amber Herold
h3. Additional work 
60 3 Amber Herold
61 4 Amber Herold
* Edit project`_`1`_`2/config.php to assign the same prefix to your processing db as defined in your MyQL setup.
62 2 Amber Herold
63
64
<pre>
65 1 Amber Herold
$DEF_PROCESSING_PREFIX = "ap";
66 2 Amber Herold
</pre>
67
68 4 Amber Herold
* Enable the processing plug-in by uncommenting out the following line in the file`dbem_1_5_1/config.php`
69 2 Amber Herold
70
<pre>
71 1 Amber Herold
addplugin("processing");
72 2 Amber Herold
</pre>
73
74 4 Amber Herold
* Copy `dbem_1_5_1/processing/config_processing.php.template` to `dbem_1_5_1/processing/config_processing.php`
75
* Edit `dbem_1_5_1/processing/config_processing.php`
76 2 Amber Herold
77
78
<pre>
79 1 Amber Herold
$PROCESSING_DB_HOST = "your_db_host";
80
$PROCESSING_DB_USER = "usr_object";
81
$PROCESSING_DB_PASS = ""; 
82
$PROCESSING_DB = "";
83 2 Amber Herold
</pre>
84
85 1 Amber Herold
    Remember that the last line should be kept empty as this will be set dynamically.
86
87
We will not include the processing host or cluster registration now. It is covered in the last part of this document.
88
89 2 Amber Herold
h2. 4. Create a test project and processing database
90
91
h3. Follow the instructions on "how to create new project":http://emg.nysbc.org/documentation/leginon/bk02ch06.php in the Leginon Manual.
92
93
h3. Additional work 
94
95
# Click on a project name on the webpage `http://your_host/project_1_2/project.php`. This will take you to a new webpage `http://your_host/project_1_2/getproject.php?pId=1`.  The number following "pId=" depends on the project id automatically assigned to the project.
96
# At the end of the Info table, you should see:
97
98
99
<pre>
100 1 Amber Herold
processing db*: not set (create processing db) *db name* ap1
101 2 Amber Herold
</pre>
102
103 7 Amber Herold
You can create the default numbered style database ap... or give it a new name with the same prefix.  If you want to specify a database name that does not use the default prefix, please note that your db user specified in the config.php in project_1_2 needs to have the necessary privileges for that database.  You may additionally want to change the value assigned to $DEF_PROCESSING_PREFIX in project_1_2/config.php if you want to use your new prefix all the time.
104 6 Amber Herold
# Enter the processing database name and click "create processing db".
105
# The page should refresh and display the linked database like this:
106 2 Amber Herold
107
108
<pre>
109 1 Amber Herold
processing db: ap1
110 2 Amber Herold
</pre>
111
112 1 Amber Herold
113 7 Amber Herold
See next section on trouble shooting if you get the original page instead.
114
115 1 Amber Herold
* *Repeat the above process for all your projects.*
116
117
If you want all your processing databases combined in one single database (not recommended, as this becomes large very fast), just use the same name for all your projects.
118
119
The above procedure not only creates the database, but also create some of the tables that you need to start processing.
120 2 Amber Herold
121
h3. Trouble Shooting 
122
123 1 Amber Herold
If the 'getproject.php' webpage remains unchanged, your processing database link is not accepted.  This is usually
124
caused by an incorrect setting such as:
125 2 Amber Herold
# The mysql user does not have the privileges to create the named database.
126 7 Amber Herold
    See Section-5.3 on what you should do.  To try it again after your mysql correction, you should repeat Section-5.2 to allow reinitialization from the project_1_2 web page and then try 7.2 again.
127 2 Amber Herold
# You have accessed an earlier version of project web page after you reinitialized the install table in your existing project database.
128 1 Amber Herold
    The install table in the project database is set to deny further changes once any project web page is accessed.  As a result, required table property changes and new table insertion would fail.
129
    Here is how to fix it:
130 2 Amber Herold
# Repeat Seciont-5.2 to allow reinitialization.
131
# Try Section-7.2 again.
132 1 Amber Herold
133 14 Amber Herold
h2. 5 Processing server side installation 
134 2 Amber Herold
135
h3. Follow "Leginon processing-server-side installation on Linux":http://emg.nysbc.org/documentation/leginon/bk02ch04s05.php 
136
137
h3. Additional work 
138
139 4 Amber Herold
* Go to your Leginon installation directory (typically /usr/lib/python2.4/site-packages/Leginon/)
140
* Edit project.py. Change line 5 to:
141 2 Amber Herold
142
143
<pre>
144 1 Amber Herold
use_processingdb_table = True
145 2 Amber Herold
</pre>
146
147 4 Amber Herold
* Add the appionData module to sinedon.cfg, which you have already modified during the Leginon installation.
148 2 Amber Herold
149
150
<pre>
151 1 Amber Herold
[appionData]
152
user:	usr_object
153 2 Amber Herold
</pre>
154
155
156 1 Amber Herold
  `[`Note`]` The module names in brackets are case sensitive and need to be exact.
157
The user name needs to match the name for which privileges have been granted on the `ap%` databases.
158
159 2 Amber Herold
h3. Download and install pyappion 
160 1 Amber Herold
161 9 Amber Herold
h4. [[Appion_Installation_Manual#Processing-server-needed-and-not-covered-by-Leginon-installation|Make Sure you have these packages installed]] 
162 2 Amber Herold
163
h4. "Download pyappion":http://appion.googlecode.com/files/pyappion-0.5.3-r3665.tgz 
164
165 10 Amber Herold
h4. [[Start_with_existing_CentOS_installation#5.-Compile-and-setup-python-(pyappion)-programs|Compile and Setup Instructions]] 
166 2 Amber Herold
167
h3. Install external packages 
168 1 Amber Herold
Follow instructions for the individual packages.
169 2 Amber Herold
Instruction for compiling Xmipp for OpenMPI is "here":http://code.google.com/p/appion/wiki/AppionInstall_CentOS#Compile_Xmipp_for_OpenMPI 
170 1 Amber Herold
171 2 Amber Herold
h2. 6. Initialize Leginon with its admin tools 
172
173
h3. DO NOT follow Leginon administration tool instructions.  We only need to upload images for appion processing, which is much simpler.
174
175
# Go to the webpage `http://your_host/dbem_1_5_1/addinstrument.php`
176
# Create a fake TEM instrument like this:
177
178
179
<pre>
180 1 Amber Herold
name: my_scope
181
hostname: whatever
182
type: Choose TEM
183 2 Amber Herold
</pre>
184
185
# Create a fake CCDCamera instrument like this:
186
187
<pre>
188 1 Amber Herold
name: my_scope
189
hostname: whatever
190
type: Choose CCDCamera
191
</pre>
192 2 Amber Herold
193
`[`Note`]` If you use Leginon, and still want to upload non-Leginon images, make sure that you create a pair of fake instruments like these on a host solely for uploading.  It will be a disaster if you don't, as the pixelsize of the real instrument pair will be overwritten by your upload.
194 1 Amber Herold
195
h2. 7. Start your first Appion session from the web 
196 2 Amber Herold
197
h3. Upload Images to a new session 
198 4 Amber Herold
199 11 Amber Herold
# Go to webpage `http://your_host/dbem_1_5_1/`  This is the general starting point for Leginon.
200
# Follow the link for the Project DB (bottom right)
201
# Select your test project, click on the name.  This takes you to the same page you created for the processing database.
202
# Below the experiment heading you will find a link that says "upload images to new session".  This takes you to your first Appion processing page, where you can use the web gui and instruction to upload your images.
203
# Once all fields are filled in click on "Just Show Command", which will bring you to a page that displays a command line.  Copy, paste, and execute this command into your text terminal, which will use the processing server programs and the database behind it.
204 1 Amber Herold
`[`Note`]` All images uploaded in an experiment session should have the same pixel size because they cannot easily be divided into groups during processing.
205
206
h3. Good starting point for future reference 
207 4 Amber Herold
208 11 Amber Herold
* If you have been successful so far, the image you uploaded will show up in the viewers.  You may choose one of
209 1 Amber Herold
the viewers provided by dbem`_`1`_`5`_`1 as your starting point to select your session, to view the images, and to start processing by following the link at the top-left corner called "processing".
210
211 4 Amber Herold
h2. Remote Appion Processing through ssh
212
 
213 1 Amber Herold
A more advanced way to run appion script is done through an ssh session.  This is equivalent of having you ssh into a computer and starting the appion processes.
214
215
There are two kinds of appion processes.  The first is a single-node process that can be run on a stand-alone workstation or the head-node of a computer cluster without PBS.  The second is a multiple-node process that requires PBS to run on the cluster.  When you use "Just Show Command" option, it is always a single-node process, but if you run through ssh it could be either, depending on the demand of the process.  For example,
216
imageuploader.py is always run as single-node process while maxlikeAlignment.py is either run on single-node with "Just Show Command" or as a PBS job submission when you run through ssh.
217
218 2 Amber Herold
h3. Install and setup ssh2 extension for php at your web-server 
219
220 1 Amber Herold
This enables an ssh session initialized from appionweb
221 11 Amber Herold
# Download and install the ssh2 extension for php as instructed at its download site:
222
* http://www.php.net/manual/en/ssh2.installation.php
223 1 Amber Herold
    Install openssl and compile libssh first before compiling php-ssh2 extension. Once you get the final ssh2.so, add it in your php.ini
224
225
The extension module is added to php in the same way as does the php-mrc module we distribute for the viewing mrc images through php.  To check whether it worked and for alternative way to make php recognize the module used in newer php, see http://emg.nysbc.org/documentation/leginon/bk02ch04s07.php under the section *Check php information* and *Alternative approach if mrc module does not show up in info.php output*
226
227 2 Amber Herold
h3. configuration at web server side 
228
229 1 Amber Herold
*IMPORTANT*: What we refer here as _your`_`processing`_`cluster_ should not be taken literally.  For example, if you access your cluster through network with a name "bestclusterever", you should name your cluster configuration php file *bestclusterever.php*, not *your`_`processing`_`cluster.php*.
230 11 Amber Herold
# Go to your_dbem`_`1`_`5`_`1/processing directory
231
# Copy *default`_`cluster.php* to *_your`_`processing`_`cluster.php_*
232
# Edit *_[http://code.google.com/p/appion/wiki/ClusterPhpSettings your`_`processing`_`cluster.php]_* to correspond to your cluster configuration
233
# Edit *config`_`processing.php* to add your cluster and, if desired, a stand-alone workstation as processing_host and register the your_processing_cluster.php you just created.
234 2 Amber Herold
235
236
<pre>
237 1 Amber Herold
$PROCESSING_HOSTS[]="your_stand-alone_processing_workstation";
238
$PROCESSING_HOSTS[]="your_processing_cluster";
239
$CLUSTER_CONFIGS= array (
240
  'your_processing_cluster'
241
);
242 2 Amber Herold
</pre>
243
244
245
h3. Setting up PBS on processing server side 
246
247 1 Amber Herold
Following [http://code.google.com/p/appion/wiki/AppionInstall_CentOS#9._Install_a_PBS_job_submission_system instruction to set up PBS on your processing server].
248
249 2 Amber Herold
h3. Testing 
250
251
h4. Testing php-ssh2 installation
252
253 1 Amber Herold
Check your info.php as you did with mrctool installation.  Corrected installed extension should show up in the output of info.php.  Reference http://emg.nysbc.org/documentation/leginon/bk02ch04s07.php under the section *Check php information* and *Alternative approach if mrc module does not show up in info.php output*.
254 2 Amber Herold
255
h4. Testing ssh log in 
256
257 1 Amber Herold
Use the top right form on the processing page to log in as if doing an ssh session. The page will acknowledge that you have been logged in if the setup is correct.  You will be able to edit description of a run and to hide failed runs when logged in. The option for submitting the job appears at the bottom of the processing form whenever available.
258 2 Amber Herold
259
h4. Testing simple appion processing job submission 
260
261 1 Amber Herold
Upload template is a good example to try:
262 11 Amber Herold
# Create a small mrc image as your template to be uploaded.
263
# Select Upload template in Import tools in the appionweb processing menu of an existing session.
264
# Type in the needed information.
265
# When you finish filling the form, instead of clicking on "Just show command", choose your processing host and run the job by clicking "Upload Template".
266
# For simple process such as this, the webpage will take a little while to refresh until the job is completed.
267
# If you get the message "Template is uploaded", the process is successful and if you refresh the page you will find the template in the available list. 
268 2 Amber Herold
269
h4. Testing PBS-required appion processing job submission 
270
271 1 Amber Herold
Particle selection such as DogPicker is a good example to try:
272 11 Amber Herold
# Click on DoG Picking under Particle Selection menu
273
# Enter the required parameters
274
# When you finish filling a form for an appion processing, choose your processing host and run the job.  Pages for monitoring the job become available after the job is queued and subsequently begins running. If the status appears as "Running or Queued" at first, the setup is likely correct.
275
# After a while, the process will be completed and the status becomes "Done" when you click to have the Status updated on the page.
276 2 Amber Herold
277 11 Amber Herold
h4. Check _your_processing_cluster.php_ setup 
278 2 Amber Herold
279 11 Amber Herold
For reconstructions involving iterations of different parameters such as EMAN reconstruction by refinement, the _your_processing_cluster.php_ is used to generate the script.  Examine the script created on the web form and modify _your_processing_cluster.php_  You can copy the script to your cluster and test run/modify it until it is correct.