Appion cluster setup
Added by Harry Kao almost 14 years ago
Hi,
We are in the process of setting up computing clusters for Appion. We would like to know the hardware and network setup for Appion in Scripps. Would you post a description of your cluster setup?
Here are some specific questions:
1. Is there more than one Appion computing cluster in your lab? We are planning to set up three separate Appion clusters. In this scenario, each cluster and it's computing nodes should have access to the image and database server. Will this be a problem in terms of data i/o on the image and database servers?
2. What type of storage device is being used for storing micrographs acquired via Leginon and for Appion processing? Is it a single storage server, or storage cluster?
3. How many computing nodes is being assigned to one single Appion headnode in Scripps? And Number of CPUs per node?
Thank you.
Harry
Replies (2)
RE: Appion cluster setup - Added by Christopher Irving over 13 years ago
Here in the AMI group we are running Appion on three clusters and are in the process of getting it running on one of the Teragrid clusters. One cluster is for development and internal projects, the second cluster is part of a pilot project to give outside users access to Appion processing and the third is the Scripps institution wide cluster.
This fist cluster consists of 20 nodes, 10 with two quad core Xeon processors and 10 with two dual core Xeon processors. Each node has 16GB of memory. This cluster has direct access to our collection of primary file servers.
The second cluster is comprised 16 nodes, each with two quad core Xeon E5540 @ 2.53GHz processors and 48GB of memory. It has a dedicated file server.
The third cluster, our institution's main cluster, is used to do most of our large image processing jobs. It has 456 nodes, 200 of which have two dual core processors the rest have two quad core processors. It's storage space is handled by the Lustre file system which is a distributed, object-based file system. Data is copied to it before processing and copied back when processing completes. However, all the nodes have access to our Leginon/Appion databases.
"In this scenario, each cluster and it's computing nodes should have access to the image and database server. Will this be a problem in terms of data i/o on the image and database servers"
It depends, on the number of nodes access the filesystem and storage infrastructure. In general, for most Appion jobs the data IO is front and back loaded. The overall IO load is not very high.
If you don't mind me asking, why are you going to have three clusters?
-Christopher
RE: Appion cluster setup - Added by Harry Kao over 13 years ago
We have about 10 to 12 researcher who will be using Appion simultaneously. Currently, we have a total of 120 computing nodes which is divided into 4 clusters. Each cluster has its own headnode with about 2 to 7TB storage in raid 10, and is shared by 3 to 4 researchers. During refinement, several hundred GB of data are being transferred back and forth between the headnode and the computing nodes. Putting more than 30 computing nodes on a headnode usually results in long wait time for the data transfer. I think only a distributed storage cluster can handle this type of data i/o.
To overcome the data i/o limitation, we are planning to install Appion on each headnode, which can access the common database and image server. For data processing, as you mentioned, we might need to add a script to copy the image data to the individual headnodes in order to reduce the i/o load on the image server.
Harry