Buffer(Ingestion) Server setup » History » Revision 5
Revision 4 (Anchi Cheng, 03/11/2019 11:43 PM) → Revision 5/6 (Anchi Cheng, 10/07/2020 11:47 AM)
h1. Buffer(Ingestion) Server setup
NRAMM/SEMC uses a redundant system that keeps pre-processing (frame alignment and ctf estimation) independent between each Krios. The buffer server also has a large enough storage to hold two weeks to a month of movie frames in case primary storage needs service.
{{include(NRAMM buffer server hardware)}}
The software and configuration requirement is described here:
h2. Software requirement for preprocessing
h3. Processing:
The same basic requirement for appion processing server.
ctfFind4, motioncor2, gctf
h3. Activate the use of buffer server in leginon database.
1. Create a directory for all the data to be saved. For example /bufferdata/frames. Make it readable by the users.
2. Run myami/dbschema/tools/buffer_host_setup.py in Leginon environment. to link the buffer host and directory with the camera. This association signals Appion to find movie frames on the buffer server rather than the primary storage.
Two files are essential which we run as root all the time.
* rawtransfer.py (Found in myami/leginon of your git clone) is used to move the movies from camera computer to the buffer server and change its name to match those of the Leginon sum images
* transfermonitor.sh (Found in myami/leginon) is used to move the movies to the primary storage.
* A third program run as cron job that removes frame movies accumulate on the buffer server after they are expired (modify leginon/transfermonitor.sh to match your setup). (Not included).
h2. What to do when primary storage is down
We stop transfermonitor.sh so that it does not try to copy the frames from the buffer server.
leginon.cfg on the leginon processing server can also be changed to point the summed images to store on the buffer server. In that case, leginondb.BufferHostData should be set to be disabled during this period using phpMyAdmin. I will make a better gui for it later.
In our case, we do happen to have a backup storage server from an older system which also has a second copy of all softwares we use for data collection so we have been pointing the summed data there instead.