Remote processing server for GPU movie alignment
Added by Michael Cianfrocco over 7 years ago
Hi Anchi,
I have a question about strategies for setting up remote processing. I would like to build a single remote processing machine (not a full-on cluster) for running Leginon on 1) Krios+K2, 2) Talos+K2, and 3)T12 w/ CCD.
For movie frame alignment, if you have a remote processing server with 4 GPUs (for instance), how do you control access to the jobs to ensure that there aren't more than 4 movie alignments happening at a given time?
Specifically, I'm wondering how many GPU cards to get for on-the-fly movie alignment, with the expectation that most of the movie alignment will be performed on the fly. So, for us, that means minimally two GPUs for this server.
However, there are probably going to be times when users will want to re-run movie alignment or there is an alignment that is still finishing from a previous session, and someone wants to start a new run. Therefore, I'm wondering if the Appion job submission has a way to queue up the GPU jobs as well.
Thank you,
Mike
Replies (2)
RE: Remote processing server for GPU movie alignment - Added by Anchi Cheng over 7 years ago
So far we have either single gpu unit on a cluster which we have a gpu queue, or multiple gpu on a single stand-alone workstation where users assign gpuid manually without a scheduler.
What you need would be to install job scheduler on your multipl-gpu machine. Schedulers can be installed on a single workstation and I see from Google search multiple options for gpu queuing. As far as I understand you can specify in your job file number of gpus requested.
What I see as a missing link here is that programs like MotionCor2 requires you enter which gpuid you want to use which job scheduler may not do. With Cuda and Nvidia being the default gpu type used, I can see that it should be possible to write a python or shell module to do this.
I will let Carl comment on his effort in qpu queue definition. We do have it running on our cluster with single gpu per node.
RE: Remote processing server for GPU movie alignment - Added by Michael Cianfrocco over 7 years ago
Hi Anchi,
I see - thanks for your response. To be honest, I wasn't planning on building more than one machine that would deal with the appion processing webserver as I don't need to put this onto our cluster.
Perhaps we won't integrate this step into the processing server, only all other steps.
Thanks,
Mike