Leginon

We are using CentOS 6 with python 2.6 and I try to keep all our workstations CentOS 6. We use yum for installing system packages such as python and mysql. I haven't done the entire python 2.7.14 environment completely self-contained, but we are also using shared storage that is NFS/GPFS mounted for appion and cryo-em packages.

I recommend using a separate namespace for shared packages, such us, /sw so that you won't accidentally shadow /usr/local. There might be cases, because of networking or some other problems, that might cause your computers to not to mount /usr/local from shared storage and it would be hard to debug this.

You have a very nice 4 GPU rackmount system with 700TB+ of storage. Would you mind sharing what vendor you used for this? We are using DDN/GPFS as our primary high performance shared storage and have a mixture of Dell and Supermicro for other storage, such as /home and /sw.

Sargis, if you invest in new hardware you'll quickly discover that CentOS 6 is too old for the newer Intel (and almost certainly AMD) processors. Regarding mount points: I'm mounting /usr/local from the server to /local on the workstations. This is as per the linux filesystem standard, where /usr/local is supposed to be for locally installed but not package-managed software. Since too many vendors use /opt for this as well, /opt isn't a good choice for a mount point, either, hence /local.

Regarding hardware: sure, I'm happy to share. The rackmount system itself is from Thinkmate (www.thinkmate.com). Thinkmate actually has an entire configuration section devoted to systems approved by the SBGrid consortium:

https://www.thinkmate.com/systems/workstations/structural-biology-grid

The rackmount GPU servers can be found here, though: https://www.thinkmate.com/systems/servers/gpx

Everything is customizable, so exactly what you get will depend on your budget. The salespeople are knowledgeable and helpful, and for preliminary questions you can use the "Engineer on Call" chat at the bottom corner of the main page.

Note that if you're doing a lot of double precision calculations, the Tesla cards become cost effective, but if you're doing mostly single precision, not even close. Nvidia won't warranty GTX1070/1080 cards in a server chassis, but Thinkmate does (you will pay a bit of a premium for the GPU cards as a result, though).

For the storage array, we went with a JetStor 864S DAS (https://www.acnc.com/) populated with 10T He10 disks. JetStor because the users already had experience with this product at their previous institution and favored it, but once I got it I found it to be a very nice piece of equipment at a reasonable price.

Thanks Patrick. That is good to know. Mounting /usr/local from the server to /local on the workstations sounds good.

One final question on this issue: since I'm installing python in a nonstandard location, I'm having to compile from the source tarball anyway. The python 2.7.14 ./config informs me that

If you want a release build with all optimizations active (LTO, PGO, etc),
please run ./configure --enable-optimizations

Presumably it should be OK to fully optimize python for this application suite?

I do not think it should be a problem to compile with the optimizations. Thanks