Leginon

The destination_head option in rawtransfer.py is a filter for the following if statement sudo code:

if leginon_image_path_in_database not start_with(destination_head):
  skip

Therefore, in your new scheme, you can either not to specify the option which would just means process everything by default, or use the non-discriminating --destination_head=/EM. The head seperation by group will just be based on your lab division when you made the directories and included in the user-based leginon.cfg.

Regarding localizing frames directory, it will just be based on the same rule leginon folder is made. It is a simple string replacement of leginon to frames. Therefore, as long as you specify leginon path with the word leginon, you will get the frames path propagated fine.

The umask function we use when creating the leginon session image and frames paths preserve the umask of the parent directory. In this case, for labC, it would be /EM/labC/leginon and /EM/labC/frames. Since these two directories are created by you as the administrator, you should have the control of how it behaves.

The umask function we use when creating the leginon session image and frames paths preserve the umask of the parent directory. In this case, for labC, it would be /EM/labC/leginon and /EM/labC/frames. Since these two directories are created by you as the administrator, you should have the control of how it behaves.

We couldn't get this to work as advertised. Here are the permissions on the parent directories we are able to control by hand (i.e. those not created by leginon):

root@kraken:/EM/taylorlab/frames/18aug09c/rawdata# cd /
root@kraken:/# ls -ld EM
drwxr-xr-x 5 root root 72 Jul 19 12:57 EM
root@kraken:/# cd EM
root@kraken:/EM# ls -ld taylorlab/
drwxr-s--- 4 taylorimages taylorlab 47 Aug  9 13:24 taylorlab/
root@kraken:/EM# cd taylorlab
root@kraken:/EM/taylorlab# ls -l
total 0
drwxr-s--- 16 taylorimages taylorlab 290 Aug 17 11:43 frames
drwxr-s--- 17 taylorimages taylorlab 310 Aug 17 11:42 leginon
root@kraken:/EM/taylorlab# cd frames
root@kraken:/EM/taylorlab/frames# ls -l
total 0
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:33 18aug09b
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:33 18aug09c
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:34 18aug09d
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:34 18aug09e
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:34 18aug09f
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:38 18aug17a
drwxrwsr-x 3 taylorimages taylorlab 29 Aug 17 11:43 18aug17b
drwxrwsr-x 3 taylorimages taylorlab 29 Jul 19 12:35 18jul19g
drwxrwsr-x 3 taylorimages taylorlab 29 Jul 19 12:54 18jul19h
drwxrwsr-x 3 taylorimages taylorlab 29 Jul 19 13:05 18jul19i
drwxrwsr-x 3 taylorimages taylorlab 29 Jun  1 15:08 18jun01a
drwxrwsr-x 3 taylorimages taylorlab 29 Jun  1 15:15 18jun01b
drwxrwsr-x 3 taylorimages taylorlab 29 May 24 12:11 18may24b
drwxrwsr-x 3 taylorimages taylorlab 29 May 24 12:16 18may24c
root@kraken:/EM/taylorlab/frames# cd 18aug09c
root@kraken:/EM/taylorlab/frames/18aug09c# ls -l
total 0
drwxrwsr-x 3 taylorimages taylorlab 77 Aug 17 11:34 rawdata
root@kraken:/EM/taylorlab/frames/18aug09c# ls -l rawdata
total 556216
-rwxrwxrwx 1 taylorimages taylorlab 569561248 Aug  9 13:28 18aug09c_00001en.count.frames.mrc
drwxrwsr-x 2 taylorimages taylorlab       297 Aug 17 11:33 references

Notice that leginon and frames folders (and the parent of these folders) have no permissions for "other", yet the final mrc file is being written with world writeable permissions, and the intermediate folders are set to 755. This is not so much a problem for people who are not members of the lab. The --- other permission prevents them from viewing/copying/modifying the underlying data. The problem is that a person in the taylorlab group will be able to modify, copy, or even delete the mrc files because of the r-x permissions on the parent directories, and I can't restrict this down any further as lab members need to be able to access the images to do research.

One of the PIs has requested the specific system design criterion that none of his grad students or postdocs be able to delete raw data files (some of the research is corporate sponsored), and at this point I'm not sure how to accomplish this without introducing data access restrictions that will be onerous. It would be nice if the final mrc file could be written with rw-r----- permissions.

I just checked the code. In rawtransfer.py, after root performed rsync to put the movie files into the directory, it only did chown, not chmod. Therefore, the behavior you see would be the default behavior on your system.

If you so desire, we can enforce the desired mode as an option.

It seems prudent to have the option of restricting permissions further by default. I can't imagine there aren't other cryo-em groups facing similar challenges. In our case, we're going to be making the facility available to external research groups, which could be competitors; hence the need for raw data security. And my PI's request that there should be a distinction between people who can work with raw data and people who can edit/delete raw data seems reasonable and necessary as well.

Well do that. Stay tuned.

Added this feature in Issue #5958 and pushed to trunk and myami-beta.

I recommend using this to modify, not redefine the permission. This operate recursively from session path.

Thanks! However, it's not entirely clear how to use this. Does this now automatically set permissions to those of the parent directory?

See the Issue #5958. I repeated the option description there.