Project

General

Profile

Removing projects/sessions?

Added by Patrick Goetz over 6 years ago

In the process of testing our new Cryo-EM lab we've generated an awful lot of junk projects and sessions which are currently cluttering up the database. Is there any (user-friendly) way to delete projects and/or sessions?

Related question: When I first set up the system, the "root" directory for data collection was /cryodata. With the addition of 2 new independent research groups, the data will now be going to:

   /EM/group_A
   /EM/group_B
    ...
   /EM/group_N

All of the existing data belongs to group_A, and if possible, I need to move it to /EM/group_A. Of course the projectdb database has projects referring to the old /cryodata location. Is there any way to migrate the information in the database so that the project data can still be located after being transferred to the new /EM/group_A area?


Replies (4)

RE: Removing projects/sessions? - Added by Anchi Cheng over 6 years ago

If you have never saved any images in those sessions, you can delete them in database tables. If you have some images saved, and thus references to the SessionData entry, I think you should just hide them. This is more an administrator level thing and I think you should use phpMyAdmin. The field should already defined. You just need to change value.

As far as migrating data, you will find them in myami/dbschema/tools type, for example,

python leginon_path_change.py

to see the instruction.

There are
  • leginon_path_change.py
  • batch_leginon_path_change.py
  • appion_path_change.py
  • batch_appion_path_change.py

RE: Removing projects/sessions? - Added by Patrick Goetz over 6 years ago

Hi -

Can you explain what you mean by "saved images"? As soon as images are collected, aren't they automatically encoded in the database?

I think you should use phpMyAdmin

Is there a specific reason for this? I'm pretty comfortable running SQL commands. I really would like to get rid of sessions/projects which were just created for testing. Does this involve so many tables that it's impractical?

batch_leginon_path_change.py and batch_appion_path_change.py look like what I need, but the help is a bit confusing:

cnsit@kraken:~/leginon/myami-3.3/dbschema/tools$ python batch_leginon_path_change.py
usage: python batch_leginon_path_change.py new_path_base
this searches database based on the session name in the new_path_base
and make the image path or frame path change
new_path_base needs to be the parent directory of the sessionname
For example, /data/leginon, and exists

So, for point of clarification, if my base path changed from /cryodata to /EM/taylorlab, the command

python batch_leginon_path_change.py /EM/taylorlab

would convert the path locations in the database from /cryodata/* to /EM/taylorlab/* ?

/cryodata/frames  --> /EM/taylorlab/frames
/cryodata/leginon --> /EM/taylorlab/leginon

And one additional question: batch_leginon_path_change.py updates the database, but doesn't physically move the actual images?

RE: Removing projects/sessions? - Added by Anchi Cheng over 6 years ago

First, about "saved images".

Any image that you can see through myamiweb before the session is hidden has its metadata saved in the database, and that metadata includes reference to the session. That is what I meant by saved image.

There are other places that reference the SessionData. Calibrations, for example.

Since you are familiar to mysql command line, you can do a search of all tables in leginondb for the field `REF|SessionData|session`, and find out how many tables have the reference to the session. By reference I mean its value equals the `DEF_id` of the SessionData you want to remove. You will also find a reference of the session in projectdb.`projectexperiments`. Look for the field value of `REF|leginondata|SessionData|session`.

I need you to find out if any of your to-be-removed sessions has '%SettingsData' for a user you are keeping, You will change the user experience back to the default if you remove SettingsData for a user. Another group of tables that is critical not to clean up is '%CalibrationData'. They are shared by all users, it is not good to remove calibrations.

Second, the batch change scripts. For the example behavior you have, you should run it twice with different base path

python batch_leginon_path_change.py /EM/taylorlab/frames
python batch_leginon_path_change.py /EM/taylorlab/leginon

RE: Removing projects/sessions? - Added by Anchi Cheng over 6 years ago

Forgot to answer the last question.

The python scripts does not physically move the actual images. You do need to move the files before running the scripts. It, in fact, check if the files are in the new path first.

    (1-4/4)