Project

General

Profile

en.mrc auto regeneration

Added by Harry Kao almost 11 years ago

Hi,

In our lab, the *en.mrc images are unaligned average of a image stack from the K2 camera. These files are not used for further analysis/processing except for real time monitoring and perhaps quick view of the the contents of the stacks later on. However these files do take up significant storage space. For example, 10K stacks in 20f/s, counting mode, would produce ~500GB of the en.mrc files. In super-resolution mode, the file size will be 4X as large.

Could we delete the *en.mrc files at the end of the imaging session and have Leginon auto-regenerate the en.mrc from the image stack on demand, if an user needs to view the content of a particular stack later on, via the myamiweb?

Thank you.

Harry


Replies (6)

RE: en.mrc auto regeneration - Added by Anchi Cheng almost 11 years ago

You can certainly delete the unaligned images to save space since you will only use the aligned images for data analysis. You will just get an error from redux saying that it could not find the file.

The time you spend one regenerating the 4k image from 20 frame-integration is probably not a good time efficient strategy for an evaluation purpose which normally means browsing through files quickly.

Instead, you may want to consider creating jpeg images of these so that myamiweb can use them. The imcache server we have is probably a good option for you. See instruction in Using_imcache_to_cache_mrc_images_as_jpeg_images_of_the_default_size_on_myamiweb. It is a python program that opens a redux thread to convert all images into jpeg and put it in the directory myamiweb would look for cached jpeg images. It runs as a server querying the database at all time to wait for more data coming in.

If you want to customize it, it is a python script, so go ahead to modify it for your special case.

RE: en.mrc auto regeneration - Added by Harry Kao almost 11 years ago

Thank you for pointing me to the imached script. This is exactly what we are looking for. Would you be able to confirm that the script will not convert the past+future reference images and the frame stacks into jpegs?

Or how can we specify a different directory for the reference images from now on, so the imached won't convert them. The previous ref images have been saved under the same 'Image' directory as all the other projects, in a sub-dir marked by a certain date plus '_ref'. Another oddity is all ref images have been saved to the same dir since Nov. of 2013.

Thank you.

RE: en.mrc auto regeneration - Added by Anchi Cheng almost 11 years ago

The script only converts what can be viewed in the myamiweb imageviewer. In other words, the ones you asked about won't be converted.

The reference session creation works like this:

1. If you have write permission to a reference session created in the last 90 days, it will write the references in there.
2. If you have no write permission to any of the above reference sessions, it will create a new one for you to write in.

If your lab prefer to change the timeout, it is in leginon/correctorclient.py in the function getReferenceSession.

RE: en.mrc auto regeneration - Added by Harry Kao almost 11 years ago

We are running imcached now, and the compression ratio is great. The new situation is that We have two microscopes, how can we restrict imcached to convert only images generated by the k2 camera(F30 microscope)? Thank you.

RE: en.mrc auto regeneration - Added by Anchi Cheng almost 11 years ago

Hi, Harry,

I created a feature for this. see Issue #2666

You can download the files involved and replaces the ones in myamiweb/imcache with them.

RE: en.mrc auto regeneration - Added by Harry Kao almost 11 years ago

Hi Anchi,

Thank you for the prompt update. I installed the new scripts, and only K2 images are being converted now. The script converts about 50 images per min, and the average compression ratio is about 58 to 1. This is great.

In our case, the K2 camera name as recorded in mysql database under CameraEMData is GatanK2Counting. The other camera is just Gatan.

Thank you.

    (1-6/6)