Feature #2203
closed
default image cache using redux
Added by Jim Pulokas almost 12 years ago.
Updated almost 11 years ago.
Description
Re-implement the default image caching that was done before redux came to be. This would pre-cache jpegs for a whole session to speed up image viewing, as long as default parameters are used.
Originally, this was triggered by someone viewing an image in myamiweb. A script would be launched to convert all mrc images to jpeg images for that session. This had a drawback of large bursts of processor/disk access the first time a session is viewed, which slows things down. Now we can try a daemon process that will always be converting new mrc images as they arrive, spreading out the load over time, which will hopefully not be as noticeable.
r17337 deactivate startcache in viewers. The function itself can be cleaned up after testing imcache.
r17338 added the use of the cached image. This is only in the real-space image side. The scale, and the pixel size calculation in the reciprocal space does not work quite right, yet. Maybe we should cache the power spectrum. The default size is better at 2048 not 1024 to get more details.
r17351: added caching of power image. Note: this also depends on r17341, a change in redux that allows specifying the output of one pipeline as the input of another (previously, all pipelines had to start with a Read pipe).
info on imcachconfig.py configuration parameters: redux_maxsize1 is the initial size, before calculating power, then both power and real image get scaled to redux_maxsize2. There are 3 separate redux pipeline parameters. sizes and filenames are determined at run time, so not specified in the config redux parameters.
For now, the best way to start the imcached script is to sent your environment PYTHONPATH to whatever myami sandbox, then cd to the myamiweb/imcache directory and start imcached from there. This is how you get it to use that specific imcacheconfig.py. In the future we can think about using the standard config file format in our standard locations instead of a python module.
Got this error when cronus2 was slow. It did a few query while redux was trying. May be because it requested too many pipes before things are done. Only imcached cashed, reduxd was fine.
Running Format(139945402934416,oformat=JPEG,rgb=False)
done /home/acheng/myami/myamiweb/imcache/cachetest/12nov30d/12nov30d_d_00013gr_00038sq_v02_00011hl_v01_00002es-a.pow.jpg
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "imcached", line 130, in run
m = task['missing'][key]
KeyError: 'jpg'
r17391 makes it so that a forward and a reverse imcached can be run by specifying the option. This way, it is possible to catch up missed cache in the past while the new ones is cached as soon as it comes in.
We run it in myamiweb/imcahed as
nohup ./imcached f > /var/log/imcached.f &
and
nohup ./imcached r > /var/log/imcached.r &
respectively.
r17407 allows start_id in imcacheconfig.py to be used with the forward/reverse option if set to a non-zero number.
- Status changed from Assigned to Closed
- Target version set to Appion/Leginon 3.0.0
Also available in: Atom
PDF