Bug #2058
closedredux caching problems
0%
Description
Consolidating similar issues into one issue.
redux cache has two known issues:
- "pop from empty list" exceptions: This is likely due to multithreaded or multiprocess access to the same redux cache. The redux server is a multithreaded socket server, but no thread locks were placed around access to the cache. The fix in r17109 should take care of this, but further testing is needed to prove that it is fixed. It is also recommened to increase the cache size (both memory and disk) in pipeline.py. The small hard coded sizes are barely enough to cache a single request in some cases, leading to more frequent "empty list" exceptions.
- some images being cached that should not be: When cache is turned on, it caches everything. Some images, particularly in appion are not "constant". They are written to disk and then later modified in place. For example, a stack file that is being appended to. Images generated for the web by redux should not be cached in these cases, because the cached image will not be up to date with the latest raw data. The current fix is to turn off the cache completely in pipeline.py. Proper solution will be to allow caching only specific results.
Updated by Jim Pulokas over 12 years ago
- Status changed from New to Assigned
- Assignee set to Anchi Cheng
r17204: Requests to the redux server will now accept "cache=off" to turn off caching for that request only.
Note: You still need a global setting of cache on/off in redux.cfg. The client is only capable of disabling cache for a particular request. Client cannot enable cache if it was not already turned on in the config file.
Now how to proceed in the php code to use this in certain places only? Do we only need to use cache on the standard Leginon image viewers and everything else (appion) should have cache=off?
Updated by Dmitry Lyumkis over 12 years ago
I just tested this on cronus3/betamyamiweb, longboard/betamyamiweb, and longboard/myamiweb. cronus3 is working, but the images on longboard (e.g. the average.png file in stackhierarchy.php or the individual particle images in viewstack.php) are not being updated.
Updated by Anchi Cheng over 12 years ago
- Assignee changed from Anchi Cheng to Sargis Dallakyan
- Affected Version changed from Appion/Leginon 2.1.0 to Appion/Leginon 3.0.0
Sargis,
This means longboard does not get nightly updated and/or redux server does not restarted on the new update.
Updated by Sargis Dallakyan over 12 years ago
- Status changed from Assigned to In Test
- Assignee changed from Sargis Dallakyan to Dmitry Lyumkis
Checked on redux and betamyamiweb on longboard. It seems that svn update and redux restart works as expected. svn update for the myamiweb was not in the crontab. Modified cron/root and the updated script looks as follows:
[root@longboard myamiweb]# more /var/spool/cron/root SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin 20 10 * * * cd /srv/www/html/betamyamiweb; svn update 25 10 * * * cd /srv/www/html/myamiweb; svn update 30 10 * * * cd /usr/lib/python2.6/site-packages/redux; svn update 40 10 * * * /etc/init.d/reduxd restart
Please try again to see if this update works.
Updated by Dmitry Lyumkis over 12 years ago
No, neither the particles in viewstack.php, nor the average.png image in stackhierarchy.php are being updated during stack creation on longboard.
Updated by Dmitry Lyumkis about 12 years ago
I think redux is still caching appion stacks and average.mrc files. cronus3/betamyamiweb is not refreshing the images.
Updated by Anchi Cheng almost 12 years ago
r17544 should refresh #images by turn off redux cache when requesting info and should refresh average image of the stack by passing option of coff=1 in loadimg.php
Updated by Dmitry Lyumkis almost 12 years ago
- Status changed from In Test to Closed
tested on new stack that was made in: /ami/data00/appion/13mar29a/stacks/test. Number of images is adjusted as the stack is being made and the particles are displayed in stackviewer.