Using Redux to serve images on myamiweb¶

Redux is new for CentOS 6 (PHP 5.3).

Installation¶

• Install python filesystem abstraction for redux caching:
  

  sudo easy_install fs
  

• Install Myami packages
  

  cd myami
  ./pysetup.sh install
  

Copy the myami/myamiweb directory to your Apache web directory¶

Example:

cd myami

#CentOS example
sudo cp -vr myamiweb /var/www/html/ 

#this is temporary for setup, revert to 755 when finished with this page
sudo chmod 777 /var/www/html/myamiweb  

#if you have SELinux enabled this command will help
sudo chcon -R --type=httpd_sys_content_t /var/www/html

Configuration¶

configure redux¶

> cd /YourMyamDownload/redux
> cp redux.cfg.template redux.cfg

• You can also copy it to /etc/myami/redux.cfg if you prefer.
• Set a writable log path by the user starting it. In this example, we start redux as root and save the log in /var/log/redux.log
• Edit the redux.cfg file as follows:
   
  

  [log]
  file: /var/log/redux.log
  

  
   
• Turn on redux caching if desired:
   
  

  [cache]
  enable: yes
  path: /var/cache/myami/redux
  disksize:  500
  memsize: 500
  

  
   
  • You need to make sure the cache path exists and writable by the user that starts the redux server (reduxd)
  • Input the desired disk_cache_path, disk_cache_size, and mem_cache_size in the next few lines
  • Create the disk_cache_path before running redux if cache will be used

Test Installation¶

1. Using Redux to do simple input output (simple client with no reduxd server needed):

   redux --filename=test.jpg --oformat=PNG > test.png

   Make sure the resulting test.png is in fact an image and not an error message.
2. Run the server: run the following on a command line:

   reduxd

   leave it running for the following...
3. Test command line client connecting to the server

   redux --server_host=localhost --filename=test.jpg --oformat=PNG > test.png

   Note: The input file name is from the perspective of the reduxd server, so be sure to give it an absolute path (unlike the first test, which was not accessing the server)
4. Check the redux.log created as configured in redux.cfg. This contains the port opened by redux server that will allow myamiweb to connect to.

Assign redux client to connect to redux server in myamiweb¶

• At myamiweb/config.php
  

  define('SERVER_HOST',"localhost")
  define('SERVER_PORT',"55123");
  

  
  The server port chosen should match the value found in redux.log when the redux server is started

Setup fftw to use wisdom (myami-3.0 only)¶

This applies only if you see the messge "** Using custom copy of fftw3"* in the previous redux test.

fftw runs much faster on odd image dimension if a wisdom is saved. It is best to create the wisdom ahead of time using the stand alone script pyami/fft/fftwsetup.py. It will store the wisdom you create in your home directory with a name like 'fftw3-wisdom-hostname'. If you only start reduxd as root and fftwsetup.py is run as root, this is enough.

• Redux needs write permission at its installation location to write fftw wisdom file.
• For developer only: you can copy the wisdom file to any other home directory that wants to use it, or copy it as root to a file called /etc/fftw/wisdom which any user can access.

Using fftwsetup.py:¶

FIrst, you need to know the typical image dimension redux needs to process, especially the large ones and ones that are not powers of 2 such as those produced by Gatan K2.

cd /YourMyamDownload/pyami/fft
./fftwsetup.py 1 number_of_rows number_of_colums

For example, for K2 summit super-resolution image there are 7676 rows and 7420 columns, therefore, the command line is

./fftwsetup.py 1 7676 7420

This may take a few minutes.

Run this for as many dimensions you know will be used and move it for general use as instructed above if preferred.

Using redux server with myamiweb (recommended)¶

1. start reduxd server if not already running from the above test:

   reduxd

2. click on "[test dataset]" on your main myamiweb home page, or go directly to the URL: myamiweb/viewerxml.php
3. test by accessing your own images from Leginon in myamiweb/imageviewer.php

starting reduxd at boot¶

sudo cp -v myami/redux/init.d/reduxd /etc/init.d/
sudo service reduxd start

Alternative reduxd installation on file server rather than web server¶

This approach is less tested, but has been found necessary in the case of a very high load system (multiple microscopes acquiring data, multiple processing jobs, multiple web clients and servers, etc). The problem has been that all of this demand for image I/O is occuring over NFS, and the NFS server is not coping with the high demand. One solution we have been investigating is to run reduxd directly on the file server rather than the web server. This means reduxd has direct access to the image files (not through NFS) and the web server now accesses reduxd over the network rather than locally. This has two potential benefits: 1) less load on the NFS server, 2) only transferring JPEGs over the network, not MRCs.

How to do it:

1. Install required myami components on the file server, at least pyami, sinedon, numextension, redux, leginon, modules, and any other 3rd part packages required by those.
2. (optional) edit init.d/reduxd and use the optional high priority command line (see comment in that file)
3. Configure redux.cfg on file server and start reduxd
4. On web server, configure config.php: turn off all caching options, set redux server host to be the file server
5. skip starting reduxd on the web server
6. skip imcache as described below, since this also uses redux locally and accesses images over NFS

Using imcache to cache mrc images as jpeg images of the default size on myamiweb¶

imcache uses redux and replaces the old myamiweb caching system that used mrc2any. It queries the database leginondata and check the cache_path for uncached images and convert them to jpeg images. These jpeg images can be 94 x smaller than the mrc 4kx4k image. Therefore, using imcache will significantly increase the image load time from the image viewer. A default size power spectrum image is also cached.

Installation¶

1. Follow Web_Server_Installation with modification for redux.
2. Install pyami, sinedon, redux on the webserver machine
3. Configure sinedon.cfg for your database
4. imcache can found in your myamiweb installation in the folder myamiweb/imcache/

Configurations:¶

configure imcache¶

> cd /YourMyamiWeb/imcache
> edit imcacheconfig.py

Modify the line

cache_path = '/srv/cache/myamiweb'

to the path you want to use as the base path for the cache to be stored. The path must exist and writable by whoever starts imcached

Assign the same cache path in myamiweb¶

• At myamiweb/config.php
  

  define('ENABLE_CACHE', true);
  define('CACHE_PATH',"/srv/cache/myamiweb");
  

h2 Test Installation

1. Start imcached in its installed location with the reverse order caching
   

   ./imcached r
   

   
   If you have acquired any Leginon images prior to this, imcached will start converting them into jpeg.
2. View a cached image in the imageviewer while checking redux log. jpeg, not mrc image should be requested by the imageviwer.

Start imcache during data collection¶

1. Start imcached in its installed location with the forward order caching in the background
   

   sudo nohup ./imcached f > /var/log/imcached.f
   

2. Wait for the first image to be cached before starting the reverse caching
   

   sudo nohup ./imcached r > /var/log/imcached.r