Leginon

The cache is on an actual disk, /var/www/html/cache , where /var is a separate partition that is currently 90% full.

The machine has 24 Gb of RAM

It is a semi-dedicated webserver with a handful of shared processes, but none of them are consuming resources when the crash happens

In pipeline.py: mem_cache_size = 400*1024*1024 # 400 MB

The crash only happens when browsing from images of one FFT image to the next

I ran top with processes sorted by memory and discovered something interesting. When using redux to browse from image to image, python memory briefly rose to 1% then fell to 0. When I turned on FFT and browsed from image to image, memory use rose by about a half a percent each time I clicked an image. I stopped clicking when python hit 9%. Memory use stayed 9% despite not doing anything with the browser. Seems like a memory leak?

I tried updating to the latest trunk version of redux, but that broke everything, so I reverted to r17113.

Found major problem introduced in r17138: the cache size from config file was read as a string rather than integer, causing completely wrong cache clean up. This is probably not Scott's problem, because that was reported before r17138. I fixed this in r17291.

For Scott's problem, I am going to suspect the way that pyami.fft causes an image to hold a reference to its fft and power spectrum, which would hold on to a lot more memory than we are actually keeping track of in the caching code. My attempt to fix it is in r17292.

Got the same memory error while running Leginon Manual Application with calculating FFT on for 4kx4k images. It went fine for maybe 15 images until the same memory error on the same function of numpy.roll. Looks like this issue originate at lower level than redux.

xception in thread data binder handler thread:
Traceback (most recent call last):
  File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap
    self.run()
  File "/usr/lib64/python2.4/threading.py", line 422, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/acheng/myami/leginon/databinder.py", line 131, in handleData
    method(args)
  File "/home/acheng/myami/leginon/watcher.py", line 35, in handleEvent
    self.processEvent(pubevent)
  File "/home/acheng/myami/leginon/watcher.py", line 43, in processEvent
    self.processData(newdata)
  File "/home/acheng/myami/leginon/imagewatcher.py", line 44, in processData
    self.processImageData(idata)
  File "/home/acheng/myami/leginon/fftmaker.py", line 47, in processImageData
    pow = self.calculatePowerImage(imagedata)
  File "/home/acheng/myami/leginon/fftmaker.py", line 68, in calculatePowerImage
    pow = imagefun.power(imarray, self.settings['mask radius'])
  File "/home/acheng/myami/pyami/imagefun.py", line 101, in power
    pow = swap_quadrants(pow)
  File "/home/acheng/myami/pyami/imagefun.py", line 184, in swap_quadrants
    a = numpy.roll(a, shift1, 1)
  File "/usr/lib64/python2.4/site-packages/numpy/core/numeric.py", line 356, in roll
    res = a.take(indexes, axis)
MemoryError

I haven't had problems related to this specific issue, but I have had some problems with random redux crashes lately. I have tracked it down (I think) to the redux log. Redux seems to be OK for a long time (weeks) then will start to crash with increasing frequency until I'm restarting it multiple times a day. I have been able to fix this temporarily by removing redux.log from /var/log. To answer your next question, no it is not unusually large and no the disk partition is not close to being full.