Appion

I also got this error with DoG picking, on Garibaldi. It ran to around 80% completion before dying this way. Both makestack2 and DoGpicker call appionLoop2.py:

Traceback (most recent call last):
  File "/opt/applications/myami/2.2/bin/dogPicker.py", line 78, in <module>
    imgLoop.run()
  File "/opt/applications/myami/2.2/lib/python2.6/site-packages/appionlib/appionLoop2.py", line 61, in run
    if not self._startLoop(imgdata):
  File "/opt/applications/myami/2.2/lib/python2.6/site-packages/appionlib/appionLoop2.py", line 548, in _startLoop
    self._checkMemLeak()
  File "/opt/applications/myami/2.2/lib/python2.6/site-packages/appionlib/appionLoop2.py", line 659, in _checkMemLeak
    apDisplay.printError("Memory leak of "+str(round(memleak,2))+"MB")
  File "/opt/applications/myami/2.2/lib/python2.6/site-packages/appionlib/apDisplay.py", line 57, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception: 
 *** FATAL ERROR ***
Memory leak of 554.22MB

Opps, sorry guys, I meant to look into this last week. I've put it on my calendar to do this Monday.

Here's the directory for the crashed DoG Picker run I mentioned:


/ami/data15/appion/13mar14d/extract/dogrun1


Everyone, (Jim, Neil, Sargis - your input would be helpful)
It would be useful to have more examples of this bug happening on guppy to determine the exact cause. If you encounter a memory leak, please add it to this issue, noting the run directory. Then make sure to leave the directory intact - don't use the same run name for subsequent runs. Garibaldi cases are good to, but I may need to get JC involved to examine logs.

1. This error occurs because Appion is checking the memory logs on the processing node at /proc/meminfo. This information is not just for the Appion job running, but all processes on the node. So the memory leak could be caused by other jobs that are using the same node. For the failure that I could trace (Dipas), Frealign was also running on the node at the same time. A CL2D job was also terminated a few hours earlier by the linux Out-Of-Memory Killer, also during the Frealign run. Apparently, jobs being stopped by the OOM killer is supposed to be extremely rare since virtual memory has been in use.
2. This makes me think that perhaps Frealign is using memory in a way that looks like a memory leak to the Appion functions that looks for it. Neil, this was your function, lots of un-commented numbers and math. I wonder if this function is actually finding a memory leak, or if the function is not designed for the way the node is currently being used, and it just looks like a memory leak.
   

       #=====================
       def _checkMemLeak(self):
           """ 
           unnecessary code for determining if the program is eating memory over time
           """ 
           ### Memory leak code:
           #self.stats['memlist'].append(mem.mySize()/1024)
           self.stats['memlist'].append(mem.active())
           memfree = mem.free()
           swapfree = mem.swapfree()
           minavailmem = 64*1024; # 64 MB, size of one image
           if(memfree < minavailmem):
               apDisplay.printError("Memory is low ("+str(int(memfree/1024))+"MB): there is probably a memory leak")
   
           if(self.stats['count'] > 15):
               memlist = self.stats['memlist'][-15:]
               n       = len(memlist)
   
               gain    = (memlist[n-1] - memlist[0])/1024.0
               sumx    = n*(n-1.0)/2.0
               sumxsq  = n*(n-1.0)*(2.0*n-1.0)/6.0
               sumy = 0.0; sumxy = 0.0; sumysq = 0.0
               for i in range(n):
                   value  = float(memlist[i])/1024.0
                   sumxy  += float(i)*value
                   sumy   += value
                   sumysq += value**2
               ###
               stdx  = math.sqrt(n*sumxsq - sumx**2)
               stdy  = math.sqrt(n*sumysq - sumy**2)
               rho   = float(n*sumxy - sumx*sumy)/float(stdx*stdy+1e-6)
               slope = float(n*sumxy - sumx*sumy)/float(n*sumxsq - sumx*sumx)
               memleak = rho*slope
               ###
               if(self.stats['memleak'] > 3 and slope > 20 and memleak > 512 and gain > 2048):
                   apDisplay.printError("Memory leak of "+str(round(memleak,2))+"MB")
               elif(memleak > 32):
                   self.stats['memleak'] += 1
                   apDisplay.printWarning("substantial memory leak "+str(round(memleak,2))+"MB")
                   print "(",str(n),round(slope,5),round(rho,5),round(gain,2),")" 
   

3. Options to fix this:
   1. Run Frealign, Xmipp, and maybe dog picker and makestack against a memory analyzer to see if there is actually a memory leak. For Xmipp, this might require recompiling for debug mode, and installing analyzers. Jim, do you know how to analyze the python programs for a mem leak? I have not looked into Frealign, is it written in C?
   2. Rewrite the mem.py function that reads info from /proc/meminfo to use the job specific memory data instead. Neil, thoughts on that?
   3. Comment out the memory check and rely on the OOM Killer to do its job. The downside is that there is no indication what the error was when a job dies (without root access anyway), which is what we currently see with xmipp CL2D. The upside is that the OOM will use a highly tuned algorithm to chose which job dies, and our dog runs and make stack runs would likely survive.
4. On a side note, during our cronus2 crisis meeting with JC, do I remember JC telling us that when we run CL2D we need to make sure we ask for all the processors on the node to make sure it has enough memory?

Any other thoughts from anyone? Any other ideas for debugging/fixing from Jim or Sargis?
Thanks!

On a side note, during our cronus2 crisis meeting with JC, do I remember JC telling us that when we run CL2D we need to make sure we ask for all the processors on the node to make sure it has enough memory

This seems to help with my Garibaldi runs -- if I request the maximum 47 GB per node (even if only using one core) this excludes other processes from using the node and the memory leak never seems to happen.

This is regarding the function checkMemLeak.

The first message, which is an Appion error, Memory is low ("+str(int(memfree/1024))+"MB): there is probably a memory leak"), never appears in my log files. So there's always more than 64 MB of available memory. Not very useful finding, except for implying that mem.free() is reporting a reasonable value.

The third message, which is an Appion warning, "substantial memory leak "+str(round(memleak,2))+"MB"), very frequently appears in my log files. It appears in runs that do not crash as well as runs that do crash.

The second message, an Appion error, "Memory leak of "+str(round(memleak,2))+"MB", only appears for the runs that actually crash. I find its conditional to be particularly opaque;

if(self.stats['memleak'] > 3 and slope > 20 and memleak > 512 and gain > 2048)

I don't understand the motivation behind the definitions of rho, gain, and slope, but it seems that that the condition (rho*slope)>32 frequently happens without compromising the run.

It is a pretty crazy if statement I agree:

• rho is the correlation coefficient
• slope is calculated slope of linear regression on the amount of memory used after each image
• gain is the total increase in memory used from the beginning of the run
• memleak = rho*slope

a large slope implies memory leak (note this was a HUGE issue with sinedon bugs in 2007), but if the correlation coefficient is low that it is likely an outlier. So, multiplying rho*slope reduced the false positives for memory leaks.

Since we have not had memory leaks in a long time, we might as well disable this functionality unless a debug flag is used.

I've changed the Errors into Warnings, so all runs will continue despite having memory leaks.

Thanks Neil,
It sounds like this function can have false positives?
I was going to change the error to warning as well, but if there is truly a memory leak I'd like to know about it.
Do you think there is anything we should still be concerned about? Do you think Frealign or cl2d may have a memory leak based on what we've seen? Or do you think everything is probably working as it should be?

Ryan, the reason you see the program die for some of the messages and not all of the messages, is that the "printWarning()" function just print a warning, but the printError() function actually throws an exception in addition to printing the error. Yes, this is poorly named, the function should be named something like terminate() that takes a termination message as a parameter.