Leginon stops after taking 1 exposure
Added by Anonymous over 14 years ago
We recently tried running a queuing session, but when we tried going back to depth-first data acquisition, we noticed that Leginon seems to get held up waiting for something after taking one exposure. We're not sure what's going on, so we tracked the messages in the log and got this for each of the nodes:
Square:
4:03:27 PM Processing: waiting for 5928 to be processed
Hole Targeting:
4:04:47 PM Waiting for target list ID 9140...
4:04:46 PM Publishing targets...
Hole:
4:06:03 PM Processing: Waiting for None to be processed
4:06:03 PM output: image displayed
Preview:
4:06:29 PM Acquisition processData done
Exposure targeting:
4:08:37 PM Exposure Targeting done waiting
4:08:37 PM Target ID 11542 has been processed. Status success
Exposure:
4:08:37 PM Acquisition processData done
4:08:37 PM Continuing...
4:08:37 PM Exposure done with target list ID: (('leginon1.localdomain', 49152), 11542), status success
For some reason, the process is stopping after one exposure (which it takes fine). It both seems to think it's done and be "waiting for None" , despite there being plenty of hole targets left in the square. We've tried killing the session and starting again, reloading the atlas (when it freezes, we can't submit new square nor does it refresh), and removing as many of the "wait for..." options in the settings as possible (though we might have not tried all combinations).
The other thing we've been trying to do is take images with only one exposure setting (near focus "en"), so we thought that maybe it was waiting for "ef", but importing it and restarting it didn't solve the problem.
We now think the problem is that we have some left over settings from our queuing attempt that might be messing up the depth first search. What settings should we look for in making sure Leginon doesn't wait for processes that don't exist?
Also: we got an error in the terminal window that said:
**Datamanager size reached, removing data as needed**
I've checked the hard disk, and there's plenty of space left. We tried deleting some of the data we've already backed up to free up some space, but it didn't seem to work. Is there some way of getting rid of the error?
Thank you for the help!
Replies (7)
Re: Leginon stops after taking 1 exposure - Added by Anonymous over 14 years ago
Sorry, I should clarify that Leginon isn't crashing--it's just stopping and waiting. The computer itself also isn't crashing, so we think we can rule out a memory issue.
Re: Leginon stops after taking 1 exposure - Added by Jim Pulokas over 14 years ago
In almost all cases, you can ignore the "Datamanager size reached" message. If your machine really is running out of memory, then this message can help diagnose the cause, but in your case, I think there is just a problem with the settings for queuing and non-queuing setup.
If you are doing depth first acquisition, then you need to have Square and Hole set to "wait for a node to process". Then you need to make sure Hole Targeting and Exposure Targeting have queuing turned off. Start with making sure those are correct. If you still have the same problem, it might be easiest for you to make screen shots of each of your settings windows so we can look at it.
Jim
Re: Leginon stops after taking 1 exposure - Added by Anonymous over 14 years ago
Thank you, Jim. I agree that it's almost certainly the queuing options that are messing up the depth first search, so I'll check those settings today so that if we run into any problems tomorrow we can have plenty of time to diagnose it.
If we return to an older session (i.e. a session before we tried queuing) would the settings for queuing be "switched off" or do they persist session to session? It might allow us at least to take a look at what's different. We also have database dumps from before and a while back--would it be possible to judge from these what the settings were on the Leginon computer?
I'll still try fixing that Datamanager size reached message, though. Anchi posted on how to fix it because someone else reported seeing it, so I'll give it a go. When it says that it deletes files as needed, how do I determine which (if any) files it's going after?
Thank you again for your help!
~Carlos
Re: Leginon stops after taking 1 exposure - Added by Anonymous over 14 years ago
Hi Jim,
I went down and took some screenshots of our settings. We checked our settings according to your recommendation, but everything seemed to be in order (at least to us). I'll email you the screenshots shortly.
If you see anything weird, please let us know!
Thank you again for the help,
~Carlos
Re: Leginon stops after taking 1 exposure - Added by Jim Pulokas over 14 years ago
The only thing that looks unusual is in Hole. You have "save image to database" turned off. I have not thought about how this would cause the problem you are having, but you should try turning it on to see if it fixes the problem. The other thing you can try is to "skip automated hole finder" in both of the hole finder nodes. That will at least simplify some of the processing and lead to less memory usage if that is the problem. You will have to then pick all of your targets manually.
Let us know if any of that works.
Jim
Re: Leginon stops after taking 1 exposure - Added by Anonymous over 14 years ago
Ok, I can try that, but those settings for hole finder were working fine before, so I'm not sure how effective turning those off might be. I'll let you know if the problem persists or if we fix it!
Thanks for the help,
~Carlos
Re: Leginon stops after taking 1 exposure - Added by Anonymous over 14 years ago
Ah, sorry for doubting you Jim! Switching the checkbox to "save to database" in the Hole node did the trick. It worked after we ticked it off, so it wasn't a memory issue after all (which is great!), so it's picking holes automatically and saving images to the database. I tried increasing the memory in sinedon too, but I'm still getting the error--we're going to go back and get rid of some less than optimal data, so at least that should work.
Thank you for the help! Leginon is working beautifully now. I think we might stick with depth-first searching for a while.
~Carlos