Asynchronous processing: Problems in event linking
Added by Joel Lüthi almost 9 years ago
Hi,
We’ve currently set up a modified version of Leginon 3.1 that acquires the whole squares at a low magnification, then gives the images for stitching, processing and target finding to external tools (Fiji, Ilastik, Matlab) and finally feeds the results back into Leginon so that targets can be acquired at high magnification.
The processing part takes quite a bit longer than the image acquisition. To speed up the process, we want to separate the processing from the image acquisition. So far, overview images were acquired and given to processing while Leginon waited for results and then acquired those (see the simplified model in DNA_Fork_MSI.pdf).
Now we want to let Leginon acquire squares, start the processing in a different thread (or eventually on virtual machines) and just continue acquiring images until results come back. To implement this asynchronous processing, I wrote a new node class (a process distributor). This node should start the acquisitions, get notified when a low magnification acquisition is finished and start processing then. When a processing is finished, it should start the high magnification imaging as soon as the next low mag acquisition is finished.
To control when the process distributor starts which process, it needs to get different events from all of the nodes (see MSI_DNA_Forks_ asynchronous.pdf). I started to implement the part of the process distributor controlling the low magnification acquisition, but I’m a RuntimeError.
Specifically, the process distributor gives a targetlist to a square acquisition node and should get a AcquisitionImagePublishEvent back from the subsquare node (for each acquired image). Once the next square should be acquired, it would return a ImageProcessDoneEvent.
But linking the AcquisitionImagePublishEvent from the subsquare node back to the Process Distributor node results in the following runtime error:
manager.py at line 920, in depth
l += depth(child, map)
RuntimeError: maximum recursion depth exceeded.
I assume this is some sort of check whether the whole node structure is linear. This here though should not be linear, by design. And the process distributor would handle the complexity of distributing the tasks. Is there some way to stop Leginon from doing this check?
Alternatively, if this is an inherent part of Leginon and can’t be avoided:
Is there a way to split up the targetLists objects into many targetlist objects each containing only one target? Then the process distributor could submit one of these target lists, wait until it finished processing and then submit the next (or start high magnification processing).
I hope this almost endless description helps to understand what I intend to do and I would be very thankful for any input on how to solve this problem (or whether it should be addressed differently).
Best,
Joel
Replies (3)
RE: Asynchronous processing: Problems in event linking - Added by Anchi Cheng almost 9 years ago
I am not sure of the logic within Manager, but as far as I know it gets all events from all of the nodes. Therefore, it duplicates what process distributor does. What you are proposing may need a different event than AcquisitionImagePublishEvent and ImageProcessDoneEvent since their handling are already defined which is probably while it got into an infinit loop.
You can continue with what you do, or see if you like the following:
I have been wanting to add a pair of PauseEvent and PausedEvent in order to make a big Pause Button that is smart enough to choose how to pause leginon
for each purpose user chooses. Maybe we can use that concept here. We will also need the inverse of that event, let's call that ContinueEvent
Here is the idea:
1. Run Subsquare node with "wait for image to process" deactive. This way Subsquare will keep running until it is told to pause.
2. DNA-fork-Targeting will be still receive AcquisitionImagePublishEvent along the way so that it can do its thing.
3. When DNA-fork-Targeting has some targets ready, you can make it send the PauseEvent to Subsquare.
4. The handling of PauseEvent should be such a way that it acquires a Player lock by self.player.pause(). The advantage of this is that it is equivalent to clicking the toolbar pause button so that Subsquare node will pause at a safe point such as before it process the next target.
5. Once paused, Subsquare sends PausedEvent back to DNA-fork-Targeting, which triggers the submission of the targets.
6. When the targets are exhausted, DNA-fork-Targeting should send ContinueEvent to Subsquare which will remove the lock on self.player with self.player.play().
There is only one target list per image. Within Acquisition class, it will check which target in there is done and only work on the ones not yet done. You don't have to send individual targets. We did it that way so that you can pause (or crash Leginon) and yet recover to the target processing.
The part isn't worked out in my idea is the target adjustment. As I mentioned before, the stitched image target will not be adjustable unless you write a extended TargetTransformer that extracts from it the tile where the target is located.
Hope this makes sense
RE: Asynchronous processing: Problems in event linking - Added by Joel Lüthi almost 9 years ago
Hey Anchi,
That's an awesome idea! I've implemented these events and I'll test them as soon as I have access to the EM again (sometime next week). I need to test whether pause & continue work as expected for me, but if they do, that's a very elegant solution!
The Target Adjustment is somewhat more difficult but needs to be dealt with anyway, not matter how I implement the asynchronous processing.
I work with square grids and I have already written a square centering function that finds the center of a square at low magnifications (at 690x). I'll be trying whether I can adjust the Targets by just re-centering on the square (at low magnification) and replacing the stage-position meta-information in my mosaic image with this new value.
I'll report back once I had a chance to test the code at the EM. As always, thanks a lot for taking the time and suggesting such great solutions to my problems!
Joel
RE: Asynchronous processing: Problems in event linking - Added by Anchi Cheng almost 9 years ago
Joel,
In case you are not using them, Leginon has simulators as defined in pyscope/instruments.cfg.template In fact, we use that for basic test during installation
Test Leginon with Simulator
I added quick instruction on how to fake calibrations for them: See Fake calibrations for Leginon simulator