Leginon

After waiting 1 hour for the first dark reference I killed the app. I will try it without gain correction.

Unfortunately, it is necessary. We do not yet being able to get from DM float number image without scaling problem, so the images comes as integer without the software correction from DM. Leginon has to do the software correction. The hardware correction is done before the counting processs in counted and super resolution modes, therefore the statistics accumulates of the counting still shows gradient due to the variation of the sensor response level. You will see that if you integrate a large number of frames. We know by now that without the gain correction applied after counting, the drift correction does not work as well.

The gain correction should not have taken that long. It is only 4 times the size of the counted image, so it should only takes 4 times the time. Check your memory usage. With our older version of DM (3712 x 3840), our first long run used super resolution and it was not slow. Note that we don't use Gatan's drift correction.

It looks like Leginon chokes when I try correction of the save frames mode. I tried it in counintg and got the same problem. Is frame correction needed or can we get away with just correcting the full exposure?

Something might have changed in Gatan's software that caused this. The Leginon gain correction is only done to the integrated image. Leginon flags DM to use dose fractionation mode with no software correction on its side (set in pyscope). Therefore, from DM's point of view, it should not do any gain correction whether you specify uncorrected or corrected image in Leginon. I am afraid that some low level function does not work right that is related to the version change. We will ask Gatan about it.

Apparently there is a software bug where Gatan scripting under reports the counts in counting and which may be causing the scaling problem that you are talking about. This is also causing problems in SerialEM as well. A patch is being tested today at Brandeis.

Bob, if I understand, it freezes during the acquisition of dark references when frames are turned on? Can you confirm this also in the Manual Application. In the Manual acquisition node, in advanced settings, you can click a check box to do a dark exposure. Try this with and without frames to confirm that is the problem. I will look into how we do dark acquisition to see if it is our problem or the plugin.
Jim

Thanks for the info. Just talked to Jim and we are going to ask David Mastronarde about this since we really communicate to DM through his plug in. Could you give us the DM version you are running for reference ? By the way, in case I caused confusion, the selectrion in DM gui for non-software corrected is called unprocessed.

The other question Jim has is that were you talking about acquiring the dark reference when you say it choked ? When an image is being corrected, it does not actually say that it is at the step of doing dark correction, which is why he wondered. All my reasoning previously assume that you were using the dark reference not acquiring (preparing) it.

If you were talking about acquiring the dark reference, then you should simply not to save fames during the acquisition. Same is true for acquiring bright image. The camera configuration gui, unfortunately, is universal to all nodes. We need to find a way to disable the option in Correction Node.

By the way, in counted and super-resolution mode, you only need to "average" 1 image for dark preparation since it is always zero. For bright, we recommend average 20 images at your desired final exposure time.

We are running GMS 2.21 1586.0 This version does not require no rotation or flip for gain normalization. We do not have the patch yet.

I have a user trying to collect so I can not switch to Manual but when I try and acquire a dark reference in correction node of MSI-T with frames it fails and without save frames it proceeds.

Should we be collecting frames at all if they are not corrected?

Do not collect frames when you acquire bright and dark references. You test confirms that frame saving is not compatible with dark image acquisition.

We collect frames only during real data collection AND only of the final exposures when we have a long exposure time (2 s or more) in counted or super-resolution mode. Long exposures will be drifting, at least for us (A reminder, we do not have Polara, just TF20). The frames allow us to do post-processing that does gain correction and then correct for the drift. It makes a big difference for us. You might not need them.

Bob, when you have a chance, can you start DM, then run the script gatansocket.py which is in pyscope. When run as a stand-alone script, it should simply tell you the current DM version that is in use. It should give you a single integer value (like 340), rather than the human readable version (like GMS 2.21 1586.0). Let me know what it says.
Thanks,
Jim

Version 31100. We should have the patch I mentioned installed tomorrow.

I have confirmed it Leginon can not collect a dark reference in super resolution mode even without frames. The DM patch that I was talking about has been installed.

It looks like the choice of image size may not be available. See output below.I will try it with other sizes.

Exception in thread Thread-300:
Traceback (most recent call last):
File "/usr/lib64/python2.6/threading.py", line 522, in _bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 477, in run
self._target(*self.__args, **self.__kwargs)
File "/usr/local/lib64/python2.6/site-packages/leginon/corrector.py", line 106, in acquireDark
imagedata = self.acquireReference(type='dark', channel=channel)
File "/usr/local/lib64/python2.6/site-packages/leginon/corrector.py", line 274, in acquireReference
refimagedata = self.acquireSeriesAverage(n, type, channel)
File "/usr/local/lib64/python2.6/site-packages/leginon/corrector.py", line 220, in acquireSeriesAverage
finaldata = self.storeCorrectorImageData(finaldata, type, channel)
File "/usr/local/lib64/python2.6/site-packages/leginon/correctorclient.py", line 569, in storeCorrectorImageData
raise RuntimeError('%s: bad array shape: %s' % (type, shape,))
RuntimeError: dark: bad array shape: (1918, 1854)

1855 X 1855 4x bin did not work
3710 X 3838 2x bin worked

Wish I know what rule Gatan people are following. Sorry that you have to do this kind of testing.

It seems the correction is not applied properly. DM gives the following output.

A problem occurred acquiring an image for SerialEM:
Warning: image is larger than the supplied array

A problem occurred acquiring an image for SerialEM:
Warning: image is larger than the supplied array

A problem occurred acquiring an image for SerialEM:
Warning: image is larger than the supplied array
The "corrected" image looks worse than the uncorrected.

The choice of super resolution formats is inconsistent. The selection in the preset is larger than the correction choices.
That seems to be the problem. Correction displays 3710 X 3838 bin 1 when binning 1 for super resolution is 7420 X 7676. There is also a mismatch for binning 2 as well correction is 1856 X 1919 while 2X binning really is 3710 X 3838. Serial EM uses a more consistent nomenclature where super resolution 1x binning is called 0.5 bin. This prevents errors when selecting exactly which pixel configuration you want.

I am getting help from Tom Sha at Gatan in figuring out the rule. Please standby.

By the way, does this inconsistency in correction displays of unbinned super resolution image happens only after you apply the patch from SerialEM or before as well? Maybe we have to account for the patch change, too.

When we go to the correction node we sometimes get the full format choices for super resolution while other times it appears that there are only the choices for counting. I suspect that this inconsistent user interface may be leading to the DM errors reported above.

As for problems since the patch I am not sure as we had limited experience with Super resolution before the patch. Was your dialog withe Tom helpful?

WIth Tom's help, I think I understand the problem now.

• DM image acquisition sequence in the backend (not the front end) requires the dimension and binning input as in physical pixel number. Therefore, whether we are using counted or super-resolution mode, the binning number is the same.

• When we want to use super-resolution, the acquisition sequence returns an array twice the size by recognizing that we set the mode to super-resolution.

• Regarding the number of pixels when a physical binning does not result in integer of the full size, here is what we need to enter, using 3710 as full size example:
  • top: 0, bottom: floor( 3710 / bin ) * bin
  • bin: bin

We will have to make changes in our code to use this new definition for super resolution mode because we need to define an array size to fill by the acquisition. I do not know why super resolution "camera" does not show up sometimes. It might have to do with the last values it need to read, there should not be any communication issue if counted and linear modes already show up.

Update on this. I started an issue for flooring camera binning #2330 and also give better default square dimensions for fft purpose #2433. They have been used locally for a while now.

Hi! We just installed a K2 on our Polara and I just started to get Leginon set up for this new camera. (We upgraded to Leginon 3.0, and DMS is version 2.31.669.1)

At the moment, I have a problem with the correction images in Leginon. For the Orius it works just fine. For the K2, it takes forever and it seems it crashes. In every mode at every setting....
The following error pops up in DM: "Warning: image is larger than the supplied array", in Leginon: "error: dark: bad array shape". So I assume the same error as described above by Bob.

I suspect it has something to do with the fact that the Orius camera is square, so no problems there. The problem for the K2 on the other hand might be in the defined dimensions? E.g. a 3710 x 3838 bin 1 image in leginon is actually described as 3838 x 3710 image in DM. This swap is true for all settings. Unless Gatan changed something in their software recently, I am not sure if this can be the problem, as nobody else complained about this?

Do you have any idea what might be going on?
Thanks,
Mandy

Okay, nevermind, I just flipped the image 270 degrees as was suggested in your site. It works now. Sorry for my too fast reaction here :-)

Mandy

Hi Mandy,
The first thing I would check is if you are rotating your image 270 degrees and flipping it in DM. That is what Leginon is expecting. This would make the x and y dimensions correct. Anchi may have other thoughts but that is the easiest thing to try right away. Good luck.
Bob

Nice to have more than me answering !

Yes, about rotation and flip. I just learned that a different rotation is needed on Krios to match what is shown on Flucam (equivalent to viewing screen on other scope). I will have to make a proper config and get people to check individually. But you two have the same type of scope, so it is the same.

About DM 2.31. David Mastronarde only just informed me that he put out the proper SerialEMCCDplugin today. I am surprised that you've already got it working.
If you have problem, try the new one from ftp://bio3d.colorado.edu/SerialEM/DMpluginBeta from what I heard from David, our function calls should remain the same.