Leginon

My guess is that linear mode references were saved in the database NOT as linear mode references because the Correction Node gui didn't realize that it IS in linear mode. The camera and tem selectrion in the settings often received the connection too late to register. It will show as the first camera in the list but internally still thinks it is selected as None. We don't have a good permanent fix for this. You will have to use this workaround:

Choose another camera first in Correction node Settings and then choose back the one you really want. It triggers the registration of the real selection.

By the way, copycal.py in http://emg.nysbc.org/boards/6/topics/1496#message-1498 can be used for your linear mode calibrations, too. It is also not a good idea to do matrix calibration with full size camera. It is still quite big for correlation.

Anchi

It looks like the dimensions are transposed. See snapshot attached for correction which is different from what matrix looks for.

I will check on our scope and camera tomorrow. We have not used linear mode for a long time. In the mean time, check your instruments.cfg for the height and width for K2Linear and flip and rotate parameters in DM, please.

Also take a screenshot on the python window of Leginon client on K2 computer. It should log the camera dimension as acquired.

Screen shot form K2 computer during linear gain correction

After gain correction I tried taking an image in Linear with Navigator and got a blank image. See screen shot of Leginon window attached. I see no indication it is communicating with the linear camera in the leginon log.

I've tried it on our K2, and it works fine.

From what you posted, it appears that you do get the image and its gain references at the right dimension now. However, the gain/dark references are probably taking incorrectly.

Your exposure time is 10 s. CMOS camera like K2 linear mode has strong dark current (The part of values that increases with exposure time without beam). When linear mode is used in Leginon, we do not have HW dark subtraction in DM turned on (This could change if necessary. It was done to reduce the need to redo Gain reference acquisition for counted/super-resolution mode every time someone redo linear mode gain reference in DM as we the case when we first got the camera). As a result, the dark current alone for 10 s exposure probably saturated the 16-bit intensity range that is transferred to Leginon. Linear mode data collection needs strong beam and short exposure time to give it good signal to noise ratio. Therefore, you will do better stay at the exposure time you will eventually use during dark/bright image collection and set it to average large number of images, than take a very long exposure but do no averaging. I added a section for recommended dose rate in Using_Gatan_K2_Summit_in_Leginon to help you and future users.

I notice that DM turns Hardware Dark an Gain correction on when I run Dark correction in Leginon. If I understand your last post this is incorrect. Am I missing something?

The new gain corrected image looks better.

DM gui is not always synced with what we give it in DM scripting through Leginon. If the record mode in DM does say Linear mode, and HW Dark Correction is indeed on when Leginon collects data, turn it off. It is possible that it is displaying what is in Counted mode still. It is a bit of a mess.

With HW dark correction off, a dark image (whether as Dark Image in Leginon Correction Node, or just because you did not let the beam go through to the detector) should have about 22000 counts for 0.5 second. If HW dark correction is off in DM, it should be around 0.

I got 1750 counts from 0.5 second exposure of the 64 e/pixel/s dose rate I mentioned in the wiki after dark and gain correction in Leginon.

I turn background subtraction off and Leginon turns it on in DM in Linear mode but the mean etc for the dark image are all Zero. I think we should be safe;)

Now I get the opposite effect. Leginon turns off the Linear DM hardware background and gain correction and I get a dark ref with 42000 mean. What happened?

I don't really know. DM gui is not quite independent to internal change through scripting. I do know that we set it to off in the script. This is one thing frustrated our linear mode users. No one uses that any more. I am just glad that it does not happen to counted and superresolution mode.

You can go to pyscope/dmsem.py on the K2 computer and change the last line of this

class GatanK2Linear(Gatan.K2Base):
  name = 'GatanK2Linear'
  ed_mode = 'linear'
  hw_proc = 'none'

to

hw_proc = 'dark'

to see if it is more stable.

Still getting 47000 mean for dark reference!

What if you take a linear mode image in DM with HW dark subtraction on. Would that reset its internal value?

Still 47000 mean for dark reference.

Is dark reference for linear all Zeros? It shouldn't be if it is integrating charge rather than counting. The one time it seemed to work I got all zeros for the statistics which is confusing.

Sorry, out of idea.

Jim wonders if you get 47000 or zero if you take this image in DM (just blank the beam). (Make sure your software correction is set to uncorrect, or you won't see the difference).

When I do a blanked unprocessed with software and HW processed image in DM The statistics are as follows:
Mean 5.14
SD 42

When not HW or software corrected
Mean 47718
SD 5343

When software corrected but not HW corrected:
Mean 46389
SD 6067

The image seems not to be corrected at all

This means that the DM software Dark Reference was taken when HW Dark subtraction was activated. This is fine. The thing is to find out why Leginon's setting of hw_proc has no effect on the image delievered since you get the same counts back whether you turn it on or off. I will give it another try here to see if I can reproduce your problem. If yes, I will talk with Gatan about it. It has worked before and yesterday here at NRAMM.

Bob,

I could not repeat your problem.

We get large Mean in Leginon images without beam.

However, I did find out that the code change in pyscope/dmsem.py has no effect on the dark image intensity values I get. In addition, my software dark reference for linear mode testing in DM gives equivalent of these:

blanked unprocess with software and hw processed image:
Mean 5.14

When not HW nor software corrected
Mean 46389

When software corrected but not HW corrected
Mean 5.14

I think this means that my software dark reference is taken when hardware dark subtraction is off while your software dark reference is taken when hardware dark subtraction is on.

Since Leginon and SerialEM plug in settings has no effect on the actual imaging, it appears that internally DM just uses whatever settings used during its dark reference acquisition. Is it possible that you or another user have retaken linear mode gain reference with HW dark subtraction on between the time you got high dark value and zero dark value?

For operation, I think all you need to do is to be consistent each time you reacquire gain/dark references in DM for Linear mode. If the dark is always close to zero, it is not a bad thing for Leginon. Leginon will get a dark image with close to zero values and do the exposure time scaling according to that.

In the mean time, I will ask Gatan what is the official choice of hardware dark subtraction when acquiring linear mode gain references.

Chris Booth from Gatan replied:

In LINEAR mode, hardware background subtraction and gain normalization should be turned OFF all the time.
In LINEAR mode, software dark subtraction and gain normalization work the same way they do with other cameras. Normally they always turned on, however if Leginon is doing the same operation outside of DM this is also acceptable.

In LINEAR mode, hardware background subtraction and gain normalization should be turned OFF all the time, if you leave it ON when you collect a gain reference, it will allow this, but your software dark/gain reference will have a small effect (as you are actually doing a dark/gain subtraction followed by a second dark/gain subtraction).

Bob,

Now that we know that high values in the dark image is normal for Linear mode, I should point out that because SeriealEM plug-in and Leginon send back 16-bit integer signal, there is a chance of overflow there. If possible, reduce the exposure time (not dose dependent) to the point that the raw bright image taken in Leginon (i.e., before correction) does not have maximum of 65535. Increase beam intensity if you need to get enough dose on the specimen.

Our dark image at 0.5 s exposure has ~ 22000 counts and corrected image mean is 1750. which means our raw (bright) image has mean ~ 23750.

It will also be worthwhile remove some extreme values of pixel values. Since linear mode does not use any of Gatan's internal correction, it will lack the dead pixel correction done in the hardware correction. I am working on a tool to do this better. For now, you can use the "add Extrema" tool in Correction node. See Bright_and_Dark_reference_images. The tool will add them to the bad pixel list for the particular camera configuration. Do make sure that you are updated to r17735 where I fixed a bug related to this tool.