Leginon

Do you mean the averaged bright image has the same std as the raw image? Assuming that your computer did not run out of the memory during the 10 reference image collection, my best guess is that there are bits of stuff on your camera away from the area you took the 1k x 1k references. The bits obscured the histogram of the image intensity enough away from a normal distribution that it dominates the calculated standard deviation.

Or do you mean that the corrected (gain normalized) image has the same standard deviation as the raw image? In other words that you suspect that correction was not taken place?

I mean that the corrected (gain normalized) image has the same standard deviation as the raw image.

The correction was carried out as the honeycomb pattern was removed in the corrected image. But the normalization was not done correctly as the corrected image has the same standard deviation as the raw image.

I checked the CCD by using DM. I used DM to take images under the same condition (same beam intensity and same exposure time). The standard deviation is the same for both 4k*4k and 1k*1k image. So, the CCD itself is OK.

Thanks.

Liguo

DM does more despiking during its gain normalization. You might have spikes in your image. Save a pair raw and corrected image with Leginon and look at their histogram from your web image viewer image report (clicking the "i" icon above the image will take you to the page). Let us know if they are similar or very different.

If the spiking is random, you might want to try to do median instead of average during bright image collection to see if that helps.

I took a pair of images. Here is the snapshots of the histograms of those two images. Please take a look. Thanks. Liguo

I took another pair of images (raw and corrected). Then I analyzed the image using matlab. Then I found some differences. The min and max are the same, but the mean and rms are quite different calculated using Leginon and matlab. The mean is 14935 (Leginon) and 12679 (matlab) respectively. The rms is 2293 (Leginon) and 248 (matlab) respectively. Could it be a bug in Leginon?

Thanks.

Liguo

If it is, it is in numpy. Are you converting the mrc images from float to integer before saving?

Also, your uncorrected image has a very negative value as minimum. Is there a defect on the chip or readout? If you know where that is, you should remove them as bad pixels if you have not done so. It might have affected your normalization.

I received the mrc file you sent. The mean value is indeed 12xxx if I split it into 4 but numpy gives the mean at 14935 if not split. That does seem wrong, but I can not tell you why.

I found the following bug report that explains the problem:
http://projects.scipy.org/numpy/ticket/465
Probably float32 will not hold the sum of a very large array.
Unfortunately, the status is "wontfix".

We will have to find a different way to calculate the mean in all of our code. Luckily, we use our own arraystats module in many places, so one change there will fix many instances. A quick grep found about 300 or more calls to mean() in myami that do not use arraystats, so those need to be changed to use arraystats. And std() will need similar fix.
The quick fix is to convert all arrays to float64 before calculating mean/std. This is not optimized for time/memory, so we could also write a mean/std function in numextension that uses a float64 accumulator but can operate on an float32 array. Maybe someone has already done this and we can just use their code.

I will copy this to issue #1105