Bug #1105
closed
Incorrect Standard deviation of large images
Added by Liguo Wang almost 14 years ago.
Updated about 11 years ago.
Affected Version:
Appion/Leginon 2.1.0
Description
When I was doing CCD correction, I got an error: "***** DataManager size reached, removing data as needed ** ". I searched for the message posted by Anchi about this issue and could not find it. Would you please tell me how to fix it?
In addition, I got a serious problem with CCD correction. I ask Leginon to do 10 averages and 2 channels for each set of dark and bright references. For 1k*1k (no binning), the standard deviation is about 200 with a mean of 11,000 in a corrected blank beam image. However, for 4k*4k (no binning), the standard deviation is the same as the raw image (~2,100 with a mean of 11,000). How can I correct that?
Thanks.
I found the following bug report that explains the problem:
http://projects.scipy.org/numpy/ticket/465
Probably float32 will not hold the sum of a very large array.
Unfortunately, the status is "wontfix".
We will have to find a different way to calculate the mean in all of our code. Luckily, we use our own arraystats module in many places, so one change there will fix many instances. A quick grep found about 300 or more calls to mean() in myami that do not use arraystats, so those need to be changed to use arraystats. And std() will need similar fix.
The quick fix is to convert all arrays to float64 before calculating mean/std. This is not optimized for time/memory, so we could also write a mean/std function in numextension that uses a float64 accumulator but can operate on an float32 array. Maybe someone has already done this and we can just use their code.
- Subject changed from DataManager size reached error and CCD normalization problem to Incorrect Standard deviation of large images
- Priority changed from Immediate to High
- Tracker changed from Support to Bug
- Assignee set to Jim Pulokas
- Show in known bugs set to No
- Status changed from New to Closed
We have been using numextension version for over 2 years now.
Also available in: Atom
PDF