Project

General

Profile

Actions

Bug #1105

closed

Incorrect Standard deviation of large images

Added by Liguo Wang about 14 years ago. Updated about 11 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
12/17/2010
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 2.1.0
Show in known bugs:
No
Workaround:

Description

When I was doing CCD correction, I got an error: "***** DataManager size reached, removing data as needed ** ". I searched for the message posted by Anchi about this issue and could not find it. Would you please tell me how to fix it?

In addition, I got a serious problem with CCD correction. I ask Leginon to do 10 averages and 2 channels for each set of dark and bright references. For 1k*1k (no binning), the standard deviation is about 200 with a mean of 11,000 in a corrected blank beam image. However, for 4k*4k (no binning), the standard deviation is the same as the raw image (~2,100 with a mean of 11,000). How can I correct that?

Thanks.

Actions #1

Updated by Jim Pulokas about 14 years ago

I found the following bug report that explains the problem:
http://projects.scipy.org/numpy/ticket/465
Probably float32 will not hold the sum of a very large array.
Unfortunately, the status is "wontfix".

We will have to find a different way to calculate the mean in all of our code. Luckily, we use our own arraystats module in many places, so one change there will fix many instances. A quick grep found about 300 or more calls to mean() in myami that do not use arraystats, so those need to be changed to use arraystats. And std() will need similar fix.
The quick fix is to convert all arrays to float64 before calculating mean/std. This is not optimized for time/memory, so we could also write a mean/std function in numextension that uses a float64 accumulator but can operate on an float32 array. Maybe someone has already done this and we can just use their code.

Actions #2

Updated by Anchi Cheng almost 14 years ago

  • Subject changed from DataManager size reached error and CCD normalization problem to Incorrect Standard deviation of large images
  • Priority changed from Immediate to High
Actions #3

Updated by Anchi Cheng almost 14 years ago

  • Tracker changed from Support to Bug
  • Assignee set to Jim Pulokas
  • Show in known bugs set to No
Actions #4

Updated by Anchi Cheng about 11 years ago

  • Status changed from New to Closed

We have been using numextension version for over 2 years now.

Actions

Also available in: Atom PDF