Project

General

Profile

Actions
Copy link

Bug #3438

closed

noise subtraction in ctfdisplay normalization error

Added by Anchi Cheng about 9 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Gabriel Lander
Category:
-
Target version:
-
Start date:
08/26/2015
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 3.2
Show in known bugs:
No
Workaround:

Description

Running on the same image attached #3436 (200 kV, Cs=2 mm, apix=1.47 A)

I tried these parameters as well as the faster --numstep=50. Got the same problem below.

ctfestimate.py --runname=ctffindrun6 --rundir=/emg/data00/appion/acheng/15aug25g/ctf/ctffindrun6 --limit=1 --commit --preset=en-a --projectid=5 --session=15aug25g --no-rejects --no-wait --continue --ampcarbon=0.15 --ampice=0.07 --fieldsize=512 --medium=carbon --bin=2 --resmin=30 --resmax=4 --defstep=1000 --numstep=50 --dast=500 --expid=605 --jobtype=ctfestimate

Final params: def1: 9.95e-07 | def2: 1.01e-06 | angle: 15.7 | defratio 1.01
 ... Defocus Astig Percent Diff 0.58 -- 9.948e-07, 1.007e-06
Computing power spectra in 1024x1024 blocks...................................
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 1024 and 35 images complete in 9.4 sec
 ... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 100
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
 ... starting peak extension
Peak Extension Complete in 0.76 msec
Traceback (most recent call last):
  File "/home/acheng/myami/appion/bin/ctfestimate.py", line 449, in <module>
    imgLoop.run()
  File "/home/acheng/myami/appion/appionlib/appionLoop2.py", line 93, in run
    self.loopCommitToDatabase(imgdata)
  File "/home/acheng/myami/appion/appionlib/appionLoop2.py", line 143, in loopCommitToDatabase
    return self.commitToDatabase(imgdata)
  File "/home/acheng/myami/appion/bin/ctfestimate.py", line 349, in commitToDatabase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/home/acheng/myami/appion/appionlib/apCtf/ctfinsert.py", line 45, in validateAndInsertCTFData
    ctfvalues = runCTFdisplayTools(imgdata, ctfvalues, opimagedir, fftpath, fftfreq)
  File "/home/acheng/myami/appion/appionlib/apCtf/ctfinsert.py", line 83, in runCTFdisplayTools
    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
  File "/home/acheng/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1267, in makeCtfImages
    ctfdisplaydict = a.CTFpowerspec(imgdata, ctfdata, fftpath, fftfreq, twod=twod)
  File "/home/acheng/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1149, in CTFpowerspec
    normpowerspec = self.normalizeCtf(powerspec, twod=twod)
  File "/home/acheng/myami/appion/appionlib/apCtf/ctfdisplay.py", line 244, in normalizeCtf
    scale /= scale.max()
ValueError: zero-size array to ufunc.reduce without identity

ctf estimation is used all the time. If the new change in the background fitting is causing this much trouble, should we revert and make it more as an option for now ?

Files

Download all files
beforeUpdate.png (413 KB) beforeUpdate.png Gabriel Lander, 08/31/2015 06:13 PM
afterUpdate.png (573 KB) afterUpdate.png Gabriel Lander, 08/31/2015 06:13 PM

Related issues 3 (3 open — 0 closed)

Related to Appion - Bug #3494: difficult image for new ctfdisplay.py to do noise subtractionIn TestNeil Voss08/31/2015

Actions
Related to Appion - Bug #3495: CTF matplotlib font path ErrorNewDmitry Lyumkis08/31/2015

Actions
Related to Appion - Feature #3950: Create CTF test dataset within DockerAssignedNeil Voss02/15/2016

Actions
Actions
Copy link
 #1

Updated by Neil Voss about 9 years ago

  • Status changed from Assigned to In Test
  • Assignee changed from Neil Voss to Anchi Cheng

Hi Anchi, I do not have time to work on this right now, but I think I fixed it in r19107.

Actions
Copy link
 #2

Updated by Anchi Cheng about 9 years ago

  • Assignee changed from Anchi Cheng to Gabriel Lander

The fix works at our end.

Gabe ?

Actions
Copy link Download all files
 #3

Updated by Gabriel Lander about 9 years ago

The 2D image no longer appear to be binned by two, and the confidences that show up are much worse than previously. I'm attaching two images showing the result of the CTF results run on the same images, before & after the update.

Actions
Copy link
 #4

Updated by Anchi Cheng about 9 years ago

In order to let nysbc users keep going, I created branch 3.2 with ctfdisplay.py reverted to r19012. This way Neil can debug when he has time.

Actions
Copy link
 #5

Updated by Neil Voss about 9 years ago

Made some tweeks to trunk ctf to help with close to focus images. Needs testing. I will try to visit in free time, but very limited right now.

Actions
Copy link
 #6

Updated by Neil Voss about 9 years ago

  • Related to Bug #3494: difficult image for new ctfdisplay.py to do noise subtraction added
Actions
Copy link
 #7

Updated by Neil Voss about 9 years ago

r19128 and r19127

Actions
Copy link
 #8

Updated by Neil Voss about 9 years ago

  • Related to Bug #3495: CTF matplotlib font path Error added
Actions
Copy link
 #9

Updated by Gabriel Lander about 9 years ago

Still not working (running r19192):

====================================
Starting image 1 ( skip:9, remain:131 ) id:3771558, file: 15sep24b_03sq_02hl_011en
 ... Pixel size: 2.05044
!!! WARNING: no best CTF value for image 15sep24b_03sq_02hl_011en
 ... Xmag=76081, dstep=1.56e-05, mpix=2.05e-10
!!! WARNING: Defocus minimum is less than zero
Defocus search range: 1000 A to 32000 A (0.10 to 3.20 um)
 ... running ctf estimation at Thu Sep 24 10:54:31 2015
/gpfs/home/glander/myami/appion/bin/ctffind64.exe
15sep24b_03sq_02hl_011en.mrc
15sep24b_03sq_02hl_011en-pow.mrc
2.2,120.0,0.15,76081.2313455,15.6,2
512,40.0,8.0,1000.0,32000.0,1000.0,500.0
 ... ctf estimation completed in 61.8 sec
 ... nominal=7.0e-07, bestdef=7.0e-07,
 ... def_1=7.5e-07, def_2=8.1e-07, astig_angle=-14.4, cross_corr=0.413,

 ... reading MRC: 15sep24b_03sq_02hl_011en-pow.mrc size:(512, 512) dtype:float32
 ... writing JPEG: 15sep24b_03sq_02hl_011en-pow.jpg
 ==== Committing data to database ==== 
 ... Committing ctf parameters for 15sep24b_03sq_02hl_011en to database
 ... Reading image...
[CTF param] def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | ampcontr 0.15 | defratio 1.072
Final params: def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | defratio 1.07
 ... Defocus Astig Percent Diff 3.46 -- 7.546e-07, 8.086e-07
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/n
umeric.py:1846: DeprecationWarning: using a non-integer number instead of an integer will result in 
an error in the future
  res = empty((N,)+dimensions, dtype=dtype)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/n
umeric.py:1849: DeprecationWarning: using a non-integer number instead of an integer will result in 
an error in the future
  tmp.shape = (1,)*i + (dim,)+(1,)*(N-i-1)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/n
umeric.py:1851: DeprecationWarning: using a non-integer number instead of an integer will result in 
an error in the future
  val = zeros(newdim, dtype)
Computing power spectra in 4096x4096 blocks./gpfs/home/glander/myami/appion/appionlib/apCtf/ctfpower
.py:101: DeprecationWarning: using a non-integer number instead of an integer will result in an erro
r in the future
  cutout = image[x1:x2, y1:y2]

 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 4096 and 1 images complete in 6.45 sec
!!! WARNING: Requested resolution (3.000) is not available (4.101)
 ... Preform a rotational average and remove spikes...
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 41
/gpfs/home/glander/myami/appion/appionlib/apCtf/ctftools.py:490: DeprecationWarning: using a non-int
eger number instead of an integer will result in an error in the future
  xdataint = xdataint[:outercutsize]
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
 ... starting peak extension
Peak Extension Complete in 0.73 msec

['/gpfs/home/glander/myami/appion/appionlib', '/gpfs/home/glander/myami/appion']
Found 0 ctf values
Traceback (most recent call last):
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 451, in <module>
    imgLoop.run()
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 93, in run
    self.loopCommitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 143, in loopCommitToDatabase
    return self.commitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 351, in commitToDatabase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 45, in validateAndInsert
CTFData
    ctfvalues = runCTFdisplayTools(imgdata, ctfvalues, opimagedir, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 83, in runCTFdisplayTool
s
    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1266, in makeCtfImages
    ctfdisplaydict = a.CTFpowerspec(imgdata, ctfdata, fftpath, fftfreq, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1148, in CTFpowerspec
    normpowerspec = self.normalizeCtf(powerspec, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 243, in normalizeCtf
    scale /= scale.max()
  File "/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/nump
y/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity

Actions
Copy link
 #10

Updated by Neil Voss about 9 years ago

Hi Gabe, I added some debugging info to help diagnose the problem, because it ended up with no overlap region.

Actions
Copy link
 #11

Updated by Gabriel Lander about 9 years ago

Here's the output with the debug (also had to comment out an if statement for this to work - r19202 ===============================================

Starting image 1 ( skip:34, remain:406 ) id:3771558, file: 15sep24b_03sq_02hl_011en
 ... Pixel size: 2.05044
Found 1 ctf values
0.167 -- ctffindrun1
*** 0.167
[CTF run]   method: ctffind | runname ctffindrun1
[CTF param] def1: 7.53e-07 | def2: 8.07e-07 | angle: -16.4 | ampcontr 0.15 | defratio 1.071
[CTF stats] conf_30-10: 0.986 | conf_5peak: 0.973 | res_0.8: 6.0A | res_0.5 5.6A
 ... Xmag=76081, dstep=1.56e-05, mpix=2.05e-10
!!! WARNING: Defocus minimum is less than zero
Defocus search range: 1000 A to 32530 A (0.10 to 3.25 um)
 ... running ctf estimation at Fri Sep 25 13:44:40 2015
/gpfs/home/glander/myami/appion/bin/ctffind64.exe
15sep24b_03sq_02hl_011en.mrc
15sep24b_03sq_02hl_011en-pow.mrc
2.2,120.0,0.15,76081.2313455,15.6,2
512,40.0,8.0,1000.0,32530.3,1000.0,500.0
 ... ctf estimation completed in 60.55 sec
 ... nominal=7.0e-07, bestdef=7.5e-07,
 ... def_1=7.5e-07, def_2=8.1e-07, astig_angle=-14.4, cross_corr=0.413,

 ... reading MRC: 15sep24b_03sq_02hl_011en-pow.mrc size:(512, 512) dtype:float32
 ... writing JPEG: 15sep24b_03sq_02hl_011en-pow.jpg
 ==== Committing data to database ==== 
 ... Committing ctf parameters for 15sep24b_03sq_02hl_011en to database
15sep24b_03sq_02hl_011en
Pixelsize (A/pix) 2.05044
 ... Reading image...
[CTF param] def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | ampcontr 0.15 | defratio 1.072
Final params: def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | defratio 1.07
 ... Defocus Astig Percent Diff 3.46 -- 7.546e-07, 8.086e-07
   volts -- 120000
   amplitude_contrast -- 0.15
   cross_correlation -- 0.41284
   angle_astigmatism -- -14.37
   cs -- 2.2
   confidence_d -- 0.64253
   confidence -- 0.41284
   graph1 -- /gpfs/group/em/appion/glander/15sep24b/ctf/ctffindruntest/opimages/15sep24b_03sq_02hl_011en-pow.jpg
   defocusinit -- 7.5303e-07
   nominal -- 7e-07
   defocus1 -- 7.54591e-07
   defocus2 -- 8.08636e-07
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1846: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  res = empty((N,)+dimensions, dtype=dtype)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1849: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  tmp.shape = (1,)*i + (dim,)+(1,)*(N-i-1)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1851: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  val = zeros(newdim, dtype)
Computing power spectra in 4096x4096 blocks.
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 4096 and 1 images complete in 6.26 sec
!!! WARNING: Requested resolution (3.000) is not available (4.101)
 ... Preform a rotational average and remove spikes...
    orig pixel 2.050 freq 1.191e-04
    trim pixel 2.050 freq 1.191e-04
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 41
Elliptical CTF limits 24.6 A -->> 4.2A
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
extremaToIndex Debug
... requestVal = 508.09663437
... extremaIndex = 0
... extremaList[extremaIndex-1] = 4471.40382371
... extremaVal = extremaList[extremaIndex] = 508.09663437
... extremaList[extremaIndex+1] = 729.80361294
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0605
... index = 85
... raddata[index-1] = 0.0604
... raddata[index] = 0.0606
... raddata[index+1] = 0.0608
extremaToIndex Debug
... requestVal = 508.09663437
... extremaIndex = 0
... extremaList[extremaIndex-1] = 4471.40382371
... extremaVal = extremaList[extremaIndex] = 508.09663437
... extremaList[extremaIndex+1] = 729.80361294
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0605
... index = 85
... raddata[index-1] = 0.0604
... raddata[index] = 0.0606
... raddata[index+1] = 0.0608
extremaToIndex Debug
... requestVal = 780.95377723
... extremaIndex = 1
... extremaList[extremaIndex-1] = 508.09663437
... extremaVal = extremaList[extremaIndex] = 729.80361294
... extremaList[extremaIndex+1] = 900.31406112
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0869
... index = 197
... raddata[index-1] = 0.0866
... raddata[index] = 0.0868
... raddata[index+1] = 0.0870
extremaToIndex Debug
... requestVal = 671.81092008
... extremaIndex = 1
... extremaList[extremaIndex-1] = 508.09663437
... extremaVal = extremaList[extremaIndex] = 729.80361294
... extremaList[extremaIndex+1] = 900.31406112
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0869
... index = 197
... raddata[index-1] = 0.0866
... raddata[index] = 0.0868
... raddata[index+1] = 0.0870
extremaToIndex Debug
... requestVal = 944.66806294
... extremaIndex = 2
... extremaList[extremaIndex-1] = 729.80361294
... extremaVal = extremaList[extremaIndex] = 900.31406112
... extremaList[extremaIndex+1] = 1045.06266452
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1072
... index = 284
... raddata[index-1] = 0.1069
... raddata[index] = 0.1072
... raddata[index+1] = 0.1074
extremaToIndex Debug
... requestVal = 835.52520580
... extremaIndex = 2
... extremaList[extremaIndex-1] = 729.80361294
... extremaVal = extremaList[extremaIndex] = 900.31406112
... extremaList[extremaIndex+1] = 1045.06266452
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1072
... index = 284
... raddata[index-1] = 0.1069
... raddata[index] = 0.1072
... raddata[index+1] = 0.1074
extremaToIndex Debug
... requestVal = 1108.38234865
... extremaIndex = 3
... extremaList[extremaIndex-1] = 900.31406112
... extremaVal = extremaList[extremaIndex] = 1045.06266452
... extremaList[extremaIndex+1] = 1173.68298368
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1244
... index = 358
... raddata[index-1] = 0.1243
... raddata[index] = 0.1245
... raddata[index+1] = 0.1247
extremaToIndex Debug
... requestVal = 999.23949151
... extremaIndex = 3
... extremaList[extremaIndex-1] = 900.31406112
... extremaVal = extremaList[extremaIndex] = 1045.06266452
... extremaList[extremaIndex+1] = 1173.68298368
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1244
... index = 358
... raddata[index-1] = 0.1243
... raddata[index] = 0.1245
... raddata[index+1] = 0.1247
extremaToIndex Debug
... requestVal = 1272.09663437
... extremaIndex = 5
... extremaList[extremaIndex-1] = 1173.68298368
... extremaVal = extremaList[extremaIndex] = 1291.06529892
... extremaList[extremaIndex+1] = 1400.09913043
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1537
... index = 483
... raddata[index-1] = 0.1535
... raddata[index] = 0.1537
... raddata[index+1] = 0.1540
[85, 197, 284, 358]
[197, 284, 358, 483]
Noise mergeIndexes [85, 197, 197, 284, 284, 358, 358, 483]
 ... starting peak extension
Peak Extension Complete in 0.81 msec
 ... fitting noise section 1 of 4
 ... finished in 115.84 msec
 ... fitting noise section 2 of 4
 ... finished in 113.82 msec
 ... fitting noise section 3 of 4
 ... finished in 112.11 msec
 ... fitting noise section 4 of 4
 ... finished in 112.32 msec
section 0 mergedata (85,)
    insert from 85 to 197 (112)
    overlap from 197 to 197 (0)
Traceback (most recent call last):
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 451, in <module>
    imgLoop.run()
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 93, in run
    self.loopCommitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 143, in loopCommitToDatabase
    return self.commitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 351, in commitToDatabase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 45, in validateAndInsertCTFData
    ctfvalues = runCTFdisplayTools(imgdata, ctfvalues, opimagedir, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 83, in runCTFdisplayTools
    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1264, in makeCtfImages
    ctfdisplaydict = a.CTFpowerspec(imgdata, ctfdata, fftpath, fftfreq, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1146, in CTFpowerspec
    normpowerspec = self.normalizeCtf(powerspec, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 245, in normalizeCtf
    scale /= scale.max()
  File "/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity

Actions
Copy link
 #12

Updated by Neil Voss about 9 years ago

try this fix.

Actions
Copy link
 #13

Updated by Gabriel Lander about 9 years ago

I got the same error.

Actions
Copy link
 #14

Updated by Neil Voss about 9 years ago

Can you include the output again, I am trying to learn something.

Actions
Copy link
 #15

Updated by Gabriel Lander about 9 years ago 

 ... Time stamp: 15sep25o17
 ... Function name: ctfestimate
 ... Appion directory: /gpfs/home/glander/myami/appion
!!! WARNING: There is more than one appion directory in your PYTHONPATH
['/gpfs/home/glander/myami/appion/appionlib', '/gpfs/home/glander/myami/appion']
 ... Using split database
Connected to database: 'ap472'
 ... Committing data to database
 ... Looking up session, 15sep24b
 ... Found project id=472 for session 15sep24b in 5.72 msec
 ... Run directory: /gpfs/group/em/appion/glander/15sep24b/ctf/ctffindruntest
!!! WARNING: directory '/gpfs/group/em/appion/glander/15sep24b/ctf/ctffindruntest' already exists.
 ... Writing function log to: ctfestimate.log
 ... Uploading ScriptData....
 ... Found 1 processors on this machine
 ... Running Appion version 'trunk'
 ... Reading old done dictionary: ctfestimate.donedict
 ... Found 35 done dictionary entries
 ... Querying database for preset 'en' images from session '15sep24b' ... 
 ... Found 441 images in 27.66 sec
 ... Remove processed images
.skipping processed images
............................................................................
 ... finished skipping in 4.74 sec

!!! WARNING: skipped 34 of 441 images
 ... [[ 0 no reprocess  | 0 rejected  | 0 wrong tilt  | 0 in donedict ]]
 ... Process images old to new
 ... Running program ctffind64.exe

Beginning Main Loop

Starting image 1 ( skip:34, remain:406 ) id:3771558, file: 15sep24b_03sq_02hl_011en
 ... Pixel size: 2.05044
Found 1 ctf values
0.167 -- ctffindrun1
*** 0.167
[CTF run]   method: ctffind | runname ctffindrun1
[CTF param] def1: 7.53e-07 | def2: 8.07e-07 | angle: -16.4 | ampcontr 0.15 | defratio 1.071
[CTF stats] conf_30-10: 0.986 | conf_5peak: 0.973 | res_0.8: 6.0A | res_0.5 5.6A
 ... Xmag=76081, dstep=1.56e-05, mpix=2.05e-10
!!! WARNING: Defocus minimum is less than zero
Defocus search range: 1000 A to 32530 A (0.10 to 3.25 um)
 ... running ctf estimation at Fri Sep 25 14:17:38 2015
/gpfs/home/glander/myami/appion/bin/ctffind64.exe
15sep24b_03sq_02hl_011en.mrc
15sep24b_03sq_02hl_011en-pow.mrc
2.2,120.0,0.15,76081.2313455,15.6,2
512,40.0,8.0,1000.0,32530.3,1000.0,500.0
 ... ctf estimation completed in 60.39 sec
 ... nominal=7.0e-07, bestdef=7.5e-07,
 ... def_1=7.5e-07, def_2=8.1e-07, astig_angle=-14.4, cross_corr=0.413,

 ... reading MRC: 15sep24b_03sq_02hl_011en-pow.mrc size:(512, 512) dtype:float32
 ... writing JPEG: 15sep24b_03sq_02hl_011en-pow.jpg
 ==== Committing data to database ==== 
 ... Committing ctf parameters for 15sep24b_03sq_02hl_011en to database
15sep24b_03sq_02hl_011en
Pixelsize (A/pix) 2.05044
 ... Reading image...
[CTF param] def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | ampcontr 0.15 | defratio 1.072
Final params: def1: 7.55e-07 | def2: 8.09e-07 | angle: -14.4 | defratio 1.07
 ... Defocus Astig Percent Diff 3.46 -- 7.546e-07, 8.086e-07
   volts -- 120000
   amplitude_contrast -- 0.15
   cross_correlation -- 0.41284
   angle_astigmatism -- -14.37
   cs -- 2.2
   confidence_d -- 0.64253
   confidence -- 0.41284
   graph1 -- /gpfs/group/em/appion/glander/15sep24b/ctf/ctffindruntest/opimages/15sep24b_03sq_02hl_011en-pow.jpg
   defocusinit -- 7.5303e-07
   nominal -- 7e-07
   defocus1 -- 7.54591e-07
   defocus2 -- 8.08636e-07
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1846: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  res = empty((N,)+dimensions, dtype=dtype)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1849: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  tmp.shape = (1,)*i + (dim,)+(1,)*(N-i-1)
/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/numeric.py:1851: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  val = zeros(newdim, dtype)
Computing power spectra in 4096x4096 blocks.
 ... Computing median of power spectra series
 ... Compute PSD with fieldsize 4096 and 1 images complete in 6.2 sec
!!! WARNING: Requested resolution (3.000) is not available (4.101)
 ... Preform a rotational average and remove spikes...
    orig pixel 2.050 freq 1.191e-04
    trim pixel 2.050 freq 1.191e-04
PART 1: SETUP PARAMETERS AND ELLIPTICAL AVERAGE
 ... Number of available peaks is 41
 ... setting the number of sections to default: 4
Elliptical CTF limits 24.6 A -->> 4.2A
 ... Determine and subtract noise model
PART 2: BACKGROUND NOISE SUBTRACTION
extremaToIndex Debug
... requestVal = 508.09663437
... extremaIndex = 0
... extremaList[extremaIndex-1] = 4471.40382371
... extremaVal = extremaList[extremaIndex] = 508.09663437
... extremaList[extremaIndex+1] = 729.80361294
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0605
... index = 85
... raddata[index-1] = 0.0604
... raddata[index] = 0.0606
... raddata[index+1] = 0.0608
extremaToIndex Debug
... requestVal = 508.09663437
... extremaIndex = 0
... extremaList[extremaIndex-1] = 4471.40382371
... extremaVal = extremaList[extremaIndex] = 508.09663437
... extremaList[extremaIndex+1] = 729.80361294
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0605
... index = 85
... raddata[index-1] = 0.0604
... raddata[index] = 0.0606
... raddata[index+1] = 0.0608
extremaToIndex Debug
... requestVal = 780.95377723
... extremaIndex = 1
... extremaList[extremaIndex-1] = 508.09663437
... extremaVal = extremaList[extremaIndex] = 729.80361294
... extremaList[extremaIndex+1] = 900.31406112
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0869
... index = 197
... raddata[index-1] = 0.0866
... raddata[index] = 0.0868
... raddata[index+1] = 0.0870
extremaToIndex Debug
... requestVal = 671.81092008
... extremaIndex = 1
... extremaList[extremaIndex-1] = 508.09663437
... extremaVal = extremaList[extremaIndex] = 729.80361294
... extremaList[extremaIndex+1] = 900.31406112
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.0869
... index = 197
... raddata[index-1] = 0.0866
... raddata[index] = 0.0868
... raddata[index+1] = 0.0870
extremaToIndex Debug
... requestVal = 944.66806294
... extremaIndex = 2
... extremaList[extremaIndex-1] = 729.80361294
... extremaVal = extremaList[extremaIndex] = 900.31406112
... extremaList[extremaIndex+1] = 1045.06266452
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1072
... index = 284
... raddata[index-1] = 0.1069
... raddata[index] = 0.1072
... raddata[index+1] = 0.1074
extremaToIndex Debug
... requestVal = 835.52520580
... extremaIndex = 2
... extremaList[extremaIndex-1] = 729.80361294
... extremaVal = extremaList[extremaIndex] = 900.31406112
... extremaList[extremaIndex+1] = 1045.06266452
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1072
... index = 284
... raddata[index-1] = 0.1069
... raddata[index] = 0.1072
... raddata[index+1] = 0.1074
extremaToIndex Debug
... requestVal = 1108.38234865
... extremaIndex = 3
... extremaList[extremaIndex-1] = 900.31406112
... extremaVal = extremaList[extremaIndex] = 1045.06266452
... extremaList[extremaIndex+1] = 1173.68298368
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1244
... index = 358
... raddata[index-1] = 0.1243
... raddata[index] = 0.1245
... raddata[index+1] = 0.1247
extremaToIndex Debug
... requestVal = 999.23949151
... extremaIndex = 3
... extremaList[extremaIndex-1] = 900.31406112
... extremaVal = extremaList[extremaIndex] = 1045.06266452
... extremaList[extremaIndex+1] = 1173.68298368
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1244
... index = 358
... raddata[index-1] = 0.1243
... raddata[index] = 0.1245
... raddata[index+1] = 0.1247
extremaToIndex Debug
... requestVal = 1272.09663437
... extremaIndex = 5
... extremaList[extremaIndex-1] = 1173.68298368
... extremaVal = extremaList[extremaIndex] = 1291.06529892
... extremaList[extremaIndex+1] = 1400.09913043
... trimFreq = 0.00011907
... extremaVal*trimFreq = 0.1537
... index = 483
... raddata[index-1] = 0.1535
... raddata[index] = 0.1537
... raddata[index+1] = 0.1540
[85, 197, 284, 358]
[197, 284, 358, 483]
Noise mergeIndexes [85, 197, 197, 284, 284, 358, 358, 483]
 ... starting peak extension
Peak Extension Complete in 0.83 msec
 ... fitting noise section 1 of 4
 ... finished in 116.54 msec
 ... fitting noise section 2 of 4
 ... finished in 115.2 msec
 ... fitting noise section 3 of 4
 ... finished in 113.08 msec
 ... fitting noise section 4 of 4
 ... finished in 114.06 msec
section 0 mergedata (85,)
    insert from 85 to 197 (112)
    overlap from 197 to 197 (0)
Traceback (most recent call last):
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 451, in <module>
    imgLoop.run()
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 93, in run
    self.loopCommitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/appionlib/appionLoop2.py", line 143, in loopCommitToDatabase
    return self.commitToDatabase(imgdata)
  File "/gpfs/home/glander/myami/appion/bin/ctfestimate.py", line 351, in commitToDatabase
    ctfinsert.validateAndInsertCTFData(imgdata, self.ctfvalues, self.ctfrun, self.params['rundir'])
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 45, in validateAndInsertCTFData
    ctfvalues = runCTFdisplayTools(imgdata, ctfvalues, opimagedir, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfinsert.py", line 83, in runCTFdisplayTools
    ctfdisplaydict = ctfdisplay.makeCtfImages(imgdata, ctfvalues, fftpath, fftfreq)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1277, in makeCtfImages
    ctfdisplaydict = a.CTFpowerspec(imgdata, ctfdata, fftpath, fftfreq, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 1159, in CTFpowerspec
    normpowerspec = self.normalizeCtf(powerspec, twod=twod)
  File "/gpfs/home/glander/myami/appion/appionlib/apCtf/ctfdisplay.py", line 258, in normalizeCtf
    scale /= scale.max()
  File "/opt/applications/python/lib/python2.6/site-packages/numpy-1.9.1-py2.6-linux-x86_64.egg/numpy/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity
Actions
Copy link
 #16

Updated by Neil Voss about 9 years ago

hmm did not print what I wanted, I updated to see if I can get it to print. So, post again if you want otherwise, I have a test case back at lab I can test on Monday.

Actions
Copy link
 #17

Updated by Gabriel Lander about 9 years ago

any updates? we have 38 users here using myami/3.0 for ctf estimation now.

Actions
Copy link
 #18

Updated by Neil Voss over 8 years ago

  • Related to Feature #3950: Create CTF test dataset within Docker added
Actions
Copy link
 #19

Updated by Gabriel Lander over 8 years ago

  • Status changed from In Test to Closed
Actions
Copy link

Also available in: Atom PDF