Appion

I did not add any new warning messages, and did a grep for the warning you have and had no hits.

I only changed two lines of code and moved a section to a later part, commit:c7a84ce2 and commit:f8f8b6a7

Got the complete error message from JC.

Running Read(335544656,{'info': False, 'filename': 
'/gpfs/group/em/leginon/csandate/17apr25a/rawdata/17apr25a_G3_00027gr_00016sq_v01_00031hl_00004ed.mrc'})
Running Shape(335544720,{'shape': (512, 494)})
REDUX ERROR 1493296147 image dimensions 1023,3458 do not allow binning 
by 2,7

.....
"/opt/applications/myami/current/lib/python2.6/site-packages/redux/pipes/shape.py", 
line 103, in run
     output = pyami.imagefun.bin(output, binfactors[0], binfactors[1])
   File 
"/opt/applications/myami/current/lib/python2.6/site-packages/pyami/imagefun.py", 
line 434, in bin
     return numextension.bin(image, binning0, binning1)
ValueError: image dimensions 1023,3458 do not allow binning by 2,7

Gabe, how can there be a full K2 image at this dimension ? You should restart DM. This happens to us some times. We ask K2 to work to hard, it gets confused some times.

this is definitely an issue with the code somehow, since we have this same error showing up when we try to view older datasets.

Redux log will help to sort this out. If you give an example, I happen to have a fresh installation that I can check against. Need the exact image height and width at the minimal.

If you could upload a single micrograph, I could test it as well.

Gabe, Please post redux.log regarding the image. It should start with REQUEST and has the filepath. You should get two requests for the image, the first time is to get INFO of the image, the second time the actual image. The image request would have "oformat=JPEG" as an option. We need to know if it got the info wrong from the start.

This might be related to the binning code. I did not change it, but I saw that it was weird. It tries to find a separating bin values for each dimension (x,y), and doing this could cause your (3710x3838) image to possibly become (1023,3458) though I am not sure why.

binfactors = []
zoomfactors = []
for i in range(len(shape)):
    zoomfactors.append(float(shape[i])/float(input.shape[i]))

    ## for rgb, binning not implemented
    if is_rgb:
        binfactors.append(1)
        continue
    else:
        ## added int() to avoid future python 3 problems
        binfactors.append(int(input.shape[i] / shape[i]))

    # bin <1 not allowed (when output bigger than input)
    if binfactors[i] == 0:
        binfactors[i] = 1

    # check original shape is divisible by new shape
    if input.shape[i] % shape[i]:
        # binning alone will not work, try initial bin, then interp
        start = binfactors[i]
        for trybin in range(start, 0, -1):
            if input.shape[i] % trybin:
                continue
            binfactors[i] = trybin
            zoomfactors[i] *= binfactors[i]
            break
    else:
        # just use bin
        zoomfactors[i] = 1.0

Maybe changing the multiplication of binfactor should be done ab initial.

float(512*2)/float(3838)

Nice catch, I can try to fix, but it would have to wait until Monday. I am doing a Leginon 3.1 to 3.2 upgrade on the Northwestern JEOL today.

When I did the test I posted earlier (By the way, I fixed a typo there) from python command line, there is a warning;

/Users/acheng/miniconda/lib/python2.7/site-packages/scipy/ndimage/interpolation.py:549: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)

JC reported back

On python2.7.11/ scipy 0.17.0
>>> output = scipy.ndimage.zoom(input,zoom,order=1)
/opt/applications/python/2.7.11/gnu/lib/python2.7/site-packages/scipy/ndimage/interpolation.py:549: UserWarning: From scipy 0.13.0, the output shape of zoom() is calculated with round() instead of int() - for these inputs the size of the returned array has changed.
  "the returned array has changed.", UserWarning)
>>> print output.shape
(1024,)

On python Python2.6.6 and scipy 0.7.2
>>> output = scipy.ndimage.zoom(input,zoom,order=1)
>>> print output.shape
(1023,)

Hi Anchi, I wrote a fix yesterday, but I want to test it more for speed before I push it to trunk.

I decided to use scipy.misc.imresize instead of scipy.ndimage.zoom because of the stupid rounding error:

output = scipy.misc.imresize(output, shape, interp='bilinear')

Hi Anchi, What is the status of openCV in CentOS 6, it seems to be much faster at this task than scipy:

https://www.kaggle.com/zfturbo/test-speed-cv2-vs-scipy

import time
import numpy
import shape

input_array= numpy.ones((3710,3838))
s = shape.Shape(shape=(512,512))
t0 = time.time()
output_array = s.run(input_array,(512,512))
print time.time() - t0
print output_array.shape

Hi Niel, thanks for the update. CentOS6 provides opencv-python.x86_64 2.0.0-12.el6 package. However, this package doesn't provide cv2. I've used opencv/2.4.13 module that we have available on SEMC cluster and added it on our redux server. I've added this piece of code in try/except to make sure it can import cv2:

        import sys
        sys.stderr.write('\n**using cv2\n')

I've tested this on Gabe's image and overall timing results are not different between two versions.

[root@node02 dev]# time /root/dev/redux/bin/redux --filename=/gpfs/leginon/sargis/17mar10b/17mar10b_4s_b_00009gr_00005sq_v01_00013hl4_00006edhi-a.mrc --oformat=PNG --shape="(512,512)" > test.png
*** Using custom copy of fftw3 wrapper
calc_fftw3: 24 CPUs found, setting threads=24
**power thread lock in effectWarning:  You have not configured Images path in leginon.cfg!  Using current directory.
NOT IN MEMORY: Format(83939664,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})
IN FILE: Format(83939664,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})

real    0m21.975s
user    0m18.454s
sys     0m3.495s
[root@node02 dev]# export PYTHONPATH=/root/dev:${PYTHONPATH}
[root@node02 dev]# time /root/dev/redux/bin/redux --filename=/gpfs/leginon/sargis/17mar10b/17mar10b_4s_b_00009gr_00005sq_v01_00013hl4_00006edhi-a.mrc --oformat=PNG --shape="(512,512)" > test.png
*** Using custom copy of fftw3 wrapper
calc_fftw3: 24 CPUs found, setting threads=24
**power thread lock in effect
**using cv2
Warning:  You have not configured Images path in leginon.cfg!  Using current directory.
NOT IN MEMORY: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})
IN FILE: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})

real    0m21.817s
user    0m18.378s
sys     0m3.410s
[root@node02 dev]# time /root/dev/redux/bin/redux --filename=/gpfs/leginon/sargis/17mar10b/17mar10b_4s_b_00009gr_00005sq_v01_00013hl4_00006edhi-a.mrc --oformat=PNG --shape="(512,512)" > test.png
*** Using custom copy of fftw3 wrapper
calc_fftw3: 24 CPUs found, setting threads=24
**power thread lock in effect
**using cv2
Warning:  You have not configured Images path in leginon.cfg!  Using current directory.
NOT IN MEMORY: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})
IN FILE: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})

real    0m21.980s
user    0m18.520s
sys     0m3.432s
[root@node02 dev]# time /root/dev/redux/bin/redux --filename=/gpfs/leginon/sargis/17mar10b/17mar10b_4s_b_00009gr_00005sq_v01_00013hl4_00006edhi-a.mrc --oformat=PNG --shape="(512,512)" > test.png
*** Using custom copy of fftw3 wrapper
calc_fftw3: 24 CPUs found, setting threads=24
**power thread lock in effect
**using cv2
Warning:  You have not configured Images path in leginon.cfg!  Using current directory.
NOT IN MEMORY: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})
IN FILE: Format(24725136,{'oformat': 'PNG', 'rgb': False, 'overlay': '', 'overlaycolor': 0.10000000000000001})

real    0m21.997s
user    0m18.377s
sys     0m3.594s

Hi Sargis, I did a try/except ImportError, so it should work if opencv/cv2 is not available.

Ok display working for 3840x3584 images, but not super resolution (7680x7168)

From JC:

ok. that has been quite painful...I downgraded fs to 0.4.0 (from 2.0.1), updated myami/trunk and installed opencv-python.
It is running now.
Still reporting some issues:
Running Read(139853629143824,{'info': False, 'filename': '/gpfs/group/em/leginon/cmurin/17may23e/rawdata/17may23e_A_00018gr_00034sq_v01_00002hl_00003ed.mrc'})
Running Shape(139853629144016,{'shape': (256, 247)})
REDUX ERROR 1495651571 image dimensions 3840,3584 do not allow binning by 14,15
Traceback (most recent call last):
File "/opt/applications/myami/trunk/lib/python2.7/site-packages/redux/server.py", line 37, in run_process
result = pipeline.process(**kwargs)
File "/opt/applications/myami/trunk/lib/python2.7/site-packages/redux/pipeline.py", line 144, in process
result = pipe(result)
File "/opt/applications/myami/trunk/lib/python2.7/site-packages/redux/pipe.py", line 163, in call
return self.run(input, **self.kwargs)
File "/opt/applications/myami/trunk/lib/python2.7/site-packages/redux/pipes/shape.py", line 58, in run
outputimg = pyami.imagefun.bin(outputimg, xbin, ybin)
File "/opt/applications/myami/trunk/lib/python2.7/site-packages/pyami/imagefun.py", line 434, in bin
return numextension.bin(image, binning0, binning1)
ValueError: image dimensions 3840,3584 do not allow binning by 14,15

Oops, I flipped the binning. See commit commit:53923384d

outputimg = pyami.imagefun.bin(outputimg, xbin, ybin)

should have been:

outputimg = pyami.imagefun.bin(outputimg, ybin, xbin)

Using new test set:

import time
import numpy
from redux.pipes import shape

time.sleep(0.1)
input_array = numpy.random.random((7680,7168))
s = shape.Shape(shape=(512,512))
t0 = time.time()
output_array = s.run(input_array,(512,512))
print time.time() - t0
print output_array.shape

Not sure what 0.4.0 and 2.0.1 refer to.

Anchi, is the shape module supposed to preserve aspect ratio, because it is scaling (7680,7168) to (512,512) which be scaling differently in both x and y.

redux shape does not preserve aspect ratio. The shape input needs to give the right shape which is handled in myamiweb/inc/imageutil.inc.

However, I just checked with an image of 5120 x 3840.

The redux request is the same in trunk and 3.2 as

REQUEST: pipeline=standard&filename=/Users/acheng/testdata/leginon/17may31a/rawdata/17may31a_00001en.mrc&shape=512x384&scaletype=stdev&scalemin=-5&scalemax=5&oformat=JPEG&overlaycolor=0.35

But the trunk version gives back a 512x512 image. 3.2 gives back 512 x 384. There is a bug in your shape function.

Fixed, there was a typo in one of the shapes, I also discovered that opencv uses a backwards shape, see commit commit:b1e5cb72

Using non-square request test set:

import time
import numpy
from redux.pipes import shape

s = shape.Shape(shape=(512,478))

time.sleep(0.1)
input_array = numpy.random.random((7680,7168))
t0 = time.time()
output_array = s.run(input_array,(512,478))
print time.time() - t0
print output_array.shape

input_array = numpy.random.random((7680,7168,3))
t0 = time.time()
output_array = s.run(input_array,(512,478))
print time.time() - t0
print output_array.shape
print "done"