Appion

Hi Anchi,

PIL doesn't support tiffs with doubles - https://github.com/python-pillow/Pillow/issues/1659

I was able to convert /home/acheng/test.tif -depth 8 /gpfs/tmp/test.tif thanks to https://stackoverflow.com/questions/48944819/image-open-gives-error-cannot-identify-image-file

I used NCCAT cluster since this convert errors on semc-head. After that I can do:

>>> numpil.readInfo('/gpfs/tmp/test.tif')
{'nx': 11520, 'ny': 8184, 'nz': 60, 'compression': 'tiff_lzw'}

I asked David Mastronarde who wrote the program that made these files.

The "Bits/Sample" is set to the data size corresponding to the data type and is 8 for bytes

He had me checked with tiffinfo ( found it on buffer01). Here is what came out

For the last frame in the original /home/acheng/test.tif, tiffinfo says

TIFF Directory at offset 0x274b1978 (659233144)
  Image Width: 11520 Image Length: 8184
  Resolution: 4.85202e+08, 4.85202e+08 pixels/inch
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Rows/Strip: 2
  SMin Sample Value: 0
  SMax Sample Value: 72
  Planar Configuration: single image plane
  DateTime: 2020:08:10 17:34:41

For the same frame that he converted as tmp.tif, tiffinfo says

TIFF Directory at offset 0x2f21f7d6 (790755286)
  Subfile Type: multi-page document (2 = 0x2)
  Image Width: 11520 Image Length: 8184
  Resolution: 4.85202e+08, 4.85202e+08 pixels/inch
  Bits/Sample: 8
  Compression Scheme: LZW
  Photometric Interpretation: min-is-black
  FillOrder: msb-to-lsb
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 1
  Rows/Strip: 1
  Planar Configuration: single image plane
  Page Number: 59-60
  DocumentName: /gpfs/tmp/test.tif
  DateTime: 2020:08:10 17:34:41
  Predictor: horizontal differencing 2 (0x2)

He is look into this now. Sargis, if you have any insight on what might be wrong, please let us know.

From David Mastronarde

I have PIL in an installation of python 2.6 on our old lab servers, so I tried this.  It works with a 5-year old file from Chen Xu, not in a 2-year old file that I collected myself.  The problem must be that the latter is BIg TIFF, which I switched to 2 years ago,  I see a post from only a year ago that says PIL doesn't support Big TIFF.

The way to handle this is described in my help:

TIFF files are saved as 'big TIFF' so that files bigger than 4 GB can be written.  This format cannot be read by version 3 of libtiff.  If this causes a problem in some other software, you can override this behavior by setting an environment variable 'IMOD_ALL_BIG_TIFF' to 0.

David

We may of may not be able to do this depending on the file size users acquire.

David said that file bigger than 4 GB is needed mostly for FISE series.

Sargis, since we have installed Big TIFF version of libtiff everywhere, if you have solution to allow reading of the frames without us reverting back to old format, that would be fantastic.

Managed to install https://pypi.org/project/tifffile/. The latest version Requires: Python >=3.6. I've installed tifffile-0.12.1 instead. It was giving this warning:

>>> import tifffile
tifffile/tifffile.py:306: UserWarning: ImportError: No module named '_tifffile'. Loading of some compressed images will be very slow. Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/

cd /gpfs/tmp/pytiff/tifffile-0.12.1/build/lib.linux-x86_64-2.6
python
>>> import tifffile
>>> tif = tifffile.TiffFile('/home/acheng/test.tif')
>>> print tif.info()

TIFF file: test.tif, 629 MiB, little endian, bigtiff, 60 pages

Series 0: 60x8184x11520, uint8, IYX, 60 pages, not mem-mappable

Page 0: 8184x11520, uint8, 8 bit, minisblack, lzw
* 256 image_width (1H) 11520
* 257 image_length (1H) 8184
* 258 bits_per_sample (1H) 8
* 259 compression (1H) 5
* 262 photometric (1H) 1
* 270 image_description (85s) SerialEMCCD: Dose frac. image, scaled by 1.00  r/
* 273 strip_offsets (4092Q) (16, 5288, 7847, 10509, 13158, 15775, 18413, 21059,
* 277 samples_per_pixel (1H) 1
* 278 rows_per_strip (1H) 2
* 279 strip_byte_counts (4092Q) (5272, 2559, 2662, 2649, 2617, 2638, 2646, 2643
* 282 x_resolution (2I) (485201760, 1)
* 283 y_resolution (2I) (485201760, 1)
* 284 planar_configuration (1H) 1
* 296 resolution_unit (1H) 2
* 306 datetime (20s) 2020:08:10 17:34:35
* 339 sample_format (1H) 1
* 340 smin_sample_value (1B) 0
* 341 smax_sample_value (1B) 72

IMAGE_DESCRIPTION
SerialEMCCD: Dose frac. image, scaled by 1.00  r/f 0
  SuperRef_gabSEM_00001_1-0.dm4

tifffile.imread('/home/acheng/test.tif') takes more than a minute on node04 and +40GB RAM, but reading a page with TiffFile is fast.

I'll probably make a new .py file and implement tiff2numpy_array using tifffile. Does this needs to go to all nodes and leginon boxes or only ingest/buffer servers? tifffile might be easier to install with python 2.7.

Thanks, Sargis.

This is mainly used when people upload movies or needing to do defect correction per frame. It is not a common task. A bit slow is o.k. for now.

If you look at Issue #8036, you will see where it is used now.

By the way, there is tifffile.py in pyami, you can replace that with your modified new version.

Thanks Anchi. Replaced tifffile.py in pyami with 2017 version. That way I don't have to worry about adding it to autoinstall script. I'll work on making the rest of changes next.