Optimize speed reading TIFF volume data

[Tom Goddard]

Reading 3d images in TIFF format with Pillow is much slower than one would expect based on disk I/O speed. It is possible to increase the read speed perhaps 5-fold using something other than Pillow.

I used an example image AICS-10_10.ome.tif, 390 Mbytes, grid size 1024 by 1024 by 81, 16-bit grayscale values, 8 channels (3 wavelengths, one brightfield, 4 segmentations). This file would normally be 1.3 Gbytes but deflate compression is being used. Four channels are binary (0/1) segmentations that compress to very small size. The three fluorophore channels use only about 8-bits for most data values. Decompressing the data is part of the time of reading the file.

Timings reading AICS-10_10.ome.tif, only first 3 channels.

7.3 seconds - ChimeraX open at step 2 then switch to step 1, includes blending and graphics.
5.2 seconds - Pillow 5.4.1 with standalone script reading planes for 3 channels.
1.8 seconds - libtiff C program reading planes for 3 channels and making single 3d array
0.8 seconds - tiffile.py single Python script reading all planes of first 3 channels

The tifffile.py library is in PyPi, seems well maintained, and is specifically for optical microscopy. Seems like a good choice to replace Pillow.

[Tom Goddard]

Test of ChimeraX reading AICS-10_10.ome.tif with tifffile.py:

3.2 seconds - ChimeraX using tifffile.py to read open first 3 channels at step 2 then switch to step 1. Based on standalone tifffile.py timing with the AICS-10_10 file, total time for step 2 and step 1 read should be about 1.2 seconds (= 0.4 + 0.8). Appears that 2 seconds is used to blend and display the images. This agrees with timings of ChimeraX using Pillow and Pillow standalone where ChimeraX took 2.1 seconds more than simply reading data.

[Tom Goddard]

Switched to reading TIFF volume with tifffile.py (version 2019.3.18) instead of Pillow (5.4.1) for 5x faster read times, and likely better handling of unsual scientific value types, e.g. 16-bit. OME, ImageJ and plain TIFF files use only tifffile.py now.