Opening Imaris microscopy excruciatingly slow

[Tom Goddard]

Opening large Imaris 3D/4D microscopy data is extremely slow. This is loading a 3 Gbyte 2 channel data from Jeremy Linsley, for example file PID20170306_TgGCAMPNTR_T0_DMSO4_0_C1_Z0.ims (won't attach this since it is 3 Gbytes). It is slow loading at full resolution, and even slower loading at step 4,4,1.

The Imaris format is HDF5, ironically intended for high performance with large arrays. I think the problem is ChimeraX is reading plane by plane, or slabs of planes (map size 9K x 2K x 31, 2 channels , 6 times. Probably it is reading a big HDF5 block, extracting just one plane from it, then throwing away the big block, then reading the same block again and again for every plane. Even though it is in the macOS in memory cache it is excruciatingly slow, minutes to read one time point when this 500 Mbytes should read in seconds.

It may be that the ChimeraX code needs to take account of the brick sizes when reading HDF5 files. I thought I made the HDF5 cache size large enough (1 Gbyte) so bricks are not thrown out as soon as a plane is read but apparently this is not working. Or maybe the problem is misdiagnosed.