Volume rendering artifacts from 8-bit colors

[Tom Goddard]

Volume rendering as a 3d image sometimes appears darker at step 1 compared to step 2, or has dramatically different brightness when using z-axis planes versus x or y-axis planes. These variations in appearance can be caused by using 8-bit colors in OpenGL framebuffers and textures with many (e.g. np > 100) highly transparent planes (e.g. alpha < 5/255). The variable brightness comes from limited resolution especially in texture transparency values, for example, with 8-bit alpha ranging from 0 to 255, 100 planes of alpha = 1 gives very different appearance than alpha = 2. Likewise accumulating in the frame buffer the sum of 100 planes with 8-bit red value = 1 is dramatically different than red value = 2.

Cases of incorrect appearance are common when rendering volumes with grid size > 100. For correct appearance the range of color and alpha values (256 values for 8-bit) needs to be substantially greater than the number of planes.

I tested using 16-bit rgba in the framebuffer and textures and this gave accurate consistent coloring.

It would be useful to allow the user to render with 16-bit rgba. It probably should not be the default since it is likely to reduce rendering speed. Both graphics and volume code would need improvement to enable 16-bit RGBA.

[Tom Goddard]

DICOM lung CT 512x512x236 step 1 folder 4-24533, normal 8-bit rendering

[Tom Goddard]

Step 2, 8-bit rendering.

[Tom Goddard]

Step 1, 16-bit framebuffer and texture.

[Tom Goddard]

Step 2, 16-bit framebuffer and texture

[Tom Goddard]

Step 1, 16-bit framebuffer, 8-bit texture

[Tom Goddard]

Step 1, 8-bit framebuffer, 16-bit texture.

[Tom Goddard]

I attached 6 example images of a lung CT scan (folder 4-24533), grid size 512 x 512 x 236, 3d projection rendered with plane spacing minimum along 3 axes, voxel size .576 x .576 x 1.25, so about 500 planes, view along z axis, perspective projection. Step 1 and 2 look quite different with normal 8-bit rendering, but look nearly the same with 16-bit rendering. Both 16-bit framebuffer and 16-bit color/alpha texture contribute have significant effect on the appearance.

[Tom Goddard]

The 6 images were screen captured directly of a retina Mac laptop screen.

[Tom Goddard]

Tiled version of 6 images for easier comparison.

[Tom Goddard]

I added a tiled version of the 6 images for easier comparison.

[Tom Goddard]

Comparison of step 1 vs step 2 for 3 channel blended light microscopy, 8-bit.

[Tom Goddard]

The attached light microscopy shows step 1 is darker than step 2 -- change is quite noticeable when switching between step 1 and 2. This is 8-bit. Data is 1024 x 1024 x 81, using 2d-xyz projection so only 81 planes. Cannot try 16-bit textures because CPU blending code can only handle 8-bit, but I believe 16-bit texture blending would give brightness similar to step 2 image.

[Tom Goddard]

I tried creating an opengl context requesting a 16-bit each red, green, blue framebuffer on 2012 MacBook Pro, and context could not be created. Not sure if this is a Qt limitation or Mac OpenGL limitation. Possibly the window bit depth needs to match the opengl context.

But I can get 16-bit framebuffer rendering using an off-screen framebuffer. That is done when image save is used. It is also done when silhouette edges are enabled. The Mac laptop did provide 16-bit off-screen framebuffers. So to show 16-bit framebuffer appearance I turned on silhouette edges (which has no edge highlighting effect in volume rendering).

To use 16-bit textures I used command "volume #1 colorMode auto16".

To allow users to enable 16-bit rendering might add a graphics command option "colorDepth 16". This would cause offscreen framebuffers to use 16-bit and also make volume textures in color mode "auto" use 16-bit instead of 8-bit. I think current default volume color mode is auto8 so it would need to be changed to just "auto". Using this graphics option would require recreating existing framebuffers and volume textures.

It would be a lot of work to be able to change the default framebuffer to 16-bit while ChimeraX is running, and may not be possible to obtain such an opengl context. All opengl resources (vertex buffers, ...) would need to be recreated, and the code does not support that currently. (There is support for recreating framebuffers and bindings for switching to a shared stereo opengl context.) A simpler approach would be to always render to an offscreen 16-bit framebuffer if 16-bit enabled and blit the result to the default framebuffer. This is what happens with the current silhouette edge rendering, so all the support is available.

[Tom Goddard]

All ones map for controlled transparency / bit depth tests.

[Tom Goddard]

Python script to compute predicted brightness for alpha-blending with various framebuffer bit depths.

[Tom Goddard]

I added a colorDepth option (values 8 or 16) to the graphics command to switch offscreen framebuffers to use 8 or 16 bits per color channel. This allows testing 16-bit color depths without editing the code. On macOS 10.14.2 with Radeon Pro 580 graphics rendering speed dropped from 45 frames/sec to 20 frames/sec which switching from 8 to 16 bit color depth. Further changing 3d texture from 8-bit to 16-bit reduces frame rate to 15 frames/sec. Timings for map size 512 512 124, lung CT scan with 800 x 800 window size on Retina display (so really 1600 x 1600) with map scaled to match window width. Frame rates very sensitive to volume size in screen pixels -- fill rate limited.

Only offscreen framebuffers are changed to 16-bit depth and when switching to 16-bit an offscreen framebuffer is used. I tried requesting 16-bit color depth when creating the OpenGL context but creation fails. It made no difference if the QWindow was requested to use 16-bits (the window was created and claimed its format was 16-bits).

[Tom Goddard]

To get 16-bit image rendering both the framebuffer color depth needs to be set to 16 (graphics colorDepth 16) and the textures used by the image rendering need to be set to 16 bits (volume #1 colorMode auto16).