Add GLTF saving of textures

[Tom Goddard]

For NIH 3D model repository NIAID wants to use GLTF. Currently we output vertex colors into GLTF. Many 3D modeling software packages don't deal with vertex colors and only support texture colors. Need to check which packages. Babylonjs.com web viewer handles gltf vertex colors. Others to check out are Blender, Maya, Cinema4D, Unity. Some apps use a plugin to read gltf. Kristen Browne at NIAID has a good bit of experience wrangling the ChimeraX models into other software.

It would help make more usable exported models if ChimeraX could write texture colors into GLTF. We can currently do this in a roundabout way with OBJ format as described here:

https://www.rbvi.ucsf.edu/chimerax/data/texture-may2021/export_3d.html

[Tom Goddard]

I added texture saving in GLTF and tested it with a sphere textured with an image using the "color image" command. Here are some details and problems.

1) GLTF only handles 2D textures in JPEG or PNG formats.

2) The texture images can be within the glb file or as separate files, I put them within the glb file.

3) ChimeraX graphics.Texture does not keep a copy of the numpy color array after making the OpenGL texture to avoiding doubling memory use for volume rendering. This makes it too hard to get the actual image data. Could read back from OpenGL but seems like a bad idea. This means no ChimeraX textures (volumes or from color image command) can currently be saved in GLTF without hacking the ChimeraX code to keep the color array. I did tests by squirreling the array in texture.image_array.

4) ChimeraX volume rendering using Drawing.multitexture which allows applying several textures to different rectangles of a single drawing. My GLTF texture output does not handle that, only single textures using Drawing.texture. It could handle it but it adds more complexity.

5) The most desirable use case is probably electrostatic coloring which currently uses vertex colors. Should I add an option to instead use texture colors? Should it use a 1D texture and let the GLTF code convert it to the mandatory 2D texture? Instead of using texture coloring in ChimeraX I could just have attributes for the colormap and vertex values for GLTF to make the texture for export and the texture option would be for GLTF saving. That has the problem that those attributes take memory and almost no one will use them, and would be lost if a session is saved unless support is added.

6) Other interesting used cases might be hydrophobicity coloring, APBS electrostatics using a volume file, radial (or other geometric) coloring.

7) Maybe the most basic useful case is converting atom/bond/ribbon from vertex colors to texture colors using a color palette as done by the "color image" command. That could be done by the "save file.glb textureColors true" command with a textureColors option.

[Tom Goddard]

I can imagine some tricky code could figure out 1D textures on surfaces from the vertex colors. That could allow the gltf save to handle all the conversion to texture colors without burdening all the routines that produce the colorings to support texture colors and vertex colors. Having the many routines that do coloring offer texture coloring seems like a recipe for very rarely used broken code.

But I am not sure the 1D textures could be figured out to high enough quality. The idea would be that vertices connected by an edge in the triangulation should be close in the 1D palette that gets built up.

There seem to be 3 approaches to getting the texture colors for say an electrostatics colored surface. 1) Have ChimeraX use texture colors controlled by an option to the command that does the coloring (e.g. coulomb command). 2) Have the coloring command keep the vertex values and colormap as an attribute that GLTF save can use. 3) Have gltf save deduce the 1D texture from the vertex colors. I don't like any of these -- too much complexity that will be very rarely used.

[Tom Goddard]

Maybe a 4th option would be to use vertex colors but specific coloring routines could add a method to the Drawing that would provide vertex values and a color map.

Maybe I should start with just converting atom/bond/ribbon colors to a palette texture as the first and probably most useful case for saving gltf.

[Tom Goddard]

I added a textureColors option for saving gltf (eg. "save stuff.glb textureColors true") that converts vertex colors and instance colors of the ChimeraX models into texture colors. The default is false, so by default vertex colors are written. This option works for atoms and bonds since those use instance colors. It also applies to any Drawing where all the triangles have single colors. So it applies to ribbons. Also some surface colorings, for instance, ones where atom patches are colored. But continuous surface coloring like electrostatic potential, hydrophobicity, or radial coloring won't get converted to texture colors in the GLTF file because for those single triangles have 3 vertices with differing colors.

In the future I may make the continuous colorings derived from a color map also get converted to texture colors. In theory it only requires converting the color map (ie palette) to a texture. But by the time the gltf save is done there is no way to get the color map -- the surface has already been given per-vertex colors. So I would probably need to add a method to get the color map and the vertex values. That is not too hard although it will take more memory to remember those, also if I don't add session support for remembering those then it may not work after restoring from a session. And also some colormap colorings only color part of a surface, for instance, if sampling from a volume and the surface extends outside the volume. A texture applies to the whole surface, and since the "outside" color is not part of the colormap it won't be in the texture. The surface could be split by the gltf save to handle the part using a colormap and the rest as two nodes, but this gets very complicated. So there are several problems with trying to output texture colors for continuously colored surfaces.

[Tom Goddard]

To export volume image style rendering I need to be able to get the 2D texture images. Currently these are not kept after the OpenGL texture is made. The pixels can be read back with glGetTexImage(). Another complication is that volume rendering uses Drawing.multitexture which applies several textures to successive rectangles. The gltf code would need to make a separate node for each rectangle. These additions don't appear too hard, but it would be nice to have a use case to motivate doing the work.