Allow Quest 2 or other standalone headsets to join ChimeraX VR meetings

[Tom Goddard]

Allow researchers to use ChimeraX virtual reality and join meetings using just a Quest 2 headset without a VR-capable PC.

Most of the virtual reality market has been taken over by the stand-alone VR headset Quest 2. This has the enormous advantage that a Windows PC with expensive graphics cards is not needed. The cost of the Quest 2 is about $400 while the cost of the Windows PC and graphics is about $1500 with the graphics card alone costing as much as the Quest 2 headset.

In order to allow ChimeraX virtual reality to be more widely used it would be great if it worked with rendering on the Quest 2. The Quest 2 operating system is Android and ChimeraX will not run on it. An alternative is ChimeraX can run on any computer (Mac, Linux, Windows) and send scenes to the the Quest 2. The user could interact with the molecule models on the Quest 2, moving, zooming, and using mouse modes and communicate back to ChimeraX what was done. Operations like moving or zooming would be done entirely on the headset so it is very responsive, while operations that change what is shown would send ChimeraX the button clicks and drags and ChimeraX would send an updated scene, so much slower updates, perhaps with a 1 second delay.

This method of ChimeraX sending scenes to the Quest 2 headset could be used for multi-person meetings. ChimeraX would only run on a single computer. This is different from the current meeting method where ChimeraX runs on each participant's computer. This simplifies synchronization.

[Tom Goddard]

Implementation Ideas

Here are some thoughts on how this could be implemented.

1) The Quest 2 could run a ChimeraX scene application written in Unity that takes gets just a graphics scene description (triangles, vertices, colors, instances, textures) obtained from a network connection.

2) For meetings where participants are at different institutions the Quest 2 would connect to a public server for instance on AWS to avoid firewall problems. The controlling ChimeraX running on a PC would also connect to that server.

3) I could also be possible to connect a VR headset directly to ChimeraX as is done now to get low latency changes of the scene, like adjusting a map contour level.

4) The scene sent to the quest 2 may need to be a reduced resolution version of what ChimeraX would show on the computer since the Quest 2 has limited graphics power, maybe 500K triangles max.

5) ChimeraX or the AWS hub could keep track of the mouse modes for each participant and the head photos.

6) To assign buttons on the Quest 2 the scene app might just hand button clicks back to ChimeraX which would determine if the click was on a user interface panel. The Quest 2 app would ideally know about buttons so they could popup when pressed without a round-trip to ChimeraX.

7) This Quest 2 VR connection mechanism to ChimeraX could be in addition to current VR meeting capabilities in ChimeraX.

8) The Quest 2 app might use OpenXR (instead of SteamVR) so it can directly support Oculus and other vendor's standalone headsets.

9) Users with PC VR headsets should be able to connect via the same mechanism to the remote ChimeraX.

10) Users in a meeting with PC VR could open data from their own computer if the communication method to ChimeraX and the PC scene handling app included a way to transfer files.

11) Tom Skillman uses Photon, a Unity multi-person VR platform, and suggests looking at it as an already made solution for generic scene sharing and user interaction.

12) The Quest 2 app would need unusual custom graphics capabilities to handle ChimeraX graphics well such as instancing of atoms and bonds and green selection outlines.

13) Having ChimeraX not directly do the VR rendering could greatly improve smooth operation since ChimeraX can do longer calculation without interrupting the VR rendering.

14) The Quest 2 app should be able to stream a new scene in the background and only display it when it is completely transferred, while still showing the old scene.

[Tom Goddard]

Various graphics capabilities of ChimeraX are likely to be some trouble to implement in Unity

A) instancing of atoms/bonds
B) selection outlines
C) volume image planes rendered back to front
D) residue labels reorient to face viewer
E) picking objects is done in ChimeraX on CPU and round trip may be slow

[Tom Goddard]

Ticket #8375 describes a simpler scheme where the standalone Quest 2 headset only shows GLTF models and does not attempt to send hand-controller clicks back to ChimeraX for updating the scene.