Use OpenXR instead of SteamVR for virtual reality rendering

[Tom Goddard]

The OpenXR API for virtual reality has gained wide-spread acceptance. It allows rendering to Oculus headsets using the Oculus App without also requiring SteamVR. This is likely to make it more reliable and perhaps improve performance. Since Oculus Quest headsets are most of the current VR market using OpenXR will simplify the use of VR by not requiring Steam and SteamVR.

Headsets such as Valve Index and Vive Pro that natively use SteamVR also work with the OpenXR API since the SteamVR app handles the OpenXR protocol.

Converting ChimeraX to OpenXR from SteamVR is part of our NIAID 2023 contract.

ChimeraX currently uses the OpenVR / SteamVR API through the pyopenvr PyPi package. The OpenXR support will use the pyopenxr PyPi packages which is developed by the same person as pyopenvr (cbruns).

[Tom Goddard]

Initial port to OpenXR

I have ChimeraX using OpenXR for rendering and hand controller input but only the most basic testing has been done so far. This work took about 2 weeks following the 2000 line hello_xr example code

​https://github.com/cmbruns/pyopenxr_examples/tree/main/xr_examples/hello_xr

OpenXR requires much more boiler-plate code and I added 800 lines of pure OpenXR code in vive/src/openxr.py and modified hundreds of lines of the ChimeraX vr code in vive/src/xr.py.

To allow testing openxr by the NIAID I made a separate command "xr" that has all the same options as the "vr" command. The "xr" command uses OpenXR and the "vr" command uses the SteamVR code. When the OpenXR code is fully tested I plan to replace the SteamVR code and have the vr command use OpenXR and remove the xr command.

[Tom Goddard]

Fixed disabling vsync so mirroring to desktop does not cause stutter

The openxr rendering flickers using Quest Pro or Quest 2 with the Oculus OpenXR runtime when desktop mirroring is used on vive.cgl.ucsf.edu with Quadro P6000 GPU. Tests show that the mirror drawing does not cause the flicker if the OpenGL swap buffers call is not made, and that with no mirror drawing it still flickers if swap buffers is called. So it appears that swap buffers is causing the flicker. With the Vive Pro headset and OpenXR steamvr runtime there is no flicker with mirroring on. So it seems the problem is specific to the Oculus OpenXR runtime. I also tried drawing the mirror window before releasing the right eye swap chain texture, but still the swap buffers was done after the release, and that also flickered.

Some further tests would be useful:

1) Make sure mirroring is not locking to 60Hz vsync.

Result: Indeed with Oculus OpenXR and mirroring chimerax "graphics rate on" shows 60 fps, while with OpenXR steamvr runtime it shows 90 fps. This explains the flicker. It turns out the ChimeraX wait_for_vsync() routine no longer works on Windows. Apparently it was broken long ago by Qt 6 which resets the wait for vsync mode on every makeCurrent according to the QSurfaceFormat.swap_interval setting. I fixed the wait for vsync problem by changing the code to use QSurfaceFormat.setSwapInterval() instead of wglSwapIntervalEXT(). This fixes the problem.

2) See if no swap buffers but glFinish() causes the flicker.

Flicker was caused by mirroring locking to the display refresh rate because the chimerax code that disabled wait for vsync was not working. Fixed now.

3) See if rendering the mirrored image to the front buffer with no swap buffers causes flicker.

Not necessary since vsync problem fixed.

4) See if the Quest headset used through the OpenXR steamvr runtime flickers.

Did not flicker. Apparently the SteamVR library somehow disabled waiting for vsync, so it didn't matter that the chimerax code to disable vsync was broken.

[Tom Goddard]

Fixed sleeping bug

When the Quest 2 headset sleeps a few seconds after it is taken off, then putting it back on does not restart the rendering. Some debugging showed that rotating the view with the mouse restarted the VR rendering. The trouble was that ChimeraX did not think it needed to render because the scene had not changed. I put in a line of code so it always renders whether or not the scene has changed.

[Tom Goddard]

AirLink with Quest stutters with SteamVR or OpenXR

The Quest 2 rendering with OpenXR and AirLink at 72 Hz is still glitching about once per second when I move my head looking at a small protein 3nox. This could be an AirLink issue, or OpenXR issue, or Quest 2 issue. See if SteamVR rendering also does this over AirLink. See if using Quest Link cable also has intermittent glitches.

SteamVR rendering with Vive Pro at 90 Hz using openvr does not glitch at all.

SteamVR rendering with Quest 2 and AirLink using openvr glitches.

SteamVR rendering with Quest 2 and Quest Link cable using openvr does not glitch (maybe one glitch per minute or two).

OpenXR with Quest 2 and Quest Link cable (no steamvr) does not glitch.

So it appears that AirLink is introducing the glitches and they happen with both the OpenXR API and SteamVR API.

These tests were all with vive.cgl.ucsf.edu, Windows 10, Quadro P6000, ethernet to RBVIVR router, no wifi connection with computer, with Quest 2 headset using AirLink on RBVIVR wifi 6 showing signal strength "excellent" at 5 GHz and 573Mbps send and receive speeds. The Quest AirLink bit rate was set to 100 Mbps dynamic.

If I lower the Quest AirLink bit rate to 20 Mbps or 10 Mbps dynamic the glitching goes away although visual quality is a bit worse. At 30 Mbps I start getting glitches again. The bit rate setting is in the Quest headset when connected by AirLink, the icon bar has a facemask icon on the left that allows setting bit rate. Maybe this old Windows 10 machine is too slow at compressing the video. Should test the new Windows 11 Nvidia 3070 VR machine to see if it exhibits the same glitching.

Got similar results with AirLink at home with 802.11ac router using Quest 2 in same room as router with Windows 10 Nvidia 1080-Ti plugged into router by ethernet cable. Quest reports "excellent" signal strength, 5GHz, 866 Mbps send/receive speed for wifi. My home wifi router at 5 GHz is using 80 MHz channel width on channel 36. Glitching about once per few seconds at 100 Mbps, no glitches at 30 Mbps, a glitch every 20 seconds at 50 Mbps. The wifi was also being used for VNC (my wife) at the same time. OpenXR and SteamVR gave very similar results with AirLink. Link cable for some reason would not work (using old USBA cable), computer would sometimes recognize Quest 2, often not, dropped it quickly with a usb device malfunction warning in Windows. Might be loose usbc connection to headset or old USBA cable and old motherboard USB (circa 2017) was not up to the task. But I think I used link cable successfully in the past. Tried 4 different USB ports.

[Tom Goddard]

Meetings

Made vr meetings work with any combination of OpenXR and SteamVR, ChimeraX 1.7 or 1.6. Tested all combinations. The OpenXRCamera is not sent to joining participants as part of the scene, so participants will have to start VR or OpenXR on their own if the host is using OpenXR. This differs from how SteamVR worked where the SteamVRCamera was sent as part of the scene so would automatically try to start SteamVR when joining a meeting.

[Tom Goddard]

Keep SteamVR and OpenXR in ChimeraX 1.7

My original plan was to replace SteamVR with OpenXR. But OpenXR has not been tested enough yet, so for ChimeraX 1.7 I will keep both SteamVR activated with the "vr" command and OpenXR activated with the "xr" command. In the future I may drop SteamVR or I may continue to support both.

Currently thousands of lines of code are needlessly duplicated between SteamVR and OpenXR. If I decide to keep SteamVR then I will clean it up so that only one copy of all the common code is used.

[Tom Goddard]

Wrong colors

The colors shown in OpenXR in the VR headset are quite different from what ChimeraX shows on the desktop when not in VR. When turning on VR the mirrored desktop display changes colors dramatically to the VR colors.

The problem is my OpenXR code is choosing an sRGB color format for the framebuffer. That format means you write to it in linear color space and it internally converts to sRGB. But our ChimeraX code does everything in sRGB (including lighting calculations which are supposed to be done in linear space), so we are writing sRGB to this framebuffer that is expecting linear.

The simple solution is to try to choose a different color format for the OpenXR framebuffer. I think my code notes the available formats for SteamVR and Oculus OpenXR and one of them had limited choices that led to me choosing sRGB. Need to revisit that.