Allow Quest passthrough video for PC VR

[meghan.mccarthy@…]

Want to be able to use passthrough video on Quest 3 headsets with ChimeraX VR. This is a goal of the 2024 NIAID contract.

[Tom Goddard]

Their is a Meta OpenXR extension called XR_FB_passthrough (FB stands for Facebook) specifically to allow doing this

​https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_FB_passthrough

Several people have tried to use it online in late 2023 and early 2024 and none of them were able to get it to work from what I could see online. They can use the extension but can't get it to blend the computer generated objects and the passthrough video.

After I enable in the Oculus PC app Settings / Beta / Developer Mode and Settings / Beta / Passthrough Video over Meta Quest Link, and add code to ChimeraX OpenXR I see the passthrough extension among many others. Here is the full list

XR_KHR_D3D11_enable
XR_KHR_D3D12_enable
XR_KHR_opengl_enable
XR_KHR_vulkan_enable
XR_KHR_vulkan_enable2
XR_KHR_composition_layer_depth
XR_KHR_win32_convert_performance_counter_time
XR_KHR_convert_timespec_time
XR_KHR_composition_layer_cube
XR_KHR_composition_layer_cylinder
XR_KHR_composition_layer_equirect
XR_KHR_visibility_mask
XR_KHR_composition_layer_color_scale_bias
XR_EXT_win32_appcontainer_compatible
XR_EXT_debug_utils
XR_OCULUS_recenter_event
XR_OCULUS_audio_device_guid
XR_FB_color_space
XR_FB_display_refresh_rate
XR_META_performance_metrics
XR_META_headset_id
XR_OCULUS_ovrsession_handle
XR_EXT_hand_tracking
XR_FB_hand_tracking_aim
XR_FB_hand_tracking_capsules
XR_FB_hand_tracking_mesh
XR_METAX1_simultaneous_hands_controllers_management
XR_META_simultaneous_hands_and_controllers
XR_FB_body_tracking
XR_META_body_tracking_full_body
XR_META_body_tracking_fidelity
XR_META_body_tracking_calibration
XR_FB_keyboard_tracking
XR_FB_passthrough
XR_META_passthrough_layer_resumed_event
XR_FB_triangle_mesh
XR_META_passthrough_color_lut
XR_METAX1_passthrough_color_lut
XR_META_passthrough_preferences
XR_FB_render_model
XR_FB_spatial_entity_container
XR_FB_scene
XR_FB_spatial_entity
XR_FB_spatial_entity_storage
XR_FB_spatial_entity_query
XR_FB_spatial_entity_user
XR_FB_spatial_entity_storage_batch
XR_FB_spatial_entity_sharing
XR_META_spatial_entity_mesh
XR_META_virtual_keyboard
XR_EXT_local_floor
XR_EXT_active_action_set_priority
XR_FB_haptic_amplitude_envelope
XR_FB_haptic_pcm
XR_FB_touch_controller_pro
XR_META_touch_controller_plus
XR_FB_touch_controller_proximity
XR_OCULUS_external_camera
XR_META_detached_controllers
XR_FBX1_touch_controller_extras
XR_EXT_palm_pose
XR_METAX2_detached_controllers

I will see if I can use XR_FB_passthrough to enable passthrough with ChimeraX VR.

[Tom Goddard]

I got passthrough video working with the XR_FB_passthrough OpenXR extension. Added a "passthrough true" option to the ChimeraX xr commmand.

On Quest 2 using AirLink set to use slow 10 Mbit/sec wifi bandwidth (at home Windows 10, GTX 1080 Ti, Wifi 5 several rooms away, Quest v63 runtime, Meta Quest Link App Version 64.0.0.489.367) it stuttered badly and the pass-through updated only a few times per second and so did not track head motion well. I wonder if this means that the pass-through video is being sent to the Windows PC to composite with the rendered graphics and then sent back to the headset. That would explain horrible performance. I had hoped that the compositing is done on the headset.

I have the Meta PC app Settings / Beta / Developer Runtime Features enabled, and Passthrough Over Meta Quest Link enabled.

Next need to try with Quest Link and Quest 3 headset.

[Tom Goddard]

Passthrough video worked well with little or no lag on Quest 3 with QuestLink. Moving models showed some flicker, but walking around them showed no flicker.

Tested passthrough video on Quest 3 with link cable, Windows 11 (vizvault.cgl.ucsf.edu), Meta Quest Link App Version 64.0.0.489.367, Quest runtime v62.

Next need to try Quest AirLink on RBVIVR wifi 6 at close range.

Also should try SteamLink -- maybe does not offer XR_FB_passthrough extension.

Need to add an easy way to turn passthrough on and off while in VR. Current it is turned on with command "xr passthrough true" and off with "xr passthrough false".

[Tom Goddard]

I tried Quest AirLink and it stuttered badly updating a few times per second, completely unusable. This was with Quest 3 with v63 runtime to wifi 6 (rbvivr wifi), with Windows 11 PC connected by ethernet to the same subnet. Switching to QuestLink (usb cable) made it work smoothly with a small bit of flicker while moving models.

[Tom Goddard]

To capture an image as seen through the Quest 3 headset with pass through while using QuestLink start screen casting (oculus.com/casting from computer Chrome browser, then click Camera icon on Quest 3 dock and Cast button, takes 30 seconds to connect, uses wifi). Then uses Windows key + PRTSCRN key to capture a Windows screenshot showing the Chrome mirroring. Then crop the screenshot in Gimp to just the Quest view. I've attached an example image captured this way.

A few other approaches were unable to capture the pass-through video and only showed the molecules. In the Quest Link dock pointing at the right date and time indicator shows a panel with a screenshot icon that captures after a 5 second count down, no passthrough, image is saved in Pictures on Windows PC. Also there is an OculusMirror app

C:\Program Files\Oculus\Support\oculus-diagnostics\OculusMirror.exe

that also fails to show the pass-through video.

[Tom Goddard]

ChimeraX VR captured image with passthrough using casting.

[Tom Goddard]

Passthrough does not work with SteamLink because the Meta Quest passthrough extension XR_FB_passthrough is not provided by OpenXR. Tested with SteamVR 2.4.4 and SteamLink Quest app version 2.0.7 (release November 30, 2023) on Quest 3 with v63 runtime.

xr true passthrough true

Available openxr extensions:
XR_KHR_vulkan_enable
XR_KHR_vulkan_enable2
XR_KHR_D3D11_enable
XR_KHR_D3D12_enable
XR_KHR_opengl_enable
XR_KHR_win32_convert_performance_counter_time
XR_EXT_win32_appcontainer_compatible
XR_KHR_binding_modification
XR_KHR_composition_layer_depth
XR_KHR_visibility_mask
XR_EXT_active_action_set_priority
XR_EXT_dpad_binding
XR_EXT_frame_composition_report
XR_EXT_hand_tracking
XR_EXT_hand_joints_motion_range
XR_EXT_hp_mixed_reality_controller
XR_EXT_local_floor
XR_EXT_palm_pose
XR_FB_display_refresh_rate
XR_HTC_vive_cosmos_controller_interaction
XR_HTC_vive_focus3_controller_interaction
XR_HTC_vive_wrist_tracker_interaction
XR_MND_headless
XR_VALVE_analog_threshold
XR_HTCX_vive_tracker_interaction
XR_EXT_debug_utils


started OpenXR rendering SteamVR/OpenXR : oculus

The Facebook passthrough video extension is not supported by this instance of OpenXR

[Tom Goddard]

I added an icon "VR" that can be added to the Home Toolbar by right click (or ctrl-click) on the toolbar then select Settings and drag the Extra / Viewing category from the right "Available buttons" side of the pane to the left "Home Tab" side of the pane. Then click Save to keep this on your Home toolbar for future ChimeraX sessions. Clicking this button runs command

xr on passthrough toggle

so it will start VR using OpenXR and enable passthrough. Clicking it again will turn off or on passthrough. This allows easily turning passthrough on and off while in VR.

[Tom Goddard]

VR toolbar icon on Home Toolbar tab.

[Tom Goddard]

Implemented passthrough.

Uses new xr command option passthrough with allowed values true|false|toggle (e.g. "xr on passthrough true"). When VR initially started with no passthrough option it is off by default. Tested with Quest 3 headset and QuestLink (usb cable) connection to PC. AirLink and SteamLink do not work (details in previous comments). Non-meta headsets (e.g. Valve Index, Vive Pro, ...) will not work because pass-through uses a Meta OpenXR extension.