Try newer RealSense depth cameras for mixed reality video capture

[Tom Goddard]

Begin forwarded message:

From: Tom Goddard
Subject: ChimeraX mixed reality with RealSense camera
Date: March 8, 2022 at 6:13:13 PM PST
To: "Cruz, Phil (NIH/NIAID) [C]", Meghan McCarthy

Hi Phil, Meghan,

Here is a status updated on mixed reality VR capture with ChimeraX. Hopefully we can set this up in the BioVizLab.

Today I tried my Intel RealSense D435 depth camera that I use to make mixed reality video captures (VR objects plus room and person video) on a new Windows 11 machine with Nvidia RTX 3090 graphics, Vive Pro headset, Vive Tracker 2.0 on the camera. It all worked nicely. In fact for some reason the depth camera does not have as many pixels where it cannot find the depth that lead to visual artifacts. That improvement may be because I updated the RealSense camera firmware from version 5.12 to 5.13.0.50. Would need to test on another computer to see if it was the firmware. Could also be I just happened to move the camera so it points at a favorable background.

Intel has a couple newer depth cameras. The one I use D435 ($289) was released in July 2018. The D455 ($399) was released in June 2020. The D455 might be better for mixed VR capture. It has a wider field of view, 90 degrees vs 70 degrees horizontal, but it also has lower RGB resolution, 1280x800 vs 1920x1080. But I run at half that 960x540 because otherwise the framerate becomes choppy because combining depth and RGB for the RealSense is a compute intensive operation. The D455 was mostly intended to handle longer range 6 meters, vs 3 meters but that is not very important for VR capture. There are some other features of the D455 that may be interesting -- global shutter instead of rolling shutter may reduce mismatch between depth and RGB, better depth resolution because the camera is physically wider (12 cm wide vs 9 cm wide). I think it would take only small adjustments to ChimeraX to handle the D455.

Another new Intel depth camera, the L515 ($589) was released July 2020 and uses Lidar instead of stereo views. It is also worth trying. The main think I would be looking for is being able to get depth as many pixels as possible. The main quality problem of the mixed VR capture is that the depth camera fails to get depths for large patches, for instance plain white walls. Also framerate is a pretty serious quality issue. I think I have often captured at 15 frames per second because the camera struggles to fuse the depth and rgb at faster framerates.

Here is a link to Intel's page comparing the depth cameras

​https://www.intelrealsense.com/compare-depth-cameras/

Tom

[Tom Goddard]

VizVault setup with Intel RealSense D435 camera.