Better handling of missing background depth in augmented reality videos

[Tom Goddard]

I need a mechanism to assure complete depth information for the background when recording augmented reality videos with a depth sensing camera (RealSense D435). Currently large patches on posters (shiny material?) in the vizvault have no depth information. This causes those patches to incorrectly show through rendered models. Here is what happens. There is no depth for the true background (room wall), so instead when I walk in front of that patch it decides I am the background and sets the background depth and color to where I stood. When I then move out of the way and a model is rendered, the realsense image shows the wall, its color does not match the "background" so it doesn't use the background depth, and instead uses the last recorded depth (where I was standing) for the realsense camera image and that is in front of the computer rendered model in many cases. So we see the far wall in front of the rendered model.

One idea for getting complete background depth is to turn on the RealSense IR projector. That will help get depth info. But it interferes with the Vive VR tracking. So turn it on for a short while 10 - 30 seconds and have the people get out of view and try to record a full depth image. Need to test if the IR projector is enough. It probably is not sufficient. If some areas still did not have depth, at least we get the background color correct and we can simply assign depth as 5 meters for any pixels where depth is unknown. Alternatively we could fill the missing depth regions with a heat equation spreading of known depth values into unknown regions.

If the IR projector is not enough we could record background color while user is out of view. Then let the user walk around the places where depth was not determined to paint in the depth.

We might want to save the depth image to a file for use on subsequent sessions as long as camera position has not changed.