How can the ChimeraX VR coordinates be aligned for 2-person VR with Quest headsets?

[Tom Goddard]

In the past a two person VR meeting with both headsets in the same room could align the coordinate systems of the two headsets by drawing the VR same boundary in Steam VR. This positions the molecule in the same physical position in the room as seen by both VR headsets.

[Tom Goddard]

I tried setting identical VR boundaries with two Vive Pro headsets and SteamVR 2.7.4 and opened a model with xyz axes to visually see if the same coordinates were used in both headsets and they were. The y axis pointed vertically. The z axis pointed toward the long axis of the first edge of the bounds rectangle I drew (using "advanced mode" for drawing bounds in SteamVR in Room Setup). This z-axis direction is also the direction I pointed in and clicked during setup when it asked for my "monitor direction". It was curious that the origin of the coordinates was not centered in the rectangle. It was about 1 meter from the boundary in -z and 2 meters from the boundary in +z. But the center and orientation were the same in both headsets.

[Tom Goddard]

xyz axis model used for visualizing coordinate system in ChimeraX

[Tom Goddard]

I tried the same test with Quest 3 and Quest 2 headsets using the Meta Quest Link app, USB cables to the headsets, and SteamVR. Both Quest headsets had the boundaries created using the Oculus software by tracing the outline with the hand controller in the same way. The coordinate systems seen in VR were completely different rotated about 100 degrees relative to one another and offset in the room rectangle xy by about a meter along each axis. Furthermore when I set down the Quest 3 for a few minutes and then put it back on it showed coordinate axes with a different orientation and center (about 30 degrees and 2 meters different). That change happened even while ChimeraX was continuously running. So it appears the Quest 3 can change coordinate systems each time it wakes from sleep.

I only saw the Quest 3 change coordinates when waking from a short sleep (a few minutes), the Quest 2 always stayed the same. I saw 3 different coordinate systems for the Quest 3 during different wakes and one of them was identical to the Quest 2 coordinate system. Which coordinate system it choose could be related to what direction the headset is facing when you first put it on.

[Tom Goddard]

The Vive Pro headsets use tracking based on fixed lighthouse base stations so it is not too surprising that their coordinate systems stay fixed. The Quest headsets use inside-out headset camera tracking. It creates a point-cloud to align to the room. I don't understand why it would change its coordinate system. I would think each time you wake the headset it would match the current camera data to the previous point cloud since it needs to preserve the coordinates so it knows where the boundaries are. So I don't have any insight into why Quest 3 would be changing the application coordinate system while the application is running if the headset sleeps.

[Tom Goddard]

Next I tried the Quest 3 and Quest 2 headsets using only the Meta Quest Link app with OpenXR and no SteamVR. I used the ChimeraX "xr on" command. The behavior was the same as with SteamVR. The Quest 3 showed different coordinate systems every time I took off the headset for a minute an d put it back on with center moving by 1-2 meters and rotations of up to 100 degrees. None of the Quest 3 coordinate systems matched the Quest 2 coordinate system. The Quest 2 kept the same coordinate system when I took the headset off and put it back on a minute later on multiple tries.

[Tom Goddard]

The next test was whether Quest alignment would stay fixed when using SteamLink (wifi connection, without using Meta Quest Link app). With the Quest 3 the coordinate system changed every time I put the headset on when using SteamLink. I started ChimeraX VR with (vr on), I think SteamLink also supports OpenXR but I did not try that. With the Quest 2 the coordinate system stayed in the same position as I put on and took off the headset while ChimeraX VR continued to run. With the Quest 2 restarting ChimeraX and restarting SteamVR put the coordinate system in the same position and orientation.

[Tom Goddard]

I tested how to align two VR headset so that ChimeraX shows a molecule in the same position and orientation of the room. The way I used to do this with old Vive Pro headsets and SteamVR is by drawing the same room boundary tracing the room corners in the same order and giving the same "monitor direction". I retested this and it works with two Vive Pro headsets.

Testing the same approach with Quest 2 and Quest 3 headsets did not produce aligned coordinates. In fact, every time I took off the Quest 3 headset and put it back on it had a substantially different coordinate system (center moved by 1 meter, rotation 30-100 degrees). The Quest 2 kept its initial coordinate system. I did not have two Quest 2 headsets to test if both produced the same coordinate system. But with Quest 3 headsets it appears certain that the coordinate systems will not align. I tested the coordinate systems with 3 different connection methods 1) SteamVR + Meta Quest Link app, 2) Meta Quest Link app and OpenXR (no SteamVR), and 3) SteamVR and SteamLink (no Meta Quest Link app). All behaved the same. It seems the headset itself controls the coordinate system, not the PC software.

[Tom Goddard]

My conclusion is I found no way to align the Quest 3 coordinate systems for multi-person VR sessions. Meta created a new cloud API about a year ago designed to align the coordinate systems of multiple headsets. The user had to agree to upload their room point cloud information to Meta to use this service. I considered uploading room information derived from cameras to Meta to be invasive of privacy so I have not used it. But that is perhaps the only way to align Quest 3 coordinate systems for multi-person sessions.

The LookSee app for standalone Quest VR provides its own alignment method for multi-person sessions where each headset places two markers at the same two locations in the physical room. These are used to determine the coordinate system. I have noticed that this sometimes goes out of alignment because the markers move. That happens because the Quest 3 headset changed the coordinate system so the markers are no longer at the same position. I have not noticed this happening every time I take off and put back on the headset as is the behavior with PC VR. I'm not sure why the standalone Quest 3 would preserve the coordinate system better, and it clearly doesn't always preserve it. So it appears that also LookSee will have trouble keeping coordinates aligned in multi-person sessions with Quest 3 headsets. So also in this case using Meta's cloud alignment solution might be the only viable approach.

[Tom Goddard]

I noticed in Unity using the Meta VR SDK that the camera can have "origin" set to "eye", "floor" or "stage". Documentation suggests that "stage" may preserve the coordinate system when the headset is taken off and put back on and that may allow aligning coordinates in a multi-person meeting. The current behavior where the coordinates change looks like "floor" origin which allows the user to reorient the coordinate system by holding down the system button on the right controller. There is probably a way to set this origin type through OpenXR. I have not investigated it further.

[Tom Goddard]

Testing a LookSee 2 person meeting again in the vizvault with alignment markers placed on opposite sides of the room (corner of desk and corner of poster), the alignment remained good after taking the headsets of and waking them about 10 times. This is with v71 of the Quest headset runtime. Pressing and holding the controller system button to realign coordinates had no effect although it showed the realign icon. This was with LookSee v17. I had the display sleep set to 15 seconds and headset sleep set to 1 minute on both headsets. After I left both sleeping 5 minutes I came back and the non-host had dropped the connection and could not reconnect (socket error 113, no route to host) and I saw that the host wifi had changed from RBVI to UCSF WPA. Changing the host headset back to RBVI wifi allowed the non-host to reconnect to the meeting and alignment of both headsets was preserved.

In summary the meeting alignment with LookSee v17 and Quest runtime v71 seems to be stable.