VR user interface suggestions

[scooter@…]

From: Scooter Morris
Subject: VR comments
Date: April 24, 2024 at 5:59:52 PM PDT
To: Tom Goddard, Zach Pearson

Hi all,

So, messing around with VR and I have some comments/suggestions.

1. I wasn't able to get passthrough to work. I kept getting an error "Facebook passthrough video extension isn't support by this instance of openxr". Pretty sure the oculus app is up-to-date.

2. I really miss the ability to dock panels. Having my menu bar, tool bar, and each panel move independently is a major pain. That alone would really enhance the UI, I think. Of course, I might want to undock a panel and drag it elsewhere, but I'm not sure that's the "normal" case.

3. I found I really wanted to resize the panels. I figured out that I could resize them by resizing the menu bar, but it didn't stick -- all subsequent panels I opened were back to the standard size.

4. Something is a bit strange with shadows. They move way too much for the amount of motion of my head. If I stayed perfectly still, they worked really well, though.

5. A label mode would be really useful -- maybe in combination with selection. I mapped the middle-finger trigger to selection, which worked OK (it was a bit tricky, but I eventually was able to make it work well), but it was hard to get any feedback on what I had selected. A label that stays up for a brief period would have been useful.

6. Speaking of the panels -- for many of the things I wanted to do on the panels, a laser pointer would have been better than the cones. Is there any way to change the "cursor" when I'm pointing at the panels? Alternatively, maybe a "mouse" mode that would allow me to map a button to a laser pointer. It would only act as a selection on the panels and not on the structure, but it might still be useful to be able to use it as a pointer in VR meetings.

I did play around with a chest CT a bit and again, the interaction with the panels was more difficult that I would have wanted, but overall, I found it reasonably illustrative.

-- scooter

[Tom Goddard]

Comment #1 of the ChimeraX VR passthrough ticket #14914 explains that you have to enable passthrough over QuestLink in the Meta Quest PC app, and it requires developer mode:

After I enable in the Oculus PC app Settings / Beta / Developer Mode and
Settings / Beta / Passthrough Video over Meta Quest Link, and add code to ChimeraX
OpenXR I see the passthrough extension among many others. 

This has not yet been documented in the ChimeraX VR documentation. Tom will add it to the documentation.

[Tom Goddard]

2) I tried docking the Model Panel and Log in ChimeraX on the desktop display and then started VR and showed the user interface, but the two panels were not stacked in VR but appear as separate floating panels. I agree that stacking panels as done on the flat-screen ChimeraX desktop would be nice.

3) All shown VR panels can be resized by pressing menu buttons on both hand controllers and moving controllers apart or together. This does not effect panels that are not yet shown or have not yet been created. Each panel can be resized individually by clicking with the trigger button on the resize icon each panel has in its upper left corner, then dragging. I could not find these in the current ChimeraX VR documentation. Tom needs to update the documentation. Elaine does not use the VR so is less familiar with these features. The menu bar does not have the upper left corner resize icon because it is not a Tool window. Probably resizing individual panels would not be problematic if the panels were stacked as in suggestion (2).

4) The shadows (enabled with graphics toolbar shadow button or full lighting) are cast from a spotlight pointing in the camera direction, ie the VR headset pointing direction. The spotlight acts as a light from infinite distance. So if you just turn your head in VR the spotlight will appear to be in a different direction. That is not intuitive, you probably expect the light is attached to the VR headset. That would require a different lighting mode special for VR.

5) There is a label model in the right mouse toolbar called "Label". That makes labels that stay. I think you click on the background to get rid of them. It's not quite what you want which I think is a popup balloon that let's you quickly see the name and number of a residue and then disappears when you point somewhere else. The popup and also laser pointer features are difficult because our code to figure out what you are pointing at is slow enough that it will cause some frame drop and glitches. So you can't easily run it all the time as you point here and there. But it can be run when you press a button.

6) Laser pointer would be great. There has been a ticket for that from 4 years ago #2846. I understand you'd be happy if it just works on the user interface panels. Figuring out what the laser hits on a panel would be fast unlike figuring out what atom is pointed at. That is needed so as you move the laser around it stops on the object it hits instead of passing right through the object.

[Tom Goddard]

Overall I would say that the user interface improvements are not as high a priority as creating VR videos and tutorials to get some users trying the existing features.