Make VR picking use proximity to cone tip

[m-schmid@…]

Currently VR hand controller modes pick objects by looking at the intersection with a ray along the axis of the cone. This can accidentally pick a distant atom when the ray misses a near atom. Picking based on distance from the cone tip would probably work better.

From: "Schmid, Michael F."
Subject: pointer in VR
Date: February 2, 2023 at 3:39:30 PM PST
To: Tom Goddard

Hi-
I am using one wand to label residues of interest in VR. A frustrating behavior is that the label pointer is highly directional; if you miss an atom/residue, it may hit and label a residue way behind (a wild shot), or if you don’t hit anything, it erases all the labels you’ve made up to then. The behavior with respect to the menu seems more reasonable; it operates by proximity to the wand tip (roughly the diameter of the cone base, I estimate), not which way it’s pointed. It would be good to have labeling, at least, work this way. 
Everybody I show the VR to is blown away.
Thanks,
Mike

[Tom Goddard]

What do you mean by "The behavior with respect to the menu seems more reasonable; it operates by proximity to the wand tip". I don't think there is any ChimeraX VR capability that uses distance to the wand tip. Instead all the VR wand modes that select an atom look at the ray extending along the center line of the cone, and that can hit something pretty far away. I'd like to make it use proximity to the wand tip but do not have code that can do that. Using the mouse to pick atoms works similarly to the VR in that it looks for an atom on the 3D ray extending into the screen at the mouse position. So there is no mouse picking code that uses proximity in 2D.

This problem of difficult VR picking has been long on my list of things to improve.

[Tristan Croll]

Waaaay back at the start of the ISOLDE project (while I was first playing around with haptic devices) I tried an implementation that would change the appearance of the atom that *would* be grabbed by the device if you were to click at that moment. In terms of making things more intuitive it was great, but performance-wise it was horrible for any large model (although admittedly a big part of that was because I was stupidly changing the appearance of the *actual* atom - triggering a redraw of the whole model - rather than drawing a simple sphere over the top of it). If it could be made fast enough it seems that would be really helpful in VR... given that handsets have multiple buttons available, perhaps one could be assigned to turn on such a preview mode while it's held down? Thinking in terms of the Quest 2 controller, you could have something like "squeeze the trigger on the barrel to preview which atom will be affected, and pull the front trigger to actually do it"?

[Tom Goddard]

Yep, it would be very nice to have visual feedback showing which atom your cone is pointing at. As you say the trick is to figure out the atoms and change its appearance without making the VR stutter -- both of those problems are tricky. This is the subject of ticket #2846 from 3 years ago which also talks about having a laser-pointer to see what you would click on. This month in a funding request for VR to NIAID I specifically added implementing support for visual feedback for picking atoms. We are still adjusting that NIAID contract that we hope to get funded for 1 year starting in March. It is not just the rendering a visual cue that is slow, figuring out which atom is too slow and will have to be done in a different thread. That is not so easy since the structure could be closed while that thread is running. Then the rendering is hard. It is not practical to render something on top of what is already rendered. This would require a whole new mechanism for applying overlays to the graphics and new shaders to composite the overlays with the molecular scene. I think the only practical solution is to add an element to the scene and have the whole scene redrawn. That will not be slow unless it causes a major update to say all the atom buffers. But if it is just one new graphics object added like a laser pointer it should add a negligible amount of render time. The scene is being rerendered 90 times per second in both eyes in VR even if nothing in the scene changes because your head and hands are constantly moving slightly.

[Tom Goddard]

This ticket for picking based on proximity in VR is the same as was requested by Tristan 3 years ago, ticket #2681.

[Tom Goddard]

From: "Schmid, Michael F."
Subject: Re: pointer in VR
Date: February 2, 2023 at 11:17:45 PM PST
To: Tom Goddard 

Hi-
Sorry I was confusing you. When you turn on the menu using the top  button above the thumbwheel, you are presented with a panel equivalent to what is on  the top and right side of the window on the monitor view. If you come near the menu with  the cone tip (no matter which way it is pointing, a menu item (eg. the “label” button, or the eye icon that turns the map display on or off) pops out and can be selected. That’s an example of 3D sensing of proximity.
Thanks for working on the Quest 2 application.
Mike

[Tom Goddard]

Begin forwarded message:

From: Tom Goddard 
Subject: Re: pointer in VR
Date: February 3, 2023 at 11:42:42 AM PST
To: "Schmid, Michael F." 

Hi Mike,

Right!  I forgot the clicks on the VR user interface panels are based on proximity and do not care what direction the hand controller is pointed in.

There is an option to the "vr" command to set how far away from the cone tip to pick objects

	vr on clickRange 5

It is specified in physical data units (usually Angstroms for atomic models).  The default is 5 Angstroms.  But like you I have seen in pick atoms far away.  I just tested the clickRange and it seems like it is not working.  So I will debug it today.  The click range still requires that the atom be on the ray emanating from the hand cone.  It is not just distance from the cone tip.  But if it worked as intended I think it would alleviate many of the mis-picking problems.  I'll send you a message if I manage to fix it today.

	Tom

[Tom Goddard]

From: "Schmid, Michael F." <m-schmid@slac.stanford.edu>
Subject: Re: pointer in VR
Date: February 3, 2023 at 1:43:18 PM PST
To: Tom Goddard <goddard@sonic.net>

Thanks! One more thing; the current behavior is, if you miss all atoms, it turns off all the labels hat you already marked, so, bummer! I know you have to be able to get rid of labels somehow, but that was not optimal.
Mike

[Tom Goddard]

From: Tom Goddard 
Subject: Re: pointer in VR
Date: February 3, 2023 at 1:57:39 PM PST
To: "Schmid, Michael F." 

Yeah, unfortunately the most frequent case for labeling residues in VR is to just check some a few labels and then hide them, so the aim was to make it easy to hide them.  I understand that doesn't work for your case.  That's a trouble with VR modes, they don't have lots of options.  I've wanted to make a command that would allow you to change the default behaviors of mouse and VR modes in small ways, but it would get so little use it is hard to get that done with all the other features that people request.

	Tom

[Tristan Croll]

I'm thinking a variation on what I was suggesting could be an easy stepping stone, while helping to avoid problems with accidentally labelling the wrong thing or wiping out all labels. Thinking in terms of the Quest 2 controller layout:

• squeezing the barrel trigger would choose an atom using the existing mechanism, and add some form of temporary visual indicator. This could be as simple as "sel {atom specifier}" to use the existing ChimeraX selection highlighting, or some independent mechanism.

• as long as the barrel trigger is kept squeezed, the next action(s) performed using other buttons or the front trigger will act on that atom, regardless of the position of the controller. If you wanted to get fancy, you could include a dashed line connecting the controller back to the atom to emphasize that.

• if the barrel trigger is *not* squeezed, use the existing mechanism.