Pick nearest atom in VR

[tic20@…]

From: Tristan Croll
Subject: A suggestion for VR interaction
Date: December 12, 2019 at 2:50:12 AM PST
To: Tom Goddard

Hi Tom,

Just thinking about ways to make VR tugging as intuitive as possible... the number one complaint I had with my old haptic-based interactive MD was that it could be hard to tell exactly which atom was going to be grabbed. While I'm no longer supporting haptic devices in ISOLDE (for the time being at least - may revisit later, since they're still fun to play with), I took the following approach in year 1 of its development. The atom to be tugged was based on proximity to the tool tip rather than intersection of a ray with the drawing - I find this is much more robust when the atoms are jiggling around. I used the following code (which could probably be optimised given the simplicity of the task):

def pick_closest_to_point(session, xyz, atoms, cutoff, displayed_only = True, hydrogens = False):

Pick the atom closest to an (x,y,z) point in scene coordinates. Optionally the
selection can be limited to include only displayed atoms and/or
exclude hydrogens.

closest = None
import numpy
if displayed_only:

atoms = atoms.filter(atoms.displays)

if not hydrogens:

atoms = atoms.filter(atoms.element_names != 'H')

atomic_coords = atoms.scene_coords

from chimerax.core.geometry import find_closest_points
ignore1, ignore2, nearest = find_closest_points([xyz], atomic_coords, cutoff)
if nearest is not None:

if len(nearest):

return atoms[nearest[0]]

return

One could also perhaps consider looking for atoms within a cone projected from the tip - but that of course would add a bit of complexity to the code.

Additionally, it was *really* helpful to draw an indicator showing exactly which atom would be affected when you pressed the button. At the time I was just changing the radius and colour of the atom itself, which of course made things horribly slow (I was still coming to terms with the code-base and its conventions), but a simple coloured sphere drawn over the atom works great and would have minimal cost.

Best regards,

Tristan

[Tom Goddard]

I agree it is sometimes useful to pick the closest atom instead of along a laser pointer line in VR and it is definitely great to have visual feedback on what will be picked before you press a button.

There are a bunch of issues that make this hard to do well. First is it kills rendering speed if every frame you need to find which of 50,000 atoms, or which of 5 million triangles of a surface is near the pointer. There are various techniques to make this fast. Atoms and triangles can be put in fast spatial look-up trees (e.g. oct-trees). Another approach has the GPU remember the object id for every pixel rendered. These require a good bit of infrastructure. Then there are the adverse performance implications of highlighting the chosen atom. Moving a marker atom over it could work although you might want transparency which introduces an entire extra rendering pass if other models don't have transparency, also have to be careful to avoid recomputing scene bounds and casting shadows.

Besides performance issues there are user interface issues. If some VR modes use a laser pointer and some use proximity it is hard for a user to know how to use the controllers. Ideally the solution is a laser pointer mode shows a laser beam that hits and ends at the first object it touches. That has the same performance issues of figuring out what is hit and adding some rendering.

Quality 3D interaction software definitely does this dynamic highlighting to help pick objects. But my approach to dealing with our limited programming effort spread across far too many features is to implement ones that consistently work well -- with high performance. If that cannot be done with available development effort, then I don't want to add half-working features that help usability sometimes and impair usability in other cases. Once we add 10 features like that the usability of the program becomes very poor as the user is frequently hitting cases that the code can't handle well.

Still this picking capability is essential so it may merit high effort. ISOLDE definitely is a special case of trying to pick a moving object. So ISOLDE may want to experiment with special solutions.

I think it is worth considering that users in general many not be comfortable manipulating atomic models with continuous frenetic simulation motion. I think this may be very taxing and stressful -- I think I'm pretty good handling it but it feels uncomfortable. Our visual system is not really adapted to monitoring constant frenetic motion. So I feel the ISOLDE simulations might better work in a different way where a simulation runs only for a few, maybe 5 seconds and shows a heavily smoothed motion from start to end, and almost all user interaction is done when simulations are not running.

[Tristan Croll]

You could get most of the effect you want by reducing the simulation temperature to zero - that way, atoms will only move once you do something to pull them out of their local minimum. Over-doing the smoothing makes it look awful - moves like molasses, lags way behind your actions, and introduces *really* weird geometry artefacts during fast movements. For the zero-K case I’ve been thinking of adding some code to keep an eye on the simulation energy and quietly pause when it converges within a tolerance to avoid wasting GPU cycles. The default temperature is there because it’s good for ISOLDE’s main use case: models where the geometry is still quite a long way from ideal and/or poorly fitted to the map. There the thermal jiggle tends to actively help the model settle past local minima. Once the model is mostly well fitted I generally advise people to drop the temperature right down.

Regarding the cost of highlighting the atom: one simple approach that works for the tugging modes is to limit the atoms under consideration to those that are currently mobile. That’s typically a few thousand or less, regardless of the size of the overall model. Obviously not a general solution for other applications, though.
 

 

[Tom Goddard]

I agree the highlighting could perform adequately like in ISOLDE with only a few thousand atoms pickable.

Let me explain a little better the idea that dynamics should be limited.  I agree smoothing when tugging could be bad, maybe while tugging you behave as it does now with little smoothing.  But if I want to say flip a peptide backbone or impose restraints to get a new rotamer, it will be much less chaotic if it just glides smoothly to the end result over a couple seconds.  Alternatively it could jiggle for a few seconds then stop.  I know you don't personally have a problem watching chaotic moving molecules for hours a day, but I suspect this is a subtle but critical factor that will limit use of ISOLDE, that other's can't tolerate that.  This is akin to VR where most people seem to have no trouble with it, but after an hour it is very fatiguing for no obvious reason.  These opinions about interactive MD are based on very little experience so don't take them too seriously.  I only raise them here because the constant jiggling motion is also a key reason why picking does not work well.