Medical Science Computing contract Ideas

Feb 13, 2019

Background

What developments should we propose for a March 2019 one year subcontract with Medical Science Computing? Meghan McCarthy and Darrell Hurt say it can be broader than medical imaging and VR. Meghan mentioned next gen NIH 3D print exchange including VR/AR with those models and in-browser visualization (specifically the WebXR preliminary standard) and tutorials. She also mentioned cytoscape 3d and VR viz. Darrell said though that the focus should be VR. There was some discussion of the NIAID interest in vacine development and molecular visualization and analysis.

General Considerations

Should we propose new areas like web VR or augmented reality or machine learning?

VR has uses for demonstrations, collaboration, and analysis. Where should we focus?

ChimeraX has VR capabilities for molecular structure, electron microsocopy, optical microscopy, medical imaging. Which application area should we focus on?

Relation to our other VR projects

What do I want to do with VR in the next year? Would like to move it beyond demonstrations, advance its use for collaboration an analysis in other labs. The main ingredient needed is collaborators with specific problems. Maybe the most promising chance for that is in cryoEM atomic model building with Yifan and Adam. I need to get Dyche and Max Krummel setup up for optical microscopy VR. Also I have more to do on binding site VR for Koda-Kimble award. MSC does not have end-users so they are not the collaborators. Proposed projects need specific collaborators who will be using the tools.

What does NIAID want?

A key missing part of the picture is what exactly the MSC contract from NIAID is funding. A subcontract would of course need to be aligned with the goals of the primary contract. If we don't know those details then we may propose things that are not in the scope of what can be funded.

Potential application areas

1) Medical imaging and optical microscopy visualization

1a) Segmentation using VR

Use ​Simple ITK in ChimeraX. Darrell Hurt says NIAID has hired SimpleITK team.

Slicer ​grow from seeds segmentation, cardiac CT

Slicer ​scissor projection tool, masking to sphere (volume eraser) for hip ball joint

Machine learning segmentation methods for medical images and ​cell microscopy. Need to investigate medical imaging methods that could be used

Segger watershed segmentation, especially hand mode, an example illustrating method is ​termite gut.

Tools to adapt to VR:

• Volume eraser
• Hide dust
• Show only largest connected blobs
• Loop tracing
• Masking to surface to create binary segmentation

1b) Medical imaging and optical microscopy visualization

• ​Photorealistic lighting from Allen Cell Science.

• Presets for coloring ​different tissues
  • Horos

• Smart initial coloring (bones, different organs or tissue types)
• Machine learning to ​color organelles in cells. ​Video.

2) Molecular Modeling and Graphics in VR

2a) Building molecular models in cryoEM using VR

• ISOLDE in VR
• Rotamers
• Build structure
• Clashes
• OpenMM and MD tools
  • parameterization
  • charges
• Interactive local energy minimization

2b) Molecular Analysis in VR

• Presets
• Full menus
• Render/select by attribute (+ define attribute)
  • Residue conservation
  • Worms; ribbon color blending
• Selection improvements
  • Multi-level selection-only undo/redo (left/right arrow keys)
  • More levels to selection hierarchy (including glycosylation level)
• Axes/planes/centroids
• ESP surface coloring
  • PQR charge info
  • Add Charge equivalent
• Tool GUIs (more usable in VR)
  • AddH
  • MatchMaker
  • Clashes/Contacts
  • FindHBond
• Chimera 1 session migration
• Thermal ellipsoids
• Rotamers
• Build Structure

3) NIH 3D

3a) Web VR visualization

• Primarily for communication and education
• AltPDB. ChimeraX exports glTF scene used with AltSpace to provide Web VR collaborative molecular PDB display (ribbons, surfaces, atoms).
• ​Peptide solubility Web VR app. Solid phase protein synthesis, Steve Kent, Steven Abbott

• Could ChimeraX offer web browser connection for multiperson VR viewing-only mode? Or just export to an AltSpace or other social VR site with one button click?

3b) 3D printing needs

• Color printing format requirements
• Reduced level of detail controls
• Direct export from ChimeraX

4) VR Development

4a) User Interface Development

Task specific user interfaces

• Drug binding analysis user interface
• Medical imaging
• Optical microscopy
• cryoEM modelling

Speech recognition (controlled vocabulary and grammar)

4b) Training

Mixed reality video capture shows person and data.

• For tutorials and explaining research results to public.

How-To screen capture videos

Multiperson VR sessions for remote training, possibly group training.

Written tutorials and user documentation

Help desk

4c) Multiperson VR

Remote and same room collaborative VR discussions.

Allow any participant to bring in new data.