NIAID SOW draft Dec, 2019

Group meeting Dec 19, 2019

R01 ChimeraX proposal ~2 FTE

1. NIH 3D pipeline using ChimeraX (Eric) ~.8
   • NIH 3D Print Exchange pipeline using Chimera will be migrated to new NIH 3D pipeline for both printing and VR. New pipeline will use ChimeraX. Current scripts are those named Chimera*py in their github site.
   • Current Chimera pipeline uses these features not in ChimeraX (from Elaine); unclear if any are “must haves”:
     • PubChem fetch
     • VRML export - Meghan’s 2016 email said of X3D and VRML “we only need X3D”
     • Import of fchk, gro, mol, sdf files
     • PDB biounit fetch
     • Combine structures, maybe not needed if only for molecular surfaces
     • Coulombic coloring, requires charge assignment
   • Export of other formats (possibly Collada, GLTF (ascii), FBX, OBJ with texture colors), X3D enhancements.
   • Sequence conservation coloring
   • Fetch sequence annotations (UniProt, domains, disease-associated mutations)
   • Read and visualize segmentation models.
2. Human Biomolecular Atlas Program (HuBMAP) multiscale visualization (Tom) ~.1
   • Models from medical imaging, 3d light microscopy, 3d electron microscopy
   • Connection with Nils Gehlenborg at Harvard, HIVE
   • Tom G sent email to Phil Cruz to ask for more details about BCBB’s role in HuBMAP
   • Oct 2019 Nature overview of HuBMAP overview article:
     • “focus of HuBMAP [is] on spatial molecular mapping”
     • “We anticipate that the first round of data will be released in the summer of 2020”
     • “HuBMAP, in collaboration with other NIH programs, plans to hold a joint meeting with the Human Cell Atlas initiative to identify and work on areas of harmonization and collaboration during the spring of 2020.”
     • “To ensure that browsers and visualization tools from HuBMAP are valuable, the consortium will work closely with anatomists, pathologists, and visualization and user experience experts, including those with expertise in virtual or augmented reality.”
     • “Ultimately, we hope to catalyse novel views on the organization of tissues, regarding not only which types of cells are neighbouring one another, but also the gene and protein expression patterns that define these cells, their phenotypes, and functional interactions. In addition to encouraging the establishment of intra- and extra-consortium collaborations that align with HuBMAP’s overall mission, we envision an easily accessible, publicly available user interface through which data can be used to visualize molecular landscapes at the single-cell level, pathways and networks for molecules of interest, and spatial and temporal changes across a given cell type of interest. Researchers will also be able to browse, search, download, and analyse the data in standard formats with rich metadata that, over time, will enable users to query and analyse datasets across similar programs.”
3. Segmentation capabilities for medical imaging, 3d light microscopy, 3d electron microscopy (Tom) ~.25
   • Interactive SimpleITK use in ChimeraX
   • Allow loading, visualizing, creating, measuring and saving segmentations.
   • Support new EMDB-SFF segmentation file format from the EM Databank
4. Medical imaging (Tom) ~.25 - ~.5
   • Metadata browser for DICOM files
   • Support radiologist collaborator needs
   • Measuring changes in time progression, alignment, volume measurement, difference coloring
   • Enhanced lighting for improved perception of details.
5. Virtual reality (Conrad) ~.25 - ~.50
   • Improve multi-person VR beyond work accomplished in SOW #2
     • Any participant can bring in new model
     • Audio chat, third-party or built-in solution
     • Localizable connection server solution
   • Recording VR sessions (for VR playback? Or conventional video playback?)
6. Drug docking (Conrad)
   • VR UI
   • Turn on and off surface display
   • Show hydrogen bonds - for induced-fit docking results? (already have this and clashes for single-receptor multi-ligand situation)
   • Local minimization (use OpenMM), needs ligand parameterization, DockPrep/Antechamber
7. Documentation and training materials for above new capabilities. (Elaine) ~.4
   • NIH 3D pipeline
     • Provide ongoing advice on ChimeraX commands due to syntax and parameter differences (lighting, cartoon style, etc.)
   • Segmentation capabilities
   • Medical imaging
   • Virtual reality
8. Administration ~.1

NIAID SOW Ideas – Dec 2019

Initial discussion on next SOW goals, Dec 5, 2019 group meeting. Followed up Dec 12, 2019 with video conference with Phil Cruz, Meghan McCarthy and Darrell Hurt.

• MSC/NIAID suggestions
  • Migration of NIH print-exchange to NIH 3D (use ChimeraX)
  • Updates to ChimeraX to support pipeline (Elaine’s missing pieces)
  • See below (scripts are all on github)
  • Extended to VR/AR as well as 3D printing
    • Need to be able to handle differences
  • Allow uploads of ChimeraX session files?
  • GLTF text form support
  • Learning about python interface to ChimeraX[c]
  • Additional file types (Collada, etc.)
• previous SOW Google doc with crossouts
• previous SOW discussion on ChimeraX wiki
• feature list in DICOM viewer notes (some are done or partially done, some are linked to tickets)
• many possibilities relating to segmentation
  • see especially the ChimeraX wiki link above
  • investigate use of machine learning to identify and annotate features
  • They are extending the itk toolkit to microscopy images
  • Focus has been on medical imaging
• TomG: try to steer more toward data types with which we are more familiar, e.g. light microscopy and EM
  • Particularly as an overlap with simple itk uses for segmentation[d]
• better support for multi-person VR sessions, hard to make reliable
  • firewall issues
  • VPN
  • AWS - rendezvous service? Conrad will investigate.
  • if our own service, we don't want everybody on it
  • investigate audio services (possibly integrated)
  • enable any participant to load new data without deleting others' data
  • material for multi-person VR sessions for remote training
• advancing VR/ViewDockX capabilities
  • Dock Prep
    • Antechamber?
  • anything else?
  • A lot of general interest. Great VR use case.
  • Turn on and off surface display
  • Hydrogen bonding
  • Local minimization (use OpenMM)
    • How do we get force field for ligands?
  • Kudos for the tape measure tool
• 3D printing: enable porting workflow from Chimera to ChimeraX
  • any tie into segmentation?
  • are they still interested in pursuing this?[e]
• Tom F: avoid overlap with NIGMS grant
• Sequence analysis/MAV?
  • coloring by conservation (conserved regions may be better vaccine targets, e.g. universal flu vaccine as in the PBS VR segment from June 2019)
  • fetch annotations or info (UniProt, domains, disease-associated mutations, ...)
• hierarchy & smart level of detail in large and complex data and segmentations
  • segmentation browser[f]
  • Support transition easily between different levels of detail
  • Google-earth style for the models
  • Look at potential synergy with HubMap[g]
• DICOM enhancements
  • DICOM browser showing data hierarchy, metadata[h]
    • Show 5 most important things, but support showing more metadata at the user request
  • better/more presets for different tissue types in DICOM data
  • provide “smart” initial coloring (bones, organs, tissue types)[i]
  • provide photorealistic lighting to visualize 3D images and volumes
• Wendell Lim's engineered cells - infectious disease tie-in? Opportunity to get NIAID more interested in light (optical) microscopy. Max Krummel.
• recording VR sessions (augmented reality, etc.)
• HTML documentation for any of these new features, of course
• Educational uses of ChimeraX and VR (e.g. undergraduate education -- multi-user especially)[j]
  • Would an animation tool fit here?
• Other types of measurement (coupled with segmentation)
  • Volume measurement
  • Comparison capabilities (e.g. progression over time)
• Other sources of collaborators for medical imaging VR
  • Dmitri, other VA collaborators, Viv in addition to Beth

Other crossouts from previous SOW:

• select and/or highlight voxels in volumetric data representations. Save the selection, and apply visualization commands to just the selection.
• provide “smart” initial coloring (bones, organs, tissue types)
• provide photorealistic lighting to visualize 3D images and volumes
• additional material may be presented in video format, e.g., “How-to” screen capture videos, mixed reality video capture showing person and data for tutorials and for explaining research results to the public