SOW Deliverables

End of contract: February 29, 2020

1. Virtual Reality user interface development (Tom 348 hours, Elaine 87 hours, Eric 696 hours, Conrad 87 hours)
   1. Provide task specific user interfaces (i.e. GUI panels and/or appropriate icons) for the following areas:
      1. medical imaging
      2. cryoEM modeling -- rotamers, structure building and verification, minimization
   2. Measure and display distance between two specified points in space in VR, both molecular models and medical images.
   3. Investigate and optionally prototype control of VR sessions by speech recognition, using a controlled vocabulary and grammar. This will support better accessibility for disabled users, e.g., users with limb differences who would have trouble with some aspects of the VR hand controllers.
2. Extend medical imaging data visualization and analysis (Tom 348 hours, Elaine 87 hours)
   1. Improve native .dcm support, e.g., compatibility for segmented dicom files.
   2. Provide presets for different tissue types in CT scans.
3. Support for multiuser sessions in a virtual reality environment (Tom 87 hours, Eric 174 hours, Conrad 435 hours)
   1. The application will allow multiple ChimeraX users to enter a shared session to interact with a 3D representation of the data in virtual reality
   2. An ideal solution will allow users from both ends (remote, or in the same physical room) to start meetings on demand and to connect without opening firewalls.
4. Outreach
   1. Workshops, Training and presentations (Tom 164 hours, Elaine 87 hours)
      1. If available, the subcontractor shall attend a meeting at NIAID’s Rocky Mountain Labs location in Hamilton, Montana. This is tentatively scheduled for August 5-9, 2019, as part of a NIAID/OCICB site visit to showcase virtual reality technology for scientific data visualization to support NIAID research.
      2. The subcontractor may propose one or more additional conference events relevant to the work, at which they will showcase the application in an oral presentation and/or hands-on conference exhibit or training. Justification shall be provided for how this will benefit NIAID.
      3. A virtual help desk for users
   2. Instructional Material and tools documentation (Elaine 609 hours)
      1. Detailed instructions for all features shall be provided in a user manual.
      2. Written user guides and tutorials shall be available as an HTML page and/or a downloadable PDF.
5. Project planning, administration, and reporting (Tom 87 hours, Scooter 174 hours)
   1. A project schedule, implementation plan, and documentation plan will be provided.
   2. Reports will be provided for all subtasks described under each task.

Progress October 31, 2019

1. Virtual Reality user interface development (Tom 348 hours, Elaine 87 hours, Eric 696 hours, Conrad 87 hours)
   1. Provide task specific user interfaces (i.e. GUI panels and/or appropriate icons) for the following areas:
      1. medical imaging
      2. cryoEM modeling -- rotamers, structure building and verification, minimization
         • 'swapaa' command implemented.​documentation Largely a port of Chimera code, with the principal differences being:
           1. New code to handle the fact that Chimera treats alternate locations as separate atoms whereas ChimeraX treats them as a single atom.
           2. Refactored rotamer library handling so that the libraries are provided by separate bundles and therefore new rotamer libraries could be supplied by third parties via the Toolshed.
         • Rotamers tool largely implemented.​documentation It is also mostly a port of Chimera code, but required porting major Chimera infrastructure, e.g.:
           1. "Smart" table where the rows are directly based on ChimeraX data items (in this case rotamers) and the table knows how to show columns based on data item information and makes callbacks that provide the data items as arguments.
           2. Configurable and embeddable H-bonds GUI so that the Rotamers tool can add an H-bonds column to the rotamers table. As a "side effect" an H-bonds tool was implemented.​documentation
           3. Similar to the previous point, a configurable and embeddable clashes/contacts GUI needs to be ported/implemented. This will be the last major piece of the Rotamers tool. When completed, both embeddable GUIs will also be incorporated into the ViewDockX tool.
         • ISOLDE refining atomic models in cryoEM maps, initial test for feasibility for using with VR.
         • Segger EM segmentation tool ported to ChimeraX. GUI can be used in VR.
         • Created EM map fitting GUI - can be used in VR.
         • Volume Eraser GUI and VR mode added.
         • GUI color and size controls for markers placed on maps .
         • Initial tests of new EMDB segmentation file format. Handles surface and index map type segmentations.
         • Now use view-aligned image rendering to improve appearance of 3d image stacks viewed edge on.
      3. General VR user interface improvements. These make all desktop GUI interfaces easier to use in VR.
         • Menus and popup windows now supported in VR.
         • Every tool is its own movable, resizable and closable panel.
         • A panel of recent file thumbnails is now available in VR, accessed via toolbar.
         • Save VR hand controller button assignments and model room positions in ChimeraX sessions. Also can assign buttons in command scripts.
         • Improved zooming to avoid models receding in distance.
         • Raised buttons on hand controller cones are now customized for Vive and Oculus controllers.
         • Oculus thumbstick support added used by modes for zoom, rotate, map series.
         • Added VR interface for analyzing docking results (ViewDockX). Support for more Maestro format files added.
   2. Measure and display distance between two specified points in space in VR, both molecular models and medical images.
      • ​Tape measure VR and mouse mode added. For maps only currently.
   3. Investigate and optionally prototype control of VR sessions by speech recognition, using a controlled vocabulary and grammar. This will support better accessibility for disabled users, e.g., users with limb differences who would have trouble with some aspects of the VR hand controllers.
      • Prototype implementation for invoking menu entries using Python speech_recognition module implemented. Results were mixed: in quiet environment with a good microphone, most voice commands were properly interpreted; in an office environment with moderate noise, only about half the commands were understood.
2. Extend medical imaging data visualization and analysis (Tom 348 hours, Elaine 87 hours)
   1. Improve native .dcm support, e.g., compatibility for segmented dicom files.
      • Improved DICOM read speed to be many times faster.
   2. Provide presets for different tissue types in CT scans.
3. Support for multiuser sessions in a virtual reality environment (Tom 87 hours, Eric 174 hours, Conrad 435 hours)
   1. The application will allow multiple ChimeraX users to enter a shared session to interact with a 3D representation of the data in virtual reality (scene sharing)
      • New multi-panel VR user interface is visible to all participants in sessions.
      • Acquired and tested newly released VR headset Oculus Rift S in multiperson VR sessions.
   2. An ideal solution will allow users from both ends (remote, or in the same physical room) to start meetings on demand and to connect without opening firewalls.
4. Outreach
   1. Workshops, Training and presentations (Tom 164 hours, Elaine 87 hours)
      1. If available, the subcontractor shall attend a meeting at NIAID’s Rocky Mountain Labs location in Hamilton, Montana. This is tentatively scheduled for August 5-9, 2019, as part of a NIAID/OCICB site visit to showcase virtual reality technology for scientific data visualization to support NIAID research.
         • Tom G attended workshop, presented two ChimeraX cryoEM training tutorials, many VR demos, followed up implementing new ChimeraX VR features requested by Rocky Mt people (menu support, Oculus thumbstick controls, function key support)
      2. The subcontractor may propose one or more additional conference events relevant to the work, at which they will showcase the application in an oral presentation and/or hands-on conference exhibit or training. Justification shall be provided for how this will benefit NIAID.
         • Tom G gave cryoEM ChimeraX tutorial at ​Stanford cryoEM workshop, organized by Wah Chiu, head of National Center for Macromolecular Imaging, gave VR demos looking at cryoEM and atomic models ATP synthase.
         • Tom Ferrin gave talk on molecular VR at the ACS annual meeting including video of multiperson VR use.
         • ​UCSF Sharecase booth showing ChimeraX VR of opioids bound to the mu-opioid receptor.
      3. A virtual help desk for users
         • ​chimerax-users archive
         • Email discussions assisting multiple users in purchasing suitable VR headsets and computers for use with ChimeraX.
   2. Instructional Material and tools documentation (Elaine 609 hours)
      1. Detailed instructions for all features shall be provided in a user manual.
         • ​ChimeraX User Guide at the RBVI website note to Scooter because viewdockx was used as the test for the future documentation reorganization, the links to ViewDockX tool and viewdockx command help are broken in this website copy. The downloaded User Guide shown from ChimeraX help menu does not have this issue.
      2. Written user guides and tutorials shall be available as an HTML page and/or a downloadable PDF.
         • All new features described above for cryoEM documented in User's Guide.
         • New cryoEM ​ChimeraX tutorials made for Rocky Mountain Lab workshop.]
           1. Alpha-synuclein fibrils
           2. EM tomography of proteasomes around aggregated fibrils in neurons.
      3. Video tutorial support
         • Added VR camera fixed in room for recording tutorials.
         • Added initial Intel RealSense depth sensing camera integration with ChimeraX VR to record mixed reality tutorials.
5. Project planning, administration, and reporting (Tom 87 hours, Scooter 174 hours)
   1. A project schedule, implementation plan, and documentation plan will be provided.
      • Video conference with Phil Cruz, Aug 7 2019 reporting progress and planning Rocky Mountain Lab workshop.
   2. Reports will be provided for all subtasks described under each task.
6. Other related developments
   • Exploratorium VR cell therapies demonstration. ​Exploratorium After Dark Party, October 24, 6-10 pm. ChimeraX VR will be used to show programmed cells form 3D structures. Data from Wendell Lim's lab; the cells are genetically engineered to self-organize by signaling and adhesion.
   • ​Makers Lab VR demonstration. ChimeraX used for a virtual reality look at molecules, cells, and tissues in the UCSF Library Makers Lab, Parnassus Campus, noon-1pm November 15.2019.
   • UCSF news article highlighting ChimeraX VR and drug resistance. ​VR Drug Discovery Makes Precision Therapy a Reality and ​video from article on YouTube
   • Helped prepare VR T-cell cancer therapy 3d microscopy demonstration for Max Krummel lab.