NTT DATA UCSF NIAID SEB Subcontract 2021 Task Order #2

March 1, 2021 - February 28, 2022 from ​Google Doc, continuation of NETE contract (Aug 2020 - Feb 2021).

1. General ChimeraX improvements to support NIAID (Eric: 20%)

2. Support the NIH 3D pipeline development, including any changes to ChimeraX to support ongoing development. (Eric: 15%)

3. Improve multi-person VR (Tom: 35%)
   1. Create video tutorials in how to use multi-person VR
   2. Hold VR office hours to train researchers to use VR capabilities
   3. Allow participants to add models from local files during a VR session
   4. Set up VR meeting servers for Africa, Europe, Asia

4. Human Biomolecular Atlas Program (HuBMAP) multiscale visualization (Eric: 30%)
   1. Develop ChimeraX tool to browse and select HuBMAP datasets for loading

5. Enhance ChimeraX’s segmentation capabilities
   1. Investigate increased use of simple ITK to support segmentation (Tom: 15%)
   2. Investigate utilizing web services to support more complex machine learning-based segmentation algorithms (Tom: 15%)
   3. Investigate reading and displaying segmentation format(s) available from imaging databases (tissue microscopy, cancer imaging data, etc.) (Tom: 10%)

6. DICOM (Tom: 25%, Eric: 5%)
   1. Improve ability to visualize and analyze medical images in VR with ChimeraX
   2. Develop browser-style interface to show DICOM dataset contents and associated metadata such as sample characteristics

7. Outreach
   1. Instructional material and tools documentation. (20%)
      1. Detailed instructions for all features shall be provided in a user manual
      2. Written user guides and tutorials shall be available as an HTML page

8. Administration (10%)
   1. Submit monthly written reports of accomplishments

Meeting Minutes

3/12/2021

Phil, Philip, Darrell (2nd half); Elaine, Eric, TomG

Problem reported last time with strut disconnection in X3D exports has been fixed by Greg. Phil will test it.

Phil will try viral capsids with both a single combined surface and separate surfaces. Each may be preferred in different situations (e.g. coloring looks better with separate surfaces but they may cause 3D printing problems).

Use "graphics triangles" to report the number of triangles for better idea of export file size.

3D pipeline developers are thinking about parallelization of postprocessing. Multiple outputs for a single user input structure, may need to organize them by purpose (e.g. printing) in the interface or ask the user about purpose and then present the appropriate outputs.

We have been adding more GUIs to ChimeraX: volume tools, color key, etc. but still many more to go: dock prep, etc.

Phil asks about copy/combine functionality, not yet in ChimeraX. In the case of where the same model is split and then put back together, can instead use ChimeraX mouse mode rotate/translate selected atoms only.

Phil and Meghan want to know more about the SPOKE project and its data sources. Elaine will send them some links.

Darrell relaying concerns from people at Rocky Mt Lab that our interactions with them and with this NIAID in-house group might conflict (?) We aren't sure what this means.

Darrell suggests TomG or bigger ChimeraX team may visit NIH Bethesda for in-person collaboration and training similar to previous Chimera workshop at NIH.

Phil wants to try Virtual Desktop wireless VR with Quest 2 to PC with ChimeraX.

Next Wed 6 AM Phil will be training new postdoc in Mali to use VR. TomG can join VR meeting if desired. Will use Africa RENU VR server.

Phil attended VR in Public Health conference this week. VR will be in doctor's offices soon and has billable codes for insurance.

3/24/2021

Phil, Philip, Darrell; Elaine, Eric, TomG, Greg

Any progress on NIH3D pipeline as pertains to ChimeraX?

• Philip has incorporated the chimeraX unified py file into the workflow, says ChimeraX download from UCSF is slow. Error codes 64 (faulty usage) and 70 (internal software error) plus "expect trouble" (using gui. To avoid this, need to use nogui mode via --offscreen startup flag)
• Script calls sys.exit. Greg says zero/nonzero should be all you see, must be something with your workflow, and nonzero = error.
• NIH3D pipeline will be using cloud computing, currently not using systems with graphics hardware.
• offscreen rendering is currently slower, uses CPU rather than GPU. Can OSMESA use multiple CPU threads? Greg and possibly others will look into it.
• ambient lighting is slower, script already does size-dependent lighting
• Scooter: next step could be performance testing. Darrell's seen very slow examples with Chimera. Philip: let's try with ChimeraX, could well be different. Scooter: should try those specific examples, Phil will try to find them in the logs.

Did you want to know anything else about SPOKE?

• Phil: it was related to meeting with NASA re drug discovery to treat radiation effects
• NASA folks are combining lots of data types related to radiation effects (e.g. from exposure in space, nuclear accidents, etc.), including CT

VR stuff

• Phil reports problematic ChimeraX VR meeting with Mali last week; may have been internet connectivity to name server.
• TomG encourages use of "report a bug" and looping him in to meetings, in case he could help.
• Darrell: latency to Uganda was measured at ~350 milliseconds.
• Phil: how best to specify meeting proxy server? TomG: use Access menu in the Options of the GUI, but you have to use a command (meeting access? meeting settings?) to add a new server to the menu.
• TomG and Willan made a 60 minute long two-person VR video.
• we added a new ChimeraX videos page, includes some VR
• Video tutorials can play in ChimeraX help browser, at least on Mac, tested by Elaine.

4/8/2021

Phil, Philip, Meghan; Elaine, Eric, TomG, Scooter

Meghan: we've talked with an AR/VR group (​imisi3d.com) in Nigeria. They have downloaded and tried ChimeraX, and will do a remote meeting with NIAID folks, who will show them medical image data.

Phil: I have a collaboration with an investigator here to examine how two protein structures might interact, VR really helps with maintaining multiple constraints simultaneously. Also used HADDOCK protein-protein docking. Now have a compelling model of how hemoglobin and eNOS might fit together, will help guide peptide competition experiments. Steven Brooks et al., manuscript is in medarchive, will be submitted to Circulation.

Discussion of vaccines for influenza and covid-19 made by fusing viral proteins or their fragments to ferritin.

They have showed ChimeraX VR to "new guy" Wilbert van P and deputy director Jill Harper.

Tom G: would like to make a video showing multiperson AR/VR on SARS-CoV2 spike mutant(s). Would be great if Fauci could do it! Meghan: or get Fauci to do voiceover. Have been discussing similar with Barney (?)

ChimeraX 1.2 release is slated for end of month.

Tom G: will be working with Rodgers in Uganda to solve Mali VR meeting connection issues. SLAC also has firewall connection issues.

4/22/2021

Meghan, Philip, Darrell; Eric, TomG, Greg, Scooter

Network security

• Discussion of network breaches effecting NIH and UCSF.

NIH 3D giving users time estimates

• Good for users to know if job will takes 1 minute or 30 minutes
• Philip says another project (nefali?) output progress log, but no time estimates
• Tom says useful to get percent complete estimates for longest running step

X3D ChimeraX including meta data

• Greg Couch asked about including per-residue meta-data in X3D
• Meghan says want meta data for data attribution in GLB

Barney's covid-19 Chimera session file

• Meghan trying to use Chimera on spike model, Chimera session from Phil Cruz (on vacation), to help Barney (virus researcher)
• Chimera molecular surfaces failed on Windows
• Did not find a way to get it into ChimeraX.

Web 3D viewers

• For fastest 3d model network transmission, DRACO algorithm/format useful.
• Kristen Browne is working on a few 3D web viewers and should give a demo.
• Babylon.js web viewer, can drag and drop GLTF files for testing.

5/6/2021

Phil, Philip, Meghan, Darrell; Scooter, Elaine, Eric, Greg

Use of PDB or other databases' metadata in pipelines

For example, Eric's script uses the suggested contour level from an EMDB file. Meghan: another example is to look in the header to distinguish a virion from other viral proteins (Phil: this is in a record in the header) and direct them to different pipeline branches. Greg: metadata in scene export formats is which graphical object corresponds to which atom(s), but other metadata is best obtained from original data file, e.g. mmCIF/PDF.

Scooter: so you want to use existing metadata to influence pipeline processing... what about writing metadata into the GLTF/GLB? Phil: yes, maybe something like the structural hierarchy, overall complex -> different proteins -> chain descriptions, and which graphical objects correspond to each, similar to what Greg was describing. Greg: we would need to know what tags are required for the intended application (Enduvo, Phil says it's like powerpoint for VR, used in Uganda for training content, includes 2D assets like slides and 3D assets, movies, sound, evaluation questions).

Kristin is developing the Enduvo content with NIAID. We would like to talk directly to Kristin and Enduvo, primarily Greg to discuss what tags would be needed. Meghan and Phil: we will discuss with them and try to organize a collective meeting. Darrell: not sure which of these tasks are worth ChimeraX development. Need to strive for only what we think is highest value to both NIAID and ChimeraX user community. ChimeraX team can weigh in on what seems to be within our reasonable scope of effort. Scooter: seems worthwhile for export from ChimeraX to be showable in 3D web viewers. Meghan's idea: maybe have ChimeraX button to put the scene on NIH 3D, which would allow people to view it via the web there. Could have some limited interactivity using hierarchy metadata, e.g. turning chains on and off. Scooter: sounds like a great idea! Meghan: tradeoff between amount of metadata/interactivity in web viewer vs. file size, at some point (e.g. individual atom level) one should just use ChimeraX directly instead. (Meghan: conversely, going the other direction, could also have links from entries in NIH 3D to download the corresponding ChimeraX session files.)

Meghan likes the viewer and annotation tool in ​https://www.biodigital.com

Scooter: useful poster on glTF: ​https://kcoley.github.io/gLTF

Next time May 20: Kristin will be at the meeting

Time after that, June 3: maybe discuss PDB50 issues with Tef

5/20/2021

Phil, Philip, Kristin; TomG, Elaine, Eric, Greg

Kristin is showing her glb file 3D viewer made using babylon.js; surfaces colored by ESP and hydrophobicity, ribbons, atomic display, shadows, ambient occlusion, realistic lighting based on dome image (?), momentum spin, changing background color; she had to reduce meshes by ~80%

They are interested in metadata not only just in overall header but also in individual elements within the file.

TomG: now we have gltf export of vertex coloring, might add texture coloring. Kristin: not sure if mesh reduction can preserve texture. Many of the textures we use are black-and-white indicating where on surface a particular effect appears.

TomG: ChimeraX can write out an OBJ with uv texture coordinates and a texture image. Phil: if you provide OBJ with texture we can try it. TomG: I made these for Unity3D, will send something.

Phil: Would be useful to make models and chains as separate nodes in a hierarchy. Kristin: also grouping. Some programs like ZBrush flatten the hierarchy, but most others keep it (e.g. Maya).

Phil: Enduvo meeting last week; they want direct access to our 3D assets. Enduvo is like PowerPoint for VR. TomG: maybe we should try making an Enduvo lesson ourselves with models exported from ChimeraX with textures or vertex colors as needed. TomG: Can I get Enduvo? Phil: Cost is too high > $1000. Can introduce you to Enduvo developers. Also Enduvo is getting help from Unity developers, since it is a Unity app.

Kristen's babylon.js viewers (she wrote the Javascript) are here: ​https://3d-viewer-sandbox.s3.amazonaws.com/index.html

Tasks for TomG:

• provide instructions to export a colored molecule (atoms + ribbons) in OBJ with texture colors.
• allow models and chains to each be a distinct GLTF node.
• put textures for electrostatic coloring and hydrophobic coloring into GLTF output.

6/3/2021

Phil, Kristin, Philip, Megan, Eric, Greg, Scooter

Scooter's notes:

GLTF Metadata Ideas

• Which chains are included
• Ligand identification
• Which triangles are associated with which residue?
• Hierarchy
• Struts vs. molecular representation
• Also hydrogen bonds
• Labels
• Biological Unit

Conceptually easy to do, but we need to have agreement.

Suggestion: just do a pass and see what everyone thinks (Greg?)

UnityMol - FAIR share and interoperability of molecular models

Darrell: Scale

Idea: take advantage of the ​UnityMol pymol session reader to write a tool to import pymol sessions into ChimeraX

• Investigate and do some specification work in 4 weeks (after grant)

Eric's notes:

Phil, Philip, Kristin, Eric, Greg, Scooter, Meghan, Darrell

Greg: metadata from files vs. metadata in models. What can we add to metadata to help?

Phil: number of polygons in GLB output

Greg: could be determined from file. ChimeraX doesn't know ahead of time, would have to generate GLB and count.

Phil: what can you provide that isn't in mmCIF header?

Greg: hierarchy; h-bonds, struts

Phil: metadata may be used in Enduvo import

Greg: each GLB "object" can have a descriptive JSON dictionary associated

Meghan: if structs were labeled, they could be removed before handing to Enduvo

Phil: H-bonds if labeled, could be turned on/off easily

Scooter: time for a prototype; maybe make tags an option

Greg: file size not a real issue if hierarchy only to chains

Phil: are there standards for GLB JSON dictionaries

Greg: ad hoc, though there may be a few well known keys

Phil: ChimeraX GLB group names do show up when in Enduvo.

Greg: Those internal names could be improved.

Meghan: FAIR (reusable) scene depiction in mmCIF (​https://journals.iucr.org/d/issues/2021/06/00/qr5004/qr5004.pdf)

Scooter: Don't want each program to make its own "standard". But design by committee is slow.

Darrell: with 3D printed models, people tend to ask "what's the scale"?

Greg: may be able to use UnityMol's PyMOL session reading to read such sessions into ChimeraX(!)

NIH: that would be amazing; users have PyMOL sessions of what they want their structure to look like

Scooter: investigate feasibility -- come back to the PyMOL idea in 4 weeks (R01 renewal)

Greg: perhaps a subset

6/17/2021

Phil, Kristin, Philip, Darrell, Yong-Sok Lee (sp?); Tom G, Elaine, Eric, Greg

Tom G mentions potential hiring plans and infrastructure funding opportunities.

YS Lee's background is in computational chemistry, ligand design, QM, reaction mechanisms, may be interested in interfacing ChimeraX to GAMESS. Elaine mentioned the SEQCROW plugin to ChimeraX, Eric shared links to plugin and paper.

Tom G: what is ETA for dock prep? Eric: Dock Prep pieces (addh, charge calculations for nonstandard residues, swapaa to fix sidechains, altloc management) are available in ChimeraX but a lot of coding is needed to assemble them fully and create UI.

Tom G: Deep SSE Tracer is a machine-learning plugin developed by Jing He. We're interested in developing and interfacing with such machine-learning-related methods. Kristen: some are better than others, certainly. TG: CryoDRGN identifies multiple conformations in EM data. TG: we are also looking at using dimensionality-reduction approaches. In VR can reduce to 3D. Elaine: could think also about animating through time, using colors/sizes as additional dimensions, etc.

Kristen: question about exporting glb of an assembly of 6htx, gives an error. "index 1 is out of bounds" TG: it's a bug. Possible workaround is to use "sym" with "copies true" (requires more memory), but we should also fix the bug. The glb file could use instancing, but we are not writing it that way currently. Greg: should write glb with instancing. TG: yes, maybe as an option, but make it the default.

Phil:

• I attended a meeting and saw a poster on molecular dynamics optimization while user moves the ligand in VR, group at Bristol: ​https://gitlab.com/intangiblerealities/narupa-applications/narupa-imd
• Some issues with NIH-3D script dealing with NMR ensembles outputs whole thing when we'd just rather use the first conformation. Eric: send me an e-mail. Phil: OK!
• Enduvo has some issues with their glb reader, so let's wait until those are sorted out before sending them files from ChimeraX.
• Re export: PDB doesn't have total molecular weight, only per each entity. We remove waters and other stuff but would like export to include total molecular weight of atoms that are present (not including hydrogens or missing segments). Elaine: why not atom count instead? Darrell: let's re-examine the logic and then take a look at calculating what we need. Darrell: important thing is really the scale. TG: like dimensions in angstroms? Darrell: degree of magnification. Elaine: inherently angstroms, but we don't know how large it will be when 3D-printed. Greg: Blender units are meters. Phil: perhaps useful to export lengths of bounding box in X,Y,Z. Greg: we may need to improve our calculation of the global bounding box. Darrell: also may be rotated relative to the molecule. Elaine: how about measure inertia? But problem is that it doesn't enclose everything like a convex hull. Tom G: perhaps we can provide a similar command that encloses all atoms and/or surfaces. Phil: maybe we could use the ellipsoid to reorient the model along screen X,Y,Z. Tom G: maybe we can add the ability to ChimeraX to dump a separate XML or JSON file with the metadata instead of stuffing it all into the GLTF.

Philip: Want to run ChimeraX without accumulating history in a home directory, and also without contacting RBVI Toolshed server every fresh start. Greg: we need to add startup option to avoid phoning home. Also can make a temporary directory and set HOME environment variable to that directory so fetched files, file history, ... goes to that directory. TomG: should do more things with a "stateless" option, avoid reading/writing user data and settings. Eric: what about caching fetch data in ~/Downloads in stateless mode? TomG: Yeah, that is a problem.

7/01/2021

Phil, Philip, Darrell; Scooter, Tom G, Elaine, Eric, Greg

Scooter: our new hire Zach is coming soon

TomG recaps gltf export improvements: options to use instancing, texture colors. Now thinking about saving animations as gltf. Morph makes a large file, but maybe a protein can be defined as rigid subunits for skeletonization and "character animation." Examples: spike RBD up/down, prefusion vs. postfusion

Scooter: these are handled by babylon.js ... see also ​thin instances

Greg: ChimeraX startup options --exit and --script (and maybe --cmd, TBD) skip contacting the Toolshed

Phil: still evaluating what we need for figuring out molecule size/dimensions in the 3D pipeline

Phil: has wireless VR with Quest 2 to a desktop been tried with ChimeraX? TG: No. Quest 2 with tether works fine. Also we use Vive Pro with wireless adaptor. Phil: Vive wireless is not supported in Uganda because computers don't have a slot for the wireless card.

Phil: performance on Apple Silicon Macs? TG: we have a mac mini but not made a native ChimeraX for that cpu, am using Intel emulation, performance seems fine although native version could be faster. PyQT library not yet available for Apple Silicon. Some python benchmark reported as 3x faster on Apple M1 vs Apple Intel. Philip: Can't trust ARM vs Intel benchmarks. TG: We could benchmark python uses but have not yet. ChimeraX port to Apple M1 is probably at least 6 months away due to need for all third-party libraries to be ported.

Some discussion of how to temporarily change $HOME on linux with the unix "env" command. Philip: got around it by containerizing. Scooter: maybe should have $CHIMERAX_HOME. Greg: have ChimeraX create a temporary home (that gets removed).

7/15/2021

Phil, Philip, Meghan, Kristin; Elaine, Eric, Greg, TomG

Phil: will not be here next time; UCSF people are planning to be on campus 7/22.

Philip: 3D pipeline and ChimeraX playing well currently. Phil: helpful if ChimeraX could report molecular weight, w/o hydrogens for simplicity. Elaine: don't you want a "real" molecular weight with hydrogens? Phil: probably not much difference, just need to be up-front about how the number is calculated.

Tangents on 3D printing:

• ​Visualizing 3D imagery by mouth using candy-like models
• ​Computational design, fabrication and evaluation of rubber protein models

Meghan: working on surface coloring in GLB; Enduvo has issues with that. Another interesting app is ​IQ3Connect where instructor and students can share a session, similar to ChimeraX meetings but all works via browser and only one person can be the instructor. Also has issues with surface coloring in imported GLB, but sounds like they already know how to fix that. Phil: Can include movies, sound files, and go from scene to scene (different models). $200-$1000/month to enable hosting one meeting at a time (being the instructor and able to import models), free to join meeting. Meghan: interface is somewhat zoom-like, host can control permissions of the attendees.

We all tried joining a IQ3Connect meeting in web browser started by Phil. Meghan joined with Quest 2 headset (untethered). We can see a very shiny thick ribbon/stick model with stubby little avatars for the participants. They will explore this app further, are currently in a trial period.

Both IQ3Connect and Enduvo detect groups or subparts of the structure. TomG: ChimeraX gltf export makes a group for each model, submodel or drawing (e.g. atoms, ribbons, bonds) in ChimeraX, but does not separate chains.

TomG is interested in trying IQ3Connect with Phil and Meghan in VR on some realistic test case.

8/11/2021

Phil, Philip, Peter Steinbach, Anthony Armstrong, Kristen; Elaine, Eric, Greg, TomG, Zach, Scooter

Lately all the attention is on AlphaFold. TG described "alphafold match" and showed the 3p5b and 6vpv examples. Plans are to add "alphafold search" for when you don't have a structure, and blast search as alternative to current blat, and maybe eventually a service for running predictions. Also demo'ed Google colab.

Phil is interested in the use to fill in missing segments of experimental structures. AlphaFold was easier to use than other methods he tried and gave a reasonable conformation (based on other data) as the top hit.

Tony mentions ongoing tests using AlphaFold for building complexes, and something about putting U in the sequence to link multiple proteins (?) He also suggests an option to trim away lower-confidence residues. Currently can be done with bfactor attribute specification.

Phil: a cautionary tale is progesterone receptor 1a28 alphafold match, you can see that multiple domains even within a monomer chain are not placed as expected relative to each other.