= NIAID Guidehouse/MSC contract May 1, 2023 - Feb 29, 2024 = 1. General ChimeraX improvements to support NIAID. a. Provide support for an NIAID internal app store b. Specific improvements to assist NIAID personnel to transition away from the unsupported legacy Chimera program, e.g.: i. More GUIs (notably, copy/combine, 2D labels) c. Investigate and improve ChimeraX usability for very wide displays and touch screens i. Specific focus on the BioViz lab wall display d. Work with Enduvo to improve exporting ChimeraX scenes in various formats to their environment. 2. Support the NIH 3D pipeline development, including any changes to ChimeraX to support ongoing development. a. Put presets on github i. Improve presets documentation b. Share code interacting with ChimeraX and NIH3D on github c. Continuing support for NIH3D as needed 3. Extend virtual reality support a. Investigate molecular viewer for standalone headsets such as Quest 2. b. Investigate use of pass-through video with the Quest Pro for multi-person sessions. c. Implement a VR ergonomic toolbar and Model panel user interface. d. Migrate ChimeraX from SteamVR to OpenXR, the current API standard for cross-platform virtual reality. 4. Explore pedagogical benefits of ChimeraX in VR vs. flat screen a. Develop a single web page describing ChimeraX VR capabilities and link to existing tutorials, how-tos, equipment advice. b. Describe use cases where VR is advantageous. c. Collaborate with the University of Indiana (Katy Borner), UCSF, and NIAID to conduct a task analysis comparing VR vs. flat screen for understanding biological macromolecules 5. Medical Images a. Improve DICOM reader by including more data types such as segmentations, and making it more robust by testing against the NCI TCIA repository. b. Add ability to “fetch” studies from TCIA c. Add support for NifTI format d. Add support for NRRD files e. Improve VR experience for medical images i. Easier manipulation of windowing and leveling, especially for complex curves f. General usability improvements for using ChimeraX with medical images driven by TCIA data 6. Outreach a. Instructional material and tools documentation. i. Detailed instructions for all features shall be provided in a user manual. ii.Written user guides and tutorials shall be available as HTML pages. iii. Create videos demonstrating new capabilities. iv. Present webinar and workshop tutorials to train users on existing and new capabilities. b. Create video tutorials for how to use multi-person VR. Attendance at meetings or workshops as required by NIAID 7. Administration a. Submit monthly written reports of accomplishments == 8/24/2023 == Zach, Tom G., Tom F., Eric Kristen, Andi, Darrell, Scooter, Phil, Victor, Elaine, Greg, Bhinnata, Meghan Meghan & Darrell show off BioViz lab. 13 VR workstations, LED touch-sensitive wall display, snazzy floor. Lab event went well. Thursday 2 groups came through; David Chen VR presentation on video wall. Seen directly by many people in the room but also by others via Zoom and in remote conference rooms. They have a recording of a practice version of the presentation. Had 10 people in a VR meeting with separate computer hosting. All used Quest 2 headsets. Had to tape cables to the headsets for reliability. Worked great. Friday the headsets were used separately, with a lab person helping each. Felt it was better this way because there were more people in the lab than expected and therefore it was noisy, which made communication in a multi-person meeting difficult. The lab meeting was part of a conference and known well ahead of time, but the lab was only actually ready the day before! Now that it’s ready, they will advertise its availability. DAE files slow to load (2 minutes to get VR set up). GLB was almost instantaneous, but didn’t work in meeting. T.G. had determined it was because GLB fails to save in session, and fixed it several days ago now. Phil(?)/Dave(?) says gzipped DAE/GLB files don’t load despite being documented to do so [worked for me in later testing, maybe missing final .gz suffix?]. Phil has been investigating presets and requests: - H-bonds and struts be 16-sided - Small-molecule bonds be 32-sided - No sharp boundaries on surfaces - 16 sides for ribbon cross sections Tom F. asked if maybe those settings could be printer dependent. Phil said he just looked at meshes. Darrell noted that most printers print objects that are a foot cubed or less, so there won’t be a lot of variance in scale. 3D print questions on our mailing lists could be forwarded to Phil. The NIH folks need to decide among themselves how much they want ChimeraX to do and how much is post-processed, e.g. low-res representations generated directly or via decimation of a high-res rep? Kristen noted that low-res reps can look good in visualization software due to smoothing from lighting. There was disagreement between Phil and Kristen as to how crude the preset output was, but Kristen was likely using an older version of the presets than Phil. Greg mentioned that maybe they just wanted ChimeraX to output ideal primitives and they would work from there, but Darrell said they wouldn’t want to have to deal with tessellation issues. Scooter mentioned that additional presets would not be hard to add. Eric added that the main issue is ensuring the NIH and CGL are on the same page about the specifications for such a preset. Kristen said that there may need to be more small-molecule presets, because that’s the main category where one size definitely does not fit all due to the wide variety of such molecules. Hundreds of thousands of files need to be rerun with the new workflows. Dave Chen requested Phong lighting for volumetric rendering. T.G. noted that that (and many other modes of lighting for volumetric rendering) is covered in the R01. Phil noted that applying a surface preset repeatedly increases the triangle count (“graphics triangles #1”). Andi got funding for a hackathon for the Human Reference Atlas in the Biovisualization Lab, 10/18-19, and needs a template for the Statement of Work. Meghan: Scooter? LOL. Scooter noted that the SoW is different for academics. Tom G. gave an update on OpenXR. It works, but was harder than anticipated to get going (2 weeks). OpenXR is very verbose. The code needs shakedown testing and T.G. is thinking of putting it in the daily build alongside the SteamVR version. Then he would test it internally and when it seems possibly okay then have the NIH folks test it. Meghan was excited by the possible additional reliability provided by not having SteamVR be yet another possible point of failure. There is no information as to whether Apple Vision Pro will work with OpenXR (T.G. rates it unlikely) but it supposedly going to work with Unity. Kristen’s Pathogen AR app is in Unity. Meghan noted that T.G.’s LookSee app worked well and is well documented. Darrell and Meghan will be visiting Rocky Mountain Labs in October and wondered if T.G. was interested in also going. T.G. seemed possibly amenable and volunteered Zach as well. Meghan and Phil noted that 5 machines in the lab had bad stuttering in VR and they all had 4070TI graphics cards, and offered to send one such card for testing. The 4000 series cards are not supported by Quest/Meta. We have a 4090 card, but it’s on a Linux box and it’s not clear if it can be switched to the Dell Optiplex 7000 due to power issues. Ultimately decided that NIH would try the OpenXR version to see if that remedied the problem, and if not then ship the card. Next meeting will be next week, but could be cancelled if no discussion topics emerge. (8/3,8/17 meetings cancelled) == 7/20/2023 == Phil, Bhinnata, Victor (at end); Zach, TomG, Eric, Elaine Phil: a couple of things about using ChimeraX in the Bioviz lab. Tried to use VR meeting with Quest 2 headsets but had problems: airlink disconnections (link cables not much better, easily jiggle loose), problems joining meeting. Later investigation found timeouts in Steam and in headset. Setting Steam timeout to longer time (in !Startup/Shutdown settings) made most issues go away, but sometimes Steam forgets this and it gets reset back to 30 seconds. After resetting everything got 6 headsets working as compared to only 1 in the previous session. Our SOP from now on will be to check the timeout settings of both the headset and SteamVR beforehand. TomG: if you get mysterious ChimeraX errors even if you find out they are due to these non-ChimeraX timeouts, please report them so we can try to improve the error messages. TomG: next ChimeraX release should remove dependence on SteamVR (will use OpenXR). Hoping to do it in the fall although the release may be later near the end of the year. Phil: if I save a session after "vr on" has been used (even if turned off before saving), that session won't open on M1 mac. Even have this problem sometimes on Intel mac. TomG: it's probably trying to import the SteamVR module. Phil: Dave Chen may have contacted you about some DICOM issue. We were confused about Dave Chen, but Phil says that his problem was fixed. Dave Chen is working with irradiated minipigs. Phil: Another that we ran into was in reading certain sets of DICOM files, e.g. some of the individual slice files don't have .dcm file extensions and so ChimeraX won't open them. Would be nice if we didn't have to add the filename extensions. Zach: will open a ticket. TomG: I encourage use of the standalone headset approach, is limited in some ways, but would be more reliable for demonstrations. Phil: I did get !LookSee going and tried the minipig data at home. TomG: the passthrough video is helpful and works with standalone. Do report any problems, as it hasn't been tested that much. Phil: as you warned, can't view huge systems with too many triangles. Victor: have multiple devices that cannot use vr meeting, cannot reach (ping) the UCSF server chimerax.meeting.net. Phil: it was working this morning, or at least from other machines. TomG: I can't ping it from my computer here either (request timed out), will have to investigate further. I don't have much experience with AWS. Maybe there are transient outages. Phil: I could start a meeting on my mac just now, seems fine. Victor: do I have to use some specific network settings in ChimeraX? TomG: no specific settings are required. TomG: I can see Phil's current meeting. Victor: mine says no meeting found. Conclusion: some problem with the specific machine, as Phil and TomG can engage with this server. TomG: you don't have to use this server, it is just a convenience to (1) create a meeting name (2) provide a hub that is not firewalled. TomG: we recently updated some packages in the daily build, causing many unforeseen problems. May be worth trying 1.6 release instead of daily build. Phil: issue with VTK model not saved in session. TomG: will doublecheck and fix in daily build. Phil: will !LookSee save volume models (transparent volume rendering style, not surfaces)? TomG: yes, will make a stack of rectangles. TomG: make sure you use a recent version of ChimeraX, there was problem earlier where volume renderings weren't included in saved GLTF. However, the performance of volume style rendering with Quest 2 headset is not great, slow (low size limit). TomG: Hopefully will be better in the Quest 3. Eric: did you try the sharp-boundaries preset? Phil: want to discuss with Kristen (at a conference now) before we decide what to do. == 7/6/2023 == Phil, Kristen Browne, Meghan, Darrell, Andi Bueckle, Ziv Yaniv; Zach, TomG, Eric, Elaine, Scooter, Selina Eric: uploaded a new NIH3D bundle. Kristen: Phil was going to test some changes before we implemented. Eric: I sent you a wheel (not on toolshed) to evaluate the presets with and without sharp boundaries, and color fromatoms. Phil: haven't looked at that yet, sorry, when was the email? Kristen: 6/30, will forward it. Phil: also need the newer ChimeraX with surface speedup. Kristen: we will update that at the same time we update the presets. Phil: I'll get the newer ChimeraX for my testing purposes. Scooter: lots of DICOM progress, Zach will demo... we also have an intern Selina looking at machine-learning methods for segmenting medical images. Two methods currently: nnUNET, and another new collaboration with Johns Hopkins folks developing a method named something like universal clip (?) Kristen: is that for CT? Scooter: yes, potentially other modalities as well. Darrell: we have image segmentation experts on staff, I would like to invite Ziv Yaniv Zach demo: showing RIDER lung CT. Ziv complained that the patient was face down in the initial view, said that clinicians always say that. Zach demo: the dicom view layout options and Segmentation tool with "paintbrush" type interface. Ziv says clinicians complained that the automated segmentations were useless. Phil is very interested in what the VR interface will be. Ziv: it will also be important how the rendering is done, because visualizing the data is the basis of manual segmentation. TomG: we have simple windowing/leveling controls in VR, but there is a lot that could be improved. Ziv: take a look at existing work and use it where you can, the complex presets in other programs such as 3D Slicer and Horos. back to machine learning... Scooter: the Johns Hopkins group has a large image set. Ziv asks about the design: docker? Scooter: more likely web services. Ziv: you may be interested in https://monai.io/model-zoo.html Scooter and Selina: the universal model group already supports >20 tissues, with data from many different sources. Ziv: modality? CT. Ziv: one of the common problems is generalizability, the broader the dataset(s) the better. Discussions of AR vs. VR, and "natural" interfaces w/o hand controllers. == 6/15/2023 == Phil, Kristen Browne, Meghan, Darrell, Bhinnata, Victor Starr Kramer; Zach, TomG, Eric, Elaine, Greg "Extra" meeting to figure out suitable settings for the desired number of surface triangles (smoothness vs. file size, calculation time, etc.) for NIH3D pipeline. Phil: established that the requirements for NIH3D and Enduvo are so different that they should just be in separate workflows Phil: turns out the discrepancies between 3DPX (3D print exchange with Chimera) and NIH3D (new pipeline with ChimeraX) were due to different procedure/rules for grid size calculation. Is this in the scripts or presets? Eric: presets, but we need to decide on appropriate size limits for switches in resolution, or caps on the number of triangles or vertices. Calculations for the biggest structures can be extremely slow. Eric: Tunable parameter is grid size. Another possibility is to go to gaussian surfaces instead of molecular surfaces. TomG: Gaussian surfaces have two adjustable parameters, resolution and grid size. We also noticed that calculation time is not linear in the number of atoms. Phil: to me, long calculations are OK as along as the user's expectations are managed. Darrell: I agree, the expectations for a website like NIH3D can be different than for an interactive program like ChimeraX. Still, we don't want ridiculously huge files that many will not be able to handle. Kristen: in situations where we have to re-run everything, multiday calculations are less feasible. May also need to warn users specifically that processing will take a long time, and have some kind of indicator to us as pipeline developers that the process is still running/proceeding rather than hung. Elaine: the hard part is the nitty gritty of setting the rules and boundaries. Darrell: we will need to work on that on our end. Kristen: can we change these parameters? Elaine: they are command options. Phil: I see the preset-generated commands and then make variations on those commands and try them. TomG: I would edit the python in the bundle files. Try it on a set of test cases and evaluate the results. Eric: we should address the problem systems, make sure the appropriate grid size is used (wrong sizes found by Phil), have Kristen apply decimation and see the results, see if TomG can speed up surface calculation, and decide on sharpBoundaries true or false. Phil: advantage to false is smaller files. Eric: maybe true for visualization presets, false for printing presets. Kristen: I should take a look at decimation for both sharp true and sharp false, see what happens to those boundaries. Phil: we're only talking about surfaces so far. I also want to think more about ribbon cross-section (number of sides, etc.). Discussion of lighting ("smoothing" using surface normals). Kristen: should we show the smoothing in the viewer, or is that misleading when the model will be printed? Similarly, what if model is too big for VR use? How to manage expectations and give the right level of detail for the intended use? Darrell: we should discuss amongst ourselves outside of this meeting with the UCSF folks. TomG: that mammoth example was interesting, giving some choices for level of resolution, but that site did not give file sizes, which was problematic. People need to understand what they are getting, to avoid frustration. Darrell: maybe we could offer a printable version and a VR-ready version. Eric: In most cases we do offer two versions, or at least specific settings suitable for printing. I'm thinking sharpboundaries true except for the larger printable models. Phil: for testing, could you make an additional set of presets same as existing except with opposite sharpBoundaries settings? Can send wheels directly instead of putting on toolshed. Where are the presets loaded? Eric: use command "info path"... Phil: found it. TomG: after editing, restart ChimeraX. Phil: do you have a feel for maximum triangles for VR? TomG: Quest2 or QuestPro 900,000 without passthrough, 700,00 with passthrough. Higher-end card 20 million triangles. Kristen: I want to make scenes with lots of proteins. TomG: I'll send you a link about somebody who was making spectacular multiprotein scenes. == 6/8/2023 == Phil, Kristen Browne, Meghan, Bhinnata(?); Zach, TomG, Eric, Elaine, Greg Phil will be away on vacation (Europe) next time 3D pipeline issues: - Kristen had one rendering that came up blank, Phil did not test that case yet - Kristen: did we want to increase quality on small molecules too? Phil did not get to that yet - Kristen: EMDB 10067 was not using recommended contour level. Phil was going to investigate that one further and if needed, make a ticket for it. - Phil: there were some updates for the disconnected pseudobond issue. Eric: submitted new version of NIH3D bundle with preset that makes missing segment pseudobonds go between CA atoms. - Phil: regarding quality of surface models, did some test prints of 4 molecules of different sizes, 3DPX (old Chimera) vs. NIH3D (new ChimeraX). Still need to investigate changing grid size and sharpBoundaries parameter. Old ones from Chimera are smoother. Tom: We need to know exactly which commands/options were used to make the 3DPX surfaces. Phil: we were using the "grid" option for more robust surfaces in Chimera. TomG: in that case, ChimeraX should be able to generate the same surfaces if we give it the right option values, since it's the same algorithm. Another important factor is whether hydrogens were added. TomG: sharpBoundaries significantly increases the number of vertices. Megan sent pictures for 6eee, and TomG and Elaine both compared Chimera and ChimeraX grid surfaces, spacing 2.179, and both looked bad/chunky. Phil: I will send data files, commands that were used, etc. How to send? email? Eric: how did you count vertices? Phil: I load the files into Blender. I used to do it in Meshlab. Kristen: different software will give you different counts. == 5/25/2023 == Phil, Darrell, Kristen Browne, Andi Bueckle; Scooter, Zach, Selina Liu, TomG, Eric, Elaine Scooter describes Alex Pico's recent Wikipathways work Kristen: only one new thing for NIH3D since last time; pdbe biounit url was broken, didn't fall back to pdb. Original ticket was about obsolete entries, should make a new ticket for failure to fall back. There was an issue with vertex colors in gltf, ChimeraX default gltf save behavior was changed to not include them for single-color objects. Need to get daily build. Another issue was higher-resolution surfaces from Chimera than ChimeraX. Phil made a table comparing # vertices etc. for several example structures of different sizes, we're not sure why there are so many more triangles from the NIH3D pipeline than in ChimeraX directly. Maybe the sharp boundaries parameter? None of their colorings are per atom, only per chain, by ESP, or by MLP. So they could turn that option off (sharpBoundaries false). [I saved an image of his table but couldn't figure out how to insert it here] Phil didn't get a chance to look at the pseudobonds-not-connecting issue. Andi's writing up something but it's not yet ready for discussion. TomG: maybe the study should use the standalone VR approach with Quest 2. Andi: is this available as a build? TomG: Yes. Take a look at ChimeraX presentations on this, e.g. [https://www.rbvi.ucsf.edu/chimerax/data/looksee-mar2023/looksee.html demo1] [https://www.rbvi.ucsf.edu/chimerax/data/paxlovid-vr-mar2023/paxlovid.html demo2] [https://www.rbvi.ucsf.edu/chimerax/data/vr-plans-may2023/vr_plans.html demo3] Scooter introduces Selina Scooter says the contract was adjusted down for the 2 months time lost Phil: back to VR, one of the structural biology groups here is asking for headset recommendations. Tethered HP (Windows mixed reality?) seems robust but controller buttons are quite different and not recognized by ChimeraX currently. The worst is that that one that brings up the menu is not recognized. TomG: we have Windows mixed reality setup and it was working, but I haven't used it for a few years, maybe API changed somehow and/or you have a different version. Phil: positions of the buttons don't map correctly, both A and B depress same thing on virtual representation, thumbstick and menu button not recognized at all. Only grip and trigger and A/B (melded) are recognized. TomG: I'll try our Windows mixed reality and get back to you. Phil: obvious recommendation would be RiftS except we can't get those anymore, whereas one can still get HP Reverbs. TomG: this group already has the expensive PC? Phil: yes. Andi: we have the tracking stations on tripods, which makes them easy to move around. TomG: We hope to get ChimeraX working with OpenXR, so it wouldn't be dependent on SteamVR. TomG: I found that devices couldn't "see" each other on guest wifi, only the secure one. This may be an issue for other institutions. Phil: We are still working through wifi issues in the bioviz lab, headsets need 80mHz and router is not providing that. Andi: too hard fighting with institutional IT, I just have to use cables everywhere. Phil: Meghan is doing a great job coordinating the hardware setups here. Discussion of airlink connections. == 5/11/2023 == Phil, Scooter, Zach, Renata, Greg, Tom, Eric, Elaine, Kristin, Meghan Mehgan: Dmitry at Seattle doing VR work including with clinical applications Zach: Nifty & NRRD support. TCIA lists ChimeraX as an analysis tool -- Federick labs still sending bugs (lots of interest) Eric: pseudo-bond error (pseudo bonds don't always connect to the ribbon). Phil is going to review ticket again and see how important it is Elaine: introduction Greg: introduction, work on small molecules Tom: VR standalone, Quest Pro broken, Quest Pro very effective. Discussion about the resolution of the camera in the Quest Pro Phil: Confusion about Mac install (does Universal installer install M1?) Kristin: High priority NIH3D issues * No color modules should always use materials, not vertex coloring (small molecules use vertex colors). Possibly the GLTF writer could potentially switch from vertex coloring to material coloring if the color is all the same * Resolution of surface models is inconsistent between chimera and chimerax. Chimera is higher resolution. * Issue with shading and 3D printing (possibly drop normal vectors)