surface dust not working in a multi-line script

[esa-pekka.kumpula@…]

The following bug report has been submitted:
Platform:        Linux-6.5.0-27-generic-x86_64-with-glibc2.35
ChimeraX Version: 1.7.1 (2024-01-23 01:58:08 UTC)
Description
I am writing scripts for generating figures and I am often using "hide dust" in those scripts. The way I do it is I copy-paste segments from a .cxc file and run them from the command line. For some reason, if I run a "surface dust" command, it does nothing if it's within a multi-line script. However, if it's a single line containing only the surface dust command, it works. Everything else I have tried so far works within a multi-line script snippet, with the exception of molmap which requires a "wait 20" command after it, otherwise the next commands don't necessarily find the molmap-generated map. This does not help with dust removal though.

OpenGL version: 3.3.0 NVIDIA 535.171.04
OpenGL renderer: NVIDIA GeForce RTX 2080/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation

Python: 3.11.2
Locale: en_GB.UTF-8
Qt version: PyQt6 6.3.1, Qt 6.3.1
Qt runtime version: 6.3.2
Qt platform: xcb

XDG_SESSION_TYPE=x11
DESKTOP_SESSION=ubuntu
XDG_SESSION_DESKTOP=ubuntu
XDG_CURRENT_DESKTOP=ubuntu:GNOME
DISPLAY=:1
Manufacturer: LENOVO
Model: 30BAS0PF00
OS: Ubuntu 22.04 Jammy Jellyfish
Architecture: 64bit ELF
Virtual Machine: none
CPU: 16 Intel(R) Xeon(R) Silver 4108 CPU @ 1.80GHz
Cache Size: 11264 KB
Memory:
	               total        used        free      shared  buff/cache   available
	Mem:           125Gi       6.8Gi       111Gi       240Mi       7.2Gi       117Gi
	Swap:           31Gi          0B        31Gi

Graphics:
	65:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)	
	Subsystem: Micro-Star International Co., Ltd. [MSI] TU104 [GeForce RTX 2080 Rev. A] [1462:3722]	
	Kernel driver in use: nvidia

Installed Packages:
    alabaster: 0.7.16
    appdirs: 1.4.4
    asttokens: 2.4.1
    Babel: 2.14.0
    backcall: 0.2.0
    beautifulsoup4: 4.11.2
    blockdiag: 3.0.0
    blosc2: 2.0.0
    build: 0.10.0
    certifi: 2023.11.17
    cftime: 1.6.3
    charset-normalizer: 3.3.2
    ChimeraX-AddCharge: 1.5.13
    ChimeraX-AddH: 2.2.5
    ChimeraX-AlignmentAlgorithms: 2.0.1
    ChimeraX-AlignmentHdrs: 3.4.1
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.12.2
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.1.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.49.1
    ChimeraX-AtomicLibrary: 12.1.5
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.3.2
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.10.5
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.3.2
    ChimeraX-ChangeChains: 1.1
    ChimeraX-CheckWaters: 1.3.2
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.2.4
    ChimeraX-Clipper: 0.22.4
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.5
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.7.1
    ChimeraX-CoreFormats: 1.2
    ChimeraX-coulombic: 1.4.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-Dicom: 1.2
    ChimeraX-DistMonitor: 1.4
    ChimeraX-DockPrep: 1.1.3
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ISOLDE: 1.7.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-IUPAC: 1.0
    ChimeraX-Label: 1.1.8
    ChimeraX-LinuxSupport: 1.0.1
    ChimeraX-ListInfo: 1.2.2
    ChimeraX-Log: 1.1.6
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.9.1
    ChimeraX-Map: 1.1.4
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.1.2
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.6.1
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.12.1
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.14
    ChimeraX-ModelPanel: 1.4
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0.3
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.1
    ChimeraX-NRRD: 1.1
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.13.1
    ChimeraX-PDB: 2.7.3
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.4
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.2
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.1
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.2
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 4.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.2
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.11
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.2
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1.2
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.12.4
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1.2
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.2.2
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-TaskManager: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.4
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.33.3
    ChimeraX-uniprot: 2.3
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.3.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-vrml: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.3
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.2.1
    contourpy: 1.2.0
    cxservices: 1.2.2
    cycler: 0.12.1
    Cython: 0.29.33
    debugpy: 1.8.0
    decorator: 5.1.1
    distro: 1.7.0
    docutils: 0.19
    executing: 2.0.1
    filelock: 3.9.0
    fonttools: 4.47.2
    funcparserlib: 2.0.0a0
    glfw: 2.6.4
    grako: 3.16.5
    h5py: 3.10.0
    html2text: 2020.1.16
    idna: 3.6
    ihm: 0.38
    imagecodecs: 2023.9.18
    imagesize: 1.4.1
    ipykernel: 6.23.2
    ipython: 8.14.0
    ipython-genutils: 0.2.0
    ipywidgets: 8.1.1
    jedi: 0.18.2
    Jinja2: 3.1.2
    jupyter-client: 8.2.0
    jupyter-core: 5.7.1
    jupyterlab-widgets: 3.0.9
    kiwisolver: 1.4.5
    line-profiler: 4.0.2
    lxml: 4.9.2
    lz4: 4.3.2
    MarkupSafe: 2.1.4
    matplotlib: 3.7.2
    matplotlib-inline: 0.1.6
    msgpack: 1.0.4
    nest-asyncio: 1.6.0
    netCDF4: 1.6.2
    networkx: 3.1
    nibabel: 5.0.1
    nptyping: 2.5.0
    numexpr: 2.8.8
    numpy: 1.25.1
    openvr: 1.23.701
    packaging: 23.2
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pexpect: 4.9.0
    pickleshare: 0.7.5
    pillow: 10.2.0
    pip: 23.0
    pkginfo: 1.9.6
    platformdirs: 4.1.0
    prompt-toolkit: 3.0.43
    psutil: 5.9.5
    ptyprocess: 0.7.0
    pure-eval: 0.2.2
    py-cpuinfo: 9.0.0
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.16.1
    pynrrd: 1.0.0
    PyOpenGL: 3.1.7
    PyOpenGL-accelerate: 3.1.7
    pyopenxr: 1.0.2801
    pyparsing: 3.0.9
    pyproject-hooks: 1.0.0
    PyQt6-commercial: 6.3.1
    PyQt6-Qt6: 6.3.2
    PyQt6-sip: 13.4.0
    PyQt6-WebEngine-commercial: 6.3.1
    PyQt6-WebEngine-Qt6: 6.3.2
    python-dateutil: 2.8.2
    pytz: 2023.3.post1
    pyzmq: 25.1.2
    qtconsole: 5.4.3
    QtPy: 2.4.1
    RandomWords: 0.4.0
    requests: 2.31.0
    scipy: 1.11.1
    setuptools: 67.4.0
    sfftk-rw: 0.7.3
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.5
    sphinx: 6.1.3
    sphinx-autodoc-typehints: 1.22
    sphinxcontrib-applehelp: 1.0.8
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.6
    sphinxcontrib-htmlhelp: 2.0.5
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.7
    sphinxcontrib-serializinghtml: 1.1.10
    stack-data: 0.6.3
    superqt: 0.5.0
    tables: 3.8.0
    tcia-utils: 1.5.1
    tifffile: 2023.7.18
    tinyarray: 1.2.4
    tomli: 2.0.1
    tornado: 6.4
    traitlets: 5.9.0
    typing-extensions: 4.9.0
    tzdata: 2023.4
    urllib3: 2.1.0
    wcwidth: 0.2.13
    webcolors: 1.12
    wheel: 0.38.4
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.9

[Tom Goddard]

Can you give an example command script? If I do these commands in a script test.cxc

open 29900 from emdb
surface dust #1
save test.png

and open the script "open text.cxc" the surface dust command works and dust is hidden in the saved image.

I'm actually not quite sure why that works! Volume surfaces are not computed unless they are drawn and in a multiline script no drawing happens unless you add a command "wait 1" to draw a frame or save an image. So in my example I'd expect the surface has not been computed when "surface dust #1" is run. But "surface dust #1" automatically updates the surface whenever it changes, so maybe that is how it is working in my example.

At any rate, you'd need to give an example script to get better advice.

[esa-pekka.kumpula@…]

Hi Tom,

Thanks for the reply! I did a little more digging and I actually could make it work using a wait command before the surface dust command. It does not seem to be a problem when one opens a map, then runs surface dust (without wait). But when I open a map and do some operation to the map, it starts to be required. In my example case, I have a map and model superposed, I then generate a map of the model using molmap, and use that as a mask with the starting map, then run dust removal. If I add a wait command before surface dust, it works.

## This does not work:
close session
open 29900 from emdb
open 8gam
molmap #2 10
wait 1
volume mask #1 surfaces #3
#wait 1
surface dust #4 size 14.1
hide ~#4 models

## This works:
close session
open 29900 from emdb
open 8gam
molmap #2 10
wait 1
volume mask #1 surfaces #3
wait 1
surface dust #4 size 14.1
hide ~#4 models

The reason I sent this report now was that I’ve been doing this for some time now (starting from ChimeraX v. 1.5 I think) and I couldn’t get it to work no matter what – even the wait command did nothing. I should’ve tried the wait command with the new version too, sorry.

Cheers,
Esa-Pekka


--
Esa-Pekka Kumpula, PhD
Senior Postdoctoral Researcher
Laboratory of Structural Biology
Institute of Biotechnology
Helsinki Institute of Life Science HiLIFE
University of Helsinki
www.helsinki.fi/lsb<http://www.helsinki.fi/lsb>




From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
Date: Thursday, 18. April 2024 at 23.31
To: goddard@cgl.ucsf.edu <goddard@cgl.ucsf.edu>, Kumpula, Esa-Pekka <esa-pekka.kumpula@helsinki.fi>
Subject: Re: [ChimeraX] #14971: surface dust not working in a multi-line script
#14971: surface dust not working in a multi-line script
------------------------------------------+-------------------------
          Reporter:  esa-pekka.kumpula@…  |      Owner:  Tom Goddard
              Type:  defect               |     Status:  assigned
          Priority:  normal               |  Milestone:
         Component:  Surface              |    Version:
        Resolution:                       |   Keywords:
        Blocked By:                       |   Blocking:
Notify when closed:                       |   Platform:  all
           Project:  ChimeraX             |
------------------------------------------+-------------------------
Comment (by Tom Goddard):

 Can you give an example command script?  If I do these commands in a
 script test.cxc

 open 29900 from emdb
 surface dust #1
 save test.png

 and open the script "open text.cxc" the surface dust command works and
 dust is hidden in the saved image.

 I'm actually not quite sure why that works!  Volume surfaces are not
 computed unless they are drawn and in a multiline script no drawing
 happens unless you add a command "wait 1" to draw a frame or save an
 image.  So in my example I'd expect the surface has not been computed when
 "surface dust #1" is run.  But "surface dust #1" automatically updates the
 surface whenever it changes, so maybe that is how it is working in my
 example.

 At any rate, you'd need to give an example script to get better advice.
--
Ticket URL: <https://www.rbvi.ucsf.edu/trac/ChimeraX/ticket/14971#comment:2>
ChimeraX <https://www.rbvi.ucsf.edu/chimerax/>
ChimeraX Issue Tracker

[Tom Goddard]

Your example makes sense. If in a script a command creates a volume and then a subsequent command uses the surface of that volume, then a "wait 1" is needed before using the surface because the surface is not computed unless it is rendered. There may be exceptions to this when the surface is computed (maybe when opening a map from a file).

This is an unfortunate situation that has been reported a number of times. I put in a fix for it 2 years ago that attempted to compute the surfaces as soon as a map is made when ChimeraX is processing a script. But apparently that is not working in your example. Let me investigate why it is not working in your case and see if I can improve it so these "wait 1" tricks are not needed in scripts.

[Tom Goddard]

Fixed.

The "wait 1" is no longer needed to get volume surfaces to be computed in a script. This will be in ChimeraX daily builds dated April 19, 2024 and later. The fix for this problem from 2 years ago only made sure maps opened from files had their surface computes before running subsequent commands. The new fix makes all maps update their surfaces when needed after each command is run. The surface is only computed if its display attribute is set to True (ie if it is set to be displayed when rendered). That restriction is to handle the case of volume time series where there may be several hundred maps with only one displayed -- computing surfaces in that case would be extremely slow and possibly run out of memory.

The new fix works by recomputing any volume surfaces that are out of date after each command while a script is running. I don't expect this to significantly degrade performance. But for scripts that do things like fitting an atomic model in a map where the surface is not needed it will run slower from needlessly computing the surface. But I think it is better that cryptic "wait 1" commands are not needed.