Support chain merging in Structure.combine()

[Tristan Croll]

The following bug report has been submitted:
Platform:        Linux-3.10.0-1160.25.1.el7.x86_64-x86_64-with-glibc2.17
ChimeraX Version: 1.3.dev202110120902 (2021-10-12 09:02:44 UTC)
Description
I realise it's not what it's made for, but the new (and brilliantly fast) `Structure.combine()` method is tantalisingly close to being a wholesale replacement for ISOLDE's `merge_fragment` to add a piece into an existing chain from another model, if I just give it an empty dict for the chain mapping (with of course some wrapper code needed up-front to prevent introducing residues with the same chain ID and number). But it still puts the piece in as a new chain rather than part of the existing one, leading to funny business when saving and reloading (mmCIF comes out weird with disconnections and some duplicated residues; PDB actually seems fine). Is this sort of merging something you'd be interested in supporting centrally? If not, looks like I could whip up some ISOLDE-specific code by copying and modifying yours (if that's OK, of course!). Looks like it would mostly be a matter of modifying the call to `new_residue()` at line 385 of Structure.cpp to specify a neighbor?

Log:
> alias preview_toolshed toolshed url https://cxtoolshed-
> preview.rbvi.ucsf.edu; toolshed reload available

> alias production_toolshed toolshed url https://cxtoolshed.rbvi.ucsf.edu;
> toolshed reload available

> isolde shorthand
    
    
    Initialising ISOLDE-specific command aliases:
    Alias	Equivalent full command
    -------------------------------------------------
    st	isolde step {arguments}
    aw	isolde add water {arguments}
    awsf	isolde add water {arguments} sim false
    al	isolde add ligand {arguments}
    aa	isolde add aa $1 sel {arguments}
    ht	isolde mod his sel {arguments}
    so	setattr sel atoms occupancy {arguments}
    ab	isolde adjust bfactors {arguments}
    ss	isolde sim start sel
    rt	isolde release torsions sel {arguments}
    rd	isolde release distances sel {arguments}
    ra	rd; rt
    pf	isolde pepflip sel
    cf	isolde cisflip sel
    cbb	color bfactor {arguments}
    cbo	color byattr occupancy {arguments}
    cbc	color {arguments} bychain; color {arguments} byhet
    cs	clipper set contourSensitivity {arguments}
    

  
UCSF ChimeraX version: 1.3.dev202110120902 (2021-10-12)  
© 2016-2021 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open 3io0

3io0 title:  
Crystal structure of EtuB from Clostridium kluyveri [more info...]  
  
Chain information for 3io0 #1  
---  
Chain | Description | UniProt  
A | EtuB protein | A5N734_CLOK5  
  
3io0 mmCIF Assemblies  
---  
1| author_and_software_defined_assembly  
  

> open 3io0

3io0 title:  
Crystal structure of EtuB from Clostridium kluyveri [more info...]  
  
Chain information for 3io0 #2  
---  
Chain | Description | UniProt  
A | EtuB protein | A5N734_CLOK5  
  
3io0 mmCIF Assemblies  
---  
1| author_and_software_defined_assembly  
  

> delete #2&~:268-273

> delete #1:268-273

> ui tool show Shell

Shell commands:  
from chimerax.atomic import AtomicStructure  
m1, m2 = session.models.list(type=AtomicStructure)  
m1.combine(m2, {},m1.position)  

> hide #2 models

> save test.pdb #1

> save test.cif #1

> open test.pdb

Chain information for test.pdb #3  
---  
Chain | Description  
A | No description available  
  

> open test.cif

Chain information for test.cif #4  
---  
Chain | Description  
A | EtuB protein  
  
test.cif mmCIF Assemblies  
---  
1| author_and_software_defined_assembly  
  

> save test_session.cxs

> hide #!3 models

> hide #4 models

> ~cartoon

> cartoon

> bond #1:267@C|#1:268@N

Created 1 bond  

> bond #1:273@C|#1:274@N

Created 1 bond  

> ~cartoon

> cartoon

> close #4#3

> save test.pdb #1

> save test.cif #1

> open test.pdb

Chain information for test.pdb #3  
---  
Chain | Description  
A | No description available  
  

> open test.cif

Chain information for test.cif #4  
---  
Chain | Description  
A | EtuB protein  
  
test.cif mmCIF Assemblies  
---  
1| author_and_software_defined_assembly  
  

> hide #3 models

> select #4:272

8 atoms, 6 bonds, 2 residues, 1 model selected  

> select clear

> save test_session.cxs

> hide #1 models

> show #1 models




OpenGL version: 3.3.0 NVIDIA 465.19.01
OpenGL renderer: NVIDIA TITAN Xp/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: Dell Inc.
Model: Precision T5600
OS: CentOS Linux 7 Core
Architecture: 64bit ELF
Virutal Machine: none
CPU: 32 Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz
Cache Size: 20480 KB
Memory:
	              total        used        free      shared  buff/cache   available
	Mem:            62G         14G         22G        394M         25G         47G
	Swap:          4.9G          0B        4.9G

Graphics:
	03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [TITAN Xp] [10de:1b02] (rev a1)	
	Subsystem: NVIDIA Corporation Device [10de:11df]	
	Kernel driver in use: nvidia
Locale: ('en_GB', 'UTF-8')
PyQt5 5.15.2, Qt 5.15.2
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.9.1
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2021.10.8
    cftime: 1.5.1
    charset-normalizer: 2.0.7
    ChimeraX-AddCharge: 1.1.4
    ChimeraX-AddH: 2.1.10
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.2.2
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.30.2
    ChimeraX-AtomicLibrary: 4.1.4
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AtomSearchLibrary: 1.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.0
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.6
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.2
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.1.1
    ChimeraX-Clipper: 0.17.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5
    ChimeraX-CommandLine: 1.1.5
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.3.dev202110120902
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1.5
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.1.1
    ChimeraX-Help: 1.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ISOLDE: 1.3.dev32
    ChimeraX-ItemsInspection: 1.0
    ChimeraX-Label: 1.1
    ChimeraX-LinuxSupport: 1.0
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.4
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 2.0.3
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.4
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.2.2
    ChimeraX-ModelPanel: 1.2
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.2
    ChimeraX-OpenCommand: 1.7
    ChimeraX-PDB: 2.6.4
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-Phenix: 0.3
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-Sample: 0.1
    ChimeraX-SaveCommand: 1.5
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.4.4
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.1
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.6
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1
    ChimeraX-ToolshedUtils: 1.2
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.13.4
    ChimeraX-uniprot: 2.2
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.0.1
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-Voyager: 0.1
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.4
    cxservices: 1.1
    cycler: 0.10.0
    Cython: 0.29.24
    decorator: 5.1.0
    distro: 1.6.0
    docutils: 0.17.1
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 3.4.0
    html2text: 2020.1.16
    idna: 3.2
    ihm: 0.21
    imagecodecs: 2021.4.28
    imagesize: 1.2.0
    ipykernel: 5.5.5
    ipython: 7.23.1
    ipython-genutils: 0.2.0
    jedi: 0.18.0
    Jinja2: 3.0.1
    jsonpickle: 2.0.0
    jupyter-client: 6.1.12
    jupyter-core: 4.8.1
    kiwisolver: 1.3.2
    line-profiler: 3.3.0
    lxml: 4.6.3
    lz4: 3.1.3
    MarkupSafe: 2.0.1
    matplotlib: 3.4.3
    matplotlib-inline: 0.1.3
    msgpack: 1.0.2
    netCDF4: 1.5.7
    networkx: 2.6.2
    numexpr: 2.7.3
    numpy: 1.21.2
    openvr: 1.16.801
    packaging: 21.0
    ParmEd: 3.2.0
    parso: 0.8.2
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 8.3.2
    pip: 21.2.4
    pkginfo: 1.7.1
    prompt-toolkit: 3.0.20
    psutil: 5.8.0
    ptyprocess: 0.7.0
    pycollada: 0.7.1
    pydicom: 2.1.2
    Pygments: 2.10.0
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.15.2
    PyQt5-sip: 12.8.1
    PyQtWebEngine-commercial: 5.15.2
    python-dateutil: 2.8.2
    python-igraph: 0.9.6
    pytz: 2021.3
    pyvis: 0.1.9
    pyzmq: 22.3.0
    qtconsole: 5.1.1
    QtPy: 1.11.2
    RandomWords: 0.3.0
    rdkit-pypi: 2021.3.5.1
    requests: 2.26.0
    scipy: 1.7.1
    setuptools: 57.5.0
    sfftk-rw: 0.7.1
    six: 1.16.0
    snowballstemmer: 2.1.0
    sortedcontainers: 2.4.0
    Sphinx: 4.2.0
    sphinx-autodoc-typehints: 1.12.0
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    suds-jurko: 0.6
    tables: 3.6.1
    texttable: 1.6.4
    tifffile: 2021.4.8
    tinyarray: 1.2.3
    tornado: 6.1
    traitlets: 5.1.0
    urllib3: 1.26.7
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.37.0
    wheel-filename: 1.3.0
File attachment: test_session.cxs

test_session.cxs​

[Tristan Croll]

Added by email2trac

[Tristan Croll]

Also, closing the models after all that killed ChimeraX:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Calling [C++: N10atomstruct5ChainE] _cpp_demotion failed.
Aborted
________________________________
From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
Sent: 19 October 2021 15:32
Cc: Tristan Croll <tic20@cam.ac.uk>
Subject: Re: [ChimeraX] #5432: ChimeraX bug report submission

#5432: ChimeraX bug report submission
------------------------------------+--------------------
          Reporter:  Tristan Croll  |      Owner:  (none)
              Type:  defect         |     Status:  new
          Priority:  normal         |  Milestone:
         Component:  Unassigned     |    Version:
        Resolution:                 |   Keywords:
        Blocked By:                 |   Blocking:
Notify when closed:                 |
------------------------------------+--------------------
Changes (by Tristan Croll):

 * Attachment "test_session.cxs" added.

 Added by email2trac

--
Ticket URL: <https://www.rbvi.ucsf.edu/trac/ChimeraX/ticket/5432>
ChimeraX <https://www.rbvi.ucsf.edu/chimerax/>
ChimeraX Issue Tracker

[pett]

I guess I prefer centralized if possible, so that improvements to the code propagate to you (and anyone else that crazy :-) ). It's more complicated that just the new_residue call -- it's also a non-trivial revamp to the chain-creation code later in the routine. Doesn't feel like a change I could slip into 1.3. If it's only 1.4, maybe you'd want to copy/change the code yourself (which you are welcome to)?

Using the session you attached, I don't get a crash when closing the models, so it's some kind of internal state thing that doesn't get recreated by the session restore.

[Tom Goddard]

The crash you observed

terminate called after throwing an instance of 'std::runtime_error'

what(): Calling [C++: N10atomstruct5ChainE] _cpp_demotion failed.

Aborted

mentions _cpp_demotion which is a method of the StructureSeq class (molobject.py) and apparently tries to change a StructureSeq instance to a Sequence instance. It looks like the C++ method that raised a std::runtime_error was a Chain constructor. The C++ Chain constructor only does

_structure->change_tracker()->add_created(_structure, this);

That adds the new chain to a change track created list. Looks like some C++ standard library generates the runtime_error. Would be nice if our code that catches and reports that actually gave the text of the runtime_error message, would make it much easier to debug.

[Tom Goddard]

I see a bit more, the error "what(): Calling [C++: N10atomstruct5ChainE] _cpp_demotion failed." is the text of the C++ std::runtime_error. It appears our C++ code Chain constructor is calling back into the Python code _cpp_demotion and that probably raises a Python exception which the C++ converts to a runtime_error losing the actual Python exception information. I would expect ChimeraX ask you to report the crash the next time you started it including a Python traceback generated by faulthandler -- would be useful for debugging.

[Tristan Croll]

That one's lost, I'm afraid. Can try to reproduce it tomorrow.
________________________________
From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
Sent: 21 October 2021 09:20
Cc: goddard@cgl.ucsf.edu <goddard@cgl.ucsf.edu>; pett@cgl.ucsf.edu <pett@cgl.ucsf.edu>; Tristan Croll <tic20@cam.ac.uk>
Subject: Re: [ChimeraX] #5432: Support chain merging in Structure.combine()

#5432: Support chain merging in Structure.combine()
----------------------------------------+----------------------------
          Reporter:  Tristan Croll      |      Owner:  Eric Pettersen
              Type:  enhancement        |     Status:  feedback
          Priority:  normal             |  Milestone:
         Component:  Structure Editing  |    Version:
        Resolution:                     |   Keywords:
        Blocked By:                     |   Blocking:
Notify when closed:                     |   Platform:  all
           Project:  ChimeraX           |
----------------------------------------+----------------------------

Comment (by Tom Goddard):

 I see a bit more, the error "what(): Calling [C++: N10atomstruct5ChainE]
 _cpp_demotion failed." is the text of the C++ std::runtime_error.  It
 appears our C++ code Chain constructor is calling back into the Python
 code _cpp_demotion and that probably raises a Python exception which the
 C++ converts to a runtime_error losing the actual Python exception
 information.  I would expect ChimeraX ask you to report the crash the next
 time you started it including a Python traceback generated by faulthandler
 -- would be useful for debugging.

--
Ticket URL: <https://www.rbvi.ucsf.edu/trac/ChimeraX/ticket/5432#comment:5>
ChimeraX <https://www.rbvi.ucsf.edu/chimerax/>
ChimeraX Issue Tracker

[pett]

Upon further reflection, preservation of custom attributes depends explicitly on the added atoms, bonds, residues, etc. being at the end of the corresponding collections, and in the same order, so supporting this request would be a significant undertaking. I think you'll have to go the copy-and-modify route. If you can reproduce the crash, open a ticket for that...

[pett]

On the other hand, once you have combined the models, you should just be able to form the polymeric bond between the chains and the two chains will be automatically combined into one. You might have to change the resulting chain ID (I believe it gets the chain ID of the N-terminal fragment) and change residue numbering. If that's not working, let me know.