Problems with MMTF bondAtomList

[pbarletta@…]

The following bug report has been submitted:
Platform:        Linux-5.4.0-65-generic-x86_64-with-debian-bullseye-sid
ChimeraX Version: 1.1 (2020-09-09 22:22:27 UTC)
Description
I created this file using the chemfiles library and I'm certain bondAtomList is present in this file. Here's the output from the mmtf python library, when I parse "sal.mmtf" and check its bond_atom_list attribute:
array([  80,   82,   99,  101,  118,  120,  129,  131,  153,  155,  172, 
       174,  196,  198,  218,  220,  242,  244,  257,  259,  271,  273, 
       290,  292,  301,  303,  320,  322,  341,  343,  353,  355,  375, 
       377,  394,  396,  401,  403,  441,  443,  451,  453,  470,  472, 
       485,  487,  501,  503,  517,  519,  533,  535,  548,  550,  560, 
       562,  580,  582,  601,  603,  617,  619,  641,  643,  657,  659, 
       676,  678,  686,  688,  698,  700,  710,  712,  727,  729,  746, 
       748,  757,  759,  767,  769,  787,  789,  807,  809,  818,  820, 
       825,  827,  839,  841,  853,  855,  870,  872,  881,  883,  905, 
       907,  924,  926,  946,  948,  953,  955,  975,  977,  992,  994, 
      1008, 1010, 1023, 1025, 1043, 1045, 1063, 1065, 1073, 1075, 1083, 
      1085, 1093, 1095, 1112, 1114, 1119, 1121, 1126, 1128, 1140, 1142, 
      1155, 1157, 1169, 1171, 1190, 1192, 1204, 1206, 1211, 1213, 1221, 
      1223, 1235, 1237, 1252, 1254, 1274, 1276, 1291, 1293, 1341, 1343, 
      1348, 1350, 1372, 1374, 1379, 1381, 1398, 1400, 1412, 1414, 1483, 
      1485, 1494, 1496, 1513, 1515, 1529, 1531, 1539, 1541, 1583, 1585, 
      1593, 1595, 1605, 1607, 1615, 1617, 1634, 1636, 1648, 1650, 1658, 
      1660, 1668, 1670, 1675, 1677, 1691, 1693, 1705, 1707, 1716, 1718, 
      1731, 1733, 1745, 1747, 1764, 1766, 1778, 1780, 1793, 1795, 1812, 
      1814, 1831, 1833, 1838, 1840, 1854, 1856, 1873, 1875, 1883, 1885, 
      1897, 1899, 1916, 1918, 1926, 1928, 1942, 1944, 1954, 1956, 1970, 
      1972], dtype=int32)

In addition bondAtomList is an optional field according to https://github.com/rcsb/mmtf/blob/master/spec.md.

Let me know if there's anything I can do.

Log:
UCSF ChimeraX version: 1.1 (2020-09-09)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open /home/pbarletta/labo/20/grilla/sal.mmtf

Traceback (most recent call last):  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/open_command/dialog.py", line 150, in _qt_safe  
if data_format is None else " format " +
StringArg.unparse(data_format.nicknames[0])))  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/core/commands/run.py", line 31, in run  
results = command.run(text, log=log)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/core/commands/cli.py", line 2805, in run  
result = ci.function(session, **kw_args)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/open_command/cmd.py", line 100, in cmd_open  
return Command(session, registry=registry).run(provider_cmd_text, log=log)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/core/commands/cli.py", line 2805, in run  
result = ci.function(session, **kw_args)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/open_command/cmd.py", line 154, in provider_open  
name or model_name_from_path(fi.file_name)), provider_kw)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/open_command/cmd.py", line 389, in collated_open  
return func(*func_args, **func_kw)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/mmtf/__init__.py", line 33, in open  
return mmtf.open_mmtf(session, data, file_name, **kw)  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/mmtf/mmtf.py", line 49, in open_mmtf  
pointers = _mmtf.parse_MMTF_file(filename, session.logger, coordsets)  
RuntimeError: MsgPack MAP does not contain required entry bondAtomList  
  
RuntimeError: MsgPack MAP does not contain required entry bondAtomList  
  
File "/usr/lib/ucsf-chimerax/lib/python3.7/site-
packages/chimerax/mmtf/mmtf.py", line 49, in open_mmtf  
pointers = _mmtf.parse_MMTF_file(filename, session.logger, coordsets)  
  
See log for complete Python traceback.  
  




OpenGL version: 4.6 (Core Profile) Mesa 20.2.6
OpenGL renderer: Mesa Intel(R) HD Graphics 630 (KBL GT2)
OpenGL vendor: Intel
Manufacturer: Dell Inc.
Model: XPS 15 9560
OS: KDE neon 20.04 focal
Architecture: 64bit ELF
CPU: 8 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Cache Size: 6144 KB
Memory:
	              total        used        free      shared  buff/cache   available
	Mem:           30Gi        12Gi       841Mi       2.4Gi        17Gi        15Gi
	Swap:          34Gi       106Mi        33Gi

Graphics:
	00:02.0 VGA compatible controller [0300]: Intel Corporation HD Graphics 630 [8086:591b] (rev 04)	
	DeviceName:  Onboard IGD	
	Subsystem: Dell HD Graphics 630 [1028:07be]
PyQt version: 5.12.3
Compiled Qt version: 5.12.4
Runtime Qt version: 5.12.9
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.8.0
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2020.6.20
    chardet: 3.0.4
    ChimeraX-AddH: 2.1.1
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.1
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.6.1
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 1.0.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.0
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.0
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.1
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-CommandLine: 1.1.3
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.1
    ChimeraX-CoreFormats: 1.0
    ChimeraX-coulombic: 1.0.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-DataFormats: 1.0
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.0
    ChimeraX-Hbonds: 2.0
    ChimeraX-Help: 1.0
    ChimeraX-HKCage: 1.0
    ChimeraX-IHM: 1.0
    ChimeraX-ImageFormats: 1.0
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0
    ChimeraX-Label: 1.0
    ChimeraX-LinuxSupport: 1.0
    ChimeraX-ListInfo: 1.0
    ChimeraX-Log: 1.1.1
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Map: 1.0.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.0
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 1.1
    ChimeraX-MDcrds: 2.0
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.0
    ChimeraX-mmCIF: 2.2
    ChimeraX-MMTF: 2.0
    ChimeraX-Modeller: 1.0
    ChimeraX-ModelPanel: 1.0
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.0
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0
    ChimeraX-OpenCommand: 1.2.1
    ChimeraX-PDB: 2.1
    ChimeraX-PDBBio: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0
    ChimeraX-PubChem: 2.0
    ChimeraX-Read-Pbonds: 1.0
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.0
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.2
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SeqView: 2.2
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.0
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.0
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.0.4
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.0
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.0
    ChimeraX-ToolshedUtils: 1.0
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.2.3
    ChimeraX-uniprot: 2.0
    ChimeraX-ViewDockX: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.3
    comtypes: 1.1.7
    cxservices: 1.0
    cycler: 0.10.0
    Cython: 0.29.20
    decorator: 4.4.2
    distlib: 0.3.1
    distro: 1.5.0
    docutils: 0.16
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 2.10.0
    html2text: 2020.1.16
    idna: 2.10
    ihm: 0.16
    imagecodecs: 2020.5.30
    imagecodecs-lite: 2020.1.31
    imagesize: 1.2.0
    ipykernel: 5.3.0
    ipython: 7.15.0
    ipython-genutils: 0.2.0
    jedi: 0.17.2
    Jinja2: 2.11.2
    jupyter-client: 6.1.3
    jupyter-core: 4.6.3
    kiwisolver: 1.2.0
    line-profiler: 2.1.2
    lxml: 4.5.1
    MarkupSafe: 1.1.1
    matplotlib: 3.2.1
    msgpack: 1.0.0
    netifaces: 0.10.9
    networkx: 2.4
    numexpr: 2.7.1
    numpy: 1.18.5
    numpydoc: 1.0.0
    openvr: 1.12.501
    packaging: 20.4
    parso: 0.7.1
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 7.1.2
    pip: 20.2.2
    pkginfo: 1.5.0.1
    prompt-toolkit: 3.0.7
    psutil: 5.7.0
    ptyprocess: 0.6.0
    pycollada: 0.7.1
    pydicom: 2.0.0
    Pygments: 2.6.1
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.12.3
    PyQt5-sip: 4.19.19
    PyQtWebEngine-commercial: 5.12.1
    python-dateutil: 2.8.1
    pytz: 2020.1
    pyzmq: 19.0.2
    qtconsole: 4.7.4
    QtPy: 1.9.0
    RandomWords: 0.3.0
    requests: 2.24.0
    scipy: 1.4.1
    setuptools: 49.4.0
    sfftk-rw: 0.6.6.dev0
    six: 1.15.0
    snowballstemmer: 2.0.0
    sortedcontainers: 2.2.2
    Sphinx: 3.1.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 1.0.3
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.4
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2020.6.3
    tinyarray: 1.2.2
    tornado: 6.0.4
    traitlets: 5.0.4
    urllib3: 1.25.10
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.34.2
File attachment: sal.mmtf

sal.mmtf​

[pbarletta@…]

Added by email2trac

[pett]

Reported by Patricio Barletta

[Greg Couch]

FYI, I rebuilt the mmtf extension with the latest mmtf-cpp source and now it crashes with a segmentation fault when I try to open sal.mmtf. Updating msgpack didn't help. I'll look at little deeper there. I'm also considering trying to resurrect the all Python version of the mmtf reader. We switched to the mmtf C++ code because the Python code was too slow.

[pbarletta@…]

Thanks for the update. You're right, the problem is in mmtf.

I'm debugging the mmtf header files, particularly *msgpack_decoder.hpp *and*
map_decoder.hpp*. My sample file has issues with the groupList  attribute,
but for some reason, mmtf complains about the bondAtomList parameter. If I
comment out this line in *msgpack_decoder.hpp*:

md.decode("groupList", true, data.groupList);

MMTF is able to read this *sal.mmtf* example, although it does complain
that the *groupList* attribute wasn't parsed.

I'm going to keep looking into it and open an issue in the mmtf cpp repo.
They don't seem to pay much attention to it, perhaps I should look into
other formats.

Thanks!

On Fri, Mar 5, 2021 at 4:11 AM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[pbarletta@…]

Scratch that. I was using the last release. I cloned from github and now it
works, I don't get any segmentation fault from mmtf and the
hasConsistentData mmtf functinon returns *True*, so your error might come
from Chimera itself.

On Fri, Mar 5, 2021 at 11:53 AM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Greg Couch]

So just to confirm that we're using the same mmtf-cpp repository, I used ​https://github.com/rcsb/mmtf-cpp. Is that the one you used? And which version of msgpack are you using? I tried the version that was current when we first made the mmtf bundle (msgpack-c had msgpack.hpp) and the 3.3.0 msgpack-cpp version that requires boost.

And an aside about MMTF. ChimeraX's mmCIF reader is very fast and mmCIF files have more metadata in them that the MMTF files. So for many purposes, you're better off use mmCIF.

[Greg Couch]

So ChimeraX is segfaulting when it tries to compare entity.type with a string. Is your mmtf file missing the entity's type?

[pbarletta@…]

Yes, that's the mmtf I'm using and the msgpack version is 3.3.0, the latest
release.

Yes, it's missing. Although that too is an optional field.

On Fri, Mar 5, 2021 at 3:30 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Greg Couch]

The entityList is optional. The entity type isn't optional. But the only case where the entity type currently matters in ChimeraX is if it is a polymer. Because, if it is a polymer, then the entity sequence is important metadata.

[pbarletta@…]

mmm..., *entityList* is an array and* type *is one of the elements. Given
that *entityList* is marked as an optional field, I take that type is
optional.
Anyway, I'll add the type so I don't get into trouble.
Thanks.

On Fri, Mar 5, 2021 at 3:46 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Greg Couch]

Looking the spec, ​https://github.com/rcsb/mmtf/blob/master/spec.md#entitylist, it shows that the entityList is optional. It is an arrays of entity objects. None of the items in an entity object are listed as optional.

[Greg Couch]

I see that the "bug" is in the ChimeraX reader. There are no entities in the sal.mmtf file, and that confuses ChimeraX. Every chain should have an associated entity. I'll see if I can relax that requirement or generate an error message. It shouldn't crash.

[Greg Couch]

I've updated ChimeraX to use the latest mmtf-cpp source code and made it handle missing entityList gracefully.

Looking at your data, it would really benefit from entity information. Is it really two chains or one? And if it is one, then there are clearly missing residues that should be in the entity's sequence. Otherwise, it should be two distinct chains (and two entities). It would also help if you gave a chain identifier to the chain. All mmCIF files from the PDB have non-empty chain identifiers -- typically A for a single chain.

[pbarletta@…]

Will do. Thanks again for your help.

On Fri, Mar 5, 2021, 5:13 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote: