Restoring new session in older ChimeraX: _pickle.UnpicklingError: invalid load key, '\x04'.

[goddard@…]

The following bug report has been submitted:
Platform:        Windows-10-10.0.19041
ChimeraX Version: 1.1 (2020-09-09 22:22:27 UTC)
Description
This error happens opening a session in ChimeraX 1.1 that was saved with a current ChimeraX daily build.  This should give an easy to understand error message that the session cannot be opened because it was written a newer ChimeraX.

Log:
UCSF ChimeraX version: 1.1 (2020-09-09)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open C:\cygwin64\home\goddard\ucsf\chimerax\src\bundles\vive\test.cxs format
> session

Traceback (most recent call last):  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-packages\chimerax\ui\gui.py",
line 629, in customEvent  
func(*args, **kw)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\ui\widgets\htmlview.py", line 440, in defer  
cxcmd(session, topic)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\ui\widgets\htmlview.py", line 456, in cxcmd  
run(session, cmd)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\commands\run.py", line 31, in run  
results = command.run(text, log=log)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\commands\cli.py", line 2805, in run  
result = ci.function(session, **kw_args)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\open_command\cmd.py", line 100, in cmd_open  
return Command(session, registry=registry).run(provider_cmd_text, log=log)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\commands\cli.py", line 2805, in run  
result = ci.function(session, **kw_args)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\open_command\cmd.py", line 154, in provider_open  
name or model_name_from_path(fi.file_name)), provider_kw)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\open_command\cmd.py", line 390, in collated_open  
return func(*func_args, **func_kw)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core_formats\\__init__.py", line 26, in open  
return cxs_open(session, data, **kw)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\session.py", line 925, in open  
session.restore(stream, path=path, resize_window=resize_window)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\session.py", line 608, in restore  
version = serialize.pickle_deserialize(stream)  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\serialize.py", line 57, in pickle_deserialize  
return unpickler.load()  
_pickle.UnpicklingError: invalid load key, '\x04'.  
  
_pickle.UnpicklingError: invalid load key, '\x04'.  
  
File "C:\Program Files\ChimeraX-1.1\bin\lib\site-
packages\chimerax\core\serialize.py", line 57, in pickle_deserialize  
return unpickler.load()  
  
See log for complete Python traceback.  
  




OpenGL version: 3.3.0 NVIDIA 456.71
OpenGL renderer: GeForce GTX 1080 Ti/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: Gigabyte Technology Co., Ltd.
Model: Z270X-Gaming K7
OS: Microsoft Windows 10 Home (Build 19041)
Memory: 17,129,693,184
MaxProcessMemory: 137,438,953,344
CPU: 8 Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz"
PyQt version: 5.12.3
Compiled Qt version: 5.12.4
Runtime Qt version: 5.12.9
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.8.0
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2020.6.20
    chardet: 3.0.4
    ChimeraX-AddH: 2.1.1
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.1
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.6.1
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 1.0.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.0
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.0
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.1
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.0
    ChimeraX-Clipper: 0.15.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-CommandLine: 1.1.3
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.1
    ChimeraX-CoreFormats: 1.0
    ChimeraX-coulombic: 1.0.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-DataFormats: 1.0
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.0
    ChimeraX-Hbonds: 2.0
    ChimeraX-Help: 1.0
    ChimeraX-HKCage: 1.0
    ChimeraX-IHM: 1.0
    ChimeraX-ImageFormats: 1.0
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0
    ChimeraX-ISOLDE: 1.1.0
    ChimeraX-Label: 1.0
    ChimeraX-ListInfo: 1.0
    ChimeraX-Log: 1.1.1
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Map: 1.0.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.0
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 1.1
    ChimeraX-MDcrds: 2.0
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.0
    ChimeraX-mmCIF: 2.2
    ChimeraX-MMTF: 2.0
    ChimeraX-Modeller: 1.0
    ChimeraX-ModelPanel: 1.0
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.0
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0
    ChimeraX-OpenCommand: 1.2.1
    ChimeraX-PDB: 2.1
    ChimeraX-PDBBio: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0
    ChimeraX-PubChem: 2.0
    ChimeraX-Read-Pbonds: 1.0
    ChimeraX-RealSense: 1.5
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.0
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.2
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SeqView: 2.2
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.0
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.0
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.0.4
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.0
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.0
    ChimeraX-ToolshedUtils: 1.0
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.2.3
    ChimeraX-uniprot: 2.0
    ChimeraX-ViewDockX: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.3
    comtypes: 1.1.7
    cxservices: 1.0
    cycler: 0.10.0
    Cython: 0.29.20
    decorator: 4.4.2
    distlib: 0.3.1
    docutils: 0.16
    filelock: 3.0.12
    funcparserlib: 0.3.6
    gdcm: 2.8.8
    grako: 3.16.5
    h5py: 2.10.0
    html2text: 2020.1.16
    idna: 2.10
    ihm: 0.16
    imagecodecs: 2020.5.30
    imagecodecs-lite: 2020.1.31
    imagesize: 1.2.0
    ipykernel: 5.3.0
    ipython: 7.15.0
    ipython-genutils: 0.2.0
    jedi: 0.17.2
    Jinja2: 2.11.2
    jupyter-client: 6.1.3
    jupyter-core: 4.6.3
    kiwisolver: 1.2.0
    line-profiler: 2.1.2
    lxml: 4.5.1
    MarkupSafe: 1.1.1
    matplotlib: 3.2.1
    msgpack: 1.0.0
    netifaces: 0.10.9
    networkx: 2.4
    numexpr: 2.7.1
    numpy: 1.18.5+mkl
    numpydoc: 1.0.0
    openvr: 1.12.501
    packaging: 20.4
    parso: 0.7.1
    pickleshare: 0.7.5
    Pillow: 7.1.2
    pip: 20.2.2
    pkginfo: 1.5.0.1
    prompt-toolkit: 3.0.7
    psutil: 5.7.0
    pycollada: 0.7.1
    pydicom: 2.0.0
    Pygments: 2.6.1
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.12.3
    PyQt5-sip: 4.19.19
    PyQtWebEngine-commercial: 5.12.1
    pyrealsense2: 2.35.2.1937
    python-dateutil: 2.8.1
    pytz: 2020.1
    pywin32: 228
    pyzmq: 19.0.2
    qtconsole: 4.7.4
    QtPy: 1.9.0
    RandomWords: 0.3.0
    requests: 2.24.0
    scipy: 1.4.1
    setuptools: 49.4.0
    sfftk-rw: 0.6.6.dev0
    six: 1.15.0
    snowballstemmer: 2.0.0
    sortedcontainers: 2.2.2
    Sphinx: 3.1.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 1.0.3
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.4
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2020.6.3
    tinyarray: 1.2.2
    tornado: 6.0.4
    traitlets: 5.0.4
    urllib3: 1.25.10
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.34.2
    WMI: 1.5.1

[goddard@…]

Here is the session file.

test.cxs​

[goddard@…]

Added by email2trac

[Greg Couch]

The problem is that the LZ4 compression is not understood by previous versions of ChimeraX. And we don't have a way of fixing that without having users download a new version of ChimeraX. And in that case, there's nothing to do because the daily build already understands LZ4 compression. The only thing we can do is to put in a FAQ saying to use unlz4 to compress the session file before trying to restore it the previous version.

For the future, we could redesign the session file format, version 4, to do the compression after the "# ChimeraX Session version 4\n" line at the start of the file. That might confuse users who try compressing the session file, and don't get much of an effect. But I doubt that would be an issue.

So the question is, do we go to version 4 and compression inline? Then older versions of ChimeraX would recognize the fact that they can not open the session file and give a better error message: "Unable to restore session: need newer version of ChimeraX to restore session"

[Tom Goddard]

Of course we can't fix ChimeraX 1.1. But we should make ChimeraX 1.2 correctly report when a user tries to open future incompatible ChimeraX sessions in 1.2. I don't follow understand why that means changing the current session file format handling of compression -- I think we should definitely not do that so close to the 1.2 release. Why is compression involved at all in this? Wouldn't ChimeraX 1.2 simply uncompress lz4 if the file is identified to be lz4 compressed and then check to see if the session can be restored?

[Tom Goddard]

Maybe you mean if we juggled around lz4 compression we could make ChimeraX 1.1 give a good error message. I don't think we need to fix ChimeraX 1.1, just make sure 1.2 does not have the same problem.

[Greg Couch]

The ChimeraX daily build (1.2) does not have the problem because it recognizes the session file as being compressed with lz4. So there is nothing to be done if we don't care that ChimeraX 1.1 gives a traceback. And yes, I was suggesting a way to use LZ4 compression and have ChimeraX 1.1 give an error message instead.

[goddard@…]

I'd say if ChimeraX 1.2 will not have this problem that is good enough.  But is that the case?  What if a better compression choice comes along than lz4 and we use that in 1.3? Will ChimeraX 1.2 throw a traceback or will it give an error message that a user can understand -- like "the file X.cxs is not in a recognized format -- if it was saved in a newer ChimeraX version it may require a more recent ChimeraX version to open".

[Greg Couch]

If a better compression algorithm comes along and we want older versions of ChimeraX to give an error message instead of a callback, then the compression needs to be inlined. And if we do that now with LZ4 for ChimeraX 1.2, then ChimeraX 1.1 would give an error message.

[Greg Couch]

The bug is probably that the session file was written using the ChimeraX

daily build. Unfortunately, this can only be fixed by upgrading your
version of ChimeraX. Starting with ChimeraX 1.2, if this situation
happens, an error message is generated rather than a Python traceback.