Opened 4 years ago

Last modified 4 years ago

#6365 assigned defect

Loading pickled file in thread crashes

Reported by: Tristan Croll Owned by: Greg Couch
Priority: normal Milestone:
Component: Core Version:
Keywords: Cc: chimera-programmers
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description 

The following bug report has been submitted:
Platform:        Windows-10-10.0.19041
ChimeraX Version: 1.4.dev202203110419 (2022-03-11 04:19:32 UTC)
Description
Last time you used ChimeraX it crashed.
Please describe steps that led to the crash here.
For some reason loading a picked file (OpenMM `ForceField` object and two version strings) in a separate Python thread is no longer thread-safe (has been working for years, now crashes intermittently). Can get rid of the threading (was only there to get some marginally better startup time), but I think worth noting.

Windows fatal exception: access violation

Thread 0x000092b8 (most recent call first):
  File "", line 228 in _call_with_frames_removed
  File "", line 1173 in create_module
  File "", line 565 in module_from_spec
  File "", line 666 in _load_unlocked
  File "", line 986 in _find_and_load_unlocked
  File "", line 1007 in _find_and_load
  File "C:\Users\tristan\AppData\Local\UCSF\ChimeraX\1.4\site-packages\chimerax\isolde\openmm\forcefields.py", line 66 in _background_load_ff
  File "C:\Program Files\ChimeraX-daily\bin\lib\concurrent\futures\thread.py", line 52 in run
  File "C:\Program Files\ChimeraX-daily\bin\lib\concurrent\futures\thread.py", line 77 in _worker
  File "C:\Program Files\ChimeraX-daily\bin\lib\threading.py", line 910 in run
  File "C:\Program Files\ChimeraX-daily\bin\lib\threading.py", line 973 in _bootstrap_inner
  File "C:\Program Files\ChimeraX-daily\bin\lib\threading.py", line 930 in _bootstrap

Current thread 0x00000020 (most recent call first):
  File "C:\Program Files\ChimeraX-daily\bin\lib\site-packages\chimerax\ui\gui.py", line 318 in event_loop
  File "C:\Program Files\ChimeraX-daily\bin\lib\site-packages\ChimeraX_main.py", line 867 in init
  File "C:\Program Files\ChimeraX-daily\bin\lib\site-packages\ChimeraX_main.py", line 1018 in 
  File "C:\Program Files\ChimeraX-daily\bin\lib\runpy.py", line 87 in _run_code
  File "C:\Program Files\ChimeraX-daily\bin\lib\runpy.py", line 197 in _run_module_as_main
===== Log before crash start =====
> isolde shorthand
    
    
    Initialising ISOLDE-specific command aliases:
    Alias	Equivalent full command
    -------------------------------------------------
    st	isolde step {arguments}
    aw	isolde add water {arguments}
    awsf	isolde add water {arguments} sim false
    al	isolde add ligand {arguments}
    aa	isolde add aa $1 sel {arguments}
    ht	isolde mod his sel {arguments}
    so	setattr sel atoms occupancy {arguments}
    ab	isolde adjust bfactors {arguments}
    ss	isolde sim start sel
    rt	isolde release torsions sel {arguments}
    rd	isolde release distances sel {arguments}
    ra	rd; rt
    pf	isolde pepflip sel
    cf	isolde cisflip sel
    cbb	color bfactor {arguments}
    cbo	color byattr occupancy {arguments}
    cbc	color {arguments} bychain; color {arguments} byhet
    cs	clipper set contourSensitivity {arguments}
    

  
UCSF ChimeraX version: 1.4.dev202203110419 (2022-03-11)  
© 2016-2022 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> isolde start


===== Log before crash end =====

Log:
> isolde shorthand
    
    
    Initialising ISOLDE-specific command aliases:
    Alias	Equivalent full command
    -------------------------------------------------
    st	isolde step {arguments}
    aw	isolde add water {arguments}
    awsf	isolde add water {arguments} sim false
    al	isolde add ligand {arguments}
    aa	isolde add aa $1 sel {arguments}
    ht	isolde mod his sel {arguments}
    so	setattr sel atoms occupancy {arguments}
    ab	isolde adjust bfactors {arguments}
    ss	isolde sim start sel
    rt	isolde release torsions sel {arguments}
    rd	isolde release distances sel {arguments}
    ra	rd; rt
    pf	isolde pepflip sel
    cf	isolde cisflip sel
    cbb	color bfactor {arguments}
    cbo	color byattr occupancy {arguments}
    cbc	color {arguments} bychain; color {arguments} byhet
    cs	clipper set contourSensitivity {arguments}
    

  
UCSF ChimeraX version: 1.4.dev202203110419 (2022-03-11)  
© 2016-2022 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  




OpenGL version: 3.3.0 NVIDIA 497.29
OpenGL renderer: NVIDIA GeForce RTX 2080/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation

Locale: en_GB.cp1252
Qt version: PyQt6 6.2.3, Qt 6.2.3
Qt platform: windows

Manufacturer: Notebook                        
Model: P7xxTM1
OS: Microsoft Windows 10 Education (Build 19041)
Memory: 68,654,501,888
MaxProcessMemory: 137,438,953,344
CPU: 16 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
OSLanguage: en-GB

Installed Packages:
    -himerax-isolde: 1.4.dev0
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.9.1
    backcall: 0.2.0
    blockdiag: 3.0.0
    certifi: 2021.10.8
    cftime: 1.6.0
    charset-normalizer: 2.0.12
    ChimeraX-AddCharge: 1.2.3
    ChimeraX-AddH: 2.1.11
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2.1
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.2.3
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.36.2
    ChimeraX-AtomicLibrary: 6.1.1
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.1
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.0
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.6.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.2
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.2.2
    ChimeraX-Clipper: 0.18.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.1
    ChimeraX-CommandLine: 1.2.1
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.4.dev202203110419
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1.5
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.1.2
    ChimeraX-Help: 1.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ISOLDE: 1.4.dev0
    ChimeraX-ItemsInspection: 1.0
    ChimeraX-Label: 1.1
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 2.0.6
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.7
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.5.3
    ChimeraX-ModelPanel: 1.3.2
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.2
    ChimeraX-OpenCommand: 1.8
    ChimeraX-PDB: 2.6.6
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0.1
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.4.8
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.1
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.8
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.0.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1
    ChimeraX-ToolshedUtils: 1.2.1
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.16.2
    ChimeraX-uniprot: 2.2
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.1.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.4
    comtypes: 1.1.10
    cxservices: 1.1
    cycler: 0.11.0
    Cython: 0.29.26
    debugpy: 1.5.1
    decorator: 5.1.1
    docutils: 0.17.1
    entrypoints: 0.4
    filelock: 3.4.2
    fonttools: 4.30.0
    funcparserlib: 1.0.0a0
    grako: 3.16.5
    h5py: 3.6.0
    html2text: 2020.1.16
    idna: 3.3
    ihm: 0.26
    imagecodecs: 2021.11.20
    imagesize: 1.3.0
    ipykernel: 6.6.1
    ipython: 7.31.1
    ipython-genutils: 0.2.0
    jedi: 0.18.1
    Jinja2: 3.0.3
    jupyter-client: 7.1.0
    jupyter-core: 4.9.2
    kiwisolver: 1.3.2
    line-profiler: 3.4.0
    lxml: 4.7.1
    lz4: 3.1.10
    MarkupSafe: 2.1.0
    matplotlib: 3.5.1
    matplotlib-inline: 0.1.3
    msgpack: 1.0.3
    nest-asyncio: 1.5.4
    netCDF4: 1.5.8
    networkx: 2.6.3
    numexpr: 2.8.1
    numpy: 1.22.1
    openvr: 1.16.802
    packaging: 21.3
    ParmEd: 3.4.3
    parso: 0.8.3
    pickleshare: 0.7.5
    Pillow: 9.0.0
    pip: 21.3.1
    pkginfo: 1.8.2
    prompt-toolkit: 3.0.28
    psutil: 5.9.0
    pycollada: 0.7.2
    pydicom: 2.2.2
    Pygments: 2.11.2
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.7
    PyQt6-commercial: 6.2.3
    PyQt6-sip: 13.2.0
    PyQt6-WebEngine-commercial: 6.2.1
    python-dateutil: 2.8.2
    pytz: 2021.3
    pywin32: 303
    pyzmq: 22.3.0
    qtconsole: 5.2.2
    QtPy: 2.0.1
    RandomWords: 0.3.0
    requests: 2.27.1
    scipy: 1.7.3
    setuptools: 59.8.0
    sfftk-rw: 0.7.1
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    Sphinx: 4.3.2
    sphinx-autodoc-typehints: 1.15.2
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    suds-community: 1.0.0
    tables: 3.7.0
    tifffile: 2021.11.2
    tinyarray: 1.2.4
    tornado: 6.1
    traitlets: 5.1.1
    urllib3: 1.26.8
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.37.1
    wheel-filename: 1.3.0
    WMI: 1.5.1

Change History (9)

comment:1 by Eric Pettersen, 4 years ago

Cc: chimera-programmers added
Component: Unassigned → Core
Owner: set to Greg Couch
Platform: → all
Project: → ChimeraX
Status: new → assigned
Summary: ChimeraX bug report submission → Loading pickled file in thread crashes

comment:2 by Tristan Croll, 4 years ago

Thread is spawned at ​https://github.com/tristanic/isolde/blob/6abd2668edef97ab87120e5f2d9e5758afdf5f03/isolde/src/openmm/forcefields.py#L181, and crashes at ​https://github.com/tristanic/isolde/blob/6abd2668edef97ab87120e5f2d9e5758afdf5f03/isolde/src/openmm/forcefields.py#L66. Still the same version of OpenMM as ChimeraX 1.3, and I haven't done anything to this code in a long long time. I've tried deleting the old pickle files so they're regenerated on the next ISOLDE start... regeneration worked fine, but the crash still happens intermittently.

comment:3 by Tom Goddard, 4 years ago

You are using Qt 6 in this report. I believe Qt is not thread safe and all calls must be done from the main thread. So if unpickling somehow makes a Qt call that could explain why it is crashing with Qt 6.

comment:4 by Eric Pettersen, 4 years ago

In particular, logging will cause changes in the Log tool. If your thread is logging anything (or doing anything else that might cause Qt activity) you can use session.ui.thread_safe to make the call in the main thread, e.g.:

session.logger.status("force field loaded")

becomes:

session.ui.thread_safe(session.logger.status, "force field loaded")

comment:5 by Tristan Croll, 4 years ago

This was done with the barebones multiprocessing.ThreadPoolExecutor, but yes - point taken. There *is* a print in the thread (which hasn't actually caused noticeable trouble in the past, but I can see how it could) - but the crash when it happens is always in the pickle.load() step. The ForceField class knows nothing whatsoever about ChimeraX and the other two components of the pickle are simple strings, so as far as I can see there's nothing that *should* be causing a crash. Still, it should be fine for me to just completely remove the threading here - this dates back to a time when I was loading the entire forcefield at once (including ~15k ligands) which was slow enough to be really annoying, but these days the ligands are all kept separate and only added to the forcefield on demand. Core forcefield load time is not particularly bad - a few seconds on the first load from XML files, under a second for subsequent loads from the pickle.

comment:6 by Greg Couch, 4 years ago

Please do not use pickle data in session files. It is not secure and can not be made secure.

Besides the threading iessue, I see two needed fixes for this bug: (1) better documentation on what can be in session data, and (2) any use of pickle needs to emit a warning or pickle needs to be disabled.

For (1) session data should be POD (plain old data), like what can be in a JSON file. ChimeraX extends what can be POD, see ​https://www.cgl.ucsf.edu/chimerax/docs/devel/core/serialize.html. And session data can include references to instances of classes, but those have to follow the ChimeraX state protocol. Some documentation for that is in https://www.rbvi.ucsf.edu/chimerax/docs/devel/tutorials/tutorial_tool_qt.html#sessions -- see take_snapshot, restore_snapshot, and get_class.

comment:7 by Eric Pettersen, 4 years ago

Tristan isn't using pickled data in session files.

in reply to:  8 ; comment:8 by Tristan Croll, 4 years ago 

None of this is saved in a session (.cxs) file. When a new ISOLDE session is started for the first time, the OpenMM forcefield is loaded from its XML files; the resulting ForceField object is pickled to a file in ISOLDE's cache directory (ChimeraX/{version}/isolde_data) and that's used by preference in future loads.
________________________________
From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
Sent: 15 March 2022 17:03
Cc: chimera-programmers@cgl.ucsf.edu <chimera-programmers@cgl.ucsf.edu>; gregc@cgl.ucsf.edu <gregc@cgl.ucsf.edu>; Tristan Croll <tic20@cam.ac.uk>
Subject: Re: [ChimeraX] #6365: Loading pickled file in thread crashes

#6365: Loading pickled file in thread crashes
------------------------------------+------------------------
          Reporter:  Tristan Croll  |      Owner:  Greg Couch
              Type:  defect         |     Status:  assigned
          Priority:  normal         |  Milestone:
         Component:  Core           |    Version:
        Resolution:                 |   Keywords:
        Blocked By:                 |   Blocking:
Notify when closed:                 |   Platform:  all
           Project:  ChimeraX       |
------------------------------------+------------------------

Comment (by Greg Couch):

 Please do not use pickle data in session files.  It is not secure and can
 not be made secure.

 Besides the threading iessue, I see two needed fixes for this bug: (1)
 better documentation on what can be in session data, and (2) any use of
 pickle needs to emit a warning or pickle needs to be disabled.

 For (1) session data should be POD (plain old data), like what can be in a
 JSON file.  ChimeraX extends what can be POD, see
 https://www.cgl.ucsf.edu/chimerax/docs/devel/core/serialize.html.  And
 session data can include references to instances of classes, but those
 have to follow the ChimeraX state protocol.  Some documentation for that
 is in
 https://www.rbvi.ucsf.edu/chimerax/docs/devel/tutorials/tutorial_tool_qt.html#sessions
 -- see take_snapshot, restore_snapshot, and get_class.

--
Ticket URL: <https://www.rbvi.ucsf.edu/trac/ChimeraX/ticket/6365#comment:6>
ChimeraX <https://www.rbvi.ucsf.edu/chimerax/>
ChimeraX Issue Tracker

comment:9 by Greg Couch, 4 years ago

Ah. It's a relief that's it's not compromising the session data. My hackles are lowered.

Note: See TracTickets for help on using tickets.