Opened 3 months ago

Closed 3 months ago

#18597 closed defect (can't reproduce)

Device loss detected in Present() and crash while running Boltz

Reported by: chimerax-bug-report@… Owned by: Tom Goddard
Priority: normal Milestone:
Component: Structure Prediction Version:
Keywords: Cc:
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description 

The following bug report has been submitted:
Platform:        Windows-10-10.0.22000
ChimeraX Version: 1.11.dev202508112234 (2025-08-11 22:34:36 UTC)
Description
Last time you used ChimeraX it crashed.
Please describe steps that led to the crash here.
Windows fatal exception: code 0x8001010d

Current thread 0x000041c4 (most recent call first):
  File "C:\Program Files\ChimeraX 1.11.dev202508112234\bin\Lib\site-packages\chimerax\ui\gui.py", line 368 in event_loop
  File "C:\Program Files\ChimeraX 1.11.dev202508112234\bin\Lib\site-packages\chimerax\core\__main__.py", line 1058 in init
  File "C:\Program Files\ChimeraX 1.11.dev202508112234\bin\Lib\site-packages\chimerax\core\__main__.py", line 1223 in 
  File "", line 88 in _run_code
  File "", line 198 in _run_module_as_main
===== Log before crash start =====
> ui tool show Boltz

> boltz predict protein
> ASTSDRDTLLAVKKDWGSPPQLKTWDPAAPNHCNWTGVTCATGGGGVVSGLTLSSMKLTGSVPASVCALKSLTHLDLSYDNLTGDFPGAALYACAGLTFLDLSNNQFSGPLPLDIDRLSPAMEHLNLSTNSFAGEVPPAVGGFPALRSLLLDTNSFTGAYPAAEISKLTGLQMLTLADNEFAPAPVPTEFSKLTNLTYLWMGGMNLTGEIPEAFSSLKELTLFSMASNQLTGSIPAWVWQHQKLQYIYLFDNALSGELTRSVTALNLLQIDLSTNQLTGDIPEDFGNLKNLTILFLYNNQLSGTIPASIGLLPQLKDIRLFQNQLSGELPPELGKHSPLGNLEVSINNLSGPLRESLCANGKLFDIVAFNNSFSGELPAELGDCITINNLMLHNNNFSGDFPEKIWSFPNLTLVMVQNNSFTGTLPAQISPKMARIEIGNNRFSGSFPASAPALKVLHAENNRLGGELPPDMSKLANLTDLSVPGNQISGSIPTSIKLLQKLNSLDMRGNRLSSAIPPGSIGLLPALTMLDLSDNEITGNIPSDVSNVFNLLNLSSNQLTGEVPAQLQSAAYDQSFLGNRLCARADSGTNLPMCPAG
> protein VMQVLDKGSSGGRGPGKGNCDWTKPGHPGCPPKSSS name k7_013498 samples 5

Running Boltz prediction of 2 proteins with 633 residues on gpu  
Using multiple sequence alignment server https://api.colabfold.com  

QDxgiVSyncService: DXGI Factory is no longer Current  

Device loss detected in Present()  

Running boltz prediction failed with exit code 1:  
command:  
C:\Users\wei/boltz2\Scripts\boltz.exe predict
C:\Users\wei/Desktop/boltz_k7_013498\k7_013498.yaml --use_msa_server
--accelerator gpu --no_kernels --diffusion_samples 5  
stdout:  
Boltz version 2.1.1  
Checking input data.  
Processing 1 inputs with 1 threads.  
Generating MSA for C:\Users\wei\Desktop\boltz_k7_013498\k7_013498.yaml with 2
protein entities.  
Running structure prediction for 1 input.  
Predicting: | | 0/? [00:00", line 198, in _run_module_as_main  
File "", line 88, in _run_code  
File "C:\Users\wei\boltz2\Scripts\boltz.exe\\__main__.py", line 7, in   
File "C:\Users\wei\boltz2\Lib\site-packages\click\core.py", line 1157, in
__call__  
return self.main(*args, **kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\click\core.py", line 1078, in main  
rv = self.invoke(ctx)  
^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\click\core.py", line 1688, in
invoke  
return _process_result(sub_ctx.command.invoke(sub_ctx))  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\click\core.py", line 1434, in
invoke  
return ctx.invoke(self.callback, **ctx.params)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\click\core.py", line 783, in
invoke  
return __callback(*args, **kwargs)  
^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\boltz\main.py", line 1219, in
predict  
trainer.predict(  
File "C:\Users\wei\boltz2\Lib\site-
packages\pytorch_lightning\trainer\trainer.py", line 859, in predict  
return call._call_and_handle_interrupt(  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-
packages\pytorch_lightning\trainer\call.py", line 68, in
_call_and_handle_interrupt  
trainer._teardown()  
File "C:\Users\wei\boltz2\Lib\site-
packages\pytorch_lightning\trainer\trainer.py", line 1005, in _teardown  
self.strategy.teardown()  
File "C:\Users\wei\boltz2\Lib\site-
packages\pytorch_lightning\strategies\strategy.py", line 536, in teardown  
self.lightning_module.cpu()  
File "C:\Users\wei\boltz2\Lib\site-
packages\lightning_fabric\utilities\device_dtype_mixin.py", line 82, in cpu  
return super().cpu()  
^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\torch\nn\modules\module.py", line
1150, in cpu  
return self._apply(lambda t: t.cpu())  
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\torch\nn\modules\module.py", line
928, in _apply  
module._apply(fn)  
File "C:\Users\wei\boltz2\Lib\site-packages\torchmetrics\metric.py", line 891,
in _apply  
this._defaults[key] = fn(value)  
^^^^^^^^^  
File "C:\Users\wei\boltz2\Lib\site-packages\torch\nn\modules\module.py", line
1150, in   
return self._apply(lambda t: t.cpu())  
^^^^^^^  
torch.AcceleratorError: CUDA error: unknown error  
CUDA kernel errors might be asynchronously reported at some other API call, so
the stacktrace below might be incorrect.  
For debugging consider passing CUDA_LAUNCH_BLOCKING=1  
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.  
  
  

Device loss detected in Present()  


===== Log before crash end =====

Log:
UCSF ChimeraX version: 1.11.dev202508112234 (2025-08-11)  
© 2016-2025 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  




OpenGL version: 3.3.0 NVIDIA 572.96
OpenGL renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation

Python: 3.11.9
Locale: zh_CN.cp936
Qt version: PyQt6 6.9.1, Qt 6.9.0
Qt runtime version: 6.9.1
Qt platform: windows

Manufacturer: LENOVO
Model: 90TY000BCD
OS: Microsoft Windows 11 家庭中文版 (Build 22000)
Memory: 34,119,708,672
MaxProcessMemory: 137,438,953,344
CPU: 20 12th Gen Intel(R) Core(TM) i7-12700KF
OSLanguage: zh-CN

Installed Packages:
    alabaster: 1.0.0
    appdirs: 1.4.4
    asttokens: 3.0.0
    babel: 2.17.0
    beautifulsoup4: 4.13.4
    blockdiag: 3.0.0
    blosc2: 3.6.1
    build: 1.2.2.post1
    certifi: 2025.8.3
    cftime: 1.6.4.post1
    charset-normalizer: 3.4.3
    ChimeraX-AddCharge: 1.5.19
    ChimeraX-AddH: 2.2.7
    ChimeraX-AlignmentAlgorithms: 2.0.2
    ChimeraX-AlignmentHdrs: 3.6.1
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 3.0
    ChimeraX-AlphaFold: 1.0.1
    ChimeraX-AltlocExplorer: 1.1.2
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Aniso: 1.3.2
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.60.11
    ChimeraX-AtomicLibrary: 14.1.21
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.4
    ChimeraX-BasicActions: 1.1.3
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 3.0.0
    ChimeraX-Boltz: 1.1
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.2
    ChimeraX-BuildStructure: 2.13.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.6.0
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.4
    ChimeraX-ChangeChains: 1.1
    ChimeraX-CheckWaters: 1.5
    ChimeraX-ChemGroup: 2.0.2
    ChimeraX-Clashes: 2.3
    ChimeraX-ColorActions: 1.0.5
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.8
    ChimeraX-CommandLine: 1.3.0
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.11.dev202508112234
    ChimeraX-CoreFormats: 1.2
    ChimeraX-coulombic: 1.4.5
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.4
    ChimeraX-Dicom: 1.2.7
    ChimeraX-DistMonitor: 1.4.2
    ChimeraX-DockPrep: 1.1.4
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.4.1
    ChimeraX-Hbonds: 2.5.3
    ChimeraX-Help: 1.3
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.4
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-IUPAC: 1.0
    ChimeraX-KVFinder: 1.7.1
    ChimeraX-Label: 1.1.14
    ChimeraX-ListInfo: 1.2.2
    ChimeraX-Log: 1.2
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.9.2
    ChimeraX-Map: 1.3
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.2.2
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.17
    ChimeraX-MedicalToolbar: 1.1
    ChimeraX-Meeting: 1.0.1
    ChimeraX-Minimize: 1.2
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.16
    ChimeraX-MMTF: 2.2
    ChimeraX-ModelArchive: 1.0
    ChimeraX-Modeller: 1.5.22
    ChimeraX-ModelPanel: 1.5.1
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0.3
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0.1
    ChimeraX-MutationScores: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.2
    ChimeraX-NMRSTAR: 1.0.2
    ChimeraX-NRRD: 1.2
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.15.1
    ChimeraX-OrthoPick: 1.0.1
    ChimeraX-PDB: 2.7.10
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.4
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1.3
    ChimeraX-ProfileGrids: 1.1.4
    ChimeraX-PubChem: 2.2
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.2
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.6.4
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.3.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 4.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.2
    ChimeraX-Scenes: 0.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.3
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-Segmentations: 3.5.7
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.17.2
    ChimeraX-Shape: 1.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.2.1
    ChimeraX-ShowSequences: 1.0.3
    ChimeraX-SideView: 1.0.1
    ChimeraX-SimilarStructures: 1.0.1
    ChimeraX-Smiles: 2.1.2
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.19.1
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.2.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.5.2
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-TaskManager: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.2.3
    ChimeraX-ToolshedUtils: 1.2.4
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.48
    ChimeraX-Umap: 1.0
    ChimeraX-uniprot: 2.3.1
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDock: 1.2.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-vrml: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.5
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.2.3
    comtypes: 1.4.11
    contourpy: 1.3.3
    coverage: 7.10.3
    cxservices: 1.2.3
    cycler: 0.12.1
    Cython: 3.1.2
    debugpy: 1.8.16
    decorator: 5.2.1
    docutils: 0.21.2
    executing: 2.2.0
    filelock: 3.18.0
    fonttools: 4.59.0
    funcparserlib: 2.0.0a0
    glfw: 2.9.0
    grako: 3.16.5
    h5py: 3.14.0
    html2text: 2024.2.26
    idna: 3.10
    ihm: 2.2
    imagecodecs: 2024.6.1
    imagesize: 1.4.1
    iniconfig: 2.1.0
    ipykernel: 6.29.5
    ipython: 8.26.0
    ipywidgets: 8.1.7
    jedi: 0.19.1
    Jinja2: 3.1.6
    jupyter_client: 8.6.3
    jupyter_core: 5.8.1
    jupyterlab_widgets: 3.0.15
    kiwisolver: 1.4.9
    line_profiler: 5.0.0
    lxml: 5.3.1
    lz4: 4.3.2
    Markdown: 3.8.2
    MarkupSafe: 3.0.2
    matplotlib: 3.10.1
    matplotlib-inline: 0.1.7
    msgpack: 1.1.0
    ndindex: 1.10.0
    nest-asyncio: 1.6.0
    netCDF4: 1.6.5
    networkx: 3.3
    nibabel: 5.2.0
    nptyping: 2.5.0
    numexpr: 2.11.0
    numpy: 1.26.4
    nvidia-nvjitlink-cu12: 12.9.86
    OpenMM: 8.2.0
    OpenMM-CUDA-12: 8.2.0
    openvr: 1.26.701
    packaging: 25.0
    ParmEd: 4.2.2
    parso: 0.8.4
    pep517: 0.13.1
    pickleshare: 0.7.5
    pillow: 10.4.0
    pip: 25.2
    pkginfo: 1.12.1.2
    platformdirs: 4.3.8
    pluggy: 1.6.0
    prompt_toolkit: 3.0.51
    psutil: 7.0.0
    pure_eval: 0.2.3
    py-cpuinfo: 9.0.0
    pycollada: 0.8
    pydicom: 2.4.4
    Pygments: 2.18.0
    pynmrstar: 3.3.5
    pynrrd: 1.0.0
    PyOpenGL: 3.1.9
    PyOpenGL-accelerate: 3.1.9
    pyopenxr: 1.1.4501
    pyparsing: 3.2.3
    pyproject_hooks: 1.2.0
    PyQt6-commercial: 6.9.1
    PyQt6-Qt6: 6.9.1
    PyQt6-WebEngine-commercial: 6.9.0
    PyQt6-WebEngine-Qt6: 6.9.1
    PyQt6_sip: 13.10.2
    pytest: 8.4.1
    pytest-cov: 6.2.1
    python-dateutil: 2.9.0.post0
    pytz: 2025.2
    pywin32: 310
    pyzmq: 27.0.1
    qtconsole: 5.5.2
    QtPy: 2.4.3
    qtshim: 1.2
    RandomWords: 0.4.0
    requests: 2.32.4
    roman-numerals-py: 3.1.0
    scipy: 1.14.0
    setuptools: 80.9.0
    sfftk-rw: 0.8.1
    six: 1.17.0
    snowballstemmer: 3.0.1
    sortedcontainers: 2.4.0
    soupsieve: 2.7
    Sphinx: 8.2.3
    sphinx-autodoc-typehints: 3.1.0
    sphinxcontrib-applehelp: 2.0.0
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 2.0.0
    sphinxcontrib-htmlhelp: 2.1.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 2.0.0
    sphinxcontrib-serializinghtml: 2.0.0
    stack-data: 0.6.3
    superqt: 0.7.5
    tables: 3.10.2
    tcia_utils: 1.5.1
    tifffile: 2025.3.13
    tinyarray: 1.2.5
    tornado: 6.5.2
    traitlets: 5.14.3
    typing_extensions: 4.14.1
    tzdata: 2025.2
    urllib3: 2.5.0
    wcwidth: 0.2.13
    webcolors: 24.11.1
    wheel: 0.45.1
    wheel-filename: 1.4.2
    widgetsnbextension: 4.0.14
    WMI: 1.5.1

Change History (2)

comment:1 by Eric Pettersen, 3 months ago

Component: Unassigned → Structure Prediction
Owner: set to Tom Goddard
Platform: → all
Project: → ChimeraX
Status: new → assigned
Summary: ChimeraX bug report submission → Boltz: some kind of GPU problem

comment:2 by Tom Goddard, 3 months ago

Resolution: → can't reproduce
Status: assigned → closed
Summary: Boltz: some kind of GPU problem → Device loss detected in Present() and crash while running Boltz

Claims online suggest "Device loss detected in Present()" means the GPU or driver had a problem and I guess the OpenGL context is lost. After Boltz subprocess was started the device loss message was issued from Qt in ChimeraX. Then Boltz gave an error "torch.AcceleratorError: CUDA error: unknown error". Then ChimeraX crashed.

My guess is the graphics driver for the Nvidia 3060 Ti (8 GB) wiped out, probably because of using too much memory for the 633 residue dimer prediction being made by Boltz. That probably caused the Qt OpenGL context to be lost leading to the ChimeraX crash.

Note: See TracTickets for help on using tickets.