Opened 4 years ago

Closed 3 years ago

#6083 closed defect (limitation)

CUDA/OpenCL errors: CUDA_ERROR_OUT_OF_MEMORY

Reported by: zke@… Owned by: Tristan Croll
Priority: normal Milestone:
Component: Third Party Version:
Keywords: Cc: Tom Goddard
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description 

The following bug report has been submitted:
Platform:        Linux-3.12.74-60.64.40-default-x86_64-with-glibc2.19
ChimeraX Version: 1.3rc202112040407 (2021-12-04 04:07:47 UTC)
Description
(Describe the actions that caused this problem to occur here)

Log:
UCSF ChimeraX version: 1.3rc202112040407 (2021-12-04)  
© 2016-2021 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open 7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif

Summary of feedback from opening
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif  
---  
warnings | Skipping chem_comp category: Missing column 'type' near line 1176  
Missing entity information. Treating each chain as a separate entity.  
Invalid residue range for struct_conf "16": invalid entity "B", near line 183  
Invalid residue range for struct_conf "17": invalid entity "B", near line 184  
Invalid residue range for struct_conf "18": invalid entity "B", near line 185  
Invalid residue range for struct_conf "19": invalid entity "B", near line 186  
Invalid residue range for struct_conf "20": invalid entity "B", near line 187  
25 messages similar to the above omitted  
Invalid sheet range for struct_sheet_range "50 1" near line 1048  
Invalid sheet range for struct_sheet_range "51 1" near line 1049  
Invalid sheet range for struct_sheet_range "52 1" near line 1050  
Invalid sheet range for struct_sheet_range "53 1" near line 1051  
Invalid sheet range for struct_sheet_range "54 1" near line 1052  
122 messages similar to the above omitted  
Missing or incomplete entity_poly_seq table. Inferred polymer connectivity.  
Skipping chem_comp category: Missing column 'type' near line 49604  
Skipping chem_comp category: Missing column 'type' near line 49765  
  
Chain information for
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif #1  
---  
Chain | Description  
A B C | No description available  
  

QXcbConnection: XCB error: 1 (BadRequest), sequence: 170, resource id: 53,
major code: 131 (Unknown), minor code: 47  

failed to create compose table  

> isolde start

> set selectionWidth 4

ISOLDE: Checking and correcting nomenclature for (pseudo)symmetric side
chains...  
ISOLDE: Corrected atom nomenclature of 104 residues in model #1 to IUPAC-IUB
standards.  
Chain information for
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif  
---  
Chain | Description  
1.2/A 1.2/B 1.2/C | No description available  
  

> addh

Summary of feedback from adding hydrogens to
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif #1.2  
---  
notes | No usable SEQRES records for
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif (#1.2) chain A;
guessing termini instead  
No usable SEQRES records for
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif (#1.2) chain B;
guessing termini instead  
No usable SEQRES records for
7SBK_delta_3closed_mutated_isolde5_real_space_refined_000.cif (#1.2) chain C;
guessing termini instead  
Chain-initial residues that are actual N termini: /A ALA 27, /B ALA 27, /C ALA
27  
Chain-initial residues that are not actual N termini: /A ASP 80, /A THR 114,
/A ASN 137, /A GLU 154, /A ASN 188, /A ALA 260, /A ASN 487, /A ASN 641, /A GLN
690, /A LYS 854, /A ALA 944, /B ASP 80, /B THR 114, /B ASN 137, /B GLU 154, /B
ASN 188, /B ALA 260, /B ASN 487, /B ASN 641, /B GLN 690, /B LYS 854, /B ALA
944, /C ASP 80, /C THR 114, /C ASN 137, /C GLU 154, /C ASN 188, /C ALA 260, /C
ASN 487, /C ASN 641, /C GLN 690, /C LYS 854, /C ALA 944  
Chain-final residues that are actual C termini:  
Chain-final residues that are not actual C termini: /A PHE 1148, /A ALA 67, /A
ASP 111, /A PHE 135, /A TYR 144, /A LEU 179, /A LEU 244, /A TYR 473, /A PRO
621, /A GLN 675, /A VAL 826, /A SER 937, /B PHE 1148, /B ALA 67, /B ASP 111,
/B PHE 135, /B TYR 144, /B LEU 179, /B LEU 244, /B TYR 473, /B PRO 621, /B GLN
675, /B VAL 826, /B SER 937, /C PHE 1148, /C ALA 67, /C ASP 111, /C PHE 135,
/C TYR 144, /C LEU 179, /C LEU 244, /C TYR 473, /C PRO 621, /C GLN 675, /C VAL
826, /C SER 937  
2569 hydrogen bonds  
/A PHE 1148 is not terminus, removing H atom from 'C'  
/B PHE 1148 is not terminus, removing H atom from 'C'  
/C PHE 1148 is not terminus, removing H atom from 'C'  
12 hydrogens added  
  

> hide HC

Done loading forcefield  

> open /fs/gpfs41/lv01/fileset02/pool/pool-
> briggs/zke/MPIB/process/20211019_Delta/phenix/delta_job372_enhancer_51k_3closed.mrc

Opened delta_job372_enhancer_51k_3closed.mrc as #2, grid size 324,324,324,
pixel 0.93, shown at level 0.00819, step 2, values float32  

> volume #2 step 1

Opened delta_job372_enhancer_51k_3closed.mrc as #1.1.1.1, grid size
324,324,324, pixel 0.93, shown at step 1, values float32  

> set bgColor white

> show cartoons

Launching using CUDA failed with the below message. Falling back to using
OpenCL.  
  
Error creating array TabulatedFunction: CUDA_ERROR_OUT_OF_MEMORY (2)  

Traceback (most recent call last):  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/openmm/openmm_interface.py", line 1606, in start_sim  
self._prepare_sim()  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/openmm/openmm_interface.py", line 1564, in
_prepare_sim  
s = self._simulation = app.Simulation(self.topology, self._system,  
File "/fs/pool/pool-
bmapps/hpcl5S12/app/soft/CHIMERAX/1.3-beta/lib/python3.9/site-
packages/openmm/app/simulation.py", line 105, in __init__  
self.context = mm.Context(self.system, self.integrator, platform,
platformProperties)  
File "/fs/pool/pool-
bmapps/hpcl5S12/app/soft/CHIMERAX/1.3-beta/lib/python3.9/site-
packages/openmm/openmm.py", line 5210, in __init__  
this = _openmm.new_Context(*args)  
openmm.OpenMMException: Error creating array TabulatedFunction:
CUDA_ERROR_OUT_OF_MEMORY (2)  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/isolde.py", line 2749, in _start_sim_or_toggle_pause  
self.start_sim()  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/isolde.py", line 2804, in start_sim  
sm.start_sim()  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/openmm/openmm_interface.py", line 677, in start_sim  
sh.start_sim()  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/openmm/openmm_interface.py", line 1611, in start_sim  
self._prepare_sim()  
File "/fs/home/zke/.local/share/ChimeraX/1.3/site-
packages/chimerax/isolde/openmm/openmm_interface.py", line 1564, in
_prepare_sim  
s = self._simulation = app.Simulation(self.topology, self._system,  
File "/fs/pool/pool-
bmapps/hpcl5S12/app/soft/CHIMERAX/1.3-beta/lib/python3.9/site-
packages/openmm/app/simulation.py", line 105, in __init__  
self.context = mm.Context(self.system, self.integrator, platform,
platformProperties)  
File "/fs/pool/pool-
bmapps/hpcl5S12/app/soft/CHIMERAX/1.3-beta/lib/python3.9/site-
packages/openmm/openmm.py", line 5210, in __init__  
this = _openmm.new_Context(*args)  
openmm.OpenMMException: Error uploading array TabulatedFunction:
clEnqueueWriteBuffer (-4)  
  
openmm.OpenMMException: Error uploading array TabulatedFunction:
clEnqueueWriteBuffer (-4)  
  
File "/fs/pool/pool-
bmapps/hpcl5S12/app/soft/CHIMERAX/1.3-beta/lib/python3.9/site-
packages/openmm/openmm.py", line 5210, in __init__  
this = _openmm.new_Context(*args)  
  
See log for complete Python traceback.  
  




OpenGL version: 3.3.0 NVIDIA 440.33.01
OpenGL renderer: Tesla K80/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: FUJITSU
Model: PRIMERGY CX2570 M1
OS: SLES 12.1 n/a
Architecture: 64bit ELF
Virutal Machine: none
CPU: 36 Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
Cache Size: 46080 KB
Memory:
	             total       used       free     shared    buffers     cached
	Mem:          504G       503G       1.2G       1.7G       176K       386G
	-/+ buffers/cache:       116G       387G
	Swap:          31G        12G        19G

Graphics:
	07:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) [102b:0522]	
	Subsystem: Fujitsu Technology Solutions Device [1734:11cc]	
	Kernel driver in use: mgag200
Locale: ('en_US', 'UTF-8')
PyQt5 5.15.2, Qt 5.15.2
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.9.1
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2021.10.8
    cftime: 1.5.1.1
    charset-normalizer: 2.0.9
    ChimeraX-AddCharge: 1.2.2
    ChimeraX-AddH: 2.1.11
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.2.3
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.31
    ChimeraX-AtomicLibrary: 4.2
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AtomSearchLibrary: 1.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.0
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.6.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.2
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.2.2
    ChimeraX-Clipper: 0.17.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5
    ChimeraX-CommandLine: 1.1.5
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.3rc202112040407
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1.5
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.1.2
    ChimeraX-Help: 1.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ISOLDE: 1.3
    ChimeraX-ItemsInspection: 1.0
    ChimeraX-Label: 1.1
    ChimeraX-LinuxSupport: 1.0
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.4
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 2.0.4
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.4
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.2.6
    ChimeraX-ModelPanel: 1.2.1
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.2
    ChimeraX-OpenCommand: 1.7
    ChimeraX-PDB: 2.6.5
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0.1
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.4.6
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.1
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.6.1
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1
    ChimeraX-ToolshedUtils: 1.2
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.13.7
    ChimeraX-uniprot: 2.2
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.0.1
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.4
    cxservices: 1.1
    cycler: 0.11.0
    Cython: 0.29.24
    decorator: 5.1.0
    distro: 1.6.0
    docutils: 0.17.1
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 3.6.0
    html2text: 2020.1.16
    idna: 3.3
    ihm: 0.21
    imagecodecs: 2021.4.28
    imagesize: 1.3.0
    ipykernel: 5.5.5
    ipython: 7.23.1
    ipython-genutils: 0.2.0
    jedi: 0.18.0
    Jinja2: 3.0.1
    jupyter-client: 6.1.12
    jupyter-core: 4.9.1
    kiwisolver: 1.3.2
    line-profiler: 3.3.0
    lxml: 4.6.3
    lz4: 3.1.3
    MarkupSafe: 2.0.1
    matplotlib: 3.4.3
    matplotlib-inline: 0.1.3
    msgpack: 1.0.2
    netCDF4: 1.5.7
    networkx: 2.6.3
    numexpr: 2.8.0
    numpy: 1.21.2
    openvr: 1.16.801
    packaging: 21.3
    ParmEd: 3.2.0
    parso: 0.8.3
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 8.3.2
    pip: 21.2.4
    pkginfo: 1.7.1
    prompt-toolkit: 3.0.23
    psutil: 5.8.0
    ptyprocess: 0.7.0
    pycollada: 0.7.1
    pydicom: 2.1.2
    Pygments: 2.10.0
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.6
    PyQt5-commercial: 5.15.2
    PyQt5-sip: 12.8.1
    PyQtWebEngine-commercial: 5.15.2
    python-dateutil: 2.8.2
    pytz: 2021.3
    pyzmq: 22.3.0
    qtconsole: 5.1.1
    QtPy: 1.11.3
    RandomWords: 0.3.0
    requests: 2.26.0
    scipy: 1.7.1
    setuptools: 57.5.0
    sfftk-rw: 0.7.1
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    Sphinx: 4.2.0
    sphinx-autodoc-typehints: 1.12.0
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2021.4.8
    tinyarray: 1.2.3
    tornado: 6.1
    traitlets: 5.1.1
    urllib3: 1.26.7
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.37.0
    wheel-filename: 1.3.0

Change History (4)

comment:1 by Eric Pettersen, 4 years ago

Cc: Tom Goddard added
Component: Unassigned → Third Party
Owner: set to Tristan Croll
Platform: → all
Project: → ChimeraX
Status: new → assigned
Summary: ChimeraX bug report submission → CUDA/OpenCL errors

comment:2 by Eric Pettersen, 4 years ago

Reported by Zunlong Ke

comment:3 by Tom Goddard, 4 years ago

Summary: CUDA/OpenCL errors → CUDA/OpenCL errors: CUDA_ERROR_OUT_OF_MEMORY

comment:4 by Tristan Croll, 3 years ago

Resolution: → limitation
Status: assigned → closed

Sorry I never responded at the time - seems there's a handful of tickets from this period that I missed addressing. Anyway, while I'd like to add some code to give a more user-friendly message when the GPU runs out of memory, that will just be window-dressing.

Note: See TracTickets for help on using tickets.