The following bug report has been submitted:
Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core
ChimeraX Version: 1.1rc202008312018 (2020-08-31 20:18:51 UTC)
Description
There seems to be a fairly severe memory leak when `_mmcif.extract_CIF_tables()` is used to parse a structure-factor file (but not an atomic mmCIF file):
(in the shell)
{{{
import tracemalloc
tracemalloc.start()
before = tracemalloc.take_snapshot()
}}}
(command line)
{{{
open 7jnu structurefactors true
close
}}}
(shell)
{{{
after = tracemalloc.take_snapshot()
top_stats = after.compare_to(before, 'lineno')
for stat in top_stats[:10]:
print(stat)
}}}
... returns the following output:
{{{
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/mmcif/mmcif.py:731: size=42.5 MiB (+42.5 MiB), count=662380 (+662380), average=67 B
<frozen importlib._bootstrap>:219: size=2520 KiB (+2520 KiB), count=26883 (+26883), average=96 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/ribbon.py:608: size=1381 KiB (+1381 KiB), count=2 (+2), average=691 KiB
/home/tic20/.local/share/ChimeraX/1.1/site-packages/chimerax/clipper/maps/map_handler_base.py:318: size=723 KiB (+723 KiB), count=10 (+10), average=72.3 KiB
<frozen importlib._bootstrap_external>:525: size=536 KiB (+536 KiB), count=5901 (+5901), average=93 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molc.py:78: size=531 KiB (+531 KiB), count=7462 (+7462), average=73 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molarray.py:358: size=360 KiB (+360 KiB), count=9 (+9), average=40.0 KiB
<__array_function__ internals>:6: size=348 KiB (+348 KiB), count=69 (+69), average=5166 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molc.py:162: size=337 KiB (+337 KiB), count=7 (+7), average=48.1 KiB
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molc.py:194: size=163 KiB (+163 KiB), count=45 (+45), average=3716 B
}}}
As I said up top, it seems specific to structure factor files:
(command line, continuing from the above)
{{{
open 7jnu
close
}}}
(shell)
{{{
after2 = tracemalloc.take_snapshot()
top_stats2 = after2.compare_to(before)
for stat in top_stats1[:10]:
print(stat)
}}}
... returns:
{{{
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/mmcif/mmcif.py:731: size=42.5 MiB (+42.5 MiB), count=662380 (+662380), average=67 B
<frozen importlib._bootstrap>:219: size=2499 KiB (+2499 KiB), count=26620 (+26620), average=96 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/ribbon.py:608: size=1381 KiB (+1381 KiB), count=2 (+2), average=691 KiB
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molc.py:78: size=918 KiB (+918 KiB), count=12913 (+12913), average=73 B
<frozen importlib._bootstrap_external>:525: size=536 KiB (+536 KiB), count=5901 (+5901), average=93 B
/opt/UCSF/ChimeraX/lib/python3.7/tracemalloc.py:127: size=371 KiB (+371 KiB), count=4313 (+4313), average=88 B
/opt/UCSF/ChimeraX/lib/python3.7/tracemalloc.py:185: size=307 KiB (+307 KiB), count=4905 (+4905), average=64 B
/opt/UCSF/ChimeraX/lib/python3.7/tracemalloc.py:472: size=255 KiB (+255 KiB), count=4647 (+4647), average=56 B
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/chimerax/atomic/molarray.py:358: size=192 KiB (+192 KiB), count=7 (+7), average=27.4 KiB
/opt/UCSF/ChimeraX/lib/python3.7/tracemalloc.py:113: size=188 KiB (+188 KiB), count=2000 (+2000), average=96 B
}}}
(i.e. no further memory leaked from mmcif).
Might be coincidence, but the number of memory blocks leaked (662380) is just a little more than ten times the number of reflection lines in the file (65726). When opening structure factor files, `mmcif.get_cif_tables()` is called from here: https://github.com/tristanic/chimerax-clipper/blob/1bbe2621ee873b5824e7a314fff9aba39dd6c2d2/src/io/cif_sf_read.py#L154.
Log:
UCSF ChimeraX version: 1.1rc202008312018 (2020-08-31)
© 2016-2020 Regents of the University of California. All rights reserved.
How to cite UCSF ChimeraX
> ui tool show Shell
/opt/UCSF/ChimeraX/lib/python3.7/site-packages/IPython/core/history.py:226:
UserWarning: IPython History requires SQLite, your history will not be saved
warn("IPython History requires SQLite, your history will not be saved")
> open 7jnu structureFactors true
Summary of feedback from opening 7jnu fetched from pdb
---
warning | WARNING: multiple experimental reflection datasets found:
F_meas_au, F_meas_sigma_au,
intensity_meas, intensity_sigma
Automatically choosing "F_meas_au, F_meas_sigma_au".
notes | Resolution: 2.3999931581295297
7jnu title:
New condition to crystallize Thermoanaerobacter ethanolicus Secondary alcohol
dehydrogenases I86A mutant with 2-pentanol and 3-methylcylohexanol [more
info...]
Chain information for 7jnu
---
Chain | Description
1.2/A 1.2/B 1.2/C 1.2/D | Secondary-alcohol dehydrogenase
Non-standard residues in 7jnu #1.2
---
2SL — (2S)-pentan-2-ol ((S)-2-pentanol)
NAP — nadp nicotinamide-adenine-dinucleotide phosphate
(2'-monophosphoadenosine 5'-diphosphoribose)
ZN — zinc ion
> close #1
Deleting Crystallographic maps (7jnu-sf.cif)
Deleting (LIVE) 2mFo-DFc
Deleting (LIVE) mFo-DFc
Deleting (LIVE) 2mFo-DFc_smooth_5
> open 7jnu
7jnu title:
New condition to crystallize Thermoanaerobacter ethanolicus Secondary alcohol
dehydrogenases I86A mutant with 2-pentanol and 3-methylcylohexanol [more
info...]
Chain information for 7jnu #1
---
Chain | Description
A B C D | Secondary-alcohol dehydrogenase
Non-standard residues in 7jnu #1
---
2SL — (2S)-pentan-2-ol ((S)-2-pentanol)
NAP — nadp nicotinamide-adenine-dinucleotide phosphate
(2'-monophosphoadenosine 5'-diphosphoribose)
ZN — zinc ion
> close #1
OpenGL version: 3.3.0 NVIDIA 450.51.06
OpenGL renderer: TITAN Xp/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: Dell Inc.
Model: Precision T5600
OS: CentOS Linux 7 Core
Architecture: 64bit ELF
CPU: 32 Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz
Cache Size: 20480 KB
Memory:
total used free shared buff/cache available
Mem: 62G 6.2G 45G 221M 10G 55G
Swap: 4.9G 0B 4.9G
Graphics:
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [TITAN Xp] [10de:1b02] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:11df]
Kernel driver in use: nvidia
PyQt version: 5.12.3
Compiled Qt version: 5.12.4
Runtime Qt version: 5.12.9
Installed Packages:
alabaster: 0.7.12
appdirs: 1.4.4
Babel: 2.8.0
backcall: 0.2.0
blockdiag: 2.0.1
certifi: 2020.6.20
chardet: 3.0.4
ChimeraX-AddH: 2.1.1
ChimeraX-AlignmentAlgorithms: 2.0
ChimeraX-AlignmentHdrs: 3.2
ChimeraX-AlignmentMatrices: 2.0
ChimeraX-Alignments: 2.1
ChimeraX-Arrays: 1.0
ChimeraX-Atomic: 1.6.1
ChimeraX-AtomSearch: 2.0
ChimeraX-AxesPlanes: 2.0
ChimeraX-BasicActions: 1.1
ChimeraX-BILD: 1.0
ChimeraX-BlastProtein: 1.0.1
ChimeraX-BondRot: 2.0
ChimeraX-BugReporter: 1.0
ChimeraX-BuildStructure: 2.0
ChimeraX-Bumps: 1.0
ChimeraX-BundleBuilder: 1.0
ChimeraX-ButtonPanel: 1.0
ChimeraX-CageBuilder: 1.0
ChimeraX-CellPack: 1.0
ChimeraX-Centroids: 1.1
ChimeraX-ChemGroup: 2.0
ChimeraX-Clashes: 2.0
ChimeraX-Clipper: 0.15.0
ChimeraX-ColorActions: 1.0
ChimeraX-ColorGlobe: 1.0
ChimeraX-CommandLine: 1.1.3
ChimeraX-ConnectStructure: 2.0
ChimeraX-Contacts: 1.0
ChimeraX-Core: 1.1rc202008312018
ChimeraX-CoreFormats: 1.0
ChimeraX-coulombic: 1.0.1
ChimeraX-Crosslinks: 1.0
ChimeraX-Crystal: 1.0
ChimeraX-DataFormats: 1.0
ChimeraX-Dicom: 1.0
ChimeraX-DistMonitor: 1.1
ChimeraX-DistUI: 1.0
ChimeraX-Dssp: 2.0
ChimeraX-EMDB-SFF: 1.0
ChimeraX-ExperimentalCommands: 1.0
ChimeraX-FileHistory: 1.0
ChimeraX-FunctionKey: 1.0
ChimeraX-Geometry: 1.1
ChimeraX-gltf: 1.0
ChimeraX-Graphics: 1.0
ChimeraX-Hbonds: 2.0
ChimeraX-Help: 1.0
ChimeraX-HKCage: 1.0
ChimeraX-IHM: 1.0
ChimeraX-ImageFormats: 1.0
ChimeraX-IMOD: 1.0
ChimeraX-IO: 1.0
ChimeraX-ISOLDE: 1.0.2
ChimeraX-Label: 1.0
ChimeraX-LinuxSupport: 1.0
ChimeraX-ListInfo: 1.0
ChimeraX-Log: 1.1.1
ChimeraX-LookingGlass: 1.1
ChimeraX-Map: 1.0.1
ChimeraX-MapData: 2.0
ChimeraX-MapEraser: 1.0
ChimeraX-MapFilter: 2.0
ChimeraX-MapFit: 2.0
ChimeraX-MapSeries: 2.0
ChimeraX-Markers: 1.0
ChimeraX-Mask: 1.0
ChimeraX-MatchMaker: 1.1
ChimeraX-MDcrds: 2.0
ChimeraX-MedicalToolbar: 1.0.1
ChimeraX-Meeting: 1.0
ChimeraX-MLP: 1.0
ChimeraX-mmCIF: 2.2
ChimeraX-MMTF: 2.0
ChimeraX-Modeller: 1.0
ChimeraX-ModelPanel: 1.0
ChimeraX-ModelSeries: 1.0
ChimeraX-Mol2: 2.0
ChimeraX-Morph: 1.0
ChimeraX-MouseModes: 1.0
ChimeraX-Movie: 1.0
ChimeraX-Neuron: 1.0
ChimeraX-Nucleotides: 2.0
ChimeraX-OpenCommand: 1.2.1
ChimeraX-PDB: 2.1
ChimeraX-PDBBio: 1.0
ChimeraX-Phenix: 0.1
ChimeraX-PickBlobs: 1.0
ChimeraX-Positions: 1.0
ChimeraX-PresetMgr: 1.0
ChimeraX-PubChem: 2.0
ChimeraX-Read-Pbonds: 1.0
ChimeraX-Registration: 1.1
ChimeraX-RemoteControl: 1.0
ChimeraX-ResidueFit: 1.0
ChimeraX-RestServer: 1.0
ChimeraX-RNALayout: 1.0
ChimeraX-RotamerLibMgr: 2.0
ChimeraX-RotamerLibsDunbrack: 2.0
ChimeraX-RotamerLibsDynameomics: 2.0
ChimeraX-RotamerLibsRichardson: 2.0
ChimeraX-SaveCommand: 1.2
ChimeraX-SchemeMgr: 1.0
ChimeraX-SDF: 2.0
ChimeraX-Segger: 1.0
ChimeraX-Segment: 1.0
ChimeraX-SeqView: 2.2
ChimeraX-Shape: 1.0.1
ChimeraX-Shell: 1.0
ChimeraX-Shortcuts: 1.0
ChimeraX-ShowAttr: 1.0
ChimeraX-ShowSequences: 1.0
ChimeraX-SideView: 1.0
ChimeraX-Smiles: 2.0
ChimeraX-SmoothLines: 1.0
ChimeraX-SpaceNavigator: 1.0
ChimeraX-StdCommands: 1.0.4
ChimeraX-STL: 1.0
ChimeraX-Storm: 1.0
ChimeraX-Struts: 1.0
ChimeraX-Surface: 1.0
ChimeraX-SwapAA: 2.0
ChimeraX-SwapRes: 2.0
ChimeraX-TapeMeasure: 1.0
ChimeraX-Test: 1.0
ChimeraX-Toolbar: 1.0
ChimeraX-ToolshedUtils: 1.0
ChimeraX-Tug: 1.0
ChimeraX-UI: 1.2.3
ChimeraX-uniprot: 2.0
ChimeraX-ViewDockX: 1.0
ChimeraX-Vive: 1.1
ChimeraX-VolumeMenu: 1.0
ChimeraX-VTK: 1.0
ChimeraX-WavefrontOBJ: 1.0
ChimeraX-WebCam: 1.0
ChimeraX-WebServices: 1.0
ChimeraX-Zone: 1.0
colorama: 0.4.3
comtypes: 1.1.7
cxservices: 1.0
cycler: 0.10.0
Cython: 0.29.20
decorator: 4.4.2
distlib: 0.3.1
distro: 1.5.0
docutils: 0.16
filelock: 3.0.12
funcparserlib: 0.3.6
grako: 3.16.5
h5py: 2.10.0
html2text: 2020.1.16
idna: 2.10
ihm: 0.16
imagecodecs: 2020.5.30
imagecodecs-lite: 2020.1.31
imagesize: 1.2.0
ipykernel: 5.3.0
ipython: 7.15.0
ipython-genutils: 0.2.0
jedi: 0.17.2
Jinja2: 2.11.2
jupyter-client: 6.1.3
jupyter-core: 4.6.3
kiwisolver: 1.2.0
line-profiler: 2.1.2
lxml: 4.5.1
MarkupSafe: 1.1.1
matplotlib: 3.2.1
msgpack: 1.0.0
netifaces: 0.10.9
networkx: 2.4
numexpr: 2.7.1
numpy: 1.18.5
numpydoc: 1.0.0
openvr: 1.12.501
packaging: 20.4
parso: 0.7.1
pexpect: 4.8.0
pickleshare: 0.7.5
Pillow: 7.1.2
pip: 20.2.2
pkginfo: 1.5.0.1
prompt-toolkit: 3.0.7
psutil: 5.7.0
ptyprocess: 0.6.0
pycollada: 0.7.1
pydicom: 2.0.0
Pygments: 2.6.1
PyOpenGL: 3.1.5
PyOpenGL-accelerate: 3.1.5
pyparsing: 2.4.7
PyQt5-commercial: 5.12.3
PyQt5-sip: 4.19.19
PyQtWebEngine-commercial: 5.12.1
python-dateutil: 2.8.1
pytz: 2020.1
pyzmq: 19.0.2
qtconsole: 4.7.4
QtPy: 1.9.0
RandomWords: 0.3.0
requests: 2.24.0
scipy: 1.4.1
setuptools: 49.4.0
sfftk-rw: 0.6.6.dev0
six: 1.15.0
snowballstemmer: 2.0.0
sortedcontainers: 2.2.2
Sphinx: 3.1.1
sphinxcontrib-applehelp: 1.0.2
sphinxcontrib-blockdiag: 2.0.0
sphinxcontrib-devhelp: 1.0.2
sphinxcontrib-htmlhelp: 1.0.3
sphinxcontrib-jsmath: 1.0.1
sphinxcontrib-qthelp: 1.0.3
sphinxcontrib-serializinghtml: 1.1.4
suds-jurko: 0.6
tables: 3.6.1
tifffile: 2020.6.3
tinyarray: 1.2.2
tornado: 6.0.4
traitlets: 5.0.0
urllib3: 1.25.10
wcwidth: 0.2.5
webcolors: 1.11.1
wheel: 0.34.2
![[ChimeraX Issue Tracking System]](/trac/ChimeraX/chrome/site/ChimeraX-icon.svg){kind=icon}
Confirmed that there is a leak. Was able to read 49449 mmCIF files before running out of memory on a 394 GB machine.