Memory leak opening and closing atomic models

[goddard@…]

The following bug report has been submitted:
Platform:        macOS-12.4-arm64-arm-64bit
ChimeraX Version: 1.5.dev202206081745 (2022-06-08 17:45:34 UTC)
Description
ChimeraX leaks a lot of memory opening and closing atomic models.  This makes it painful to run nogui processing scripts on large collections of atomic models because they need to be divided into batches.

I've attached a Python script that demonstrates the memory leak.  Here are some test results opening and closing various models 100 times in nogui mode.

# Starting chimerax in nogui on macOS 12.4 gives 68 Mbytes memory use reported by Mac Activity Monitor.
# For small structure 17gs (0.38 MB cif file) memory use goes to 145 Mbytes.
# For larger structure 7azs (42 MB cif file) memory use goes to 526 Mbytes.
# For a PDB format file 1s72 (9 MB pdb file) memory use goes to 1450 Mbytes.
# For EMDB map 1080 (4 MB ccp4 file) memory use goes to 71 Mbyte.
# For 7azs opening without adding to open models memory use goes to 280 Mbytes.




Log:
UCSF ChimeraX version: 1.5.dev202206081745 (2022-06-08)  
© 2016-2022 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  




OpenGL version: 4.1 Metal - 76.3
OpenGL renderer: Apple M1 Max
OpenGL vendor: Apple

Python: 3.9.11
Locale: UTF-8
Qt version: PyQt6 6.3.0, Qt 6.3.0
Qt runtime version: 6.3.0
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 32 GB
      System Firmware Version: 7459.121.3
      OS Loader Version: 7459.121.3

Software:

    System Software Overview:

      System Version: macOS 12.4 (21F79)
      Kernel Version: Darwin 21.5.0
      Time since boot: 20 days 11:47

Graphics/Displays:

    Apple M1 Max:

      Chipset Model: Apple M1 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 32
      Vendor: Apple (0x106b)
      Metal Family: Supported, Metal GPUFamily Apple 7
      Displays:
        Color LCD:
          Display Type: Built-in Liquid Retina XDR Display
          Resolution: 3456 x 2234 Retina
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: No
          Connection Type: Internal


Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    appnope: 0.1.3
    Babel: 2.10.1
    backcall: 0.2.0
    blockdiag: 3.0.0
    certifi: 2021.10.8
    cftime: 1.6.0
    charset-normalizer: 2.0.12
    ChimeraX-AddCharge: 1.2.3
    ChimeraX-AddH: 2.1.11
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2.1
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.4.2
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.2
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.39.2
    ChimeraX-AtomicLibrary: 7.0
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.1
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.7
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.2
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.2.3
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.1
    ChimeraX-CommandLine: 1.2.3
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.5.dev202206081745
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.1
    ChimeraX-DistMonitor: 1.1.5
    ChimeraX-Dssp: 2.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.2
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.1.2
    ChimeraX-Help: 1.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0
    ChimeraX-Label: 1.1
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.1
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 2.0.6
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.7
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.5.5
    ChimeraX-ModelPanel: 1.3.2
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.2
    ChimeraX-OpenCommand: 1.9
    ChimeraX-PDB: 2.6.6
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0.1
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.6
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.1
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.8
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.0.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1
    ChimeraX-ToolshedUtils: 1.2.1
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.20.2
    ChimeraX-uniprot: 2.2
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.1.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.4
    cxservices: 1.2
    cycler: 0.11.0
    Cython: 0.29.26
    debugpy: 1.6.0
    decorator: 5.1.1
    docutils: 0.17.1
    entrypoints: 0.4
    filelock: 3.4.2
    fonttools: 4.33.2
    funcparserlib: 1.0.0a1
    grako: 3.16.5
    html2text: 2020.1.16
    idna: 3.3
    ihm: 0.27
    imagecodecs: 2022.2.22
    imagesize: 1.3.0
    ipykernel: 6.6.1
    ipython: 7.31.1
    ipython-genutils: 0.2.0
    jedi: 0.18.1
    Jinja2: 3.0.3
    jupyter-client: 7.1.0
    jupyter-core: 4.10.0
    kiwisolver: 1.4.2
    line-profiler: 3.4.0
    lxml: 4.7.1
    lz4: 3.1.10
    MarkupSafe: 2.1.1
    matplotlib: 3.5.1
    matplotlib-inline: 0.1.3
    msgpack: 1.0.3
    nest-asyncio: 1.5.5
    netCDF4: 1.5.8
    networkx: 2.6.3
    numexpr: 2.8.1
    numpy: 1.22.1
    openvr: 1.16.802
    packaging: 21.0
    ParmEd: 3.4.3
    parso: 0.8.3
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 9.0.1
    pip: 21.3.1
    pkginfo: 1.8.2
    prompt-toolkit: 3.0.29
    psutil: 5.9.0
    ptyprocess: 0.7.0
    pycollada: 0.7.2
    pydicom: 2.2.2
    Pygments: 2.11.2
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.8
    PyQt6: 6.3.0
    PyQt6-Qt6: 6.3.0
    PyQt6-sip: 13.3.1
    PyQt6-WebEngine: 6.3.0
    PyQt6-WebEngine-Qt6: 6.3.0
    python-dateutil: 2.8.2
    pytz: 2022.1
    pyzmq: 22.3.0
    qtconsole: 5.3.0
    QtPy: 2.0.1
    requests: 2.27.1
    scipy: 1.7.3
    setuptools: 59.8.0
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    Sphinx: 4.3.2
    sphinx-autodoc-typehints: 1.15.2
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    suds-community: 1.0.0
    tables: 3.7.0
    tifffile: 2021.11.2
    tinyarray: 1.2.4
    tornado: 6.1
    traitlets: 5.1.1
    urllib3: 1.26.9
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.37.1
    wheel-filename: 1.3.0
File attachment: test_memleak.py

test_memleak.py​

[goddard@…]

Added by email2trac

[Eric Pettersen]

According to tracemalloc, the top 10 differences after opening and closing 7azs once are:

Top 10 differences:
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/atomic/molc.py:80: size=3504 KiB (+3504 KiB), count=53655 (+53655), average=67 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/atomic/molarray.py:159: size=2525 KiB (+2525 KiB), count=11 (+11), average=230 KiB
<frozen importlib._bootstrap>:228: size=549 KiB (+549 KiB), count=6520 (+6520), average=86 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/atomic/molarray.py:86: size=261 KiB (+261 KiB), count=21 (+21), average=12.5 KiB
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/atomic/molarray.py:1393: size=40.4 KiB (+40.4 KiB), count=171 (+171), average=242 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py:510: size=24.4 KiB (+24.4 KiB), count=393 (+393), average=63 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/orderedset.py:36: size=20.4 KiB (+20.4 KiB), count=134 (+134), average=156 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/_internal.py:255: size=19.8 KiB (+19.8 KiB), count=169 (+169), average=120 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/graphics/drawing.py:255: size=14.2 KiB (+14.2 KiB), count=26 (+26), average=558 B
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/atomic/molobject.py:1461: size=10.1 KiB (+10.1 KiB), count=9 (+9), average=1146 B

[Tom Goddard]

The tracemalloc info in the previous comment is not too revealing. The top two memory increases (3.5 MB and 2.5 MB) are the Python that fetches a C++ molecular attribute, and the Python that slices a numpy array for a atomic model Collection (ie array data). This does not reveal which molecular attributes were allocated and not freed. It may be that some attributes were not freed because the Python Structure instance does not get deleted, or it could be that the Python wrapper for certain atomic C++ attributes incorrectly sets the reference count so that those attribute values never get deleted.

[Eric Pettersen]

It took me two days to track this down, but it's fixed now. The key and values of the metadata dictionary attribute had an extra reference count.

[Tom Goddard]

Cool. I can believe that was har to find! No doubt memory is still leaking. In my original test reporting this the descriptions says I opened and closed 7azs 100 times and got a 0.5 Gbyte memory increase. Did you measure what the leakage is after your fix?

[Eric Pettersen]

Seems to be 0.3 GB now. I nuked the first leak on the tracemalloc list, but I need to deal with at least the second one too and possibly some of the others.

[Eric Pettersen]

The remaining memory leak was a false flag. The drawing code keeps references to pseudobond groups that don't get released until the next frame draw, so memory use seems to accumulate unless you put a "wait 1" in the loop. With "wait 1" in the script loop, memory use before and after execution (100 open/close of 7azs) was:

With GUI: 253.5 / 293.3
No GUI: 136.9 / 163.5

Have attached a revised script with the wait command, as well as some useful memory debugging code (commented out) for reference

Am reassigning to T.G. so he can assess if he wants to improve the drawing code's behavior. It would seem kind of important since anyone writing a structure-processing script will likely trip over this gotcha.

[Eric Pettersen]

has "wait 1" in loop

[Tom Goddard]

Can you provide more details on "The drawing code keeps references to pseudobond groups". Where are the references? And what code removes them? I'm guessing maybe it is the redraw manager which tracks which drawings have changed so it can decide whether to update shadows based on whether any of those drawings cast shadows. If that is it, then it would be pretty easy to fix by not remembering the drawings and just remember if shadows need updating.

[Eric Pettersen]

As per the discussion in the hallway, I will work up a script that demonstrates the pseudobond group references and that T.G. can use to determine if code changes eliminates the refs.

[Eric Pettersen]

Okay, uploaded a script that can be run in nogui mode. It opens and closes 7azs and demonstrates that there are three pseudobond groups with live references after that, and shows the referrer for the first live pbg. Then it does a "wait 1" and shows that the references go away.

[Tom Goddard]

Fixed the pseudobond group references mentioned in the previous comment. The graphics code kept a set of drawings whose shape has changed so that on the next frame it can decide if shadows need to be updated. I changed it so it only keeps a flag indicating if shadows need updating.

Reassigning to Eric in case there is anything more he wants to do to reduce memory leaks.

[Eric Pettersen]

Fixed another leak (in residues.uniq_sequences) where I was again bitten by the list insert/append difference (append [only] needs a DECREF).

Also modified the test script (attached) to open/close the structure once before starting tracemalloc so that structure-related module imports aren't counted as "leaks". This also implies that when running the script to monitor increased memory usage that you should open/close the structure once to get a baseline that includes the module imports.

The remaining leaks (if that's what they are) are smaller than the module imports and not obviously within the ChimeraX code itself, so I am closing this ticket. If another large leak shows up, another ticket could be opened.

Fix: ​https://github.com/RBVI/ChimeraX/commit/237b3070d8ff4005d4aabed5fba82441ae5dc173

[Tom Goddard]

Good work!