MemoryError despite seemingly moderate memory usage

[m-schmid@…]

The following bug report has been submitted:
Platform:        Windows-10-10.0.18362
ChimeraX Version: 1.2.dev202103272340 (2021-03-27 23:40:01 UTC)
Description
This time it didn't crash until 64% memory. One thing; the file is an unsigned integer 8-bit file, but the error says "data type int32". Is this the internal representation of the data?

Log:
UCSF ChimeraX version: 1.2.dev202103272340 (2021-03-27)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  
Unable to load numpy_formathandler accelerator from OpenGL_accelerate  

> vr true

started SteamVR rendering  

> open X:/TOXOPLASMA/CONOID_HEIGHT/protruded-eman/TPE-16-axy150u8.hdf

Opened TPE-16-axy150u8.hdf as #101, grid size 1024,1024,300, pixel 14.2, shown
at step 1, values uint8  

> volume #101 region 0,0,0,1023,1023,299 step 4

> volume #101 region 0,0,0,1023,1023,299 step 4

> volume #101 step 1

> vr false

> volume #101 level 92

> volume #101 change image level 81,0 level 99.99,0.8 level 212,1

> color height #1 center 0,0,0 axis z palette rainbow

> color height #1 center 0,0,0 axis z palette rainbow

> color height #1 center 0,0,0 axis z palette rainbow

> close session

> open X:/TOXOPLASMA/CONOID_HEIGHT/protruded-eman/TPE-16-axy150u8.hdf

Opened TPE-16-axy150u8.hdf as #1, grid size 1024,1024,300, pixel 14.2, shown
at step 1, values uint8  

> volume #1 region 0,0,0,1023,1023,299 step 4

> volume #1 region 0,0,0,1023,1023,299 step 4

> volume #1 step 1

> color height #1 center 0,0,0 axis z palette rainbow

> volume #1 level 92

> volume #1 change image level 81,0 level 99.99,0.8 level 212,1

> surface dust #1 size 200

> volume #1 level 91

An error occurred in drawing the scene. Redrawing graphics is now stopped to
avoid a continuous stream of error messages. To restart graphics use the
command "graphics restart" after changing the settings that caused the error.  
  
Traceback (most recent call last):  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\core\updateloop.py", line 73, in draw_new_frame  
view.draw(check_for_changes = False)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 165, in draw  
self._draw_scene(camera, drawings)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 214, in _draw_scene  
self._update_projection(camera, vnum)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 886, in _update_projection  
near_far = self.near_far_distances(camera, view_num)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 901, in near_far_distances  
near, far = self._near_far_bounds(cp, vd)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 914, in _near_far_bounds  
b = self.drawing_bounds(allow_drawing_changes = False)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\view.py", line 608, in drawing_bounds  
dm.cached_drawing_bounds = b = self.drawing.bounds()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 947, in bounds  
dbounds = [d.bounds() for d in self.child_drawings()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 947, in <listcomp>  
dbounds = [d.bounds() for d in self.child_drawings()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 947, in bounds  
dbounds = [d.bounds() for d in self.child_drawings()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 947, in <listcomp>  
dbounds = [d.bounds() for d in self.child_drawings()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 964, in bounds  
sb = self.geometry_bounds()  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\chimerax\graphics\drawing.py", line 990, in geometry_bounds  
vshown = numpy.unique(self.triangles[tmask,:])  
File "<__array_function__ internals>", line 5, in unique  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\numpy\lib\arraysetops.py", line 261, in unique  
ret = _unique1d(ar, return_index, return_inverse, return_counts)  
File "C:\Program Files\ChimeraX 1.2.dev202103272340\bin\lib\site-
packages\numpy\lib\arraysetops.py", line 314, in _unique1d  
ar = np.asanyarray(ar).flatten()  
MemoryError: Unable to allocate 697. MiB for an array with shape (182828412,)
and data type int32  
  




OpenGL version: 3.3.0 NVIDIA 456.71
OpenGL renderer: GeForce GTX 1080/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: ASUS
Model: All Series
OS: Microsoft Windows 10 Pro (Build 18363)
Memory: 17,086,476,288
MaxProcessMemory: 137,438,953,344
CPU: 16 Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz
OSLanguage: en-US
Locale: ('en_US', 'cp1252')
PyQt5 5.15.2, Qt 5.15.2
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.9.0
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2020.12.5
    cftime: 1.4.1
    chardet: 3.0.4
    ChimeraX-AddCharge: 1.0
    ChimeraX-AddH: 2.1.4
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.12
    ChimeraX-AtomicLibrary: 3.0
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AtomSearchLibrary: 1.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 1.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.5.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.1
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.1
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.2.1
    ChimeraX-CommandLine: 1.1.3
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.2.dev202103272340
    ChimeraX-CoreFormats: 1.0
    ChimeraX-coulombic: 1.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.1
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.0
    ChimeraX-Hbonds: 2.1
    ChimeraX-Help: 1.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.0
    ChimeraX-ImageFormats: 1.1
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-Label: 1.0
    ChimeraX-ListInfo: 1.1
    ChimeraX-Log: 1.1.2
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.0.2
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.0
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 1.1
    ChimeraX-MDcrds: 2.2
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.3
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.0
    ChimeraX-ModelPanel: 1.0.1
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0
    ChimeraX-OpenCommand: 1.5
    ChimeraX-PDB: 2.3.1
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0.1
    ChimeraX-PubChem: 2.0.1
    ChimeraX-ReadPbonds: 1.0
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.4
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SeqView: 2.3
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.0
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.0.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.3
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.0.1
    ChimeraX-ToolshedUtils: 1.1
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.7.1
    ChimeraX-uniprot: 2.1
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.3
    comtypes: 1.1.7
    cxservices: 1.0
    cycler: 0.10.0
    Cython: 0.29.21
    decorator: 4.4.2
    distlib: 0.3.1
    docutils: 0.16
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 2.10.0
    html2text: 2020.1.16
    idna: 2.10
    ihm: 0.17
    imagecodecs: 2020.5.30
    imagesize: 1.2.0
    ipykernel: 5.3.4
    ipython: 7.18.1
    ipython-genutils: 0.2.0
    jedi: 0.17.2
    Jinja2: 2.11.2
    jupyter-client: 6.1.7
    jupyter-core: 4.7.1
    kiwisolver: 1.3.1
    line-profiler: 2.1.2
    lxml: 4.6.2
    lz4: 3.1.0
    MarkupSafe: 1.1.1
    matplotlib: 3.3.2
    msgpack: 1.0.0
    netCDF4: 1.5.4
    networkx: 2.5
    numexpr: 2.7.3
    numpy: 1.19.2
    numpydoc: 1.1.0
    openvr: 1.14.1501
    packaging: 20.9
    ParmEd: 3.2.0
    parso: 0.7.1
    pickleshare: 0.7.5
    Pillow: 7.2.0
    pip: 21.0.1
    pkginfo: 1.5.0.1
    prompt-toolkit: 3.0.18
    psutil: 5.7.2
    pycollada: 0.7.1
    pydicom: 2.0.0
    Pygments: 2.7.1
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.15.2
    PyQt5-sip: 12.8.1
    PyQtWebEngine-commercial: 5.15.2
    python-dateutil: 2.8.1
    pytz: 2021.1
    pywin32: 228
    pyzmq: 22.0.3
    qtconsole: 4.7.7
    QtPy: 1.9.0
    RandomWords: 0.3.0
    requests: 2.24.0
    scipy: 1.5.2
    setuptools: 50.3.2
    sfftk-rw: 0.6.7.dev1
    six: 1.15.0
    snowballstemmer: 2.1.0
    sortedcontainers: 2.2.2
    Sphinx: 3.2.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 1.0.3
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.4
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2020.9.3
    tinyarray: 1.2.3
    tornado: 6.1
    traitlets: 5.0.5
    urllib3: 1.25.11
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.36.0
    wheel-filename: 1.3.0
    WMI: 1.5.1

[Tom Goddard]

The memory error is making an int32 array of triangles (3 integer indices into a list of vertices). So the int32 data type is unrelated to your uint8 map data type. The problem surely is that ChimeraX is being limited to 4 Gbytes on your machine with 16 Gbytes and we need to figure out why Windows is limiting it.

[m-schmid@…]

I tried opening an old session file that had 9 maps open at once, and used to work. Bug report submitted.

On 4/6/21, 11:12 AM, "ChimeraX" <ChimeraX-bugs-admin@cgl.ucsf.edu> wrote:

    #4462: MemoryError despite seemingly moderate memory usage
    ----------------------------------+-------------------------
              Reporter:  m-schmid@…   |      Owner:  Tom Goddard
                  Type:  defect       |     Status:  assigned
              Priority:  normal       |  Milestone:
             Component:  Volume Data  |    Version:
            Resolution:               |   Keywords:
            Blocked By:               |   Blocking:
    Notify when closed:               |   Platform:  all
               Project:  ChimeraX     |
    ----------------------------------+-------------------------

    Comment (by Tom Goddard):

     The memory error is making an int32 array of triangles (3 integer indices
     into a list of vertices).  So the int32 data type is unrelated to your
     uint8 map data type.  The problem surely is that ChimeraX is being limited
     to 4 Gbytes on your machine with 16 Gbytes and we need to figure out why
     Windows is limiting it.

    --
    Ticket URL: <https://plato.cgl.ucsf.edu/trac/ChimeraX/ticket/4462#comment:2>
    ChimeraX <http://www.rbvi.ucsf.edu/chimerax/>
    ChimeraX Issue Tracker

[goddard@…]

Hi Mike,

  This is very likely a problem with your Windows 10 computer, nothing to do with ChimeraX.  Windows 10 is simply not giving ChimeraX the memory it asks for even though it is using far less than the 16  Gbytes you have installed.

  Looking online there are various reasons this might happen.  First is your Windows 10 Pro 32-bit or 64-bit?  If it is 32-bit then no process can use more than 4 Gbytes.  I didn't even know they make 32-bit operating systems anymore.  If things used to work for you then unless Windows 10 got reinstalled it must be 64-bit.  But check it.

  Next online people describe a msconfig setting that can limit maximum memory use.  Here's an example of that discussion.  You can check whether the maximum memory use is being limited.

https://answers.microsoft.com/en-us/windows/forum/windows_10-performance-winpc/only-4gb-usable-of-16gb-on-windows-10-64-bit/93c2c8c0-867d-4882-bd16-778de7630cb6 <https://answers.microsoft.com/en-us/windows/forum/windows_10-performance-winpc/only-4gb-usable-of-16gb-on-windows-10-64-bit/93c2c8c0-867d-4882-bd16-778de7630cb6>

Again I have no idea how that would have changed on your computer.

  Another problem people discuss online is that they have 16 Gbytes installed but apparently only 4 Gbytes can be used due to memory failure.  Sometimes they reinstall the memory to get it to work.

  I think ChimeraX 1.1 used to work for you on these files.  If you run ChimeraX 1.1 and it now gets out of memory errors, since ChimeraX 1.1 did not change, it is very likely that Windows 10 or computer somehow has a memory problem.  There is a small possibility that you saved some setting like changing the maximum volume data cache size in ChimeraX that could increase memory use.  But when you run ChimeraX it fails when Task Manager says not too much memory is being used -- so it all points to Windows 10 or the computer hardware having a memory problem.

	Tom

[m-schmid@…]

Hi-
Thanks. Quite a few things to try.
First I will try to run a ChimeraX earlier version on that session file.
Well, that didn’t work either. Bug report(s?) sent.
Next I’ll try rebooting the computer.
Mike

From: Tom Goddard <goddard@sonic.net>
Date: Tuesday, April 6, 2021 at 3:34 PM
To: "Schmid, Michael F." <m-schmid@slac.stanford.edu>
Cc: "ChimeraX-bugs@cgl.ucsf.edu" <ChimeraX-bugs@cgl.ucsf.edu>
Subject: Re: [ChimeraX] #4462: MemoryError despite seemingly moderate memory usage

Hi Mike,

  This is very likely a problem with your Windows 10 computer, nothing to do with ChimeraX.  Windows 10 is simply not giving ChimeraX the memory it asks for even though it is using far less than the 16  Gbytes you have installed.

  Looking online there are various reasons this might happen.  First is your Windows 10 Pro 32-bit or 64-bit?  If it is 32-bit then no process can use more than 4 Gbytes.  I didn't even know they make 32-bit operating systems anymore.  If things used to work for you then unless Windows 10 got reinstalled it must be 64-bit.  But check it.

  Next online people describe a msconfig setting that can limit maximum memory use.  Here's an example of that discussion.  You can check whether the maximum memory use is being limited.

https://answers.microsoft.com/en-us/windows/forum/windows_10-performance-winpc/only-4gb-usable-of-16gb-on-windows-10-64-bit/93c2c8c0-867d-4882-bd16-778de7630cb6

Again I have no idea how that would have changed on your computer.

  Another problem people discuss online is that they have 16 Gbytes installed but apparently only 4 Gbytes can be used due to memory failure.  Sometimes they reinstall the memory to get it to work.

  I think ChimeraX 1.1 used to work for you on these files.  If you run ChimeraX 1.1 and it now gets out of memory errors, since ChimeraX 1.1 did not change, it is very likely that Windows 10 or computer somehow has a memory problem.  There is a small possibility that you saved some setting like changing the maximum volume data cache size in ChimeraX that could increase memory use.  But when you run ChimeraX it fails when Task Manager says not too much memory is being used -- so it all points to Windows 10 or the computer hardware having a memory problem.

Tom

[Tom Goddard]

Rebooting the computer fixed the problem as reported in #4473.