LZ4 compression instead of gzip for maps in session files

[goddard@…]

The following bug report has been submitted:
Platform:        macOS-10.15.7-x86_64-i386-64bit
ChimeraX Version: 1.2.dev202011130113 (2020-11-13 01:13:13 UTC)
Description
Might be desirable to change including of maps in sessions to use lz4 compression or use no compression.  Currently it uses gzip compression which is slow but gives a good size reduction.  LZ4 will be much faster, probably nearly as fast as no compression but give a factor of 2 less size reduction.  Already the entire session file is lz4 compressed so maybe no volume compression is needed size the map data will get lz4 compressed as part of the whole session.

The gzip compession has the advantage of smaller file size.  So it is not clear whether the default should optimize file size or speed.  It might be worth timing a typical case to see how big the speed difference is writing and reading gzip vs lz4 compresssed maps in sessions.

Log:
UCSF ChimeraX version: 1.2.dev202011130113 (2020-11-13)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open /Users/goddard/Downloads/6ugw.pdb format pdb

6ugw.pdb title:  
Crystal structure of the FC fragment of PF06438179/GP1111 an infliximab
biosimilar In A C-centered orthorhombic crystal form, lot A [more info...]  
  
Chain information for 6ugw.pdb #1  
---  
Chain | Description  
A | PF-06438179/GP1111 FC  
  
Non-standard residues in 6ugw.pdb #1  
---  
ACT — acetate ion  
BMA — β-D-mannose  
FUC — α-L-fucose  
GAL — β-D-galactose  
MAN — α-D-mannose  
NAG — N-acetyl-D-glucosamine  
ZN — zinc ion  
  

> molmap #1 5

Opened 6ugw.pdb map 5 as #2, grid size 48,61,43, pixel 1.67, shown at level
0.0987, step 1, values float32  

> save test2.cxs




OpenGL version: 4.1 ATI-3.10.18
OpenGL renderer: AMD Radeon Pro Vega 20 OpenGL Engine
OpenGL vendor: ATI Technologies Inc.Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro15,3
      Processor Name: 8-Core Intel Core i9
      Processor Speed: 2.4 GHz
      Number of Processors: 1
      Total Number of Cores: 8
      L2 Cache (per Core): 256 KB
      L3 Cache: 16 MB
      Hyper-Threading Technology: Enabled
      Memory: 32 GB
      Boot ROM Version: 1037.147.4.0.0 (iBridge: 17.16.16610.0.0,0)

Software:

    System Software Overview:

      System Version: macOS 10.15.7 (19H2)
      Kernel Version: Darwin 19.6.0
      Time since boot: 18 days 18:36

Graphics/Displays:

    Intel UHD Graphics 630:

      Chipset Model: Intel UHD Graphics 630
      Type: GPU
      Bus: Built-In
      VRAM (Dynamic, Max): 1536 MB
      Vendor: Intel
      Device ID: 0x3e9b
      Revision ID: 0x0002
      Automatic Graphics Switching: Supported
      gMux Version: 5.0.0
      Metal: Supported, feature set macOS GPUFamily2 v1

    Radeon Pro Vega 20:

      Chipset Model: Radeon Pro Vega 20
      Type: GPU
      Bus: PCIe
      PCIe Lane Width: x8
      VRAM (Total): 4 GB
      Vendor: AMD (0x1002)
      Device ID: 0x69af
      Revision ID: 0x00c0
      ROM Revision: 113-D2060I-087
      VBIOS Version: 113-D20601MA0T-016
      Option ROM Version: 113-D20601MA0T-016
      EFI Driver Version: 01.01.087
      Automatic Graphics Switching: Supported
      gMux Version: 5.0.0
      Metal: Supported, feature set macOS GPUFamily2 v1
      Displays:
        Color LCD:
          Display Type: Built-In Retina LCD
          Resolution: 2880 x 1800 Retina
          Framebuffer Depth: 24-Bit Color (ARGB8888)
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: No
          Connection Type: Internal

Locale: ('en_US', 'UTF-8')
PyQt version: 5.15.1
Compiled Qt version: 5.15.1
Runtime Qt version: 5.15.1
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    appnope: 0.1.0
    Babel: 2.9.0
    backcall: 0.2.0
    biopython: 1.78
    blockdiag: 2.0.1
    certifi: 2020.11.8
    cftime: 1.2.1
    chardet: 3.0.4
    ChimeraX-AddH: 2.1.3
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.1
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.9.4
    ChimeraX-AtomicLibrary: 1.1
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AtomSearchLibrary: 1.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 1.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.2
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.0
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.1
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.1
    ChimeraX-Cloth: 1.0
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-CommandLine: 1.1.3
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.2.dev202011130113
    ChimeraX-CoreFormats: 1.0
    ChimeraX-coulombic: 1.0.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-DataFormats: 1.0.1
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.0
    ChimeraX-Hbonds: 2.1
    ChimeraX-Help: 1.0
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.0
    ChimeraX-ImageFormats: 1.1
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0
    ChimeraX-Label: 1.0
    ChimeraX-ListInfo: 1.1
    ChimeraX-Log: 1.1.1
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Map: 1.0.2
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.0
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 1.1
    ChimeraX-MDcrds: 2.1
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.0
    ChimeraX-mmCIF: 2.2
    ChimeraX-MMTF: 2.0
    ChimeraX-Modeller: 1.0
    ChimeraX-ModelPanel: 1.0
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.0
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0
    ChimeraX-OpenCommand: 1.3.1
    ChimeraX-PDB: 2.2
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0.1
    ChimeraX-PubChem: 2.0.1
    ChimeraX-Read-Pbonds: 1.0
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.2.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SeqView: 2.2.1
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.0
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.0.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.2.2
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.0
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Tempy: 0.6.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.0
    ChimeraX-ToolshedUtils: 1.0
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.3.1
    ChimeraX-uniprot: 2.0
    ChimeraX-ViewDockX: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.3
    comtypes: 1.1.7
    cxservices: 1.0
    cycler: 0.10.0
    Cython: 0.29.21
    decorator: 4.4.2
    distlib: 0.3.1
    docutils: 0.16
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 2.10.0
    html2text: 2020.1.16
    idna: 2.10
    ihm: 0.17
    imagecodecs: 2020.5.30
    imagecodecs-lite: 2020.1.31
    imagesize: 1.2.0
    ipykernel: 5.3.4
    ipython: 7.18.1
    ipython-genutils: 0.2.0
    jedi: 0.17.2
    Jinja2: 2.11.2
    jupyter-client: 6.1.7
    jupyter-core: 4.6.3
    kiwisolver: 1.3.1
    line-profiler: 2.1.2
    lxml: 4.5.2
    lz4: 3.1.0
    MarkupSafe: 1.1.1
    matplotlib: 3.3.2
    MolecularDynamicsViewer: 1.1
    msgpack: 1.0.0
    netCDF4: 1.5.4
    netifaces: 0.10.9
    networkx: 2.5
    numexpr: 2.7.1
    numpy: 1.19.2
    numpydoc: 1.1.0
    openvr: 1.14.1501
    packaging: 20.4
    parso: 0.7.1
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 7.2.0
    pip: 20.2.3
    pkginfo: 1.5.0.1
    prompt-toolkit: 3.0.8
    psutil: 5.7.2
    ptyprocess: 0.6.0
    pycollada: 0.7.1
    pydicom: 2.0.0
    Pygments: 2.7.1
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.15.1
    PyQt5-sip: 12.8.1
    PyQtWebEngine-commercial: 5.15.1
    python-dateutil: 2.8.1
    pytz: 2020.4
    pyzmq: 20.0.0
    qtconsole: 4.7.7
    QtPy: 1.9.0
    RandomWords: 0.3.0
    requests: 2.24.0
    scipy: 1.5.2
    Send2Trash: 1.5.0
    SEQCROW: 0.20
    setuptools: 50.3.0
    sfftk-rw: 0.6.7.dev1
    six: 1.15.0
    snowballstemmer: 2.0.0
    sortedcontainers: 2.2.2
    Sphinx: 3.2.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 1.0.3
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.4
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2020.9.3
    tinyarray: 1.2.3
    tornado: 6.1
    traitlets: 5.0.5
    urllib3: 1.25.11
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.35.1

[Tom Goddard]

Saving a session including maps with 300 by 300 by 300 map with float values (EMDB 22000) and fit atomic model 6x2b with 21000 atoms took 3.4 seconds, opening the session 0.8 seconds, session file size 100 Mbytes, original map size 108 Mbytes. Gzip compression on the original map from the shell only reduces it to 100 Mbytes, so it doesn't compress much due to noise.

In this case of an experimental map with noise gzip compression does nothing but slow down save and open.

[Tom Goddard]

Saving a session with a molmap map, pdb 6xsb, molmap #1 3 grid .4, map size 334,345,462 taking 210 Mbytes as MRC, session file is 40 Mbytes saved in 2.8 seconds, opened in 0.8 seconds, compressed map is 39 Mbytes.

So with a molmap map the compression is a factor of 5.

With same molmap instead not compressed, and just getting LZ4 compression of entire file, saved session is 43 Mbytes, saved in 0.5 seconds, restored in 0.5 seconds. Small increase in size with big increase in performance.

Seems that uncompressed map is much better choice. With experimental map (emdb 22000) test from previous comment, save was 0.5 seconds, restore 0.25 seconds, file size 109 Mbytes.

[Tom Goddard]

Fixed.

Made maps included in sessions not compressed. Overall session file compression with LZ4 does a much better job, 5 times faster write, 3 times faster read, with similar file size.