rainbow command ignores palette alpha values

[Elaine Meng]

The following bug report has been submitted:
Platform:        macOS-13.0.1-arm64-arm-64bit
ChimeraX Version: 1.5 (2022-11-24 00:03:27 UTC)
Description
Rainbow command is ignoring palette alpha values.

Log:
UCSF ChimeraX version: 1.5 (2022-11-24)  
© 2016-2022 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> open 4hhb

Summary of feedback from opening 4hhb fetched from pdb  
---  
note | Fetching compressed mmCIF 4hhb from
http://files.rcsb.org/download/4hhb.cif  
  
4hhb title:  
The crystal structure of human deoxyhaemoglobin At 1.74 angstroms resolution
[more info...]  
  
Chain information for 4hhb #1  
---  
Chain | Description | UniProt  
A C | Hemoglobin subunit alpha | HBA_HUMAN  
B D | Hemoglobin subunit beta | HBB_HUMAN  
  
Non-standard residues in 4hhb #1  
---  
HEM — protoporphyrin IX containing Fe (HEME)  
PO4 — phosphate ion  
  

> rainbow chains palette rgba(0.0,1.0,1.0,1.0):rgba(0.0,0.0,1.0,0.2)

> rainbow chains palette #00ffffff:#0000ff20




OpenGL version: 4.1 Metal - 83
OpenGL renderer: Apple M1 Max
OpenGL vendor: Apple

Python: 3.9.11
Locale: UTF-8
Qt version: PyQt6 6.3.1, Qt 6.3.1
Qt runtime version: 6.3.2
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,2
      Model Number: MK1H3LL/A
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 32 GB
      System Firmware Version: 8419.41.10
      OS Loader Version: 8419.41.10

Software:

    System Software Overview:

      System Version: macOS 13.0.1 (22A400)
      Kernel Version: Darwin 22.1.0
      Time since boot: 55 days, 17 hours, 30 minutes

Graphics/Displays:

    Apple M1 Max:

      Chipset Model: Apple M1 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 32
      Vendor: Apple (0x106b)
      Metal Support: Metal 3
      Displays:
        Color LCD:
          Display Type: Built-in Liquid Retina XDR Display
          Resolution: 3456 x 2234 Retina
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: No
          Connection Type: Internal


Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    appnope: 0.1.3
    asttokens: 2.1.0
    Babel: 2.11.0
    backcall: 0.2.0
    blockdiag: 3.0.0
    build: 0.8.0
    certifi: 2021.10.8
    cftime: 1.6.2
    charset-normalizer: 2.1.1
    ChimeraX-AddCharge: 1.4
    ChimeraX-AddH: 2.2.1
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2.1
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.6
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.3
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-ArtiaX: 0.3
    ChimeraX-Atomic: 1.41.5
    ChimeraX-AtomicLibrary: 8.0.3
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.3
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.1
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.7.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.3.1
    ChimeraX-ChangeChains: 1.0.2
    ChimeraX-CheckWaters: 1.3
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.2.4
    ChimeraX-Clipper: 0.19.0.dev0
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.2
    ChimeraX-CommandLine: 1.2.4
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.5
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.1
    ChimeraX-DistMonitor: 1.3
    ChimeraX-DockPrep: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.2
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-Label: 1.1.7
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.2
    ChimeraX-Map: 1.1.2
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.1
    ChimeraX-MatchMaker: 2.0.9
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.8
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.6
    ChimeraX-ModelPanel: 1.3.6
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.1
    ChimeraX-MouseModes: 1.1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.9.1
    ChimeraX-PDB: 2.6.8
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PhenixUI: 1.0.4
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0.1
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.7.2
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.1
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.10
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.1.3
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.1
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.24.3
    ChimeraX-uniprot: 2.2.1
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.1.4
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.1
    ChimeraX-WebServices: 1.1.0
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.5
    cxservices: 1.2
    cycler: 0.11.0
    Cython: 0.29.32
    debugpy: 1.6.4
    decorator: 5.1.1
    Deprecated: 1.2.13
    docutils: 0.19
    entrypoints: 0.4
    executing: 1.2.0
    filelock: 3.7.1
    fonttools: 4.38.0
    funcparserlib: 1.0.1
    future: 0.18.2
    grako: 3.16.5
    graphtools: 1.5.2
    h5py: 3.7.0
    html2text: 2020.1.16
    idna: 3.4
    ihm: 0.33
    imagecodecs: 2022.7.31
    imageio: 2.22.1
    imagesize: 1.4.1
    importlib-metadata: 5.1.0
    ipykernel: 6.15.3
    ipython: 8.4.0
    ipython-genutils: 0.2.0
    jedi: 0.18.1
    Jinja2: 3.1.2
    joblib: 1.2.0
    jupyter-client: 7.3.4
    jupyter-core: 5.1.0
    kiwisolver: 1.4.4
    line-profiler: 3.5.1
    llvmlite: 0.39.1
    lxml: 4.9.1
    lz4: 4.0.2
    MarkupSafe: 2.1.1
    matplotlib: 3.5.2
    matplotlib-inline: 0.1.6
    mrcfile: 1.4.3
    msgpack: 1.0.4
    nest-asyncio: 1.5.6
    netCDF4: 1.6.0
    networkx: 2.8.5
    numba: 0.56.2
    numexpr: 2.8.4
    numpy: 1.23.1
    openvr: 1.23.701
    packaging: 21.3
    pandas: 1.4.3
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pexpect: 4.8.0
    phate: 1.0.8
    pickleshare: 0.7.5
    Pillow: 9.2.0
    pip: 22.2.2
    pkginfo: 1.8.3
    platformdirs: 2.5.4
    prompt-toolkit: 3.0.33
    psutil: 5.9.1
    ptyprocess: 0.7.0
    pure-eval: 0.2.2
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.12.0
    PyGSP: 0.5.1
    pynndescent: 0.5.7
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.9
    PyQt6-commercial: 6.3.1
    PyQt6-Qt6: 6.3.2
    PyQt6-sip: 13.4.0
    PyQt6-WebEngine-commercial: 6.3.1
    PyQt6-WebEngine-Qt6: 6.3.2
    pyqtgraph: 0.13.1
    python-dateutil: 2.8.2
    pytz: 2022.6
    PyWavelets: 1.4.1
    PyYAML: 6.0
    pyzmq: 24.0.1
    qtconsole: 5.3.1
    QtPy: 2.3.0
    RandomWords: 0.4.0
    requests: 2.28.1
    s-gd2: 1.8
    scikit-image: 0.19.3
    scikit-learn: 1.1.2
    scipy: 1.9.0
    scprep: 1.2.1
    Send2Trash: 1.8.0
    setuptools: 59.8.0
    setuptools-scm: 7.0.5
    sfftk-rw: 0.7.2
    six: 1.16.0
    sklearn: 0.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    Sphinx: 5.1.1
    sphinx-autodoc-typehints: 1.19.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    stack-data: 0.6.2
    starfile: 0.4.11
    superqt: 0.3.3
    tables: 3.7.0
    tasklogger: 1.2.0
    threadpoolctl: 3.1.0
    tifffile: 2022.7.31
    tinyarray: 1.2.4
    tomli: 2.0.1
    torch: 1.12.1
    tornado: 6.2
    tqdm: 4.64.1
    traitlets: 5.3.0
    typing-extensions: 4.3.0
    urllib3: 1.26.13
    wcwidth: 0.2.5
    webcolors: 1.12
    wheel: 0.37.1
    wheel-filename: 1.4.1
    wiggle: 0.2.2
    wrapt: 1.14.1
    zipp: 3.11.0

[Tom Goddard]

Fixed.

The rainbow command is the same as "color sequential". In general, the color command changes atom and ribbon colors without altering their transparency unless you use the transparency option or explicitly specify a color with transparency, such as rgba(1.0, 1.0, 0.0, 0.5) or #ffff0080. Palettes arguments do not know whether transparency was explicitly specified. I've made rainbow and color sequential use the palette transparency if the palette has at least one transparent color, otherwise it preserves the existing transparencies of atoms and ribbons. If the transparency option is specified it overrides the palette transparency values. This behavior seems the best. It allows using the transparency values in a palette, and also allows keeping current transparency values by using an opaque palette or using the palette and making the atoms and ribbons opaque by adding the option "transparency 0".

The color sequential documentation should describe the setting of transparency with palettes.

[Elaine Meng]

Rainbow (color sequential) is only one of several commands that use palettes:  coulombic, mlp, color byattribute, color  sample, color electrostatic, color gradient, color radial, color cylindrical, color height, maybe others.  Is this fixed for all such uses, or only for rainbow? 

These all refer to the same palette documentation, which in turn says that all types of color specifications (rgba, hex, and other methods that explicitly assign transparency) can be used to define the palette.

If you mean it should explain that the "transparency" command option will take precedence, yes, I will definitely add that information.

[Tom Goddard]

My changes apply only to the rainbow and "color sequential" commands.  I'd suggest that this text in the color and rainbow docs

"The transparency option sets transparency to the specified percent, where 0% is completely opaque and 100% is completely transparent. Existing transparency is not changed unless this option is used or the colorname explicitly specifies alpha (opacity). An alpha value overrides this option if both are given. Note that the transparency command can be used to set transparency apart from color. There are limitations in rendering transparency."

be augmented to describe how specifying the palette option effects transparency.

I looked at a few other commands that take a palette option to see what they do to transparency.   Commands mlp, "color sample" (coloring by map value), "color electrostatic", "color gradient" ignore palette transparency and uses the single color transparency for the surface or the specified transparency option value.  The coulombic command, color byattribute, color radial, color cylindrical, color height always use the palette transparency and never keep the current transparency.  The colorings that automatically update when the surface shape changes will not be able to preserve current surface transparency.  In summary we seem to have 3 different ways to handle transparency with palettes.  Modulating transparency is so rarely used that we haven't given it much attention.

[Elaine Meng]

I was surprised that any palette-using commands did not honor their transparency.  In my mental representation of how it worked, it would be exactly analogous to simple single coloring, it uses any transparencies in the specified color (or colors in the case of a palette) unless overridden with a separate transparency option.

To try to make everything clearer in the overly long color.html page, I have separate sections documenting simple coloring vs. palette-using coloring.  The text you quoted is from the simple coloring section.  I didn't want to be overly duplicative (since the main problem with this color.html file is too many words) so the palette-coloring sections simply referred back to that single-coloring transparency explanation.  I thought it was clear enough that the transparency option would override whether or not you had a single color or multiple colors, but you are saying it is not.

However, I did not realize palette transparency is mostly ignored.  So  I will have to think about how to change the palette documentation in general, because all of those commands refer back to the same palette documentation and the same description of transparency.  Now I have to explain that rainbow is the only one that does NOT ignore palette transparency, so it is more complicated.

This assumes you are not going to "fix" transparent-palette handling for all those other commands.  If you do plan to change any other commands, let me know.

[Tom Goddard]

It would be useful to document the palette transparency behavior.  I agree that the transparency option overriding the palette transparencies is pretty obvious, still worth saying.  And it is somewhat complex because there are at least 3 different ways commands are handling palette transparency.

When you use command "color blue" this preserves existing transparency.  When you say "rainbow palette redblue" I would probably expect it to behave similarly, changing only colors not transparency.  But if you use a palette option or a single color that is not opaque I would expect it would replace the current transparencies.  That is how I made rainbow work -- it wasn't doing that before -- in fact it had different behavior depending on whether rainbow residues or rainbow chain was used.  I think this is probably the most consistent and intuitive and flexible way for palettes to change transparency.  The other commands that use palettes don't support the two main use cases: keeping current transparencies, or replacing current transparencies.  They only allow one of those possibilities.  Ideally they would allow both (where possible, ie when volume surfaces don't change shape).  But it is so rarely used I don't plan on fixing those now.  We could make another ticket to document the commands that handle palette transparency not too so well.

[Elaine Meng]

I'm too confused about which commands act which way to document it correctly.  Is it true that all of these ignore palette transparency?

coulombic, mlp, color byattribute, color  sample, color electrostatic, color gradient, color radial, color cylindrical, color
height

Which of them ignore pre-existing transparency? 

[Tom Goddard]

The commands you list all ignore the current surface transparency. Some of them use the palette transparency and some do not, as noted in comment 4 above:

"The coulombic command, color byattribute, color radial, color cylindrical, color height always use the palette transparency and never keep the current transparency."

"Commands mlp, "color sample" (coloring by map value), "color electrostatic", "color gradient" ignore palette transparency and uses the single color transparency for the surface or the specified transparency option value."

[Elaine Meng]

Ugh, it is repugnant and messy, but hopefully thorough and correct.  Instead of referring to a single description, now I have separate descriptions of palette transparency handling and how it interacts with the transparency option (if any) and pre-existing transparency (if any) in many different places:

simple coloring (single color)
rainbow (color sequential)
color by attribute
color by map value (sample, electrostatic, gradient)
color by distance (height, cylindrical, radial) (diff from byattribute since there is no transparency option)
coulombic
mlp

Looks like
(A) docs were wrong for simple coloring, transparency option wins, not the color def.  I fixed that too.
(B)  Surface Color colorwells don't show opacity in the Color Editor so I didn't have to do anything with that page.

[Tom Goddard]

Thanks!  The Surface Color panel should allow transparency in its color buttons.  That was an oversight (the default color button disallows transparency).  I've fixed that.

[Elaine Meng]

Maybe it shouldn't?  If I understood correctly, half of the options (map, electrostatic, gradient) will ignore the transparency anyway.

[Tom Goddard]

Right. Sorry to fiddle with this after you worked on the docs. I think it would be better to make mlp, "color sample" (coloring by map value), "color electrostatic", "color gradient" all use the palette transparency, and if the transparency option is specified then it overrides the palette. That would make all the command besides rainbow and its equivalent color sequential behave in the same way. Is that ok?

[Elaine Meng]

I'm OK with re-revising the docs since that would be nicer behavior -- let me know if you do it -- just note that "mlp" does not have a "transparency" option currently.

[Tom Goddard]

Ok, I'll have it changed before I leave work today, including adding a transparency option to mlp.

[Elaine Meng]

Also the color radial/height/cylindrical command does not have a transparency option either.

[Elaine Meng]

The coulombic command also does not have a transparency option.

[Tom Goddard]

Ok, I added transparency options to mlp, color radial, color cylindrical, color height, and made them use the palette transparency values if the transparency option is not specified.

I did not add a transparency option to coulombic. That is Eric's code and I'll let him add that if he wants to. I don't think these transparency options are essential. You can always issue a transparency command after the coloring command. It is just a convenience.