Bundles not installing

[pett]

The following bug report has been submitted:
Platform:        macOS-13.3.1-arm64-arm-64bit
ChimeraX Version: 1.7.dev202306142048 (2023-06-14 20:48:12 UTC)
Description
Installing the bundles in the daily build claims sucessful installation but then says "No change in list of installed bundles" and the capabilities of that bundle don't show up and the bundle does not appear in the list of installed bundles.  I have experienced this with the NIHPresets bundle and Elaine with the PhenixUI bundle.


Log:
UCSF ChimeraX version: 1.7.dev202306142048 (2023-06-14)  
© 2016-2023 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> toolshed show

Downloading bundle ChimeraX_NIHPresets-1.1.11-py3-none-any.whl  

Errors may have occurred when running pip:  

pip standard error:  
\---  
  
[notice] A new release of pip is available: 23.0 -> 23.1.2  
[notice] To update, run:
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/MacOS/ChimeraX -m pip
install --upgrade pip  
\---  

pip standard output:  
\---  
Looking in indexes: https://pypi.org/simple,
https://cxtoolshed.rbvi.ucsf.edu/pypi/  
Processing /Users/pett/Downloads/ChimeraX_NIHPresets-1.1.11-py3-none-any.whl  
Requirement already satisfied: ChimeraX-Core~=1.3 in
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-
packages (from ChimeraX-NIHPresets==1.1.11) (1.7.dev202306142048)  
Requirement already satisfied: ChimeraX-PresetMgr~=1.0 in
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-
packages (from ChimeraX-NIHPresets==1.1.11) (1.1)  
Requirement already satisfied: ChimeraX-Struts>=1.0.1,~=1.0 in
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-
packages (from ChimeraX-NIHPresets==1.1.11) (1.0.1)  
Requirement already satisfied: ChimeraX-Geometry~=1.0 in
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-
packages (from ChimeraX-Struts>=1.0.1,~=1.0->ChimeraX-NIHPresets==1.1.11)
(1.3)  
Requirement already satisfied: ChimeraX-Arrays~=1.0 in
/Users/chimera/Applications/ChimeraX_Daily.app/Contents/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-
packages (from ChimeraX-Geometry~=1.0->ChimeraX-Struts>=1.0.1,~=1.0->ChimeraX-
NIHPresets==1.1.11) (1.1)  
Installing collected packages: ChimeraX-NIHPresets  
Successfully installed ChimeraX-NIHPresets-1.1.11  
\---  

Successfully installed ChimeraX-NIHPresets-1.1.11  
No change in list of installed bundles  




OpenGL version: 4.1 Metal - 83.1
OpenGL renderer: Apple M1 Max
OpenGL vendor: Apple

Python: 3.11.2
Locale: UTF-8
Qt version: PyQt6 6.4.2, Qt 6.4.2
Qt runtime version: 6.4.3
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: Mac Studio
      Model Identifier: Mac13,1
      Model Number: Z14J0008FLL/A
      Chip: Apple M1 Max
      Total Number of Cores: 10 (8 performance and 2 efficiency)
      Memory: 32 GB
      System Firmware Version: 8422.100.650
      OS Loader Version: 8422.100.650

Software:

    System Software Overview:

      System Version: macOS 13.3.1 (a) (22E772610a)
      Kernel Version: Darwin 22.4.0
      Time since boot: 27 days, 22 hours, 38 minutes

Graphics/Displays:

    Apple M1 Max:

      Chipset Model: Apple M1 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 24
      Vendor: Apple (0x106b)
      Metal Support: Metal 3
      Displays:
        PHL 279P1:
          Resolution: 5120 x 2880 (5K/UHD+ - Ultra High Definition Plus)
          UI Looks like: 2560 x 1440 @ 60.00Hz
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Rotation: Supported


Installed Packages:
    alabaster: 0.7.13
    appdirs: 1.4.4
    appnope: 0.1.3
    asttokens: 2.2.1
    Babel: 2.12.1
    backcall: 0.2.0
    beautifulsoup4: 4.11.2
    blockdiag: 3.0.0
    blosc2: 2.0.0
    build: 0.10.0
    certifi: 2022.12.7
    cftime: 1.6.2
    charset-normalizer: 3.1.0
    ChimeraX-AddCharge: 1.5.11
    ChimeraX-AddH: 2.2.5
    ChimeraX-AlignmentAlgorithms: 2.0.1
    ChimeraX-AlignmentHdrs: 3.3.1
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.9.3
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.3
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.45.7
    ChimeraX-AtomicLibrary: 10.0.7
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.3.2
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.10.3
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.3.2
    ChimeraX-ChangeChains: 1.0.2
    ChimeraX-CheckWaters: 1.3.1
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.2.4
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.3
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.7.dev202306142048
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.4.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-Dicom: 1.2
    ChimeraX-DistMonitor: 1.4
    ChimeraX-DockPrep: 1.1.2
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-Label: 1.1.7
    ChimeraX-ListInfo: 1.2
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.2
    ChimeraX-Map: 1.1.4
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.1
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.12
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.9
    ChimeraX-ModelPanel: 1.3.7
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.0
    ChimeraX-NRRD: 1.0
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.10.2
    ChimeraX-PDB: 2.7.2
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PhenixUI: 1.1.11
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.1
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.2
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 3.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.1
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.8.3
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.1
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.12
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1.2
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.2
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.29.1
    ChimeraX-uniprot: 2.2.2
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.2.1
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.1
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.1.3
    contourpy: 1.1.0
    cxservices: 1.2.2
    cycler: 0.11.0
    Cython: 0.29.33
    debugpy: 1.6.7
    decorator: 5.1.1
    docutils: 0.19
    executing: 1.2.0
    filelock: 3.9.0
    fonttools: 4.40.0
    funcparserlib: 1.0.1
    grako: 3.16.5
    h5py: 3.8.0
    html2text: 2020.1.16
    idna: 3.4
    ihm: 0.35
    imagecodecs: 2023.3.16
    imagesize: 1.4.1
    ipykernel: 6.21.1
    ipython: 8.10.0
    ipython-genutils: 0.2.0
    ipywidgets: 8.0.6
    jedi: 0.18.2
    Jinja2: 3.1.2
    jupyter-client: 8.0.2
    jupyter-core: 5.3.1
    jupyterlab-widgets: 3.0.7
    kiwisolver: 1.4.4
    line-profiler: 4.0.2
    lxml: 4.9.2
    lz4: 4.3.2
    MarkupSafe: 2.1.3
    matplotlib: 3.6.3
    matplotlib-inline: 0.1.6
    msgpack: 1.0.4
    nest-asyncio: 1.5.6
    netCDF4: 1.6.2
    networkx: 2.8.8
    nibabel: 5.0.1
    nptyping: 2.5.0
    numexpr: 2.8.4
    numpy: 1.23.5
    openvr: 1.23.701
    packaging: 21.3
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 9.3.0
    pip: 23.0
    pkginfo: 1.9.6
    platformdirs: 3.5.3
    prompt-toolkit: 3.0.38
    psutil: 5.9.4
    ptyprocess: 0.7.0
    pure-eval: 0.2.2
    py-cpuinfo: 9.0.0
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.14.0
    pynrrd: 1.0.0
    PyOpenGL: 3.1.6
    PyOpenGL-accelerate: 3.1.6
    pyparsing: 3.0.9
    pyproject-hooks: 1.0.0
    PyQt6-commercial: 6.4.2
    PyQt6-Qt6: 6.4.3
    PyQt6-sip: 13.4.1
    PyQt6-WebEngine-commercial: 6.4.0
    PyQt6-WebEngine-Qt6: 6.4.3
    python-dateutil: 2.8.2
    pytz: 2023.3
    pyzmq: 25.1.0
    qtconsole: 5.4.0
    QtPy: 2.3.1
    RandomWords: 0.4.0
    rdkit: 2022.9.5
    requests: 2.31.0
    scipy: 1.9.3
    setuptools: 67.4.0
    setuptools-scm: 7.0.5
    sfftk-rw: 0.7.3
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.4.1
    sphinx: 6.1.3
    sphinx-autodoc-typehints: 1.22
    sphinxcontrib-applehelp: 1.0.4
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.1
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    stack-data: 0.6.2
    tables: 3.8.0
    tcia-utils: 1.2.0
    tifffile: 2022.10.10
    tinyarray: 1.2.4
    tomli: 2.0.1
    tornado: 6.3.2
    traitlets: 5.9.0
    typing-extensions: 4.6.3
    tzdata: 2023.3
    urllib3: 2.0.3
    wcwidth: 0.2.6
    webcolors: 1.12
    wheel: 0.38.4
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.7

[Zach Pearson]

Do the bundles get sent to the system local Python like in #9191?

[pett]

AFAICT, no, they do not. It took me awhile to check because there is both a /usr/bin/python3 and /usr/local/bin/python3 on my machine.

[Zach Pearson]

They don't go to ~/.local/lib/python3.11 either?

[pett]

There is no such file on my (or Elaine's) machine -- they're Macs.

[Zach Pearson]

OH, right. On Mac I think that directory is ~/Library/Python/3.11/lib/python/site-packages/

[pett]

Looking at ~/Library/Application Support/ChimeraX/1.7/site-packages, it seems I was able to install the PhenixUI bundle back on June 5th, so the problematic change must have happened since then.

[pett]

Yes, it did go into ~/Library/Python/3.11/lib/python/site-packages/! Good call.

[Zach Pearson]

Great! I was hoping it was the same, so maybe it has a common cause. I wonder if some environment variable or configuration we used to set to send things to our Python directories changed in 3.10 or 3.11 -- I'll look!

[Zach Pearson]

Installing the bundles in the daily build

Just to clarify, is this a command-line devel install / Makefile install, a ChimeraX command line devel install, or a ChimeraX command line toolshed install?

[pett]

It was "toolshed install" from the ChimeraX command line (PhenixUI bundle) and from the Toolshed page (NIHPresets bundle), which I'm guessing is "toolshed install" under the hood.

[Zach Pearson]

Interestingly, the pip install command from the core correctly sends packages to ~/Library/Application Support/ChimeraX/1.7/site-packages

[Zach Pearson]

In that script, I set the environment variable PYTHONUSERBASE to app_dirs.user_data_dir before installing packages and restore it afterwards, and also take pains to ensure that I use the Python executable instead of the ChimeraX executable.

[Zach Pearson]

I uninstalled the blastprotein bundle and did a toolshed install of it and verified that it went to

~/Library/Python/3.11/lib/python/site-packages

while pip install sent it to

~/Library/Application Support/ChimeraX/1.7/site-packages

And the only difference is that I set PYTHONUSERBASE in core/src/commands/pip.py but that environment variable is not set in the install code in toolshed_utils/src/__init__.py. I've abstracted the code that saves, sets, and resets PYTHONUSERBASE into a context manager in core/src/python_utils.py. Any invocations of pip in our codebase should be under these two lines:

from chimerax.core.python_utils import chimerax_user_base
with chimerax_user_base():
   invoke_pip()

There's one hitch, which is that whatever directory is set as PYTHONUSERBASE, pip will install to $PYTHONUSERBASE/lib/python/site-packages. This first came up in #8927 but was easily mitigated.

Since that ticket whenever pip is invoked we check to see if the old site-packages directory is real or a symbolic link, and if it's real we migrate it to that new expected directory -- it becomes ~/Library/Application Support/ChimeraX/1.7/lib/python/site-packages -- and then we symbolically link the old directory at ~/Library/Application Support/ChimeraX/1.7/site-packages to the new one to avoid breaking existing code. It's just a fool's errand to expend too much energy fighting what Python clearly wants to be the behavior of pip.

In Python 3.9, the python in lib/python/site-packages had a version number but it looks like it no longer does in 3.11. I'll double check the behavior on Windows before tomorrow's build.

[Zach Pearson]

It occurs to me that toolshed utils can't depend on the core, so let me rework what I wrote...

[Zach Pearson]

Er, actually, it imports the core all over the place. Nevermind x 2

[Zach Pearson]

Fixed for Windows, too.