Installing a bundle replaces PyPi packages with different versions

[goddard@…]

The following bug report has been submitted:
Platform:        macOS-15.5-arm64-arm-64bit
ChimeraX Version: 1.9 (2024-12-11 19:11:19 UTC)
Description
I installed a new bundle (locscalesurfer) and it appears it has installed numpy with the version locscalesurfer listed as a dependency 1.24.4 while ChimeraX 1.9 seems to come with newer numpy 1.26.4.

I'm wondering if we have a strategy for ChimeraX contributed bundles to avoid having them replace PyPi dependencies ChimeraX uses or that other contributed bundles use.  I don't see any easy way to handle this.  But it would be good to at least have some ChimeraX documentation that describes the problem.

Log:
UCSF ChimeraX version: 1.9 (2024-12-11)  
© 2016-2024 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> toolshed install
> /Users/goddard/Downloads/ChimeraX_locscalesurfer-0.1-py3-none-any.whl

Errors may have occurred when running pip:  

pip standard error:  
\---  
ERROR: pip's dependency resolver does not currently take into account all the
packages that are installed. This behaviour is the source of the following
dependency conflicts.  
chimerax-deepssetracer 1.1 requires numpy~=2.0.1, but you have numpy 1.24.4
which is incompatible.  
chimerax-deepssetracer 1.1 requires torch~=2.1.2, but you have torch 2.2.2
which is incompatible.  
chimerax-bundlebuilder 1.4.0 requires numpy==1.26.4, but you have numpy 1.24.4
which is incompatible.  
blosc2 3.0.0 requires numpy>=1.25.0, but you have numpy 1.24.4 which is
incompatible.  
\---  

pip standard output:  
\---  
\---  

Installed ChimeraX-locscalesurfer (0.1)  

> ui tool show Shell

[Repeated 1 time(s)]




OpenGL version: 4.1 Metal - 89.4
OpenGL renderer: Apple M2 Ultra
OpenGL vendor: Apple

Python: 3.11.4
Locale: en_US.UTF-8
Qt version: PyQt6 6.7.1, Qt 6.7.1
Qt runtime version: 6.7.3
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: Mac Studio
      Model Identifier: Mac14,14
      Model Number: Z1800003VLL/A
      Chip: Apple M2 Ultra
      Total Number of Cores: 24 (16 performance and 8 efficiency)
      Memory: 64 GB
      System Firmware Version: 11881.121.1
      OS Loader Version: 11881.121.1

Software:

    System Software Overview:

      System Version: macOS 15.5 (24F74)
      Kernel Version: Darwin 24.5.0
      Time since boot: 19 hours, 6 minutes

Graphics/Displays:

    Apple M2 Ultra:

      Chipset Model: Apple M2 Ultra
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 60
      Vendor: Apple (0x106b)
      Metal Support: Metal 3
      Displays:
        PHL 278B1:
          Resolution: 3840 x 2160 (2160p/4K UHD 1 - Ultra High Definition)
          UI Looks like: 1920 x 1080 @ 60.00Hz
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Rotation: Supported


Installed Packages:
    alabaster: 1.0.0
    alphashape: 1.3.1
    anyio: 4.7.0
    appdirs: 1.4.4
    appnope: 0.1.4
    asttokens: 3.0.0
    auditwheel: 6.1.0
    autocommand: 2.2.2
    babel: 2.16.0
    backports.tarfile: 1.2.0
    beautifulsoup4: 4.12.3
    biopython: 1.83
    blockdiag: 3.0.0
    blosc2: 3.0.0
    build: 1.2.1
    certifi: 2023.11.17
    cftime: 1.6.4.post1
    charset-normalizer: 3.4.0
    ChimeraX-AddCharge: 1.5.18
    ChimeraX-AddH: 2.2.6
    ChimeraX-AlignmentAlgorithms: 2.0.2
    ChimeraX-AlignmentHdrs: 3.5
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.16.1
    ChimeraX-AlphaFold: 1.0.1
    ChimeraX-AltlocExplorer: 1.1.2
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.58.8
    ChimeraX-AtomicLibrary: 14.1.11
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.4
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 3.0.0
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.13.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.4.0
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.4
    ChimeraX-ChangeChains: 1.1
    ChimeraX-CheckWaters: 1.4
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.3
    ChimeraX-Clipper: 0.24.0
    ChimeraX-clix: 0.1.4
    ChimeraX-ColorActions: 1.0.5
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.6
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.9
    ChimeraX-CoreFormats: 1.2
    ChimeraX-coulombic: 1.4.4
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-DeepSSETracer: 1.1
    ChimeraX-Dicom: 1.2.6
    ChimeraX-DistMonitor: 1.4.2
    ChimeraX-DockPrep: 1.1.3
    ChimeraX-Dssp: 2.0
    ChimeraX-EMalign: 0.1.1
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.4.1
    ChimeraX-Hbonds: 2.5
    ChimeraX-Help: 1.3
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.3
    ChimeraX-ISOLDE: 1.9
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-IUPAC: 1.0
    ChimeraX-KVFinder: 1.2.1
    ChimeraX-Label: 1.1.14
    ChimeraX-LigandRecognizer: 0.2
    ChimeraX-ListInfo: 1.2.2
    ChimeraX-locscalesurfer: 0.1
    ChimeraX-Log: 1.2
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.9.1
    ChimeraX-Map: 1.3
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-maskChains: 1.1
    ChimeraX-MatchMaker: 2.1.6
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.7.2
    ChimeraX-MedicalToolbar: 1.1
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.14.2
    ChimeraX-MMTF: 2.2
    ChimeraX-ModelArchive: 1.0
    ChimeraX-Modeller: 1.5.18
    ChimeraX-ModelPanel: 1.5
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0.3
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-MutationScores: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.2
    ChimeraX-NIHPresets: 1.1.19
    ChimeraX-NMRSTAR: 1.0.2
    ChimeraX-NRRD: 1.2
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.14
    ChimeraX-OrthoPick: 1.0.1
    ChimeraX-PDB: 2.7.6
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.4
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PhenixUI: 1.3.7
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1.2
    ChimeraX-PubChem: 2.2
    ChimeraX-QScore: 1.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.2
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.6.2
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.3.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 4.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.2
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-Segmentations: 3.5.6
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.14
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.2.0
    ChimeraX-ShowSequences: 1.0.3
    ChimeraX-SideView: 1.0.1
    ChimeraX-SimilarStructures: 1.0.1
    ChimeraX-Smiles: 2.1.2
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.18.1
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.2.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.5
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-TaskManager: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-TetraScape: 0.2
    ChimeraX-Toolbar: 1.2.3
    ChimeraX-ToolshedUtils: 1.2.4
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.41
    ChimeraX-Umap: 1.0
    ChimeraX-uniprot: 2.3.1
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.4.4
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-vrml: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.4
    ChimeraX-Zone: 1.0.1
    click: 8.1.7
    click-log: 0.4.0
    colorama: 0.4.6
    comm: 0.2.2
    contourpy: 1.3.1
    cxservices: 1.2.3
    cycler: 0.12.1
    Cython: 3.0.10
    debugpy: 1.8.9
    decorator: 5.1.1
    docutils: 0.21.2
    einops: 0.3.2
    executing: 2.1.0
    filelock: 3.15.4
    finufft: 2.3.1
    fonttools: 4.55.3
    fsspec: 2024.3.1
    funcparserlib: 2.0.0a0
    glfw: 2.8.0
    grako: 3.16.5
    h11: 0.14.0
    h5py: 3.12.1
    html2text: 2024.2.26
    httpcore: 1.0.7
    httpx: 0.28.1
    idna: 3.10
    ihm: 1.3
    imagecodecs: 2024.6.1
    imagesize: 1.4.1
    importlib-metadata: 4.0.1
    importlib_metadata: 8.0.0
    importlib_resources: 6.4.0
    inflect: 7.3.1
    ipykernel: 6.29.5
    ipython: 8.26.0
    ipywidgets: 8.1.5
    jaraco.context: 5.3.0
    jaraco.functools: 4.0.1
    jaraco.text: 3.12.1
    jedi: 0.19.1
    Jinja2: 3.1.4
    joblib: 1.4.2
    jupyter_client: 8.6.2
    jupyter_core: 5.7.2
    jupyterlab_widgets: 3.0.13
    kiwisolver: 1.4.7
    line_profiler: 4.1.3
    llvmlite: 0.42.0
    lxml: 5.2.2
    lz4: 4.3.3
    MarkupSafe: 3.0.2
    matplotlib: 3.9.2
    matplotlib-inline: 0.1.7
    more-itertools: 10.3.0
    mpmath: 1.3.0
    mrcfile: 1.3.0
    msgpack: 1.0.8
    ndindex: 1.9.2
    nest-asyncio: 1.6.0
    netCDF4: 1.6.5
    networkx: 3.3
    nibabel: 5.2.0
    nptyping: 2.5.0
    numba: 0.59.1
    numexpr: 2.10.2
    numpy: 1.24.4
    numpy: 1.26.4
    openvr: 1.26.701
    ordered-set: 4.1.0
    packaging: 23.2
    packaging: 24.1
    ParmEd: 4.2.2
    parso: 0.8.4
    pep517: 0.13.1
    pexpect: 4.9.0
    pillow: 10.4.0
    pip: 24.2
    pkginfo: 1.11.1
    platformdirs: 4.3.6
    platformdirs: 4.2.2
    pooch: 1.8.2
    prompt_toolkit: 3.0.48
    psutil: 6.0.0
    ptyprocess: 0.7.0
    pure_eval: 0.2.3
    py-cpuinfo: 9.0.0
    pycollada: 0.8
    pydicom: 2.4.4
    pyelftools: 0.31
    Pygments: 2.18.0
    pynmrstar: 3.3.4
    pynndescent: 0.5.12
    pynrrd: 1.0.0
    PyOpenGL: 3.1.7
    PyOpenGL-accelerate: 3.1.7
    pyopenxr: 1.0.3401
    pyparsing: 3.2.0
    pyproject_hooks: 1.2.0
    PyQt6-commercial: 6.7.1
    PyQt6-Qt6: 6.7.3
    PyQt6-WebEngine-commercial: 6.7.0
    PyQt6-WebEngine-Qt6: 6.7.3
    PyQt6-WebEngineSubwheel-Qt6: 6.7.3
    PyQt6_sip: 13.8.0
    python-dateutil: 2.9.0.post0
    pytz: 2024.2
    pyvista: 0.44.1
    pyzmq: 26.2.0
    qtconsole: 5.5.2
    QtPy: 2.4.2
    qtshim: 1.0
    RandomWords: 0.4.0
    requests: 2.32.3
    Rtree: 1.3.0
    scikit-learn: 1.4.2
    scipy: 1.10.1
    scipy: 1.14.0
    scooby: 0.10.0
    setuptools: 72.1.0
    sfftk-rw: 0.8.1
    shapely: 2.0.6
    six: 1.16.0
    sniffio: 1.3.1
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.6
    Sphinx: 8.0.2
    sphinx-autodoc-typehints: 2.2.3
    sphinxcontrib-applehelp: 2.0.0
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 2.0.0
    sphinxcontrib-htmlhelp: 2.1.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 2.0.0
    sphinxcontrib-serializinghtml: 2.0.0
    stack-data: 0.6.3
    superqt: 0.6.3
    sympy: 1.12
    tables: 3.10.1
    tcia_utils: 1.5.1
    threadpoolctl: 3.5.0
    tifffile: 2024.7.24
    timm: 0.4.12
    tinyarray: 1.2.4
    tomli: 2.0.1
    torch: 2.2.2
    torchvision: 0.17.2
    tornado: 6.4.2
    tqdm: 4.66.4
    traitlets: 5.14.3
    trimesh: 4.4.7
    typeguard: 4.3.0
    typing_extensions: 4.12.2
    typing_extensions: 4.12.2
    tzdata: 2024.2
    umap-learn: 0.5.6
    urllib3: 2.2.3
    vtk: 9.3.1
    wcwidth: 0.2.13
    webcolors: 24.6.0
    wheel: 0.43.0
    wheel: 0.43.0
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.13
    zipp: 3.19.2
    zipp: 3.19.2

[Tom Goddard]

I think the best we can do with this problem of conflicting dependencies is document the problem, and if we have any ideas for mitigating it document those too. I don't see how we can handle conflicting dependencies, and any ChimeraX bundle that lists exact version numbers for PyPi packages is likely to create conflicting dependencies.

[Zach Pearson]

If it's a bundle, it should be sticking e.g. numpy in the user site directory and not overwriting ChimeraX's internal numpy. pyKVFinder does this too, it appears harmless, but Eric would know more about it

[Tom Goddard]

From the ChimeraX Python shell sys.path shows that the user site packages directory comes before the ChimeraX install site-packages directory. In theory it would get the numpy from the user site packages, although for numpy it probably imports it before the user site-packages directory was added to sys.path so it in fact gets the ChimeraX install numpy.

If there are two versions of any PyPi package required by ChimeraX bundles (built-in or Toolshed), some code is going to get a version other than the one it wants, because only one version of a module can be loaded in Python. I think we haven't run into problems with this yet because in fact the dependencies that say they want a specific version can work with many versions. But if ChimeraX was using more Toolshed packages that pulled in more versions of old and new PyPi packages (not specifically numpy) then we eventually will run into incompatibilities.

I think we should document that the version management for PyPi packages requested by ChimeraX extensions is really not functional. This is a limitation of the Python ecosystem, not specific to ChimeraX. But it is precarious in ChimeraX because it allows extensions to replace whatever PyPi packages it wants with the versions it wants.

If Toolshed were more successful and if major versions of PyPi dependencies of often used packages (numpy comes to mind) changed then our Toolshed eco-system would be seriously broken by some tools crippling other tools by installing PyPi packages that don't allow the other tools to work. Quite possibly the Toolshed will never be successful enough for this to become a problem.

[Eric Pettersen]

Yeah, third-party bundle dependencies are really for declaring dependencies on packages that we don't ship/install with ChimeraX. Third-party bundles really shouldn't directly declare dependencies on external packages that we do ship/install with ChimeraX, they should just work with the versions we provide.

The real sticking point is when these packages that a bundle depends on that we don't ship/install in turn declare dependencies on packages that we do ship/install, but at different version numbers. That's what happens with pyKVFinder -- it declares a dependency on a different version of numpy. I don't know of any good solution for that.