Toolshed Python 3.8 bundle won't install in Python 3.7

[goddard@…]

The following bug report has been submitted:
Platform:        Darwin-19.6.0-x86_64-i386-64bit
ChimeraX Version: 1.0 (2020-06-04 23:15:07 UTC)
Description
Trying to install the hkcage bundle in ChimeraX 1.0 from toolshed gives this error.

The bundle was built with the ChimeraX daily build which is using Python 3.8 while ChimeraX 1.0 uses Python 3.7.  But the bundle_info.xml does not reference Python 3.8 nor does the wheel file name "ChimeraX_HKCage-1.1-py3-none-any.whl" and it will work fine on Python 3.7.  It is not clear how the Python 3.8 dependency appeared, nor how to say any Python 3 will suffice.

Log:
UCSF ChimeraX version: 1.0 (2020-06-04)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> toolshed show

Downloading bundle ChimeraX_HKCage-1.1-py3-none-any.whl  
Looking in indexes: https://pypi.org/simple,
https://cxtoolshed.rbvi.ucsf.edu/pypi/  
Processing /Users/goddard/Downloads/ChimeraX_HKCage-1.1-py3-none-any.whl  
ERROR: Package 'ChimeraX-HKCage' requires a different Python: 3.7.7 not in
'>=3.8'  
WARNING: You are using pip version 20.1; however, version 20.2.4 is available.  
You should consider upgrading via the
'/Users/goddard/ucsf/ChimeraXOld/ChimeraX 1.0.app/Contents/MacOS/ChimeraX -m
pip install --upgrade pip' command.  
  

OpenGL version: 4.1 ATI-3.10.18
OpenGL renderer: AMD Radeon Pro Vega 20 OpenGL Engine
OpenGL vendor: ATI Technologies Inc.Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro15,3
      Processor Name: 8-Core Intel Core i9
      Processor Speed: 2.4 GHz
      Number of Processors: 1
      Total Number of Cores: 8
      L2 Cache (per Core): 256 KB
      L3 Cache: 16 MB
      Hyper-Threading Technology: Enabled
      Memory: 32 GB
      Boot ROM Version: 1037.147.4.0.0 (iBridge: 17.16.16610.0.0,0)

Software:

    System Software Overview:

      System Version: macOS 10.15.7 (19H2)
      Kernel Version: Darwin 19.6.0
      Time since boot: 31 days 14:24

Graphics/Displays:

    Intel UHD Graphics 630:

      Chipset Model: Intel UHD Graphics 630
      Type: GPU
      Bus: Built-In
      VRAM (Dynamic, Max): 1536 MB
      Vendor: Intel
      Device ID: 0x3e9b
      Revision ID: 0x0002
      Automatic Graphics Switching: Supported
      gMux Version: 5.0.0
      Metal: Supported, feature set macOS GPUFamily2 v1

    Radeon Pro Vega 20:

      Chipset Model: Radeon Pro Vega 20
      Type: GPU
      Bus: PCIe
      PCIe Lane Width: x8
      VRAM (Total): 4 GB
      Vendor: AMD (0x1002)
      Device ID: 0x69af
      Revision ID: 0x00c0
      ROM Revision: 113-D2060I-087
      VBIOS Version: 113-D20601MA0T-016
      Option ROM Version: 113-D20601MA0T-016
      EFI Driver Version: 01.01.087
      Automatic Graphics Switching: Supported
      gMux Version: 5.0.0
      Metal: Supported, feature set macOS GPUFamily2 v1
      Displays:
        Color LCD:
          Display Type: Built-In Retina LCD
          Resolution: 2880 x 1800 Retina
          Framebuffer Depth: 24-Bit Color (ARGB8888)
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: No
          Connection Type: Internal

PyQt version: 5.12.3
Compiled Qt version: 5.12.4
Runtime Qt version: 5.12.8

[Tom Goddard]

hkcage wheel file built with ChimeraX daily build has direct Python 3.8 dependency

[Tom Goddard]

I attached the hkcage wheel that somehow had a direct Python 3.8 dependency.

I am going to update the hkcage bundle on Toolshed by building it with ChimeraX 1.0.

[Tom Goddard]

I added hkcage bundle version 1.2 where I built the wheel with ChimeraX 1.1.1 / Python 3.7 and this new version installed from Toolshed in ChimeraX 1.0.

When a developer installs a pure Python bundle using ChimeraX 1.2 (Python 3.8) and says in bundle_info.xml dependencies it works with ChimeraX 1.0 (Python 3.7) I think it should in fact install in 1.0, so there should be no explicit Python 3.8 dependency, since the ChimeraX-Core dependency determines what Python version is required.

[Greg Couch]

Added limitedAPI argument to BundleInfo in bundle_info.xml. So in this case you would add limitedAPI="3.7". That changes the generated wheel name and the python_requires argument to setuptools. For Python code there is no verification that the code actually corresponds to the earlier version of Python. For C code, it defines Py_LIMITED_API to limit the C API appropriately. So, in theory, it is possible to create a binary bundle that will continue to work with newer versions of Python without recompiling. In practice, the Python unicode APIs are not part of the limited API, so we have to use the version-specific API.

[Tom Goddard]

This seems like an error prone solution. It is too easy to forget to put this tag in their bunde_info.py. The bundle lists the requirements such as ChimeraX-Core ~= 1.0 and does not list any Python requirement so I think for pure Python bundles the best assumption is that the code only needs the Python included in ChimeraX core 1.0 or newer. So "devel build" should not add a Python >= 3.8 dependency.

If the above is not feasible, then the Toolshed needs to not offer to install a bundle that says it requires ChimeraX-Core ~= 1.0 if it also says it requires Python >= 3.8 since ChimeraX 1.0 uses Python 3.7. As it is now ChimeraX attempts to install it and it fails in a cryptic way. Just not offering to install it is also likely to be very confusing. So in fact, probably Toolshed needs to not let you upload a bundle that claims ChimeraX 1.0 compatibility and also requires Python 3.8. These checks seem hard to implement which brings me back to thinking that the devel command should not have put the Python >= 3.8 requirement into the wheel in the first place if the user did not explicitly specify that requirement in bundle_info.xml.

[Greg Couch]

Adding the Python >= 3.8 requirement catches errors at bundle installation time instead of runtime. If the developer is super careful and checks that their code works with earlier versions of Python, then they can attest to that in their bundle_info.xml file. But if the developer is "lazy" and just tests with the current version of ChimeraX, then the bundle is generated as being dependent on that version. Your solution of building hkcage with an earlier version of ChimeraX is the right solution -- it is the same solution we use for binary compatibility on Linux. In general, building on an older version is forwards compatible, and building on a new version is not backwards compatible.

Should Python 3.8 bundles be listed as being available for ChimeraX 1.1? No, they shouldn't, but that is a separate problem where the toolshed needs to infer or be given additional information to figure it out.

We have a similar problem on the Mac with wheels that needs 10.14 on 10.13 systems, and that should be resolvable by the toolshed. The incompatible ABI issue also affects Linux and Windows, but that isn't discovered until the bundle is used because there is no wheel metadata to help. An example is our recent change to the C++ compiler on Linux. That has the potential of breaking things in subtle ways and there is no wheel metadata for that.

Should the error message be better? Yes, it should. I will be looking into that.

[Eric Pettersen]

I guess I feel that the default behavior of bundle builder should be the "usually correct" behavior. Since it is rare that a developer will use a feature of Python 3 that is specific to a minor version number, I feel that the default behavior for a pure Python bundle should be to not restrict it to a particular minor version number. For savvy developers that do use such version-number-specific features (and are likely aware they are doing so), they can add the correct dependency to bundle_info.xml.

For non-pure-Python bundles, I think the current behavior is fine.

[Eric Pettersen]

Already got snagged with the behavior with the AddH bundle I uploaded to the ToolShed. It won't install into the production build because Bundle Builder declares it to depend on Python 3.8 when in fact it has no such dependency. I stand by my assertion that Bundle Builder should not be adding that restriction to pure-Python bundles.

[Greg Couch]

You're wearing me down. Even though it is wrong to say that a Python-only bundle works with all versions of Python 3 when it has been only tested with Python 3.8, we can assume that the savvy developer will add conditional guards around version specific code. That way the default can be Python 3. And, of course, all ChimeraX developers are savvy ones.

[Eric Pettersen]

Putting guards in the code is one way the "savvy" developer could go. My thinking is that they would put limitedAPI="3.8" (or whatever their bundle actually needs) in their bundle_info.xml.