macOS build problems on Big Sur caused by wheel file names

[Tom Goddard]

On macOS Big Sur, 11.x, the ChimeraX build fails because bundle wheel file names have suffix -macosx_11_0_x86_64.whl and pip won't install that wheel saying it is an unknown platform, and wants suffix -macosx_10_16_x86_64.whl. This is because Python considers the macOS version to be 10.16 as reported by import platform ; platform.mac_ver() instead of 11.4. Online posts say that is because

"Applications compiled for Big Sur get back "11.0" as the operating system version. Applications compiled for earlier versions get "10.16". This is so logic that assumes 10 as the prefix will not break."

​https://stackoverflow.com/questions/65290242/pythons-platform-mac-ver-reports-incorrect-macos-version/65402241

We are using the standard Python 3.9.6 binary distribution compiled for macOS 10.9 I think so it reports 10.16. To work around this problem ChimeraX can be compiled on Big Sur setting environment variable SYSTEM_VERSION_COMPAT=1

env SYSTEM_VERSION_COMPAT=1 make build-from-scratch

This causes macOS to always report the macOS version as 10.16 and the ChimeraX build then makes all wheels with suffix -maxosx_10_16_x86_64.whl and these install without problems.

The ChimeraX core wheel file name uses the 10_16 suffix because it determines the wheel file name using mk/os.make (variable BIN_PLATFORM) and our Makefile explicitly uses 10_16 for macOS versions >10.

I think it would be more sensible to have ChimeraX produce wheel file names with 11_0 in the suffix and for the next macOS Monterey (12.0) use 12_0.

[Greg Couch]

Currently we "export MACOSX_DEPLOYMENT_TARGET=10.13" when building the MacOS binaries. So all of our wheels should be 10_13, but I see that they are 10_14, so there's another bug to investigate. I think bundle builder should default to our build default and allow for it to be overridden.

[Tom Goddard]

I agree if we target the nightly builds at 10.13 then the wheels should have os version 10_13 in the name.

My description for this ticket is about developer builds on Big Sur where MACOSX_DEPLOYMENT_TARGET is not set and I think the wheel file names should have 11_0 in that case.

[Tom Goddard]

The stackoverflow post

​​https://stackoverflow.com/questions/65290242/pythons-platform-mac-ver-reports-incorrect-macos-version/65402241

mentions that Python built for macOS 10.x but running on Big Sur will report macOS version 11.4 if environment variable SYSTEM_VERSION_COMPAT=0 is set. In other words, not setting that environment variable is not equivalent to setting it to 0.

I tried a build on Big Sur using

env SYSTEM_VERSION_COMPAT=0 make build-from-scratch >& make.out

with our os.make modified so BIN_PLATFORM uses 11_0 and all wheel file names use 11_0 and the build works. So I am going to change os.make and recommend this recipe for doing builds on Big Sur.

[Tom Goddard]

The SYSTEM_VERSION_COMPAT=0 version also worked on macOS 12.0 beta (Monterey) in tests by Zach Pearson.

I have added a description of this to the Programming Manual in the compiling section.

[Tom Goddard]

I think the SYSTEM_VERSION_COMPAT=0 solution is adequate. We could eliminate the need for this by using a Python binary build made on macOS 11 when ChimeraX is built on macOS 11 or newer. But currently we use standard Python binary distributions and I am reluctant to instead compile our own Python since that is prone to have missing modules due to missing dependencies, or may create modules linking to non-system libraries that will prevent signing. So I don't think the complication is worth the trouble.

Another method to avoid the developer needing to know about this environment variable is to set it in the top level Makefile. It looks like GNU make allows "export SYSTEM_VERSION_COMPAT = 0" which will set that environment variable for all subprocesses.

[Tom Goddard]

The SYSTEM_VERSION_COMPAT variable is not needed on macOS 11 with arm64 CPU apparently because we use a different standard Python distribution that was built on macOS 11.

[Tom Goddard]

I moved the SYSTEM_VERSION_COMPAT=0 into variable APP_BOOTSTRAP_PIP in config.make so it is not needed to set the environment variable on the make command-line.

[Tom Goddard]

Fixed.

Builds on macOS 11 now work with "make build-from-scratch" using wheel file names with the correct macOS version number, ie suffix -macosx_11_0_x86_64.whl.