Make universal Intel/ARM Mac distribution

[Tom Goddard]

We'd like to distribute Mac ChimeraX as a single Intel / ARM64 universal distribution. Most Mac apps are distributed as universal packages so the user does not need to know if they have an Intel or M1/M2 processor. It requires making "fat binaries" for each shared library and executable that contain both Intel and ARM versions.

We may be able to combine our current Mac Intel ChimeraX and Mac ARM64 ChimeraX using the "lipo" command-line tool that merges Intel and Arm64 binaries to make universal binaries.

To make this work we need the Intel and ARM64 ChimeraX to use identical versions and packaging for all the dependent libraries. Currently there are some different versions and packaging because of Python libraries we use that are distributed Intel but not for ARM64 via PyPi where we use Conda to provide ARM64 versions. In some cases (e.g. PyTables) the conda versions are not putting the shared libraries with package as PyPi does it. Other difficulties are occur where Intel and ARM versions of the PyPi packages are not identical. An example of this is numpy where the Intel version uses an older gfortran library than the Arm64 version. Numpy does not provide a universal wheel.

We'll need to assess whether we can make universal versions of all the libraries we use, and how much work it will be.

[Tom Goddard]

Zach may be the one to work on the universal Mac distribution. But since I have looked into some, I've assigned it to me for now.

[Tom Goddard]

This ticket is one part of our Mac ARM distribution effort, ticket #4663.

[Tom Goddard]

I made a script to traverse the ChimeraX ARM64 and Intel builds and see what is different. Script is make_universal.py in build_tools repository under macosx_package.

The biggest problems is that PyTables from Conda (ARM64) needs to be packaged to put libraries in the same place as PyTables from PyPi (Intel). This should be pretty easy.

Numpy and SciPy are using different fortran and gcc libraries on ARM64 and Intel but probably can include both. Need to test that.

The following file differences can probably be ignored, using the ARM64 file. OpenMM lists a different build version in its Python. Imagecodecs uses DOS line endings in ARM64 and unix in Intel for some files. Our rest server code has a difference that looks like a bug including some private key on ARM64 that is not included on Intel.

The make_universal.py will need some special case handling for the numpy and and scipy problems and ignorable python file differences.

# PyTables libraries installed by prereqs/pips for ARM64
# These libraries should be included in site-packages/tables as they are for the PyPi pytables.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libblosc.1.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libblosc.1.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libssl.3.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libssl.3.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libcrypto.3.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libcrypto.3.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libc++.1.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libc++.1.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libsnappy.1.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libsnappy.1.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libhdf5.200.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/libhdf5.200.dylib is not

# numpy Intel and ARM64 come from PyPi but are different wheels, no universal wheel available
# Using same numpy version 1.22.1, but ARM64 uses libgfortan.5.dylib while Intel uses libgfortran.3.dylib,
# ARM64 uses libgcc_s.2.dylib, while Intel uses libgcc_s.1.dylib.
# May be able to include both libraries and correct one will be used.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/distutils/__config__.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/distutils/__config__.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/include/numpy/_numpyconfig.h and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/core/include/numpy/_numpyconfig.h file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/__config__.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/__config__.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgfortran.5.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgfortran.5.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgcc_s.2.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/numpy/.dylibs/libgcc_s.2.dylib is not

# SciPy comes from PyPi for Intel and ARM64 but different wheels
# Same as numpy, same version but uses different libgfortan and libgcc_s versions.
# Probably can just copy both into universal build.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/_distributor_init.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/_distributor_init.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/__config__.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/__config__.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/.dylibs/libgfortran.5.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/.dylibs/libgfortran.5.dylib is not
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/.dylibs/libgcc_s.2.dylib is a file but ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/scipy/.dylibs/libgcc_s.2.dylib is not

# OpenMM comes from Conda, build numbers are different for Intel vs ARM64 although version is the same
# Can probably ignore this.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openmm/version.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/openmm/version.py have different contents

# imagecodecs ARM64 wheel comes from Conda, Intel wheel from PyPi
# Differences are only DOS vs unix line endings.  Can detect these are the same.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/numcodecs.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/numcodecs.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/__init__.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/__init__.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/_imagecodecs.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/_imagecodecs.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/imagecodecs.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/imagecodecs.py file sizes differ
chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/__main__.py and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/imagecodecs/__main__.py file sizes differ

# rest_server includes private key on ARM64 but not on Intel.
# This looks like a mistake.

chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/rest_server/server.pem and ChimeraX_intel.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/rest_server/server.pem file sizes differ

[Tom Goddard]

The make_universal.py script ignored some files which had differences, files or directories that ended in any of these

__pycache__, debugpy, RECORD, WHEEL, METADATA, direct_url.json, _CodeSignature, imagecodecs/licenses, .a, .c, .dist-info

Some of those (RECORD, WHEEL, METADATA, direct_url.json, .dist-info) are PyPi repository meta-data. Using the ARM64 version of the metadata on Intel Mac may lead to some problems if you try to pip install new packages into ChimeraX it will have the wrong information about which versions (Intel vs ARM64) are installed for the effected packages. I am not sure if that is going to cause havoc say with Toolshed installs.

The universal build can probably safely exclude pycache. But maybe it should have those for fastest load time. Not sure if the user drags the app to Applications whether Python will be able to make the .pyc files. Need to check. Also not sure how Intel vs ARM64 .pyc files differ. Maybe there is just a timestamp in the file.

The _CodeSignature directories can be stripped since we will notarize the universal build.

The imagecodecs/licenses had differing DOS vs unix line endings.

The .a archive files should be made universal. But I'm not sure if lipo handles those and can combine them. If we leave them out then it might prevent plugin developers from linking to them.

Not sure what the .c files are included for. Probably not important.

ChimeraX-Shell is pulling in debugpy via qtconsole and ipykernel. I think it had tons of line ending differences (DOS vs unix) but need to look again.

[Tom Goddard]

In the preceding tests comparing ARM64 to Intel ChimeraX I made the Intel ChimeraX use the same universal Python used by the ARM64 ChimeraX. Otherwise there are a bunch more differences between the Intel and universal Pythons (like different versions of the thread library). The universal Python worked in a simple test opening a PDB on an Intel Mac.

[Tom Goddard]

I made a universal Mac build and tested on Intel and M1 machines (descartes.cgl.ucsf.edu and my M1 laptop). Testing was minimal, opening atomic model and map, save a session and a cmap file and restore files. Distribution size of universal dmg is about 43% larger than the Intel Mac dmg distribution, show electrostatic surface coloring. I notarized the universal build using the notarize.py script.

569 Mbytes universal, uncompressed 1.6 Gbytes
398 Mbytes Intel, uncompressed 1.1 Gbytes
365 Mbytes ARM, uncompressed 1.0 Gbytes

To handle cases where ARM and Intel had different library names I include both libraries and the correct one is found at least in the PyTables case I tested).

Our Mac ARM build has many libraries that are already universal but some of the ChimeraX Intel libraries compiled on ARM don't work due to missing symbols, for example atomic_lib/_load_libs*.so is missing chutil.so symbols. The ChimeraX build indicates the source of the problem is that libchutil.a only had ARM objects as shown here in the make output

INFO: g++ -dynamiclib -undefined dynamic_lookup -arch arm64 -arch x86_64 -g -Wl,-rpath,@loader_path -Wl,-install_name,@rpath/libatomstruct.dylib atomic_cpp/atomstruct_cpp/Atom.o atomic_cpp/atomstruct_cpp/AtomTypes.o atomic_cpp/atomstruct_cpp/AtomicStructure.o atomic_cpp/atomstruct_cpp/Bond.o atomic_cpp/atomstruct_cpp/Chain.o atomic_cpp/atomstruct_cpp/ChangeTracker.o atomic_cpp/atomstruct_cpp/CompSS.o atomic_cpp/atomstruct_cpp/CoordSet.o atomic_cpp/atomstruct_cpp/MolResId.o atomic_cpp/atomstruct_cpp/PBGroup.o atomic_cpp/atomstruct_cpp/PBManager.o atomic_cpp/atomstruct_cpp/Point.o atomic_cpp/atomstruct_cpp/Pseudobond.o atomic_cpp/atomstruct_cpp/Residue.o atomic_cpp/atomstruct_cpp/Ring.o atomic_cpp/atomstruct_cpp/RingCalc.o atomic_cpp/atomstruct_cpp/Sequence.o atomic_cpp/atomstruct_cpp/Structure.o atomic_cpp/atomstruct_cpp/StructureSeq.o atomic_cpp/atomstruct_cpp/destruct.o atomic_cpp/atomstruct_cpp/search.o atomic_cpp/atomstruct_cpp/seq_assoc.o -L/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/lib -Lsrc/lib -L/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/arrays/lib -lpyinstance -lelement -latomstruct_tmpl -lioutil -lchutil -llogger -larrays -o src/lib/libatomstruct.dylib
ld: warning: ignoring file /Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/lib/libioutil.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/lib/libchutil.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
ld: warning: ignoring file /Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/lib/liblogger.a, building for macOS-x86_64 but attempting to link with file built for macOS-arm64

[Zach Pearson]

You may need to add '-arch arm64 -arch x86_64' to the CC and CXX definitions in the Darwin section of mk/os.make

[Tom Goddard]

Next steps are

1) Decide if the 570 Mbyte universal distribution versus the 400 Mbytes for Intel / ARM specific distributions is best for users (longer download, but don't have to choose CPU type).
2) If we want to distribute universal build, have daily builds create it after ARM and Intel Mac builds succeed, then notarize the result.
3) Initially we would offer this as a techpreview build, then promote it to daily build once it has been tested by users for about a month.

[Tom Goddard]

Since most users will have ARM Macs in the future the file size comparison should be between ARM distribution 365 Mbytes and universal 569 Mbytes, a 56% size increase. On my fast home network connection (Sonic gigabit fiber) the universal took 22 seconds to download at 200 Mbits second (wifi). Users in other countries might get about 1/10 that download speed, so about 2.5 minutes for ARM build or 4 minutes for universal.

The Safari web browser identifies the platform as Mac Intel even on Mac ARM machines so if we offer separate ARM and Intel builds the user will need to choose since we do not know which they need. We could even offer all 3 builds with universal at the top with a description that says works on all Macs, while the the ARM / Intel descriptions say smaller download but only works on M1/M2 or Intel Macs. This has the drawback that the user has to choose, but at least the first choice allows them not to know what kind of Mac CPU they have.

[Tom Goddard]

LZMA compression of the dmg files can make it 30% smaller, ticket #7274, but requires macOS 10.15.

[Tom Goddard]

The notarized Mac universal wheel with LZMA compression I made yesterday from our nightly Mac ARM and Intel builds was just 380 Mbytes. This is smaller than the Intel Mac dmg which currently uses older zlib compression which is 429 Mbytes. Our nightly Mac ARM distribution is 256 Mbytes with LZMA compression. So if we only distributed the universal build it would still be smaller than our current Intel only Mac distribution.

[Tom Goddard]

I added a nightly build platform to make the Mac universal build and it is in the Technology Preview section of the download page as of yesterday.

[Tom Goddard]

Eric notes that the universal build shows a crossed out application icon on macOS 10.15.7.

Tom:

It worked correctly on macOS 12 Intel and ARM machines.   The ChimeraX.app/Contents/Info.plist text file differs on Intel and ARM and when non-binary files differ the universal build includes the ARM file.  And that file has lines that specify the minimum macOS version

	<key>LSMinimumSystemVersion</key>
	<string>11.0.0</string>

Could you try changing that 11.0.0 to 10.14.0 and see if that makes it happy?

Eric:

That removed the circle/slash but did not get it to run:

Sep 23 12:00:19 hal2-client4 com.apple.xpc.launchd[1] (com.apple.xpc.launchd.oneshot.0x1000000f.ChimeraX[2942]): removing service since it exited with consistent failure - OS_REASON_CODESIGNING | When validating /Applications/ChimeraX_techpreview.app/Contents/MacOS/ChimeraX:
	  The code contains a Team ID, but validating its signature failed.
	Please check your system log.
Sep 23 12:00:19 hal2-client4 com.apple.xpc.launchd[1] (com.apple.xpc.launchd.oneshot.0x1000000f.ChimeraX[2942]): Binary is improperly signed.

Tom:

Editing the the Info.plist probably invalidates the code signature.  I will have to make the change in the universal build before notarizing.

[Tom Goddard]

I put a new Mac universal build on the download page (September 3, 12:49 pm) that has the minimum OS 10.13 (that is the Intel Mac build target). Eric can you give it a try?

[Greg Couch]

Remember Qt6 has a minimum of 10.14.

On 9/23/2022 1:31 PM, ChimeraX wrote:

[Eric Pettersen]

That worked fine. Opened a large structure. Ran "debug test". Ran "coulombic". Everythingl worked.

[Tom Goddard]

Done.

Mac universal builds have been on the download page about a month.