config.make sets SSL_CERT_FILE messing up curl fetch during build on Mac

[Tom Goddard]

I've made the ChimeraX build fetch third party libraries from plato using curl. But on Mac this fails because chimerax/mk/config.make sets SSL_CERT_FILE to a Python location which does not exist when the build starts. Why is our code setting SSL_CERT_FILE to an a location that does not even exist until the build installs openssl?

From mk/config.make:

ifeq ($(OS),Darwin)
export SSL_CERT_FILE = $(PYSITEDIR)/certifi/cacert.pem
endif

Error in prereqs fetching p7zip:

curl --silent --show-error -O https://www.rbvi.ucsf.edu/chimerax/data/prereqs/p7zip/p7zip_9.20.1_src_all.tar.bz2
curl: (77) error setting certificate verify locations:

CAfile: /Users/goddard/ucsf/chimerax/build/lib/python3.7/site-packages/certifi/cacert.pem
CApath: none

[Greg Couch]

It's because the Mac doesn't support OpenSSL, but Python uses/needs it. If I remember correctly, Apple's cryptography public API is missing pieces that Python needs.

[goddard@…]

Not sure what "python needs it" means.  Python has not even been built when this build error occurs.  If SSL_CERT_FILE needs to be set it seems it should be set specifically where it is needed, not in config.make where it gets set to a file that doesn't exist until the openssl module is built.

For now I have worked around the problem using curl --insecure to not verify certificates, not the best solution.

I have not yet checked in this curl fetching code for all the prereqs -- have only done a complete test build on Mac and need to test on Windows and Linux before I check it in.

[Greg Couch]

I'm sure it was easier to set SSL_CERT_FILE globally than delay it until after Python is built. Python has no support for the Apple cryptography APIs, so to get SSL to work, it uses OpenSSL. OpenSSL needs a list of root certificates (Certificate Authorities) to be able to validate certificates and that is what the SSL_CERT_FILE is for. For backwards compatiblity, Apple has a vestigial OpenSSL (0.9.?), but has removed the header files from XCode, so it is not possible to compile against it anymore -- and that version of OpenSSL is so old that it doesn't work with modern web servers.

You could change curl on the Mac to be "env -u SSL_CERT_FILE curl".

[Greg Couch]

I see that the --insecure option is given to curl in mk/config.make. I'd prefer to use --cert-status instead. And it works on essex without --insecure, which is 10.14.6. If there is a version of macOS where it doesn't work, we could add the right certificate to the repository and use --cacert.