Cloud builds

[Zach Pearson]

We want the ability to build ChimeraX in the cloud, download the program created to Plato, and present the downloaded artifact as a daily build installer.

I've been investigating some providers such as GitHub Actions, Semaphore CI, Azure, and CircleCI and I'll note findings here.

[Zach Pearson]

Quick summary of available options

Cloud Providers	Linux	Docker	macOS x86_64	macOS arm64!*	Windows	Self-Hosted
GitHub Actions	Ubuntu 18,20,22	Set up on Linux VM	Y	Y	Server 2019, 2022	Y
SemaphoreCI	Ubuntu 18,20	Set up on Linux VM	Y	Y	No; need minGW on Linux	Y
CircleCI	Ubuntu 20,22	Native	Most xcode versions since 10.3.0	<-- See last cell	Server 2019,2022	Y
Azure	Ubuntu 18,20,22	Set up on Linux VM	Y	Y	Server 2019 and 2022	Y
Jenkins	Y	Y	Y	Y	Y	Only self-hosted
TravisCI	Y	Runs on Linux VM but 1st class script support	All versions of xcode back to 7.3	<-- See last cell	Server 1809(?)	Enterprise only

• Cross-compile from Intel VM with appropriate Xcode version.

[Zach Pearson]

Given that all of the platforms seem to have some kind of Windows support except SemaphoreCI, and using it would require some change to our existing build system I'd vote to use some other platform.

That's somewhat disappointing, since Semaphore has super ergonomic monorepo support and I would be a strong advocate for them if they supported Windows.

[Zach Pearson]

Also, it's not really "in the cloud" if you have to host it yourself is it? Jenkins is out for me.

[Zach Pearson]

Overview of "Monorepo" options where I define good monorepo support as any service that lets you _easily_ make rules like "only build this directory or run its test if a file inside it changes"

Cloud Providers
GitHub Actions	Can use path selectors but by hand
CircleCI	Has path selectors, dynamic configurations, and promotions "continuations"
Azure	Complex hand-rolled configuration including directories and excluding others
TravisCI	None?

I think CircleCI is the way to go for now from the standpoint of their flexibility. Their free tier is also more generous: 6000 build minutes vs 1300

If that's still not enough, GitHub's free tier is basically limitless.

[Zach Pearson]

​https://docs.github.com/en/actions/creating-actions/creating-a-composite-action Could use composite actions if we went with GitHub Actions.

[Zach Pearson]

Extremely happy to report that I was able to build ChimeraX using GitHub actions, download the artifact it created to my local machine, and run it successfully.

[goddard@…]

Very cool!  Can we see the files on Github that you used to do this?

[Zach Pearson]

Yes. ​https://github.com/zjp/ChimeraX/tree/develop/.github/workflows

[goddard@…]

Wow!  Pretty simple.  I think you said sometimes Github has fails getting our prereqs off plato.  How often?  If the build fails is there a way to look at the build tree to debug on the virtual machine that did the build?  Is there a way to use the same scripts but build on virtual machines on plato?

We should discuss at developer meeting your findings when Greg is back.

[Zach Pearson]

That doesn't really happen anymore, or at least it hasn't for the last few runs I tried.

You can use ssh, but it requires some setup:

​https://github.com/marketplace/actions/debugging-with-ssh

There's a way to use self-hosted runners, yes, and those can be in Docker.

But if we're looking at building in the cloud we should go all the way. :)

[Zach Pearson]

The macOS pipeline also works, though the resulting app is not notarized.

But don't worry, it's ​not hard to notarize a macOS app with GitHub Actions.

Also, compressing ChimeraX with xzip (tar -Jcvf} results in a file that is 228mb vs 388mb for the dmg we currently distribute, .tar.xz files can be opened with stock macOS (to my knowledge), and the .tar.xz removes an installation step because it expands into a directly runnable ChimeraX.app

[goddard@…]

Great progress.  I wonder if dmg handle other compression algorithms like xzip.  It is best to deliver Mac apps the way they are usually delivered.

[goddard@…]

The hdiutil man page says dmg uses zlib compression with compression level 1 (fastest, least compressed).  It can also make a dmg with bzip2 which should give much better compression, slower to compress, hopefully not slower to decompress, would need testing.

[Zach Pearson]

Sure, that's a good point. I think the only other application I've seen come as a bare .tar.xz is XCode.

[Zach Pearson]

Over the last few days I've been experimenting with cloud builds using GitHub Actions and found it to be a satisfactory platform. I'm able to build the macOS version of ChimeraX and the Ubuntu versions. Both versions work when downloaded to the local machine (the Ubuntu 20 version was tested by downloading to WSL).

I have been experimenting with a Windows build using ​this template to set up Cygwin. This appears to work, except that the GitHub Actions runner appears to use different versions of either Visual Studio or the Windows SDK -- have not yet investigated, but will today. Long story short, vsvars.sh sources successfully then the build system complains that it's not set properly, but I believe that once that's in place a build should work.

Another problem cropped up, which is that for Ubuntu 18.04

The next _possible_ steps are this:

• Add jobs that build the versions currently built on singularity
• Move the build_tools repo to GitHub using GitHub secrets for storage of sensitive information
  • Convert the build scripts to actions that can be used in the ChimeraX repo
• Download build artifacts back to Plato for distribution
  • Alternatively, ​GitHub has a way to track artifact downloads so we could replace current links with the direct download link from GitHub.
• Automated changelog based on commit titles?
• Think about how we can slightly loosen the coupling on the build steps. Do we really need to spend 20 minutes setting up ChimeraX's Python to run unit tests over bundles?
• How can we set up actions that only test a bundle when it changes?

[Zach Pearson]

Meant to say that the problem with 18.04 is that it seems to not want to install PyQt6-Qt6 on the RBVI repo but will evidently do it just fine on my fork.

[Zach Pearson]

I've been monitoring the CentOS builds on Github Actions for the past few days, looking at where they fail to get the last few missing dependencies in. Today I added a commit to get ImageMagick so that we can make the ChimeraX icon. This is one of the last steps in building ChimeraX, after all of the prereqs, all of the bundles, and the ChimeraX executable. Once it builds the CentOS build should be working.

After that the last platform to work on is Windows. It is currently not building some of the source-only pip prerequsites correctly (ihm, ParmEd, and tinyarray).

[Greg Couch]

Does this handle all of the Linux variants or just CentOS 8? (Should have called it RHEL 8, might be too late to change.)

As for using the the result of the cloud build, we'll probably need multiple artifacts. The package, a corresponding .tar.gz (for local remote installs that don't go through a package manager), and a copy of the generated documentation to put on the web. The generated documentation, vdocs/user, is a flushed out version of what is in the distribution. Perhaps the readthedocs version of the documentation would be sufficient, but that is different from what we currently do.

[Zach Pearson]

I want to say yes but I forget the finer differences between the Generic build and the Rocky 8 build. Don't we build the Generic Linux build with Rocky 8?

Ubuntu 20.04 and 22.04 are working. I'm switching out CentOS 8 Stream for Rocky 8, and then Rocky 9 should follow closely behind that.

Zipping the user and developer docs for ourselves won't be hard, but readthedocs should be sufficient for developer documentation if we agree to use it. One of its advantages is that it's usually going to be newer than what we get from GitHub (but never behind GitHub).

[Greg Couch]

The "generic" build is currently done on Debian 10. Forgot that vdocs/user is the expanded user documentation (duh), so that needs to be copied back. The documentation is only needed from one build. We're currently using the base Linux variant.

[Zach Pearson]

I had a lot of trouble getting Rocky 9 and Debian 10 to build. In both cases ffmpeg was the culprit. My Rocky 9 container gets stuck compiling parts of libvpx unless I bump libvpx to 1.12 (I just tried the newest version and it worked; I didn't work down through the versions to find the earliest working one).

My Debian 10 container complains that it needs libogg1.1, even though it's installed.

Did you see these errors on bare metal / in Singularity? If so how did you work around them?

[Greg Couch]

I do not see recall seeing those errors. Any workarounds are either in the singularity definition files for the platform, or in our Makefiles. Our singularity images have not been updated in a long time, it could be that the problem is with recent versions of Rocky 9 or Debian 10.

[Zach Pearson]

OK, I was able to get Rocky 8 and 9 building on my fork, and Windows, and it'll be tested on the main repo tonight.

The hangup with Windows is that neither the windows-2019 (Windows Server 2019) nor the windows-latest (Windows Server 2022) runner use the same versions for e.g. the Windows 10 SDK that we use, but ChimeraX still builds with the versions that are available (you can look at vsvars.sh to see what the options are).

[Zach Pearson]

​https://github.com/RBVI/ChimeraX/actions/runs/3712314879 first successful run with every platform except Debian 10.

[goddard@…]

Cool!  Have you tested each of the resulting distributions?  Does the build make the installers (.rpm, .deb, .dmg, .exe) or just the app directory tree?

[Zach Pearson]

I have not, besides the Ubuntu 20 and macOS builds ​a few months ago.

It doesn't make the installers (yet), just the app directory tree. What's being built right now is also not ready for distribution. The macOS application is not notarized, and no build uses commercial PyQt6.

When we agree on a good method for getting the right PyQt6 package into the build runner, and a good method for notarization (it's possible to do it in GitHub Actions), we can slot building the installers in afterward and download that as an artifact for distribution.

[goddard@…]

Ok, we should discuss the pros and cons of having our installers created on the cloud.  My previous comment was less about distribution of ChimeraX and more about testing to see if your builds are working.  For that it is fine to use PyQt GPL as long as we don't distribute that.  And it would be fine not to notarize, although maybe in the latest macOS it won't allow running in any way if not notarized?

[Greg Couch]

Could we use a local "cloud"? What would be needed for that?

[Zach Pearson]

You can; there is documentation for setting up self hosted runners ​here. If I understand it correctly the runner application has access to the underlying machine.

[Zach Pearson]

I was provoked to revisit this by the recent kerfuffle with the wheel.

I've taken the time to fix the issues that were causing the cloud nightly builds to fail. There is still a nightly build on GitHub that mirrors the exact environment of the ChimeraX build (well, except that the Windows build takes place on MinGW instead of Cygwin), but I've added a new set of workflow files:

.github/workflow-skeleton.yml -------------------- Defines workflows for Ubuntu 20.04, Ubuntu 22.04, Rocky 8 (Docker on Ubuntu 20.04), 
                                                   macos-latest (an arm64 Mac runner), and Windows 2022. 
.github/{bundle,prereq,srcapp}-file-filters.yml -- Defines file filters used to detect changes to files in specific directories
.github/workflows/prereqs.yml -------------------- Detects changes in the prereqs folder using the file filters in 
                                                   .github/prereq-file-filters.yml, then rebuilds that prereq for all platforms
.github/workflows/bundle-changes.yml ------------- Should eventually do the same thing as prereqs.yml, for bundles
.github/workflows/srcapps.yml -------------------- Ditto, for src/apps
.github/workflows/check-for-changes.yml ---------- Runs each of the 3 preceding workflows, then builds and tests ChimeraX every
                                                   time someone pushes to the develop branch
.github/workflows/build-and-test.yml ------------- Will eventually abstract the build and test part of the preceding workflow, 
                                                   so it can be reused when prereqs, bundles, or apps change

We can now tell whether the next daily build will fail 15 minutes after a push, rather than biting our nails until the next day waiting to see if there was a syntax error or build failure.

What's set up now is cloud testing. I don't consider it cloud building because we're not taking artifacts back to Plato for distribution. When Greg gets back from Vacation I'll discuss with him how to set up a technology preview of ChimeraX built by GitHub CI.

Then we can close this ticket.

[Zach Pearson]

I've enabled tech preview builds of macOS arm64 and all the flavors of Ubuntu we support, but those Ubuntu builds aren't using generic Linux bundles yet.

[Zach Pearson]

We now produce nightly builds of all platforms. Further bugs should be opened for problems with platform specific builds if they arise.