Replace Opal web services

[Greg Couch]

The Opal web services use the suds_jurko pypi package from 2014. That package prevents us from updating to newer versions of setuptools (>=58.0). We have long talked about switching away from Opal. Now is that time.

[Zach Pearson]

A while ago I started a feature branch called decommission_opal for this purpose.

Searching the codebase for 'Opal', it's referenced in blastprotein, bogus, modeller, webservices. Additionally, the alphafold bundle uses a CGI script instead of the REST services.

Checklist:

---

alphafold
blastprotein ​Commit c05fe0
bogus Should consider removing bogus, but it's not functional so it's not necessary to worry about.
modeller
webservices Fix is simple: Just remove the opal file.

[Zach Pearson]

We should consider removing the Opal code from the blastprotein bundle for 1.3 -- it's been unused for a few weeks. If not reverted, it would be removed by ​c05fe0.

[Zach Pearson]

As of today Modeller is using the new backend and a pull request has been opened which will convert the alignments bundle to the new backend, remove OpalJob from the webservices bundle, and remove suds-community from prereqs/pips.

[Zach Pearson]

Blocking ChimeraX in PyPi and related tickets since we need to close this to bump setuptools.

[Zach Pearson]

I found that I could install poetry and suds-community at the same time. If we did want to use poetry we could begin that process even before Eric commits his changes to webservices. I personally think it's a great tool and I encourage others to skim its website: ​https://python-poetry.org/

This ticket is about updating our build tools being blocked. Tangentially related is that, for M1 packaging, we will need to either modify the bundle builder so that it creates universal shared objects when ext_modules is specified, or since it should be possible to put a package on PyPi or Conda without using ChimeraX to build it, we could move that out to pyproject.toml and build.py or setup.py.

I would strongly advocate for the latter approach -- I like the efficiency of solving two issues at once.

[pett]

Is the idea that poetry would be used in lieu of prereqs/pips, or would it be broader than that?

[Zach Pearson]

No, prereqs/pips could still be used.

From my perspective the question is about choosing the backend that best supports building our Python packages (core and our bundles) for publishing and inclusion in the desktop ChimeraX distribution.

My thinking is:

1. if we are able to solve installing a PyPi package within ChimeraX (1.4 milestone), and
2. we solve installing dependencies with 'devel install' or 'pip install' (1.4 milestone) from within ChimeraX, and
3. since for pure packages a setup.py build is being deprecated, and setup.cfg will be deprecated, and
4. therefore we must support pyproject.toml even for setuptools, and
5. we can pack as much custom metadata as we want into pyproject.toml

That would seem to solve all of the problems I can think of which would require the continued use of bundle_builder. The bulk of the PyPi packaging project reduces to properly defining setup.py (build.py for poetry) and pyproject.toml. Building M1 universal wheels reduces to properly defining those extensions in setup.py or build.py.

There are aesthetic differences between the two -- poetry is more full featured, with better support for pyproject.toml in my opinion.

[Zach Pearson]

Clarification: The setup.py deprecation is a pretty big undertaking because generating and executing one is how bundle_builder works.

[Zach Pearson]

A possible advantage of poetry is its dependency resolver, but pip has also supposedly had one since 20.3: ​https://twitter.com/whitequark/status/1324428822423478273

[Greg Couch]

The discussion does not appear to be related to Opal anymore.

[Zach Pearson]

That's true and not true at the same time. Each ticket has a request and a point; this ticket's request is to replace Opal, but its point is to clear a blocker that's keeping us from updating a build tool. If there's a viable alternative, which itself has a lot of knock-on effects and its own balance with regards to buy-in and benefit, it's related enough to me to mention that in the ticket. We still want Opal gone, but this could downgrade it from 'blocker' to 'major'.

[pett]

Muscle/Clustal-Omega services ported.

[Zach Pearson]

Great! Thank you! I'll look at getting suds-community out of our prereqs.