Support pyproject.toml

[Zach Pearson]

We should not make this a high priority as many infrastructure projects in the Python ecosystem are in the process of attaining PEP621 compliance, but as part of our project to release a PyPi and/or Conda version of ChimeraX, I think it would be good for us and for our bundle developers to support pyproject.toml as an alternative to bundle_info.xml. I see several benefits to this:

1) pyproject.toml is at least as extensible as XML. We need only to look for (at minimum) a [tool.chimerax] header when we read it in to get all of our custom metadata
2) We and our users could roll ChimeraX-specific project configuration data into a consolidated file, as I have already done in our repo for yapf, flake8, etc.
3) TOML files are simpler to write than XML files and have a much greater signal-to-noise ratio.

Among others.

If we did support pyproject.toml as a means to write bundle specifications, then we would have to implement the following:

• require tomli>=2 for Python versions <=3.11, which will have a built-in toml library.
• Update the bundle builder to read toml files in a way compatible with existing XML tags
• Make a stylistic decision whether to use additional headings, e.g. [tool.chimerax.classifiers] or just [tool.chimerax] for more complex data, such as providers.

My list is likely short owing to general inexperience with our bundle build system.

As to ecosystem support:

• setuptools is planning to support pyproject.toml alongside setup.{py,cfg}
• Common development tools such as pyright, yapf, pytest, and tox already support configuration through a project's top-level pyproject.toml

I imagine an outside developer would have several additional categories customizing tools in their local repo, but for an example of what pyproject.toml would look like for blastprotein:

# vim: set expandtab ts=4 sw=4:
# === UCSF ChimeraX Copyright ===
# Copyright 2021 Regents of the University of California.
# All rights reserved.  This software provided pursuant to a
# license agreement containing restrictions on its disclosure,
# duplication and use.  For details see:
# http://www.rbvi.ucsf.edu/chimerax/docs/licensing.html
# This notice must be embedded in or attached to all copies,
# including partial copies, of the software or any revisions
# or derivations thereof.
# === UCSF ChimeraX Copyright ===
[build-system]
requires = ["setuptools"]
build-backend = ["setuptools.build_meta"]

[project]
name="chimerax.blastprotein"
version="2.0"
authors = [{ name = "UCSF RBVI", email = "chimerax@cgl.ucsf.edu" }]
synopsis = "Search PDB/NR/AlphaFold using BLAST"
description = """
Interface to the RBVI web service to search the PDB, NR, or AlphaFold
databases using BLAST.  Results are shown in a tool when a graphical
interface is available; otherwise, the results are logged.
"""
classifiers = [
    "Development Status :: 5 - Stable",
    "Environment :: MacOS X :: Aqua",
    "Environment :: Win32 (MS Windows)",
    "Environment :: X11 Applications",
    "Framework :: ChimeraX",
    "Intended Audience :: Science/Research",
    "License :: Free for non-commercial use",
    "Operating System :: MacOS :: MacOS X",
    "Operating System :: Microsoft :: Windows :: Windows 10",
    "Operating System :: POSIX :: Linux",
    "Programming Language :: Python :: 3",
    "Topic :: Scientific/Engineering :: Visualization",
    "Topic :: Scientific/Engineering :: Chemistry",
    "Topic :: Scientific/Engineering :: Bio-Informatics",
]
requires-python = ">= 3.8"
dependencies = [
     "chimerax.core ~= 1.0"
]

[project.optional-dependencies]
gui = [ 'ChimeraX-UI >= 1.0' ]
webservices = [ 'ChimeraX-WebServices >= 1.0' ]
alignments = [ 'ChimeraX-UI >= 1.0', 'ChimeraX-Alignments >= 2.0' ]
alphafold = [ 'ChimeraX-AlphaFold >= 1.0' ]

[project.dependency-links]
# Tell pip to look at the toolshed, too
dependency-links = "https://cxtoolshed.rbvi.ucsf.edu"

[project.urls]
Home = "https://www.rbvi.ucsf.edu/chimerax/"
Documentation = "https://www.cgl.ucsf.edu/chimerax/docs/user/index.html"
Source = "https://www.github.com/RBVI/ChimeraX"

[tool.chimerax]
bundle-name="ChimeraX-BlastProtein"
min-session-version="1"
max-session-version="2"
categories = ["Sequence"]
chimerax-classifiers = [
    "Category :: Sequence",
    "Tool :: Blast Protein :: Sequence :: Search PDB/NR/AlphaFold using BLAST",
    "Command :: blastprotein :: Sequence :: Search PDB/NR/AlphaFold using BLAST",
    "Command :: blastprotein pull :: Sequence :: Get results for a finished BlastProtein job",
    "Command :: blastpdb :: Sequence :: Search PDB/NR/AlphaFold using BLAST"
]

Running python3 -m build in the blastprotein folder in this configuration will successfully generate a wheel, though since setuptools support is immature it's labelled UNKNOWN-0.0.0-py3-none-any.whl

[Greg Couch]

This in intriguing. The Python package build system is undergoing lots of changes right now. The immediate concern is that setup.py is deprecated in newer versions of setuptools, so the bundle_builder bundle will need to change "soon" if we are going to track setuptools. pyproject.toml might help.

[Zach Pearson]

Depending on how quickly setuptools can support it and how dire our needs are to upgrade setuptools, it could work out better to go with setup.cfg, but regardless I'm interested in learning more about the bundle builder and working with you on it, since I anticipate a need to understand it well for packaging anyway.

[Zach Pearson]

​Initial support has been merged into setuptools! According to ​this issue, there is a regression concerning not being able to include data files (​setuptools PR pending here). I'll make it a priority to deprecate suds so we can bump setuptools, but if we have bundles that use data files I'd propose waiting for a little bit.

[pett]

If by "data files' you mean the equivalent of the DataFiles tag in our bundle_info files, 41 bundles use that tag. If you mean the equivalent of the ExtraFiles tag, 7 bundles use that.

[Zach Pearson]

Stopgap: convert pyproject.toml to bundle_info.xml on the fly using ​pyproject_to_bundle_info.py (not yet a complete implementation).

[Zach Pearson]

If there are no objections I think it's appropriate to put the metadata writer that I was going to put in core inside bundle builder and bump the minor version. After discussion, where it seemed confusing that it was in core, I think that's the most appropriate place to put it. Bundle builder is also one of the closest things to an independent package we have, and using it as a build requirement is a natural extension of its current functionality.

[Zach Pearson]

Simple bundles -- with nothing but Python classifiers and a handful of ChimeraX Tool or Command classifiers -- can rely on toml if they want to by depending on bundle_builder and calling bundle_builder.write_bundle_classifiers().

[Zach Pearson]

As of ​this commit we now support pyproject.toml and editable bundle installs, but not by default. If a bundle_info.xml file exists, the devel command uses it.