OpenMM on PyPi

[Tristan Croll]

The following bug report has been submitted:
Platform:        Windows-10-10.0.22631
ChimeraX Version: 1.8.dev202404160016 (2024-04-16 00:16:48 UTC)
Description
FYI: the OpenMM project recently hired some consultants for the migration to PyPI, and it looks like they're very close to being ready to go. They have a set of release-candidate wheels at https://github.com/openmm/openmm-wheels/releases/tag/8.1.1-6 if you want to try them out. I was part of the early discussions and pushed on the need for ABI compatibility (i.e. for them to work with GCC-toolset-10 for the Linux builds), but I have a feeling that hasn't happened with the current candidate builds. Now would probably be a good time to get involved in the discussion, while they're still actively working on it. 

OpenGL version: 3.3.0 NVIDIA 552.22
OpenGL renderer: NVIDIA GeForce RTX 3070 Laptop GPU/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation

Python: 3.11.4
Locale: en_GB.cp1252
Qt version: PyQt6 6.6.1, Qt 6.6.1
Qt runtime version: 6.6.3
Qt platform: windows

Manufacturer: HP
Model: HP ZBook Studio 15.6 inch G8 Mobile Workstation PC
OS: Microsoft Windows 11 Pro (Build 22631)
Memory: 34,007,068,672
MaxProcessMemory: 137,438,953,344
CPU: 16 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
OSLanguage: en-GB

Installed Packages:
    alabaster: 0.7.16
    appdirs: 1.4.4
    asttokens: 2.4.1
    Babel: 2.14.0
    beautifulsoup4: 4.12.3
    blockdiag: 3.0.0
    blosc2: 2.0.0
    build: 1.2.1
    certifi: 2024.2.2
    cftime: 1.6.3
    charset-normalizer: 3.3.2
    ChimeraX-AddCharge: 1.5.16
    ChimeraX-AddH: 2.2.6
    ChimeraX-AlignmentAlgorithms: 2.0.2
    ChimeraX-AlignmentHdrs: 3.4.3
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.12.5
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.1.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.56.1
    ChimeraX-AtomicLibrary: 14.0.3
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.4
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.4
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.12.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.4
    ChimeraX-ChangeChains: 1.1
    ChimeraX-CheckWaters: 1.4
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.2.4
    ChimeraX-Clipper: 0.23.0
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.5
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.8.dev202404160016
    ChimeraX-CoreFormats: 1.2
    ChimeraX-coulombic: 1.4.3
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-Dicom: 1.2
    ChimeraX-DistMonitor: 1.4.2
    ChimeraX-DockPrep: 1.1.3
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.2
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ISOLDE: 1.8.dev0
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-IUPAC: 1.0
    ChimeraX-Label: 1.1.9
    ChimeraX-ListInfo: 1.2.2
    ChimeraX-Log: 1.1.6
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.9.1
    ChimeraX-Map: 1.2
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.1.3
    ChimeraX-MCopy: 1.0
    ChimeraX-MDcrds: 2.7
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.14.1
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.15
    ChimeraX-ModelPanel: 1.5
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0.3
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.1
    ChimeraX-NMRSTAR: 1.0.2
    ChimeraX-NRRD: 1.1
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.13.4
    ChimeraX-PDB: 2.7.5
    ChimeraX-PDBBio: 1.0.1
    ChimeraX-PDBLibrary: 1.0.4
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1.1
    ChimeraX-PubChem: 2.2
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.2
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.4.1
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.2
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 4.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.2
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-Segmentations: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.11.2
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.3
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1.2
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.16.3
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.2
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.5
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-TaskManager: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.4
    ChimeraX-Topography: 1.0
    ChimeraX-ToQuest: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.37.2
    ChimeraX-uniprot: 2.3
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.4
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-vrml: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.3
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.2.2
    comtypes: 1.4.1
    contourpy: 1.2.1
    cxservices: 1.2.2
    cycler: 0.12.1
    Cython: 3.0.10
    debugpy: 1.8.1
    decorator: 5.1.1
    docutils: 0.20.1
    executing: 2.0.1
    filelock: 3.13.4
    fonttools: 4.51.0
    funcparserlib: 2.0.0a0
    glfw: 2.7.0
    grako: 3.16.5
    h5py: 3.11.0
    html2text: 2024.2.26
    idna: 3.7
    ihm: 1.0
    imagecodecs: 2024.1.1
    imagesize: 1.4.1
    ipykernel: 6.29.2
    ipython: 8.21.0
    ipywidgets: 8.1.2
    jedi: 0.19.1
    Jinja2: 3.1.3
    jupyter-client: 8.6.0
    jupyter-core: 5.7.2
    jupyterlab-widgets: 3.0.10
    kiwisolver: 1.4.5
    line-profiler: 4.1.2
    lxml: 5.2.1
    lz4: 4.3.3
    MarkupSafe: 2.1.5
    matplotlib: 3.8.4
    matplotlib-inline: 0.1.7
    msgpack: 1.0.8
    nest-asyncio: 1.6.0
    netCDF4: 1.6.5
    networkx: 3.3
    nibabel: 5.0.1
    nptyping: 2.5.0
    numexpr: 2.10.0
    numpy: 1.26.4
    openvr: 1.26.701
    packaging: 24.0
    ParmEd: 4.2.2
    parso: 0.8.4
    pep517: 0.13.1
    pillow: 10.3.0
    pip: 24.0
    pkginfo: 1.10.0
    platformdirs: 4.2.0
    prompt-toolkit: 3.0.43
    psutil: 5.9.8
    pure-eval: 0.2.2
    py-cpuinfo: 9.0.0
    pycollada: 0.8
    pydicom: 2.3.0
    pygments: 2.17.2
    pynmrstar: 3.3.4
    pynrrd: 1.0.0
    PyOpenGL: 3.1.7
    PyOpenGL-accelerate: 3.1.7
    pyopenxr: 1.0.3401
    pyparsing: 3.1.2
    pyproject-hooks: 1.0.0
    PyQt6-commercial: 6.6.1
    PyQt6-Qt6: 6.6.3
    PyQt6-sip: 13.6.0
    PyQt6-WebEngine-commercial: 6.6.0
    PyQt6-WebEngine-Qt6: 6.6.3
    python-dateutil: 2.9.0.post0
    pytz: 2024.1
    pywin32: 306
    pyzmq: 26.0.0
    qtconsole: 5.5.1
    QtPy: 2.4.1
    RandomWords: 0.4.0
    requests: 2.31.0
    scipy: 1.13.0
    setuptools: 69.5.1
    sfftk-rw: 0.8.1
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.5
    sphinx: 7.2.6
    sphinx-autodoc-typehints: 2.0.1
    sphinxcontrib-applehelp: 1.0.8
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.6
    sphinxcontrib-htmlhelp: 2.0.5
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.7
    sphinxcontrib-serializinghtml: 1.1.10
    stack-data: 0.6.3
    superqt: 0.6.3
    tables: 3.8.0
    tcia-utils: 1.5.1
    tifffile: 2024.1.30
    tinyarray: 1.2.4
    tornado: 6.4
    traitlets: 5.14.2
    typing-extensions: 4.11.0
    tzdata: 2024.1
    urllib3: 2.2.1
    wcwidth: 0.2.13
    webcolors: 1.13
    wheel: 0.43.0
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.10
    WMI: 1.5.1

[Tom Goddard]

I tried the Mac ARM wheel OpenMM-8.1.1-cp311-cp311-macosx_11_0_arm64.whl and it worked fine with the ChimeraX tug mouse mode in the current ChimeraX 1.8 code after I took out the code that sets the OPENMM_PLUGIN_DIR environment variable (#15082).

While I could try Mac Intel, Windows and Linux, I suspect it will work and if it does not the OpenMM team will fix it, so I am not inclined to spend the time.

Regarding getting them to distribute the Linux PyPi openmm with different C++ ABIs -- good luck. Usually PyPi wheels are strictly for Python module use, not linking your own C++ code. Sometimes like with numpy you can link to C which has a portable C ABI, but I don't know of any example where PyPi wheels support this with C++. Tristan, can you use OpenMM entirely via Python if some new Python OpenMM APIs were added? Or does your use in ISOLDE require linking in C++?

[Tom Goddard]

On Linux they have two OpenMM wheels, one with everything except the CUDA plugins (about 10 Mbytes) and another with just the cuda plugins (about 1 Mbyte). I guess we could include both on Linux so that users with CUDA could get the higher performance. The two wheels are described in the openmm-wheels github readme.

​https://github.com/openmm/openmm-wheels

[Tristan Croll]

Looks like they are actually using the gcc-toolset-10 compiler, but the
current builds weren't compiled with `std=c++11` so symbol resolution is
failing on anything involving std::string (and probably elsewhere - that's
just the first thing it hits). I've gotten back to them on that, but also
(re-)opened the discussion on implementing the features I need in OpenMM to
allow me to work purely through the Python API.

On Wed, May 1, 2024 at 11:52 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>

[Tristan Croll]

Turns out avoiding the c++11 ABI was a deliberate choice, to maintain
compatibility with the manylinux2014 standard (see
https://discuss.python.org/t/how-to-set-glibcxx-use-cxx11-abi-for-manylinux2014-and-manylinux2010-wheels/10551).
Looks like they need to keep it that way because the openmm-torch package
uses the pytorch C++ API, and pytorch is manylinux2014-compliant.

On Thu, May 2, 2024 at 2:27 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>

[Tristan Croll]

I see two possible minimal-effort ways around this:

1. ChimeraX defines
_GLIBCXX_USE_CXX11_ABI=0 when building (and adds that to the defines in
bundle builder). Shouldn’t change anything about how ChimeraX performs nor
require any other code changes (see
https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html), and
has the advantage that it maximises compatibility should anyone else see
the need to use the C++ API from a PyPI package.
2. I reconfigure things so each of my compiled libraries depends on only
one of OpenMM or ChimeraX, then add the relevant
_GLIBCXX_USE_CXX11_ABI definition for each.

On Thu, 2 May 2024 at 16:22, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>

[Tom Goddard]

I don't know what the implications of _GLIBCXX_USE_CXX11_ABI=0 would be. Greg handles our Linux ChimeraX builds and perhaps can comment on whether that change would have adverse consequences. I'm not sure if other Toolshed bundles compiles against our ChimeraX C++ atomic library. If so, they would also need to change their builds to have a compatible non-CXX11 ABI.

[Greg Couch]

I'm in favor of option 2. I want to be able to switch ChimeraX to C++17, like Qt 6 has.

[Tristan Croll]

That's working, albeit still with one minor hack: I had to put a symlink to
libOpenMM.so in ChimeraX/lib for ISOLDE to be able to link the library at
runtime. I've asked them to implement Zach's request at
https://github.com/openmm/openmm/issues/3820 to provide get_include() and
get_lib() methods, which if I understand correctly should take care of that.

On Thu, May 2, 2024 at 11:19 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>

[Tom Goddard]

We plan on starting release candidates for ChimeraX 1.8 next week and releasing around June 1, 2024. Should we stick with our current OpenMM 8.0.0 taken from Conda? Or should we try to use the PyPi OpenMM 8.1.1? I'm inclined to stick with 8.0.0 because there is not enough time to catch problems before the release.

[Tristan Croll]

I'm flexible either way, but I admit it would be nice to have the new
version in place - still hoping to get some active development happening
again, and there are some new features in there that would be great to
experiment with.

On another note: Peter's not all that keen on implementing the get_lib()
method due to worries about getting the implementation right for different
scenarios (they still want to support a separate conda package, and people
building from source)... but that should be reasonably easy to work around.
At runtime I can add some code to ISOLDE's custom_init to find and add the
OpenMM lib directory to the search path. A little more difficult at compile
time... is there (or is it possible to add) something like a
CHIMERAX_BASE_DIR macro that I can use in the <LibraryDir> and <IncludeDir>
entries?

On Fri, May 3, 2024 at 7:07 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>

[Tom Goddard]

We can put OpenMM 8.1.1 in the daily builds after the ChimeraX 1.8 release to experiment with its new capabilities.

I'm not sure about having ChimeraX tell you its base directory. Also what do you mean by "base directory"? If you mean the top install directory for chimerax, there is then the issue that Linux, Mac, Windows all use different directory structures within the ChimeraX app, so maybe you really want a path to the ChimeraX lib directory? I'm all not sure what you mean about adding the OpenMM lib directory to the "search path". What search path? Isn't the issue that the operating system runtime library loader has to find the library?

When I installed the OpenMM candidate pypi wheel on Mac the OpenMM libraries ended up in site-packages at

ChimeraX.app/Contents/lib/python3.11/site-packages/OpenMM.libs/lib

When you build your ISOLDE library linking against OpenMM can you give it a relative runtime loader path to find the OpenMM libraries, since ISOLDE is also under site-packages -- oops, right ISOLDE is in the user's home directory, not the app.

A possible solution is that we just make the symbolic links to OpenMM libraries in the ChimeraX lib directory. But do symbollic links exist/work on Windows?

Another idea -- if the OpenMM libraries are already loaded by the process then I think the runtime loader is not going to search for them. So maybe if you just import openmm before importing ISOLDE libraries that solves the problem.

[Tom Goddard]

One further note about the original base directory question. I think python sys.executable gives the full path to the ChimeraX executable.

[Zach Pearson]

Replying to Tristan Croll:

...for different
scenarios (they still want to support a separate conda package, and people
building from source)...

Fine by me. We can have a workaround in ChimeraX.

[Tom Goddard]

On Aug 20, 2024, at 12:39 PM, Zach Pearson <zjp@…> wrote:

That’s unfortunate, thanks for the info. Updating OpenMM isn’t too hard, but it would’ve been nice to have. I’ve put some effort into learning GitHub Actions, so we can fully automate it now.

— Zach

On 20 Aug 2024, at 12:30, Tristan Croll <tcroll@…> wrote:

Hi Zach,

In principle, yes - I’ve tried it out (in Linux) and it’s pretty straightforward to make it work. In practice it might be a touch premature: my efforts to get Altos to give them help in this migration have hit seemingly endless roadblocks (actually had an engineer put a bunch of work in, but it’s been stuck pending internal code review for months now). Meanwhile they hired some outside consultants whose solution was apparently super convoluted and unmaintainable. If I can’t find a way to get them some tangible help soon, there’s a substantial chance they’ll abandon the whole PyPI foray and go back to Conda only.

— Tristan

On Tue, 20 Aug 2024 at 19:51, Zach Pearson <zjp@…> wrote:
Hi Tristan,

Have you tried out the PyPi version of OpenMM with Isolde? Would it work in ChimeraX instead of our current workflow of getting it from Conda and repackaging it?

— Zach

Altos Labs UK Limited | England | Company reg 13484917
Registered address: 3rd Floor 1 Ashley Road, Altrincham, Cheshire, United Kingdom, WA14 2DT