Unable to install bundle with mdtraj (source code package) dependency

[jacknicoludis@…]

This is not so much a bug, but help with an error when trying to install a
new bundle. I am working on a bundle to read .h5 trajectory files (based on
the example for loading .xyz files on the ChimeraX webpage). I have written
the bundle, built it, but when I try to install it I get the following (the
full message from the devel install step):

```
devel install /Users/jack.nicoludis/trjh5
running build
running build_py
running bdist_wheel
installing to build/bdist.macosx-10.13-x86_64/wheel
running install
running install_lib
creating build/bdist.macosx-10.13-x86_64/wheel
creating build/bdist.macosx-10.13-x86_64/wheel/chimerax
creating build/bdist.macosx-10.13-x86_64/wheel/chimerax/trjh5
copying build/lib/chimerax/trjh5/io.py ->
build/bdist.macosx-10.13-x86_64/wheel/chimerax/trjh5
copying build/lib/chimerax/trjh5/__init__.py ->
build/bdist.macosx-10.13-x86_64/wheel/chimerax/trjh5
running install_egg_info
running egg_info
writing ChimeraX_trjh5.egg-info/PKG-INFO
writing dependency_links to ChimeraX_trjh5.egg-info/dependency_links.txt
writing requirements to ChimeraX_trjh5.egg-info/requires.txt
writing top-level names to ChimeraX_trjh5.egg-info/top_level.txt
reading manifest file 'ChimeraX_trjh5.egg-info/SOURCES.txt'
writing manifest file 'ChimeraX_trjh5.egg-info/SOURCES.txt'
Copying ChimeraX_trjh5.egg-info to
build/bdist.macosx-10.13-x86_64/wheel/ChimeraX_trjh5-0.1-py3.7.egg-info
running install_scripts
creating
build/bdist.macosx-10.13-x86_64/wheel/ChimeraX_trjh5-0.1.dist-info/WHEEL
creating 'dist/ChimeraX_trjh5-0.1-py3-none-any.whl' and adding
'build/bdist.macosx-10.13-x86_64/wheel' to it
adding 'chimerax/trjh5/__init__.py'
adding 'chimerax/trjh5/io.py'
adding 'ChimeraX_trjh5-0.1.dist-info/METADATA'
adding 'ChimeraX_trjh5-0.1.dist-info/WHEEL'
adding 'ChimeraX_trjh5-0.1.dist-info/top_level.txt'
adding 'ChimeraX_trjh5-0.1.dist-info/RECORD'
removing build/bdist.macosx-10.13-x86_64/wheel
Distribution is in
/Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-any.whl
[toolshed install](help:user/commands/toolshed.html#install)
/Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-any.whl

[toolshed install
/Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-
any.whl](cxcmd:toolshed install
/Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-any.whl)

Traceback (most recent call last):
File
"/Applications/ChimeraX-1.1.1.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/toolshed_utils/__init__.py",
line 489, in _pip_install
results = _run_pip(command, logger)
File
"/Applications/ChimeraX-1.1.1.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/chimerax/toolshed_utils/__init__.py",
line 540, in _run_pip
raise RuntimeError(s)
RuntimeError: Looking in indexes: https://pypi.org/simple,
https://cxtoolshed.rbvi.ucsf.edu/pypi/
Processing
/Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-any.whl
Requirement already satisfied: ChimeraX-Core~=1.1.1 in
/Applications/ChimeraX-1.1.1.app/Contents/lib/python3.7/site-packages (from
ChimeraX-trjh5==0.1) (1.1.1)
Requirement already satisfied: cython>=0.28 in
/Applications/ChimeraX-1.1.1.app/Contents/lib/python3.7/site-packages (from
ChimeraX-trjh5==0.1) (0.29.20)
Collecting mdtraj>=1.9.3
Using cached mdtraj-1.9.4.tar.gz (2.1 MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
WARNING: --use-feature=2020-resolver no longer has any effect, since it is
now the default dependency resolver in pip. This will become an error in
pip 21.0.
ERROR: Command errored out with exit status 70:
command: /Applications/ChimeraX-1.1.1.app/Contents/MacOS/ChimeraX
/Applications/ChimeraX-1.1.1.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip
install --ignore-installed --no-user --prefix
/private/var/folders/66/scl1xcmn6v9gncjw3wbp69qc0000gp/T/pip-build-env-37344ypk/overlay
--no-warn-script-location --no-binary :none: --only-binary :none: -i
https://pypi.org/simple --extra-index-url
https://cxtoolshed.rbvi.ucsf.edu/pypi/ -- setuptools wheel numpy
'Cython>0.28'
cwd: None
Complete output (0 lines):
----------------------------------------
ERROR: Command errored out with exit status 70:
/Applications/ChimeraX-1.1.1.app/Contents/MacOS/ChimeraX
/Applications/ChimeraX-1.1.1.app/Contents/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip
install --ignore-installed --no-user --prefix
/private/var/folders/66/scl1xcmn6v9gncjw3wbp69qc0000gp/T/pip-build-env-37344ypk/overlay
--no-warn-script-location --no-binary :none: --only-binary :none: -i
https://pypi.org/simple --extra-index-url
https://cxtoolshed.rbvi.ucsf.edu/pypi/ -- setuptools wheel numpy
'Cython>0.28' Check the logs for full command output.
```

The error indicates logs that might help figure out the issue but I don't
know where they are. I also do not know what exit status 70 refers to and
there's no documentation on this. Any suggestions?

Best,
Jack

[Greg Couch]

Csn you send me a copy of /Users/jack.nicoludis/trjh5/dist/ChimeraX_trjh5-0.1-py3-none-any.whl so I can dig into what's going on? I'd like to be able to reproduce your bug so I can confirm when it's fixed.

[jacknicoludis@…]

Hi,
I'm attaching the .whl file. I've changed the package since I first opened
this bug, but I think the bug is still the same. I think the issue is
because I am importing mdtraj in my package. I'm guessing mdtraj is not an
included package in ChimeraX. So, now the question is how can I make mdtraj
a dependency. Is it possible to install a new dependency in a bundle?

On Sat, Dec 5, 2020 at 3:41 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

MDTrajViewer-1.0-py3-none-any.whl​

[jacknicoludis@…]

Added by email2trac

[Greg Couch]

The bug is due to how mdtraj dependency installs. It calls "python path-to-pip args" instead of "python -m pip args". Almost have a fix.

Work on this has uncovered a nasty bug with numpy and Python in Windows 10 build 2004: ​https://developercommunity.visualstudio.com/content/problem/1207405/fmod-after-an-update-to-windows-2004-is-causing-a.html

[jacknicoludis@…]

I'm wondering about the status of this bug. Please let me know! Thanks!

Jack

On Tue, Dec 8, 2020 at 12:20 AM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Greg Couch]

I've been working on this, on and off. The problem comes from the fact that the mdtraj is only available in source form. I haven't been able to figure what exactly is going wrong, but I have a workaround that I need to think more about the possible side effects. I've milestoned this to make sure this is fixed before 1.2 comes out. For testing purposes, you can download the mdtraj package source and use Chimera's python to build and install it (python setup.py install). Then, when you install your bundle, the dependency will already be taken care of, and your bundle should install.

Long term, the fact that mdtraj is only available is source form is a big problem. Windows users typically do not have a compiler installed and they are the largest installed base. And for macOS and Linux, even though the compilers are often available, they are not guaranteed to be. If this were a bundle the RBVI developed, we would compile it and stick it in our repository. We do not want to be in the business of compiling third party packages, nor keeping them up-to-date, but that might be necessary for your bundle to be usable on the toolshed.

[jacknicoludis@…]

Thanks for the prompt reply. I understand the issue and will work on the
workaround for my specific application. It is possible that I could extract
the necessary aspects for loading trajectory files from mdtraj and package
them into my bundle without the need to compile mdtraj separately. We'll
see what I come up with with the limited time I am devoting to this.

The reason I am developing this bundle is because the only way that I've
found to visualize trajectory data is using the Molecular Dynamics Viewer
bundle, which only allows .pdb format trajectory files. These are not
efficient trajectory file formats, so I was hoping to support visualizing
trajectories using the mdtraj hdf5 trajectory format (or even the GROMACS
.xtc file format would be nice!). I was wondering if there are other
trajectory visualization features that are expected in future versions of
ChimeraX and what the timeline on those would be. I'd be thankful for any
information you have on the development plans for trajectory data.

Thanks!

On Mon, Dec 21, 2020 at 3:14 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Greg Couch]

Eric, could you comment on trajectory support? Thanks.

[Eric Pettersen]

ChimeraX's support for trajectories is way behind Chimera, which in turn is somewhat behind VMD. Nonetheless, ChimeraX can read trajectory coordinates in the following formats: DCD, xtc, trr, and (in the daily build) Amber NetCDF. To do this you need to open a structure that corresponds to any single frame of your trajectory (with the atoms in the same order as in the trajectory coordinates), and then open the trajectory coordinates with a second "open" command (e.g. "open trajectory.xtc structure #1") to load them into the structure. You can bring up a simple slider for playing through or recording the trajectory with the "coordset slider" command.

The code that reads the trajectory coordinates can be found in the md_crds module (read_coords.py). We hope to develop additional capabilities for MD analysis, but there are a lot of competing priorities that makes it unlikely that we'll get to it real soon. So any third-party contributions are welcome! :-)

--Eric

[jacknicoludis@…]

Awesome - thank you for the answer. I wasn't aware I could open the other
file formats you listed, but it is still somewhat limited if you are unable
to load .gro files as the topology and then an .xtc file as the trajectory,
since these are the standard outputs for GROMACS simulations. Do you know
if .gro files are planned to be supported?

On Mon, Dec 21, 2020 at 4:12 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Eric Pettersen]

Okay, I ported Chimera's .gro-reading code to ChimeraX. You will be able to open .gro files (as a structure, not a trajectory) in tomorrow's daily build. You will then be able to add .xtc/.trr trajectory coordinates to such a structure.

--Eric

[jacknicoludis@…]

Thank you! That's super helpful. If there's something I can do to help add
support for the mdtraj .hdf5 file format, I'd be happy to help. If you
could point me to where the file format reading code is, I can try to take
a stab at it.

Thank you again!

On Mon, Dec 21, 2020 at 6:51 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Eric Pettersen]

Well, if you are still using your 1.1.1 installation, the code that reads the trajectory file for the current formats is in read_coords.py in the /Applications/ChimeraX-1.1.1.app/Contents/lib/python3.7/site-packages/chimerax/md_crds folder.

It is probably best to use that code as a guideline to base your own bundle on (rather than editing it in place), because getting ChimeraX to look in that folder for reading .h5 files is another whole ball of wax and probably involves editing /Applications/ChimeraX-1.1.1.app/Contents/lib/python3.7/site-packages/ChimeraX_MDcrds-2.0.dist-info/METADATA to add the info for the .h5 data format and file reader and then rebuilding the toolshed cache with the "toolshed reload" command -- and I'm not 100% sure that's sufficient.

One "cheat code" you could use to directly use read_coords.py is to rename your .h5 trajectory to one of the other known trajectory suffixes and temporarily replace the code that reads that format with your code to read .h5. This assumes you want to eventually send me the .h5 code to add to md_crd rather than develop your own bundle.

To develop your own bundle, there is a lot of support in the Developers Guide (​https://www.cgl.ucsf.edu/chimerax/docs/devel) and particularly the "Seven Easy Steps" part, specifically the "Read a New File Format" section (​https://www.cgl.ucsf.edu/chimerax/docs/devel/tutorials/introduction.html#writing-bundles-in-seven-easy-steps).

--Eric

[Greg Couch]

Won't make 1.2

[Greg Couch]

Hoping to fix this as a side effect of dealing with other pip related bugs.