Install PyPi packages with a ChimeraX command

[Tom Goddard]

For developers writing ChimeraX Python code it would be convenient if a ChimeraX command like "pip install pandas" could install extra PyPi packages they need.

Here is an email discussion of the trouble that arises trying to install PyPi packages into ChimeraX on Windows:

​https://plato.cgl.ucsf.edu/pipermail/chimerax-users/2021-June/002297.html

While it seems like adding such command would be trivial, unfortunately I recall the Python pip module does not have a function API to install packages! Seem incredibly poorly designed only to work as a script for the pip shell command. But I also recall some additional PyPi packages adds the pip APIs.

[Greg Couch]

Plan to get this into 1.3.

[Greg Couch]

Needs to many changes to internals to be done before 1.3 release.

[Zach Pearson]

I found if I opened the shell tool I was able to pip install --upgrade pip, but the original ticket references Windows. I'll try the same command on Windows and see if it puts files in the right place. If that's the case then that could be one way to script it.

[Greg Couch]

Won't make 1.4.

[Zach Pearson]

I found that it wasn't too hard to implement this with a subprocess based approach. The key is calling subprocess with sys.executable rather than python3, because the former will choose ChimeraX's Python and the latter will choose the system Python.

Tom suggested using ​pip-api, a package that claims to provide an importable pip API. I elected not to go this route because in the end, that package also just calls subprocess to invoke the pip command line. It also does some magic to divine where pip is installed using os.environ.get("PIPAPI_PYTHON_LOCATION"), falling back to sys.executable if that env var isn't acceptable.

I wanted more control over what a user could do, the behavior of pip, and a smaller code surface.

[Zach Pearson]

It would be an interesting experiment to distribute a very minimal ChimeraX on macOS and install those pesky platform specific dependencies on first start rather than trying to make a universal build.

[Zach Pearson]

I've tested the module I made on macOS and Windows and it seems to work!

[goddard@…]

Does it install the PyPi package in the ChimeraX application site-packages or in the user directory (ie with pip option "--user")?  I'm not sure what we want it to do by default, but probably the user area since that is where Toolshed puts everything it installs I think.

[Zach Pearson]

From what I can tell it goes to the ChimeraX site-packages directory, but I agree with you that the user directory is probably better. Let me test inserting the "--user" flag and see what happens.

[Zach Pearson]

Yup, "--user" was the key.

[pett]

Nice! Is it a "toolshed" subcommand?

Also, "--user" is definitely the way to go because sometimes the app isn't writable by non-privileged users.

[Zach Pearson]

No, but should it be?

"toolshed install --pip"?
vs
"pip install"?

[goddard@…]

I kind of like "pip install pandas" as the ChimeraX command -- that was the original idea in the ticket description for the command name.  The current "toolshed install" command syntax is

	toolshed install  bundle-name [ version ] [ noDeps true | false ]

It seems a bit strange to use "toolshed install" for PyPi since nothing is coming from Toolshed and it isn't installing a bundle with a bundle_info.xml.  So I think the command "pip install" still makes more sense to me.

[Zach Pearson]

That's how it is now. You can do

pip {install, uninstall, list, check, show, search, debug}, and you've got two flags: upgrade and verbose.

If you can think of anything else useful let me know.

[goddard@…]

Fantastic!  Thanks!

[pett]

In this case I agree. For Zach's benefit, the issues/advantages/disadvantages involved when choosing a new command vs. a subcommand are:

New command:

• Not closely related to an existing command
• Might be difficult to remember/discover as a subcommand of an existing command

Subcommand:

• Fewer top-level commands makes command index less unwieldy and can aid discovery
• Is something that users of the main command would frequently be interested in doing, and they would find out about by reading the main command help
• Is something users would naturally consider as part of the main command functionality

This new command has elements from both the above, but is probably distinct enough to justify its own command.

[pett]

If it were going to be a subcommand, it would actually most naturally be a subcommand of "devel" rather than "toolshed". BTW, it is trivially easy to make it a subcommand of devel simply by naming the command "devel pip" rather than just "pip". I kind of favor "devel pip" actually, but wouldn't die if it remained "pip".

[goddard@…]

For a fair number of commands we make both a top level command and a subcommand, so both "pip install" and "devel pip install" where one is just an alias of the other.  For example there is a "vr on" command and a "device vr on" command.  In the documentation it is then grouped with other "device" commands, like "device webcam" or "device snav" (space navigator 3d mouse).  I think probably "pip" and "devel pip" is a case where allowing both command names makes sense.

[Zach Pearson]

Added a 'devel pip' alias