Bundle builder cannot install PyPi dependency pyrealsense2

[Tom Goddard]

How do I add a PyPi dependency to a Toolshed bundle? I have not been able to get it to work.

I have a bundle chimerax/src/bundles/realsense that controls a RealSense depth sensing camera that I would like to put on the Toolshed. It has a PyPi dependency pyrealsense2 that I have not been able to get bundle builder to install.

I tried adding the dependency in the realsense/bundle_info.xml as follows

<Dependencies>

<Dependency name="ChimeraX-Core" version=">=0.9"/>
<Dependency name="pyrealsense2" version=">=2.30.0.1184"/>

</Dependencies>

Then make install fails saying

pkg_resources.DistributionNotFound: The 'pyrealsense2' distribution was not found and is required by the application

Here is the full traceback

$ make install
../../../ChimeraX.app/bin/ChimeraX.exe --nogui --safemode --exit --cmd "devel install . user false exit true "
INFO:
Executing: devel install . user false exit true
INFO:
Traceback (most recent call last):

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\runpy.py", line

193, in _run_module_as_main

"main", mod_spec)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\runpy.py", line

85, in _run_code

exec(code, run_globals)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\ChimeraX_main.py", line 871, in <module>

exit_code = init(sys.argv)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\ChimeraX_main.py", line 643, in init

run(sess, cmd)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\core\commands\run.py", line 31, in run

results = command.run(text, log=log)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\core\commands\cli.py", line 2837, in run

result = ci.function(session, kw_args)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\core\commands\devel.py", line 116, in devel_install

session, test=test, debug=debug, user=user)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\core\commands\devel.py", line 146, in _run

bb = _get_builder(path, logger)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\core\commands\devel.py", line 190, in _get_builder

bb = BundleBuilder(logger, bundle_path=path)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 52, in init

self._make_setup_arguments()

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 530, in
_make_setup_arguments

for cm in self.c_modules]

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 530, in <listcomp>

for cm in self.c_modules]

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 843, in ext_mod

libraries, cpp_flags) = self._compile_options(logger, dependencies)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 749, in
_compile_options

d_inc, d_lib = self._get_bundle_dirs(logger, dep)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\chimerax\bundle_builder\bundle_builder.py", line 762, in
_get_bundle_dirs

if not get_distribution(req):

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\pkg_resources
init.py", line 481, in get_distribution

dist = get_provider(dist)

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\pkg_resources
init.py", line 357, in get_provider

return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\pkg_resources
init.py", line 900, in require

needed = self.resolve(parse_requirements(requirements))

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-

packages\pkg_resources
init.py", line 786, in resolve

raise DistributionNotFound(req, requirers)

pkg_resources.DistributionNotFound: The 'pyrealsense2==2.30.0.1184'
distribution was not found and is required by the application

BUG:
pkg_resources.DistributionNotFound: The 'pyrealsense2==2.30.0.1184'
distribution was not found and is required by the application

File "C:\cygwin64\home\goddard\chimerax\ChimeraX.app\bin\lib\site-
packages\pkg_resources
init.py", line 786, in resolve
raise DistributionNotFound(req, requirers)

_See log for complete Python traceback._

0.00% done: Initializing core
50.00% done: Initializing bundles
make: * install Error 1

[Tom Goddard]

This is on Windows 10. I am using vive.cgl.ucsf.edu for the tests.

The bundle installs correctly with pip:

$ ~/chimerax/ChimeraX.app/bin/ChimeraX.exe -m pip install pyrealsense2==2.30.0.1184
Collecting pyrealsense2==2.30.0.1184

Using cached ​https://files.pythonhosted.org/packages/f9/07/e899a24b7c86ef806befde4506c653781708948ce4ff1d21896ebd878d59/pyrealsense2-2.30.0.1184-cp37-cp37m-win_amd64.whl

Installing collected packages: pyrealsense2
Successfully installed pyrealsense2-2.30.0.1184
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Lock 2499498902728 acquired on C:\Users\goddard\AppData\Local\UCSF\ChimeraX\Cache\0.91\toolshed\bundle_info.cache.lock
Lock 2499498902728 released on C:\Users\goddard\AppData\Local\UCSF\ChimeraX\Cache\0.91\toolshed\bundle_info.cache.lock

[Tom Goddard]

When I install with pip it says it is using a cached wheel for pyrealsense2. If I uninstall pyrealsense2 and then install without using the cache it also works correctly:

$ ~/chimerax/ChimeraX.app/bin/ChimeraX.exe -m pip install --no-cache-dir pyrealsense2==2.30.0.1184
Collecting pyrealsense2==2.30.0.1184

Downloading ​https://files.pythonhosted.org/packages/f9/07/e899a24b7c86ef806befde4506c653781708948ce4ff1d21896ebd878d59/pyrealsense2-2.30.0.1184-cp37-cp37m-win_amd64.whl (8.5MB)

Installing collected packages: pyrealsense2
Successfully installed pyrealsense2-2.30.0.1184
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Lock 2149275821064 acquired on C:\Users\goddard\AppData\Local\UCSF\ChimeraX\Cache\0.91\toolshed\bundle_info.cache.lock
Lock 2149275821064 released on C:\Users\goddard\AppData\Local\UCSF\ChimeraX\Cache\0.91\toolshed\bundle_info.cache.lock

[Tom Goddard]

Changing the bundle_info.xml line to request version >= 2 or omitting the version does not change the error -- still get distribution not found.

[Conrad Huang]

That's very odd. I added the exact same dependency line to a bundle and it installed with no problems (see below). Can you try blowing away the pip cache completely? On Windows 10, it's at C:\Users\username\AppData\Local\pip\cache.

$ make install
../../../ChimeraX.app/bin/ChimeraX.exe --nogui --safemode --exit --cmd "devel install . user false exit true  "
INFO:
Executing: devel install . user false exit true
running build
running build_py
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
writing ChimeraX_SpeechRecognition.egg-info\PKG-INFO
writing dependency_links to ChimeraX_SpeechRecognition.egg-info\dependency_links.txt
writing requirements to ChimeraX_SpeechRecognition.egg-info\requires.txt
writing top-level names to ChimeraX_SpeechRecognition.egg-info\top_level.txt
reading manifest file 'ChimeraX_SpeechRecognition.egg-info\SOURCES.txt'
writing manifest file 'ChimeraX_SpeechRecognition.egg-info\SOURCES.txt'
running build_ext
running bdist_wheel
installing to build\bdist.win-amd64\wheel
running install
running install_lib
creating build\bdist.win-amd64\wheel
creating build\bdist.win-amd64\wheel\chimerax
creating build\bdist.win-amd64\wheel\chimerax\speech
copying build\lib\chimerax\speech\cmd.py -> build\bdist.win-amd64\wheel\.\chimerax\speech
copying build\lib\chimerax\speech\speech.py -> build\bdist.win-amd64\wheel\.\chimerax\speech
copying build\lib\chimerax\speech\__init__.py -> build\bdist.win-amd64\wheel\.\chimerax\speech
running install_egg_info
Copying ChimeraX_SpeechRecognition.egg-info to build\bdist.win-amd64\wheel\.\ChimeraX_SpeechRecognition-1.0-py3.7.egg-info
running install_scripts
creating build\bdist.win-amd64\wheel\ChimeraX_SpeechRecognition-1.0.dist-info\WHEEL
creating 'dist\ChimeraX_SpeechRecognition-1.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
adding 'chimerax/speech/__init__.py'
adding 'chimerax/speech/cmd.py'
adding 'chimerax/speech/speech.py'
adding 'ChimeraX_SpeechRecognition-1.0.dist-info/METADATA'
adding 'ChimeraX_SpeechRecognition-1.0.dist-info/WHEEL'
adding 'ChimeraX_SpeechRecognition-1.0.dist-info/top_level.txt'
adding 'ChimeraX_SpeechRecognition-1.0.dist-info/RECORD'
removing build\bdist.win-amd64\wheel
Distribution is in .\dist\ChimeraX_SpeechRecognition-1.0-py3-none-any.whl
INFO:
Executing: toolshed install ".\\dist\\ChimeraX_SpeechRecognition-1.0-py3-none-any.whl" userOnly false
Successfully installed ChimeraX-SpeechRecognition-1.0
Installed ChimeraX-SpeechRecognition (1.0)

0.00% done: Initializing core
50.00% done: Initializing bundles

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
$ /d/chimerax/ChimeraX.app/bin/python -m pip list
Package                          Version
-------------------------------- -----------
[...]
PyQtWebEngine-commercial         5.12.1
pyrealsense2                     2.30.0.1184
python-dateutil                  2.8.1
[...]

[Tom Goddard]

I removed the pip cache and get the same error.
I'll attach the bundle_info.xml file. Try it with chimerax/src/bundles/realsense. Maybe there is something wrong in my file.

[Tom Goddard]

Your "make install" output for SpeechRecognition does not indicate it installed pyrealsense, although I see from your following pip list command it was installed previously.

[Conrad Huang]

It looks like the <Module> tag in bundle_info.xml is the culprit. The same dependency line works in my bundle with no C module. Looking into why...

[Tom Goddard]

Thanks.  The reason I want this to work is so my realsense bundle can go on Tool Shed because very few users will have the camera that uses it and so I don't want to put pyrealsense into prereqs for something almost no one will use.  But the NIAID / MSC folks are excited to use this so I want it to be easily accessible to them through Tool Shed.  Thanks for looking at it.

[Conrad Huang]

Should be fixed in 60c893b33.

Bundle builder was looking for include and library directories from packages that the compiling bundle depend on. Now it just assumes that uninstalled bundles are not needed for compilation.

Tom, can you please confirm it works?

[Tom Goddard]

Yes, working for me.  Thanks.