Programmer documentation missing atomic and map_data classes

[Tom Goddard]

Lots of ChimeraX APIs such as Atoms are Atom are not in the programming documentation because Sphinx fails to import atomic. The problem is that Sphinx uses python3.8 to import atomic and it fails due to not finding libarrays.dylib. The chimerax.map module also cannot be imported without starting the ChimeraX app for the same reason, although that one seems to get documented, probably because the order of imports done by Sphinx get the libarrays.dylib loaded before hand.

Here is the traceback trying to import atomic

/Users/pett/src/chimerax/ChimeraX.app/Contents/bin/python3.8 -I /Users/pett/rm/test.py
Traceback (most recent call last):

File "/Users/pett/rm/test.py", line 1, in <module>

import chimerax.atomic

File "/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/atomic/init.py", line 15, in <module>

import chimerax.atomic_lib

File "/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/atomic_lib/init.py", line 27, in <module>

from . import _load_libs

ImportError: dlopen(/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/atomic_lib/_load_libs.cpython-38-darwin.so, 2): Library not loaded: @rpath/libarrays.dylib

Referenced from: /Users/pett/src/chimerax/ChimeraX.app/Contents/lib/python3.8/site-packages/chimerax/atomic_lib/lib/libatomstruct.dylib
Reason: image not found

Here is the traceback trying to import map

$ ~/ucsf/chimerax/ChimeraX.app/Contents/bin/python3.8
Python 3.8.5 (default, Dec 2 2020, 20:24:01)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import chimerax.map

Traceback (most recent call last):

File "<stdin>", line 1, in <module>
File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/map/init.py", line 31, in <module>

from ._map import contour_surface, sphere_surface_distance

ImportError: dlopen(/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/map/_map.cpython-38-darwin.so, 2): Library not loaded: @rpath/libarrays.dylib

Referenced from: /Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/chimerax/map/_map.cpython-38-darwin.so
Reason: image not found

[Tom Goddard]

The trouble is the bundle arrays is not normally imported by anyone -- it is just used by other C++ code. But it can be imported and has a load_libarrays() function which loads libarrays.dylib. That call is made currently in only two places in geometry/init.py and atomic/molobject.py. So the reason ChimeraX can import atomic but python3.8 cannot is because ChimeraX fortuitously imports geometry before atomic. Strictly this monkey business of "from chimerax import arrays; arrays.load_libarrays()" would be needed in dozens of places, anywhere another C++ module that links against it is about to be imported. I hate to litter the code with that fragile crap but not sure if there is another way.

[Tom Goddard]

Here are the 35 bundles in the distribution that depend on libarrays

chem_group
dssp
mmcif
morph
segment
atomic_lib
surface
stl
atomic
atomic
mask
map
arrays
cloth
geometry
atom_search_lib
graphics
leap_motion
coulombic
webcam
realsense
pdb
align_algs
mmtf
mlp

[Tom Goddard]

It is a bit unusual that we have a Python module that is just for other modules to link against its C++ library. Maybe it is sensible that all modules that link against arrays needs some explicit Python code to load that dependent library. Ugly, but I guess Python does not have a mechanism to handle such cross module linking because it is rare.

Another idea is that the chimerax module does the load_libarrays() on behalf of the many submodules that need it. Right now the module chimerax is really a namespace with no init.py file. But I think this is pretty screwy and we should have chimerax.version and possibly other things in the chimerax module that currently are in chimerax.core.

[Tom Goddard]

To fix the Sphinx problem Eric put into the programmer guide Makefile (docs/devel/Makefile) setting LD_LIBRARY_PATH to pick up libarrays

​https://github.com/RBVI/ChimeraX/commit/65f2ad26667fd916307d2698aab9a91c9749a23a

Let's see if that makes Sphinx happy with tomorrow's builds having all the expected developer documentation.

Currently we do not support use of ChimeraX modules outside of running the ChimeraX application and Sphinx is violating that rule. In the future I would like to provide a PyPi or Conda chimerax package that can be used without running the ChimeraX desktop app. For that to work the problem in this ticket will need a better solution than setting LD_LIBRARY_PATH.

[Eric Pettersen]

Since, strictly speaking, atomic_lib needs libarrays, I made it call load_libarrays() before loading its own libs.

[goddard@…]

Yeah, I'm afraid all 35 bundles that use libarrays should be making an explicit call to load_libarrays() before importing their C++ module.  It is just so unpleasant that the runtime linking has to be explicitly handled in the Python code that I am not yet willing to add all those calls until I determine there is no saner solution.

[Greg Couch]

Not all 35 need it. For example, mmtf imports atomic_lib, so if atomlib imports it, mmtf is good to go.

[goddard@…]

The 35 bundles have a dependency on ChimeraX-Arrays in bundle_info.xml. _mmtf has a direct dependency on libarrays by including arrays/pythonarray.h so it should not rely on atomic_lib to pull in libarrays.

_mmtf/mmtf.cpp

[Tom Goddard]

Eric's LD_LIBRARY_PATH fix has restored the programming documentation.

We want the ChimeraX Python modules to be loadable without linking errors and without relying on environment variables, so I have made another ticket #4073 to address that. When that is done the LD_LIBRARY_PATH fix for the docs can be removed.

[Tom Goddard]

Fixed.

Took out the LD_LIBRARY_PATH fix that allowed Sphinx to import modules that depended on libarrays. All modules that depend on libarrays now load it themselves, ticket #4073.