Bundle's own compiled libraries should be at top of symbol search path

[Tristan Croll]

I *guess* this is a ToolShed issue?

Short story: shared libraries contained in my bundle are at the bottom of the symbol lookup order at runtime, causing crashes when a duplicate symbol occurs.

As an experiment, I uninstalled Numpy and Scipy and replaced them with the Intel versions:

$CHIMERAX_HOME/bin/python3.6 -m pip uninstall numpy scipy
$CHIMERAX_HOME/bin/python3.6 -m pip install intel-numpy intel-scipy

... which worked fine until I attempted to load a crystallographic map using Clipper (which for legacy reasons uses FFTW2 and is built against a cut-down MKL library containing only what's necessary for its FFTW2 wrapper). This caused a segmentation fault due to Clipper attempting to call fftw_destroy_plan() from Numpy's MKL library (which installs into ChimeraX's main lib dir, and is probably using the FFTW3 wrapper - same symbol, different memory layout).

Looking at the Clipper library:

$ ldd libclipper.so 
	linux-vdso.so.1 =>  (0x00007ffc02f5d000)
	libmmdb2.so => /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/./libmmdb2.so (0x00007fc0c1670000)
	libccp4.so => /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/./libccp4.so (0x00007fc0c1437000)
	libfftw2xc.so => /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/./libfftw2xc.so (0x00007fc0c1230000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc0c101a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc0c0dfe000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc0c0a31000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc0c1df5000)
	libfftw_mkl_core.so => /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/./libfftw_mkl_core.so (0x00007fc0bd763000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fc0bd55f000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc0bd25d000)

... where libfftw2xc.so is the library containing the symbol I want.

Following what's going on using:

LD_DEBUG=all chimerax-daily 2 > ld_debug.log

... reveals that the libraries I've explicitly linked are coming in right at the bottom of the search order. For a symbol that's in my MKL library but not in Numpy's:

     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=chimerax-daily [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/bin/../lib/libpython3.6m.so.1.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libc.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libpthread.so.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libdl.so.2 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libutil.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libm.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/_mklinit.cpython-36m-x86_64-linux-gnu.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_rt.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libimf.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libsvml.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libirng.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libgcc_s.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libintlc.so.5 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/../../../libiomp5.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_core.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_thread.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_lp64.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_avx.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_vml_avx.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libGL.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libGLX.so.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libGLdispatch.so.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libX11.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libXext.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libxcb.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libXau.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMCPU.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/librt.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/simtk/openmm/../../../../libOpenMM.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libstdc++.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMPME.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libfftw3f.so.3 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libfftw3f_threads.so.3 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMOpenCL.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/usr/local/cuda/lib64/libOpenCL.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMRPMDOpenCL.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/simtk/openmm/../../../../libOpenMMRPMD.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMDrudeOpenCL.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/simtk/openmm/../../../../libOpenMMDrude.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMRPMDReference.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMDrudeReference.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/plugins/libOpenMMAmoebaReference.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/simtk/openmm/../../../../libOpenMMAmoeba.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/clipper_python.cpython-36m-x86_64-linux-gnu.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libclipper.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libclipper_cx.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/bin/../lib/libpython3.6m.so.1.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libstdc++.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libm.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libgcc_s.so.1 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libpthread.so.0 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/lib64/libc.so.6 [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libmmdb2.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libccp4.so [0]
     32103:	symbol=rfftwnd_destroy_plan;  lookup in file=/home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libfftw2xc.so [0]
     32103:	binding file /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libclipper.so [0] to /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libfftw2xc.so [0]: normal symbol `rfftwnd_destroy_plan' [CODEABI_1.0]

For the offending symbol (which appears in both libraries):

     32103:	symbol=fftw_destroy_plan;  lookup in file=chimerax-daily [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/bin/../lib/libpython3.6m.so.1.0 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/libc.so.6 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/libpthread.so.0 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/libdl.so.2 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/libutil.so.1 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/libm.so.6 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/_mklinit.cpython-36m-x86_64-linux-gnu.so [0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_rt.so [0]
     32103:	binding file /home/tic20/.local/share/ChimeraX/0.8/site-packages/chimerax/clipper/libclipper.so [0] to /opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_rt.so [0]: normal symbol `fftw_destroy_plan' [CODEABI_1.0]
     32103:	symbol=fftw_destroy_plan;  lookup in file=/opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_lp64.so [0]
     32103:	binding file /opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_lp64.so [0] to /opt/UCSF/ChimeraX-daily/lib/python3.6/site-packages/numpy/core/../../../../libmkl_intel_lp64.so [0]: normal symbol `fftw_destroy_plan'

[Greg Couch]

Symbols from an already loaded library override subsequent loads.