Opened 9 years ago

Closed 8 years ago

#506 closed defect (fixed)

pip install overrides LD_LIBRARY_PATH

Reported by: tic20@… Owned by: Conrad Huang
Priority: blocker Milestone:
Component: Tool Shed Version:
Keywords: Cc: chimera-programmers@…
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

If I define a little bash script, chimera-python, wrapping ChimeraX's Python:

#!/bin/bash

CHIMERA_HOME=/home/tic20/apps/chimerax

export LD_LIBRARY_PATH=$CHIMERA_HOME/lib:/home/tic20/apps/gcc-4.9.2/lib64

$CHIMERA_HOME/bin/python3.5 $*

... then try:

chimera-python -m pip install lxml

...

I get the output pasted below. The short story is that LD_LIBRARY_PATH appears to have been overridden, and the gcc flags have the library path hard-coded to -L/var/tmp/chimerax_build/develop/chimerax/build/lib.

Collecting lxml

Using cached lxml-3.7.0.tar.gz

Building wheels for collected packages: lxml

Running setup.py bdist_wheel for lxml ... error
Complete output from command /home/tic20/apps/chimerax/bin/python3.5 -u -c "import setuptools, tokenize;file='/tmp/pip-build-ujjcqr36/lxml/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmpln9kgi9dpip-wheel- --python-tag cp35:
Building lxml version 3.7.0.
Building without Cython.
Using build configuration of libxslt 1.1.28
Building against libxml2/libxslt in the following directory: /usr/lib64
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/lxml
copying src/lxml/builder.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/sax.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/init.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.5/lxml
creating build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/init.py -> build/lib.linux-x86_64-3.5/lxml/includes
creating build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/init.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.5/lxml/html
creating build/lib.linux-x86_64-3.5/lxml/isoschematron
copying src/lxml/isoschematron/init.py -> build/lib.linux-x86_64-3.5/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.5/lxml/includes
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building 'lxml.etree' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
creating build/temp.linux-x86_64-3.5/src/lxml
gcc -pipe -fPIC -std=gnu99 -I/var/tmp/chimerax_build/develop/chimerax/build/include -L/var/tmp/chimerax_build/develop/chimerax/build/lib -DUSE_DYLD_GLOBAL_NAMESPACE -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/home/tic20/apps/chimerax/include/python3.5m -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -w
gcc -pipe -fPIC -std=gnu99 -I/var/tmp/chimerax_build/develop/chimerax/build/include -L/var/tmp/chimerax_build/develop/chimerax/build/lib -DUSE_DYLD_GLOBAL_NAMESPACE -pthread -shared -O3 -Wall -Wextra -Wl,-rpath,/var/tmp/chimerax_build/develop/chimerax/build/lib build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -L/usr/lib64 -L/var/tmp/chimerax_build/develop/chimerax/build/lib -lxslt -lexslt -lxml2 -lrt -lz -lm -lpython3.5m -o build/lib.linux-x86_64-3.5/lxml/etree.cpython-35m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -lpython3.5m
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

----------------------------------------
Failed building wheel for lxml
Running setup.py clean for lxml

Failed to build lxml
Installing collected packages: lxml

Running setup.py install for lxml ... error

Complete output from command /home/tic20/apps/chimerax/bin/python3.5 -u -c "import setuptools, tokenize;file='/tmp/pip-build-ujjcqr36/lxml/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-yq0vr531-record/install-record.txt --single-version-externally-managed --compile:
Building lxml version 3.7.0.
Building without Cython.
Using build configuration of libxslt 1.1.28
Building against libxml2/libxslt in the following directory: /usr/lib64
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/lxml
copying src/lxml/builder.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/sax.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/init.py -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.5/lxml
creating build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/init.py -> build/lib.linux-x86_64-3.5/lxml/includes
creating build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/init.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.5/lxml/html
copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.5/lxml/html
creating build/lib.linux-x86_64-3.5/lxml/isoschematron
copying src/lxml/isoschematron/init.py -> build/lib.linux-x86_64-3.5/lxml/isoschematron
copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.5/lxml
copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.5/lxml/includes
copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.5/lxml/includes
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng
copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/rng
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl
creating build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.5/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
running build_ext
building 'lxml.etree' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/src
creating build/temp.linux-x86_64-3.5/src/lxml
gcc -pipe -fPIC -std=gnu99 -I/var/tmp/chimerax_build/develop/chimerax/build/include -L/var/tmp/chimerax_build/develop/chimerax/build/lib -DUSE_DYLD_GLOBAL_NAMESPACE -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -Isrc/lxml/includes -I/home/tic20/apps/chimerax/include/python3.5m -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -w
gcc -pipe -fPIC -std=gnu99 -I/var/tmp/chimerax_build/develop/chimerax/build/include -L/var/tmp/chimerax_build/develop/chimerax/build/lib -DUSE_DYLD_GLOBAL_NAMESPACE -pthread -shared -O3 -Wall -Wextra -Wl,-rpath,/var/tmp/chimerax_build/develop/chimerax/build/lib build/temp.linux-x86_64-3.5/src/lxml/lxml.etree.o -L/usr/lib64 -L/var/tmp/chimerax_build/develop/chimerax/build/lib -lxslt -lexslt -lxml2 -lrt -lz -lm -lpython3.5m -o build/lib.linux-x86_64-3.5/lxml/etree.cpython-35m-x86_64-linux-gnu.so
/usr/bin/ld: cannot find -lpython3.5m
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

----------------------------------------

Command "/home/tic20/apps/chimerax/bin/python3.5 -u -c "import setuptools, tokenize;file='/tmp/pip-build-ujjcqr36/lxml/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-yq0vr531-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-ujjcqr36/lxml/

Change History (4)

comment:1 by Conrad Huang, 9 years ago

I don't think the problem is LD_LIBRARY_PATH. The compilation failure is due to using compiler flags that referece our daily build directory rather than some place in the ChimeraX install tree. Unfortunately, this is an ugly problem, especially since the three platforms (Linux, Mac and Windows) all behave differently.

There are several things we need to assure:

  1. Existence of "include" directory in distribution. I think we got this one.
  2. Organization of "lib" directory in distribution so that it can be used for compilation. This one is particularly ugly since different bundles will have libraries/shared objects/DLLs in their own install location rather than a centralized one. For Windows, it's even worse since we need to have both the .dll and .lib files in order for the bundle to both run and be usable in compilations. I don't think we even include the .lib files for core .dlls, let alone putting them in the right place.
  3. Making the "sysconfig" module return the correct values. sysconfig is what "pip" uses to get the compilation flags. On Linux and Mac, sysconfig relies on the "_sysconfigdata" module, which has the build flags (with references to the daily build directory). I can't even figure out how Windows finds the compilation flags since _sysconfigdata.py is not present (yet the sysconfig module does not throw any exceptions).

These will take a bit of work to straighten out. Which brings up the question of "How did Tristan manage to compile _anything_ against ChimeraX?"

comment:2 by tic20@…, 9 years ago 

He didn't. :) The only compiled libraries I'm using have no dependencies on ChimeraX, and I only use them via Python.

T

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY

comment:3 by tic20@…, 9 years ago 

Simple workaround hack for now: if I create a symlink to the chimerax 
directory in /var/tmp/chimerax_build/develop/chimerax/build, then I can 
successfully install libxml.

On 2016-12-14 21:51, ChimeraX wrote:

comment:4 by Conrad Huang, 8 years ago

Resolution: → fixed
Status: new → closed

The ChimeraX distribution now includes include files and link libraries, and the new procedure for building bundles using only ChimeraX.app without the source tree is described in ​http://www.cgl.ucsf.edu/chimerax/docs/devel/writing_bundles.html (the "writing" part of the URL is a misnomer since the document is really more about building the bundle once the code is written). Since the new procedure uses the standard Python setuptools module for compilation, it should not have references to anything outside of ChimeraX.app itself.

Note: See TracTickets for help on using tickets.