[chimera-dev] Chimera python packaging

[Tom Goddard]

Hi Thomas,

  Packaging all the libraries needed by PyTom like Chimera does it is a lot of work.  For Chimera we build from source and include about 40 third-party libraries and programs (like Python, numpy, hdf5, PIL, ffmpeg, …) on Windows, Mac and Linux in 32-bit and 64-bit versions.  The Chimera build and packaging system is just Makefiles we wrote, a system that has probably taken about 1 programmer-year to develop and maintain.

  Even though it is a lot of work, if you want to people to use your software you have to package it for an easy install.  Requiring people to compile or install a dozen required packages will reduce your number of users I'd say by 10 to 100-fold.

  It looks like for PyTom you have about 10 required libraries and support Linux and Mac but not Windows.  Probably distributing just 64-bit would be fine -- since that is needed for big data sets.  There are plenty of problems trying to package these all up.  Here are a few that come to mind.  Your build machine may have libraries that your users will not have and when you compile third party libraries they may link against those.  For instance, fftw might see that you have a blas system library and compile for it.  So you often have to tell packages not to use such libraries.  Sometimes no configure flag is available and we patch the third party build scripts.  On Linux we only rely on Linux Standard Base (LSB) libraries.  Common libraries like tiff, zip, jpeg are ones to look out for.  On both Linux and Mac you'll need to tell your executables where to find the included shared libraries.  We use environment variables (LD_LIBRARY_PATH, DYLD_LIBRARY_PATH on Mac) which means are exectutables shell scripts that set the environment variables and then start the real executable.  The paths have to be relative paths since the user could install your package anywhere.  On the Mac there are also Frameworks to worry about.  We build Python as a Mac Framework, but maybe you would find it easier to instead use the non-framework version since that will be like Linux.  You of course have to build on old versions of Mac and Linux operating systems if you want compatibility with older operating systems.  For instance we build Mac Chimera on Mac OS 10.6 even though 10.9 is the current release.  It is a pain to keep machines with old operating systems running.  It is possible to compile on a newer operating system targeting an older operating system but we find it easier just to use the old operating system.  Even with that compatibility among different Linux distributions was a bit of a nightmare in the past.  I'm not sure if is still troublesome.  You should make sure the licenses for the libraries you want to redistribute allow that.  You'll have trouble using GPL libraries if you distribute any libraries that are not open source. This only scratches the surface of the long list of problems we have dealt with in building third-party libraries, packaging and distributing them.

	Tom


On Jan 8, 2014, at 10:59 AM, Thomas Hrabe wrote:

> Hi Tom,
> 
> I am looking for ways to ship my old software pytom (pytom.org) in a way similar to chimera.
> 
> Meaning that I want to add all required dependencies such as python, numpy, … to the whole package and ship the compiled version.
> Is there anything I have to look out for, especially for the python?
> How do you automate daily builds and all that? Do you use an integration server like jenkins?
> 
> Thank you for your support,
> Thomas
> 
> 
> ----------------------------------
> 
> Thomas Hrabe (Ph.D.)
> Sanford-Burnham 
> Medical Research Institute
> 10901 North Torrey Pines Road
> La Jolla, CA 92037, USA
> +1 858 646 3100 ext 4070
> 
> 
> 
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://plato.cgl.ucsf.edu/pipermail/chimera-dev/attachments/20140108/e1a5a3b6/attachment.html>