Toolshed install of Clipper bundle does nothing

[Tom Goddard]

Clicking the Download button on the Clipper bundle using the Toolshed interface within ChimeraX does nothing -- no feedback, no dialog asking if should be installed for all users, no new Tools / General / Clipper menu entry.

The download of the wheel works using an external web browser

​https://cxtoolshed.rbvi.ucsf.edu

Then installs with

$ cx -m pip install ~/Downloads/ChimeraX_Clipper-0.1.0-py3-none-any.whl

where cx is the Mac ChimeraX executable.

[Conrad Huang]

Is this still true? I just tried it today and it at least asked me whether to install. There was also a message in the Log saying it was downloading the bundle.

[Tom Goddard]

Seems to install now.

This clipper bundle claims it is for all platforms but it seems it include linux shared libraries so does not work on mac. Perhaps Tristan should be told how to submit his Clipper bundle to toolshed saying it is Linux only.

[Conrad Huang]

Hi, Tristan.

The ChimeraX toolshed now supports different platforms, and the bundle-building procedure has been changed to (try to) make creating bundles simpler and more reliable. The documentation is in ​http://www.cgl.ucsf.edu/chimerax/docs/devel/writing_bundles.html, which contains a link to a sample bundle that includes C code. The new scheme no longer uses setup.py directly. Instead it uses an XML file to describe bundle information and ChimeraX itself is used to build the bundles (still using setup.py internally).

Unfortunately, in this case, it looks like the Clipper libraries in the bundle are independent of ChimeraX and are copied in rather than compiled with the bundle. Is that correct? If so, it should still be possible to force platform-specific bundles using the "pure_python" attribute in the BunldeInfo element in bundle_info.xml. If you're busy with other projects, I would be happy to try to convert the Clipper source bundle to the new scheme.

It would be great if we can get separate wheels for different platforms. Even if we are missing wheels for unsupported platforms, at least users will not be surprised when nothing happens on a "successful" install.

Conrad

[tic20@…]

Hi Conrad (et al.),

Yes, that bundle was submitted in something of a hurry to have something 
ready for the Price meeting. Platform-specific builds of course need to 
be done - but right now my focus needs to be on pulling together enough 
material to make a credible bid for independent funding, so I'm not sure 
how soon I'll be able to look into it.

In terms of how the Clipper plugin is put together: the Clipper and 
Clipper-Python libraries are distributed as part of the CCP4 suite. Now 
that my (quite extensive) changes to Clipper-Python have been merged 
into the main trunk it should become somewhat easier to do 
multi-platform builds, but the big annoyance right now is that in the 
official distributions the libraries are all linked with absolute paths. 
For my bundles I tweaked the Linux build to use relative paths to make a 
fully-portable library (and did one for the Mac that is now entirely out 
of date) - but this all still needs to be formalised and extended to all 
three major operating systems. Ideally I'd like to convince the CCP4 
folks to provide this option as a compiler flag... we shall see. To be 
completely frank, my experience with large build systems is quite 
minimal - I'm learning as fast as I can, but that's in between many 
other demands on my time. The funding gods permitting, I may be able to 
get some more experienced hands on that in the future.

By the way: there's a manuscript on Clipper-Python currently being 
finalised for submission in the next few days. The ChimeraX plugin gets 
a mention - do you have any ChimeraX manuscripts 
published/in-press/submitted that could be cited?

Cheers,

Tristan





On 2017-06-29 21:42, ChimeraX wrote:

[pett]

Doesn't seem like Conrad has responded yet, but we do have a paper submitted for a special issue of Protein Science. Editors never explicitly let you know these things, but most likely it's out for review right now. The title of the paper is "UCSF ChimeraX: Meeting Modern Challenges in Visualization and Analysis".

--Eric

[tic20@…]

Perfect - I believe that's where the Clipper manuscript is headed!

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[tic20@…]

Hi Conrad,

Now that I've had the chance to look at this a bit, it looks like a 
great improvement! It will also make things somewhat easier for ISOLDE 
itself, which has a few small C++ routines. On that subject, I have a 
query: within this framework will it be possible for me to #include 
OpenMM headers and have ChimeraX find the libraries in its tree? That 
would be fantastic if so - I've been realising more and more that at 
some point I'll have to bite the bullet and switch to connecting to 
OpenMM via its C++ API. Some setup functions are simply too slow via 
Python, and if the dream of MD in VR is to go ahead then OpenMM will 
need to be running in its own dedicated thread with fully asynchronous 
communication to ChimeraX.

In a similar vein, I'm hoping to convince the others involved in 
Clipper-Python that it's worth packaging it up as a PyPI package. That 
way standardised builds of the compiled libraries would always be 
available online and the ChimeraX plugin would be pure, 
platform-independent Python.

Thanks,

Tristan


On 2017-06-29 21:42, ChimeraX wrote:

[Conrad Huang]

We have no mechanism for specifying include directories to be installed with ChimeraX in bundle_info.xml, but I believe the code that we use for building bundles (setuptools.setup) does. So all we need to do is expose that functionality. Will something like:

<IncludeDirs>

<IncludeDir name="openmm" folder="openmm/include/openmm"/>

<IncludeDirs>

be sufficient? I intend this to mean that there should be an "openmm" directory under the standard ChimeraX include directory and that the files in "openmm" can be found in the subdirectory "openmm/include/openmm" in the bundle source tree.

[tic20@…]

Yes, sounds like that should work nicely.

In other news, I have agreement with the Clipper folks that 
clipper-python will be added to the PyPI. Not sure how soon it will 
happen, but it will happen. Once it does, I'll get together a 
Python-only version of the ChimeraX plugin and get it onto the Toolshed 
as soon as possible.

On 2017-07-08 00:36, ChimeraX wrote:

[Conrad Huang]

I started implementing support for the IncludeDirs element in bundle_info.xml and ran into an issue: users may not have write-permission for ChimeraX.app, so they will not be able to install the bundle, even if they do not care about the include files. One possible solution is to put the include directory next to the user-specific site-packages directory. But this complicates the devel command since it must now look in multiple directories instead of using a single standard include directory. It also makes cleaning up messier. So here's my idea:

1. make the initial developer put include (and library) files in the bundle (specifically under the Python package corresponding to the bundle),
2. make the subsequent developer specify that they need initial bundle for compilation.

The advantages are that: (a) keeps things relatively clean, in the sense that everything for a bundle lives in a single Python package, and (b) provides some documentation in code that one bundle requires another bundle to build.

The nitty gritty details:

1. The bundle API will be extended with two calls, get_include_dirs() and get_library_dirs(), that return list of directories. These will be used as part of the compilation process to build the compiler command line. The defaults will be to return the include and lib directories under the bundle Python package, respectively for get_include_dirs() and get_library_dirs(). These methods may be overridden to use multiple directories or those with arbitrary names. (This idea was inspired by Numpy, which uses a similar setup.)
2. The initial developer will need to use the DataFiles element to put the include and library files into the right location.
3. The subsequent developer will need to add a RequiresBundle element in CModule elements that depend on the bundle include and library files. Presence of RequiresBundle triggers the use of (1) during compilation.
4. If a library needs to be compiled (as opposed to copying some system files), there must be a Library element in bundle_info.xml. Since setup.py does not support building arbitrary libraries, we will need to process all Library elements before running setup.py since the libraries will probably be needed to build the rest of the bundle.

This approach is more complicated than just dropping some files in ChimeraX.app but I think is more sustainable in the long run. What do you think?

[tic20@…]

That looks good to me. Another thing that would be useful would be a 
built-in mechanism in the Makefile to define directories that should be 
copied verbatim (and recursively) into the final wheel. For example, in 
ISOLDE I currently have three directories: molprobity_data to hold the 
Ramachandran and rotamer tables (plain text); amberff to hold custom 
AMBER forcefield files (xml); resources to hold icons etc. for the 
widget (various). I think this would be a fairly common requirement. 
Ideally a line in the Makefile like:

DATA_DIRS = molprobity_data amberff resources

... should tell ChimeraX to copy these. That was easy enough to arrange 
in the old system, but looks like it will need some adjustment to 
bundle_builder.py to make possible now.

On 2017-07-18 19:02, ChimeraX wrote:

[tic20@…]

Never mind - I see now that this can already be done via the 
bundle_info.xml. Adding fields like the following:

   <DataFiles>
     <DataFile>data_1/*</DataFile>
     <DataFile>data_2/*</DataFile>
   </DataFiles>

... seems to do what I need.

On 2017-07-21 10:14, ChimeraX wrote:

[tic20@…]

OK, I've ported ISOLDE over to the new build system. Rather nice, I have 
to say! Will try to get some intermediate solution worked out for 
Clipper while waiting for the full PyPI solution, then upload fresh 
bundles for each.

On 2017-07-21 15:07, ChimeraX wrote:

[tic20@…]

I just made a Linux-specific build of ISOLDE using the new system and 
uploaded it to the Toolshed. When I then navigate to the toolshed from 
my Mac, it can see and download the new release despite the 
Linux-specific naming.

On 2017-07-21 18:00, ChimeraX wrote:

[Conrad Huang]

Hi, Tristan.

I forgot to fix the server side. On the bundle web page, it used to replace "Download" with "Install" if the browser is in ChimeraX regardless of the bundle platform. It now checks and displays "Install" only if the bundle is platform-independent or matches the current platform; otherwise, it displays "[platform] Download".

Can you give that a try and see if it works for you?

For now, you can still click on the "[platform] Download" button, and ChimeraX still tries to install it. What do you think is the right thing to do: (a) hide the button completely, or (b) save the bundle when it does not match the current platform?

Thanks.

Conrad

[Conrad Huang]

Implemented option (b) to save bundle not targeted for current platform.

[Tom Goddard]

Ticket retargeted after milestone closed

[Conrad Huang]

ChimeraX now warns that the bundle could not be installed and logs the location of the downloaded file.