Can't upload bundle to ToolShed

[Tristan Croll]

I get the following error:

The file you submitted has app name as ChimeraX-Clipper but must be ChimeraX_Clipper.

If, in my bundle_info.xml I change the name accordingly:

<BundleInfo name="ChimeraX_Clipper" version="0.3" package="chimerax.clipper"
  	    minSessionVersion="1" maxSessionVersion="1">

... then the name of the wheel file changes, but make app-install still yields:

Successfully installed ChimeraX-Clipper-0.3

... and I get the same error on trying to upload.

[Tristan Croll]

Looks like the root cause of this is baked pretty deeply into the Wheel architecture. In site_packages/pip/_internal/wheel.py:

class Wheel(object):
    """A wheel file"""

    # TODO: maybe move the install code into this class

    wheel_file_re = re.compile(
        r"""^(?P<namever>(?P<name>.+?)-(?P<ver>.*?))
        ((-(?P<build>\d[^-]*?))?-(?P<pyver>.+?)-(?P<abi>.+?)-(?P<plat>.+?)
        \.whl|\.dist-info)$""",
        re.VERBOSE
    )

    def __init__(self, filename):
        """
        :raises InvalidWheelFilename: when the filename is invalid for a wheel
        """
        wheel_info = self.wheel_file_re.match(filename)
        if not wheel_info:
            raise InvalidWheelFilename(
                "%s is not a valid wheel filename." % filename
            )
        self.filename = filename
        self.name = wheel_info.group('name').replace('_', '-')
        # we'll assume "_" means "-" due to wheel naming scheme
        # (https://github.com/pypa/pip/issues/1150)
        self.version = wheel_info.group('ver').replace('_', '-')
        self.build_tag = wheel_info.group('build')
        self.pyversions = wheel_info.group('pyver').split('.')
        self.abis = wheel_info.group('abi').split('.')
        self.plats = wheel_info.group('plat').split('.')

        # All the tag combinations from this file
        self.file_tags = {
            (x, y, z) for x in self.pyversions
            for y in self.abis for z in self.plats
        }

... so all underscores get internally converted to minus signs, no matter what input you give to setup.py. Can't fix this from my end, and can't upload my new release until it's fixed.

[Tom Goddard]

Conrad is on vacation until January 7.

Your bundle name looks like all our other bundles. I'm wondering if there is anything unusual about your Clipper bundle, or if simply every upload to the toolshed is broken.

[Tom Goddard]

If there is nothing special about your Clipper bundle that is causing this problem, and all uploads to the toolshed are broken, then the fix is not in some low level pip code. Upload used to work, and some of our code must have changed to break it.

[tic20@…]

The requirement for the name to be "ChimeraX_<name>" rather than 
"ChimeraX-<name>" is new since the last time I uploaded a bundle. My 
point in posting the pip code is that this requirement seems to be 
fundamentally incompatible with the Wheel framework.


On 2018-12-20 18:37, ChimeraX wrote:

[Tom Goddard]

When was the last time you successfully uploaded to the toolshed?

[tic20@…]

That would be my last official release (~March/April).

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[Greg Couch]

All of our internal bundles are ChimeraX-xxxx. So that is expected to work.

[tic20@…]

From my reading, I don’t think it’s actually possible to make a wheel called “ChimeraX_xxx” without some seriously evil hacking.

My question right now is: is this something that can be fixed today? If not, I’ll wait until after the new year to release. Tomorrow was my cut-off-point - any later and I’m afraid too many people will miss the release announcement.

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[Greg Couch]

Realistically, this isn't going to be fixed today.

[tic20@…]

Ok, thanks. In that case I’m happy to wait until Conrad gets back. I can always get preview builds to interested parties for “play testing” in between.

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[Greg Couch]

Found where the error message was generated in submit_app/processwheel.py, but don't know enough django to follow how expect_app_name is set (in submit_app/views.py: expect_app_name = request.POST.get('expect_app_name')).

[Conrad Huang]

I was able to upload the DSSP bundle to the ChimeraX preview toolshed, which has some uncommitted changes. I committed those, and pulled them onto the production toolshed. Can you give it another try and see if it works any better? Thanks.

Conrad

PS The top of bundle_info.xml for DSSP is:

<BundleInfo name="ChimeraX-Dssp" version="1.0"
            package="chimerax.atomic.dssp"
            minSessionVersion="1" maxSessionVersion="1">

and the file uploaded is named "ChimeraX_Dssp-1.0-cp36-cp36m-win_amd64.whl".

[tic20@…]

Trying out Clipper on the preview toolshed: it uploads successfully, but 
it does give me the following message:

Bundle Name 	ChimeraX-Clipper
ChimeraX-Clipper (A ChimeraX_ prefix for the bundle is recommended to 
avoid name clash with Python distributions from pypi.python.org. The 
prefix will not be displayed in the toolshed browsing and download pages 
to simplify the output but is required for all other purposes.)

In bundle_info.xml:

<BundleInfo name="ChimeraX-Clipper" version="0.3" 
package="chimerax.clipper"
   	    minSessionVersion="1" maxSessionVersion="1">

The wheel file is:
ChimeraX_Clipper-0.3-cp36-cp36m-linux_x86_64.whl

I'm still pretty sure the problem is that wheel.py makes having 
underscores in the bundle name effectively impossible - you'll get the 
underscore in the wheel filename, but internally they're substituted for 
minus signs before the final package name is written.


On 2019-01-15 00:25, ChimeraX wrote:

[Conrad Huang]

Yes, there is an inherent incompatibility between pip and wheel on the use of '_' vs. '-'. I had to put in code in the toolshed to mediate between them, and just assumed that bundle names will only use '-'. One simple solution is to make bundle_builder issue a warning and replace '_' in bundle names with '-'. Does that sound reasonable?

[tic20@…]

Works for me.

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[tic20@…]

New error message:

The file you submitted problem with dependencies: dependency on 
u'ChimeraX_Atomic': no such bundle.

In my bundle_info.xml:

   <Dependencies>
     <Dependency name="ChimeraX-Core" version="==0.8"/>
     <Dependency name="ChimeraX-Atomic" version=">=1.0"/>
     <!-- <Dependency name="Clipper-Python" version=">=0.1.1"/> -->
   </Dependencies>


On 2019-01-17 19:30, ChimeraX wrote:

[tic20@…]

It *does* upload to the preview toolshed, though.

On 2019-01-18 17:18, ChimeraX wrote:

[Conrad Huang]

Code to warn and replace underscores with hyphens is in c336d7b32.

The inability to upload to the production site is an oversight on my part and should be fixed over the weekend.

[tic20@…]

Any chance this could be done today? I have a meeting tomorrow (London 
time) and would really like to have the release out ahead of that.

On 2019-01-18 18:21, ChimeraX wrote:

[Conrad Huang]

The problem should be fixed, and error messages have been updated. Give it a spin and see if it gives you another error.

[tic20@…]

Still getting this message:

The file you submitted App name given as "ChimeraX-Clipper" but must be 
"ChimeraX_Clipper".

On 2019-01-22 20:03, ChimeraX wrote:

[tic20@…]

Wheel file is at 
https://drive.google.com/open?id=1oUoAoIspNK9LKDzy6eBz_BZEclUy2oyH if 
you need to take a look.

On 2019-01-22 20:36, ChimeraX wrote:

[Conrad Huang]

Very odd. I just submitted that bundle to both the preview and production toolshed. I cancelled the preview one, but only after it accepted it and asked. I proceeded with the production one and it completed it (without requiring confirmation since it is an update). It's up on the production toolshed now. Not sure what is going on but will keep looking.

[tic20@…]

Curiouser and curiouser...

Well, in the meantime if I get all my Linux, Mac and Windows builds onto 
my Google Drive, would you mind pushing them onto the ToolShed?

On 2019-01-22 21:05, ChimeraX wrote:

[tic20@…]

If you can, the remaining links are:

Clipper:
https://drive.google.com/open?id=1l07T_iy9gGIGN-eUQEH1Ss3Sn800Ev6t
https://drive.google.com/open?id=1dNkKsMjPHiFeoQsQIwyxuarh5AJcQ6BO

ISOLDE:
https://drive.google.com/open?id=1tfLgZ59PPYwNH-Cty5QV-2jceSZXXgWl
https://drive.google.com/open?id=1pMzGSmR8LbWF2wWBzhHb0_ufAThwZj2z
https://drive.google.com/open?id=1fOo6jPW07sjP7y1GVZxguge8IT2NF9LK

On 2019-01-22 21:24, ChimeraX wrote:

[Conrad Huang]

Okay, all three platforms for both Clipper and ISOLDE are on the production toolshed.

[tic20@…]

Installs correctly on all three platforms. Thanks!

On 2019-01-22 22:47, ChimeraX wrote:

[Conrad Huang]

I think I found it. There were two places in the toolshed for submitting apps and I only fixed the '_'/'-' issue in one place. Both preview and production should have the fix now. (Not closing ticket for a couple days, just in case.)

[Conrad Huang]

This problem is fixed. Unfortunately, others have popped up.