RFE: Add wildcard option in ExtraFiles portion of bundle_info.xml

[Tristan Croll]

I'd like to suggest the following amendment to get_extrafiles() in bundle_builder.py:

    def _get_extrafiles(self, bi):
        import glob, os
        self.extrafiles = {}
        for dfs in self._get_elements(bi, "ExtraFiles"):
            pkg_name = dfs.getAttribute("package")
            files = []
            for e in self._get_elements(dfs, "ExtraFile"):
                source = e.getAttribute("source")
                filename = self._get_element_text(e)
                files.append(("file", source, filename))
            for e in self._get_elements(dfs, "ExtraFileGroup"):
                source = e.getAttribute("source")
                dirname = self._get_element_text(e)
                sourcefiles = glob.glob(source)
                if not len(sourcefiles):
                    raise RuntimeError('ExtraFileGroup pattern {} does not match any files!'.format(source))
                for sf in sourcefiles:
                    files.append(("file", sf, os.path.join(dirname, os.path.basename(sf))))
            for e in self._get_elements(dfs, "ExtraDir"):
                source = e.getAttribute("source")
                dirname = self._get_element_text(e)
                files.append(("dir", source, dirname))
            if files:
                if not pkg_name:
                    pkg_name = self.package
                self.extrafiles[pkg_name] = files
                datafiles = [(t[0], t[2]) for t in files]
                try:
                    self.datafiles[pkg_name].extend(datafiles)
                except KeyError:
                    self.datafiles[pkg_name] = datafiles

... adding an extra field ExtraFileGroup that takes a wildcard expression as its source property and a destination directory as its contents. This makes it easy to handle compiled dependencies where the headers are intermingled with the implementation source - currently one would have to list each header file independently, or copy *all* the source. This way, that's reduced to e.g.:

     <ExtraFiles>
       <ExtraFileGroup source="src/deps/clipper/*.h">include/clipper</ExtraFileGroup>

     </ExtraFiles>

[Tristan Croll]

Or perhaps even better:

    def _get_extrafiles(self, bi):
        import glob, os
        self.extrafiles = {}
        for dfs in self._get_elements(bi, "ExtraFiles"):
            pkg_name = dfs.getAttribute("package")
            files = []
            for e in self._get_elements(dfs, "ExtraFile"):
                source = e.getAttribute("source")
                filename = self._get_element_text(e)
                files.append(("file", source, filename))
            for e in self._get_elements(dfs, "ExtraFileGroup"):
                source = e.getAttribute("source")
                source_base_dir = os.path.dirname(source)
                while '*' in source_base_dir or '?' in source_base_dir:
                    source_base_dir = os.path.split(source_base_dir)[0]
                dirname = self._get_element_text(e)
                sourcefiles = glob.glob(source, recursive=True)
                if not len(sourcefiles):
                    raise RuntimeError('ExtraFileGroup pattern {} does not match any files!'.format(source))
                for sf in sourcefiles:
                    files.append(("file", sf, os.path.join(dirname, os.path.relpath(sf, source_base_dir))))
            for e in self._get_elements(dfs, "ExtraDir"):
                source = e.getAttribute("source")
                dirname = self._get_element_text(e)
                files.append(("dir", source, dirname))
            if files:
                if not pkg_name:
                    pkg_name = self.package
                self.extrafiles[pkg_name] = files
                datafiles = [(t[0], t[2]) for t in files]
                try:
                    self.datafiles[pkg_name].extend(datafiles)
                except KeyError:
                    self.datafiles[pkg_name] = datafiles

... allows me to capture the entire header tree with:

     <ExtraFiles>
       <ExtraFileGroup source="src/deps/clipper/**/*.h">include/clipper</ExtraFileGroup>
     </ExtraFiles>

(leaving out the '/' still captures a single directory non-recursively).

[Conrad Huang]

I'm a little uncomfortable with the ** syntax because it is definitely not part of the commonly accepted glob syntax. How about something like:

<ExtraFileGroup source="src/deps/clipper" pattern="*.h" recursive="true">include/clipper</ExtraFileGroup>

The user specifies the anchor (source) that corresponds to the bundle location (content of the tag), and then the glob pattern and whether it is recursive are specified separately.

[Tristan Croll]

That works for me. Was unaware that “**” is specific to Python glob.
 

 

[Conrad Huang]

Since ignorance of the law is not a defense, I plead guilty that I did not know that ** has been around for almost ten years.

The change above has been made and committed to the repository. Tomorrow's build should support ExtraFileGroup.

[Tristan Croll]

Much obliged!

On 2019-08-05 22:46, ChimeraX wrote:

[Tristan Croll]

Crashes on building - `_get_extrafiles()` needs to import os and glob.

On 2019-08-06 16:11, ChimeraX wrote:

[Conrad Huang]

Import statements have been added and committed.

I rebuilt ChimeraX after the previous change, and of course everything worked because we have no ExtraFileGroup tags in our bundles :-(