Unit testing framework?

[Tristan Croll]

API changes in the update to ChimeraX 1.1 have broken both Clipper and ISOLDE in all sorts of new and exciting ways, many of which are only coming to light after a full day of testing (a few examples below). This makes me a bit jumpy, and gets me thinking about adding some level of unit testing. The challenge here, of course, is that automatic unit testing becomes somewhat difficult in a graphical environment designed for interactivity. Just wondering if you have any solutions implemented in house, or any thoughts on the subject?

Examples:

• all the changed imports, of course (relatively easy: grep for all the imports done by Clipper/ISOLDE, try them all and list those that fail, then find the new import)
• changes to picking API (view.first_intercept replaced by view.picked_object, change to Drawing.pickable behaviour)
• dssp_cache input to match_maker.match() changed from set to dict - broke my reference-model distance and torsion restraints, and would have made it through to the release if I hadn't happened to try to use the torsion restraints for a model I'm currently helping with.
• something different about the way models are cleaned up on deletion now causes a traceback when I close a Clipper crystal structure - just picked up this evening, will tackle tomorrow.

Not complaining, as such - I can see the logic behind these changes - but without a solid testing framework it's only going to get harder for third-party developers to catch and respond to things like this.

[pett]

Reassigning to Greg since he's our test-framework expert (well, certainly has the most experience with the various relevant test frameworks). Am I correct in that you are asking for advice in setting up a test framework for ISOLDE?

I personally have tried to avoid changing APIs marked as supported, but that hasn't helped that much for two reasons. One, many useful APIs are not marked as supported yet. Two, sometimes it's irresistible and sometimes I'm prodded into changing them by others (e.g. the trigger-blocking API).

I am mystified by one of your examples. dssp_cache is not an argument to match_maker.match() -- it is created by the method itself, so I don't see how it changing from a set to a dictionary could break calls to the routine. Clearly I am missing something.

[Tristan Croll]

Yep - mainly just asking for advice on this one. Regarding the 
match_maker question - sorry, you're right. Was late and I already had 
my dev machine turned off. The instances where my code got broken are 
cases where the top-level match() method didn't do what I need, and I 
had to burrow deeper and make use of match_maker.match.align(). For 
these cases (creating a map averaged by non-crystallographic symmetry; 
restraining to match a reference template) I need to be able to align an 
arbitrary (possibly multi-chain) selection of residues as a single 
block.

On 2020-09-02 00:40, ChimeraX wrote:

[pett]

Ah. Yeah, the align() routine isn't truly public, as hinted at by its lack of a doc string. Nonetheless, you needed it and the signature did change.