Add PDB-REDO to fetch by id to get x-ray maps

[olibclarke@…]

From: Oliver Clarke via ChimeraX-users <chimerax-users@…>
Subject: [chimerax-users] Re: Difference maps in chimerax?
Date: October 23, 2023 at 11:17:07 AM PDT
To: Francisco Murphy Pérez <>
Cc: ChimeraX Users Help <chimerax-users@…>
Reply-To: Oliver Clarke <>

Hi Francisco,

The alias is for Chimera - you will need to convert the syntax to use with ChimeraX. It will not work in ChimeraX as written.

I think as a feature suggestion, it would be good to have access to PDB-REDO in Fetch by ID.

I also wonder if it would be worth having a single "Fetch from EDS" option, that fetches both the 2Fo-Fc and Fo-Fc maps, split and colored appropriately for the difference map.

Cheers
Oli

[Tom Goddard]

PDB-REDO appears to have only the structure factors .mtz file and not real-space maps. ChimeraX cannot open MTZ files without the Clipper plugin. So it doesn't make sense to include PDB-REDO fetch of MTZ in the standard ChimeraX distribution when it cannot open the file. It would make more sense to add PDB-REDO fetch to the Clipper plugin.

Also a major goal of PDB-REDO is to provide a rerefined atomic model. So I'm not clear what all the PDB-REDO would fetch, maybe both rerefined model and MTZ and compute real-space 2fo-fc and fo-fc maps?

[Tristan Croll]

While I’m not particularly a fan of PDB-REDO (in my opinion they tend to
*vastly* oversell its ability to improve models, lulling users into a false
sense of security) I can see how people might want this. Should be easy
enough to add.

If you did want to add your own native support for MTZ files etc. with
minimal maintenance overhead, these days Project Gemmi (
https://pypi.org/project/gemmi/) would probably get you there quite easily.
If I was starting ISOLDE from scratch today I’d certainly use it over
Clipper (and I’ve been seriously considering migrating to it even now).
It’s a joint effort between the company Global Phasing and the CCP4
project, and very well supported given they’ve committed to making it the
new base for all their software. Using it to take a model and MTZ and spit
out a real-space map covering your volume of choice should be pretty
straightforward.

On Mon, 23 Oct 2023 at 19:35, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>
>

[Tom Goddard]

Good to know about Gemmi and pdb-redo limitations. It is hard to support x-ray model building well and due to limited resources we have never made that a priority. So I'm going to leave making real-space maps from MTZ in your court Tristan. As usual our effort is focussed on cryoEM and also analysis of structures beyond model building.

[olibclarke@…]

Incorporation of native MTZ support would still be worth considering I think - not for X-ray model building, but for visualization of maps from both X-ray crystallography and microED... seems like GEMMI might make that a possibility? While microED is not technically cryoEM I guess, it does fall into kind of the same basket...

ISOLDE is great, but native support would be much better (e.g. ISOLDE only works with the stable releases of ChimeraX, and is really designed for model manipulation/interactivity more than pure visualization). Just my 2 cents

Cheers
Oli

[Tristan Croll]

Well… technically you don’t need ISOLDE for that, just the Clipper bundle
(ISOLDE depends on Clipper, but Clipper itself is standalone).

On Wed, 25 Oct 2023 at 12:21, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>
>
>

[olibclarke@…]

I can't seem to install that on the daily release... the released version of clipper is old, and clicking on the newer versions in the toolshed doesn't seem to do anything - apologies if I am being dim...

>> 
>> 
>> 
>> 
>> 
>> 

[olibclarke@…]

Added by email2trac

[Tristan Croll]

This might have something to do with #9861 (a bug in how the daily builds
are doing their subprocess calls to install Toolshed bundles).

On Wed, 25 Oct 2023 at 12:34, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>

[Tristan Croll]

Although come to think of it, it won’t be just that. There was a recent
bump in the version number of ChimeraX_AtomicLibrary, so even the most
recent Clipper dev build will be incompatible with the current daily builds.

On Wed, 25 Oct 2023 at 12:38, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>

[Tom Goddard]

The Clipper toolshed install is a bit confusing because it shows the 3 year old Clipper 0.15 version. That is a mistake in how the requirements for Clipper 0.15 were specified that claimed it could work with any ChimeraX 1.x version. I think if that version did not exist you might get a clearer message that there are no compatible Clipper versions for your ChimeraX. I suggest to Tristan that he delete the 0.15 Clipper distribution because has the wrong requirements and just confuses people trying to install -- I also was confused by this several months ago.

If you login on Toolshed cxtoolshed.rbvi.ucsf.edu and choose under Editor Actions / Edit this page, and go to the Release History tab there is a Delete button by each release and you can delete 0.15.

The one downside is if someone has ChimeraX 1.1 and tries to install Clipper it will say no compatible versions are available. But they really should update ChimeraX in that case so I think deleting 0.15 is better for almost all users who are not using ChimeraX 1.1.

[Tom Goddard]

While we could add gemmi I don't think we have the time, because X-ray and microED are not focus areas.

On the other hand, each time we release ChimeraX I discuss (too late) with Tristan whether we should include ISOLDE and Clipper in the standard ChimeraX distribution. I tend to think this would help users and make ISOLDE even more popular, but so far Tristan and I have not coordinated to do this. For the daily builds we would probably not include it unless the ChimeraX nightly build scripts can reliably build those packages. If not included in daily builds then users could try to install from Toolshed if a compatible version is available.

[Tristan Croll]

The ToolShed will allow me to delete the Clipper 0.15 version for Windows
and Mac, but not for Linux because StrudelScore 0.1 - 0.1.3 versions
declare it as a dependency.

Regarding including Clipper and ISOLDE in the ChimeraX distribution: that
should be *technically *straightforward (both of them build using your
standard Bundle Builder framework). The only challenge with ISOLDE (not
Clipper, which is LGPL) is the issue of commercial users - I'm not sure the
current license as written would technically allow them to have it on their
computer unless they've purchased a license. Could give their legal teams a
collective heart attack.

On Wed, Oct 25, 2023 at 7:09 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>

[Tom Goddard]

Right, the licensing of ISOLDE for commercial users might be a nightmare. So I guess we stick with the current ISOLDE distribution through the Toolshed. Although it sounds like it might be possible to include Clipper and that addresses the MTZ reading, I'm not to enthusiastic about that because the main idea that justified the extra complexity and work to include third-party code in ChimeraX builds was to allow easier access to ISOLDE. But we could discuss how hard it would be to include Clipper if others are motivated.

I made a separate ticket #10064 about being unable to delete Clipper 0.15 Linux due to StrudelScore. That is not reasonable that some other developer can restrict your ability to get rid of broken distributions.

[Tristan Croll]

So getting back to the original suggestion about fetching model and maps from PDB-REDO: to implement this in the way "structureFactors true" is added to "open XXX fromDatabase YYY" would require some addition to the ChimeraX code akin to ​https://github.com/RBVI/ChimeraX/blob/c9ee4994c39db1287dce5bd946a2279484a852b1/src/bundles/mmcif/src/mmcif.py#L283 (with very minor changes based on the fact that PDB-REDO provides the structure factors as MTZ rather than .cif). Wouldn't actually require any changes to the existing Clipper bundle.

[Tom Goddard]

It is true that PDB-REDO fetch could be in the standard ChimeraX distribution even though it needs Clipper to load the mtz file, and if Clipper is not installed it could just give an error saying you would need to install Clipper. Other options would be to make PDB-REDO part of the Clipper bundle, or make it a separate bundle that depends on Clipper.

The real issue is who wants to do the work to add it and maintain it. Since it is very closely aligned with Tristan's ISOLDE project and he is much more expert at X-ray than me, I'm inclined to leave it in his hands if he thinks it is worth pursuing. But he said in comment 2 he is not a fan of PDB-REDO.

[pett]

The ability to fetch PDB-REDO structures is now in the daily build, either through File→Fetch By ID or using the "redo:" prefix in the "open" command.

I have left the capability to fetch the model/map combo commented out for now, because ChimeraX can only read MTZ files by using the Clipper plugin, and that plugin currently only works for ChimeraX 1.7 and earlier and the daily build is 1.8. Whenever Clipper becomes available for 1.8 I will uncomment the combined fetch.

[Tristan Croll]

I’m traveling in the US this week, but should be able to get new Clipper
builds out soon after I get back.

On Wed, 20 Mar 2024 at 09:25, ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>
>

[pett]

Great!

--Eric

[r.joosten@…]

Indeed that sounds excellent.

Cheers,
Robbie

[Tristan Croll]

OK, Clipper builds are up, built against yesterday's ChimeraX nightly
builds.

On Wed, Mar 20, 2024 at 6:31 PM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

>
>
>
>
>

[pett]

Fetching the PDB-REDO MTZ maps as well will be tomorrow's daily build. I have to say, since Clipper is in charge of opening the map it's pretty wild. It reminds me of the beginning of Outer Limits episodes ("There is nothing wrong with your television. Do not attempt to adjust the picture. We are now controlling the transmission.") since Clipper does a lot more than just show the map!

[Tom Goddard]

How do I fetch an PDB-REDO MTZ map? When I try in the current daily build

open 6c08 fromDatabase redo structureFactors true

it gives an error #10051, and when I try

open 6c08 fromDatabase redo format mtz

it gives error

Format 'mtz' not supported for database 'redo'. Supported formats are: pdb or mmcif

[pett]

It's "open 6c08 fromDatabase redo format mtz". It doesn't work in the daily build because the Mac daily build hasn't worked since it got added (as per #14862). In your personal build if you reinstall the 'map' module (and 'mmcif' or 'pdb' modules for the other REDO fetches), it should work.

[Tom Goddard]

On the ChimeraX download page it shows the Mac universal, intel and arm builds were all built more recently than your March 28 commit of MTZ redo fetch. The download page says they are based on the git code with last commit at March 28 at 5:55 pm which is your MTZ redo commit. Not sure why my develper build would work when the daily build does not but indeed in my development build I get

open 6c08 fromDatabase redo format mtz

You need to install the Clipper bundle to open PDB-REDO MTZ-format map files. See the log for details.
Install the Clipper bundle from the Toolshed.