Machine-friendly REST interface

[Tristan Croll]

I've been learning how to use the existing REST ChimeraX interface, and am most of the way to convincing the PHENIX team that it's superior to XMLRPC (which it quite obviously is). At the moment it's configured to run command-line commands and report back whatever they write in the log. That's great for human interaction, but for scripting it would be great to have a (probably quite small) set of generic functions that report back a set of easily machine-readable results - e.g. open_model() reporting back a (list of) model ID(s) suitable for use in further calls; open_map(), etc..

I would have to add a few extra methods specific to ISOLDE and Clipper (for loading reflection data, etc.). I could build the whole interface into ISOLDE by subclassing the existing one and going from there, but I think it would make more sense if the bones of it were implemented in the official ChimeraX interface.

[Tom Goddard]

Good idea. There is definitely a need to get back values from ChimeraX REST calls that are not just free formatted log output.

For having Cytoscape control ChimeraX I believe Conrad added various "info" commands.

https://www.rbvi.ucsf.edu/chimerax/docs/user/commands/info.html

I guess what will be needed is a specific list of what return values are needed to best control ChimeraX via REST. For opening a model returning the id number(s) of the new model(s) there could perhaps be an option to the open command that specifies that a very specific log output should be given. I'm not sure if that approach will work in other cases -- will need a specific lists of needs to figure that out.

[Tristan Croll]

Billy Poon is responsible for most PHENIX GUI/interface issues, and may be interested in contributing. Billy: the question is, what list of functions and return values would be needed by PHENIX to get the level of control you want?

[BKPoon@…]

Hi everyone,

Some basic functions for opening files is what I had in mind initially.
This is to enable the viewing of refinement/validation results like we have
currently. We had a Phenix phone conference and the suggestion was that
these functions would be independent of file formats.
- "open_model" can open model files in either PDB or CIF format.
- "open_map" can open maps in CCP4/MRC format, and map coefficients in some
data format. For map coefficients, there is a CCTBX program that can be
called to convert the map coefficients into CCP4/MRC format.
- "center_at_xyz" (function name not finalized) would center the view at a
certain coordinate (e.g. atom) with an option for zoom level
- "color_map" (function name not finalized) would change the color of the
map. Some users have asked about difference maps for cryo-em (experimental
map - calculated model map) and this would make it easier to change colors
across different viewers.
- Also, something like "update_model" or "update_map" that could update an
existing model/map in the viewer instead of opening a new one and deleting
the old one.

The idea is that there is a consistent set of function names between Phenix
and any visualization program and we would have a translation layer that
converts these function calls into the appropriate commands for the
visualization program.

Longer term (for the new Phenix GUI), Tristan and I had a brief exchange
about making the communication bidirectional, so that ChimeraX and other
visualization programs can call functions/programs in Phenix. One initial
possibility would be for users to make atom selections in ChimeraX and then
the syntax gets translated into Phenix syntax through this interface. I
have also been working on making a consistent API for running end-user
programs in CCTBX/Phenix so that could help make it easier for ChimeraX to
call the refinement and model-building programs in Phenix. I think that
would be more beneficial to users.

Lastly, there will be a Phenix workshop in Berkeley next March (3/10-3/14),
so it would be great if Tom or someone else from ChimeraX can come. Last
time Tom came, I wasn't quite ready for these low-level infrastructure
changes, but with our migration to Python 3 and the development of a new
GUI, these improvements can be incorporated upfront and done more cleanly
(e.g. moving from RPC to REST).

--
Billy K. Poon
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909
Web: https://phenix-online.org


On Mon, Dec 17, 2018 at 2:37 AM ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu>
wrote:

[Tristan Croll]

Comments from Tom Burnley (leading the CCP-EM project from Diamond Light Source):

We don't have a set framework for this yet but my preference would be for something simple and standard so REST would be a good option. In term of actions it would be very useful in the first instance to be able to...

1) Send multiple EM maps (e.g. sharpened)

2) Send PDB files / atomic coordinates

3) Send atomic annotation e.g. atomic scores for goodness of fit etc

4) Return PDB files / updated coordinates post refinement

For maps and coordinates I assume easiest way is to pass file paths.

[Conrad Huang]

The current REST implementation is specifically designed to run ChimeraX commands and return their output, with no explicit ability to send additional data or structure return output. Obviously, those capabilities can be hacked in. Alternatively, we can extend the REST server code to support application-specific REST interfaces. The solution we go with depends on the complexity of the required functionality.

For example, to hack a solution, we would need ways for the caller to:

• send data files (not a problem if we assume the two programs are on the same host),
• run a custom command whose input are files and whose text output format is known,
• parse the command output for additional data.

An application-specific REST interface is not much harder. Right now, the REST server is hardwired to use only the "commands" parameter from the HTTP request, even though it keeps track of all the parameters received. We can restructure the class to serve as a base class so that the request processing code may be overridden and handle multiple parameters (such as PDB files, atomic annotations, etc) and return easily parsable output (e.g., XML).

To me, item 4 ("Return PDB files / updated coordinates post refinement") is the tricky one because it is unclear which side initiates the interaction. For the first three items, ChimeraX is clearly the server. For item 4, how is the transaction initiated?

[tic20@…]

Yes - for that last option I think the other package would need to be running its own server. 

 
 
Tristan Croll
Research Fellow
Cambridge Institute for Medical Research
University of Cambridge CB2 0XY
 

 

[Tristan Croll]

Suggested implementation

[Tristan Croll]

I've attached an initial implementation that I think works quite nicely. The idea is as follows:

• The server class has a method, register_server_method that allows any method matching certain criteria to be exposed to the client. The criteria are fairly straightforward:
  • The first argument must be the ChimeraX session (this will be stripped out of the signature passed to the client)
  • The remaining arguments must be JSON-serialisable types (essentially strings, ints, floats, NoneType or lists)
  • The method must return a dict of JSON-serialisable types

• A bare GET call from the client causes the the server to provide a JSON-formatted description of the available methods (function name, signature and docstring). The client's get_available_methods() function uses this to create new methods and add them to its own class, allowing them to be called and inspected just like any other method. For example, on the server side there is the method:

def test_command(session, arg1:'whatever', arg2:'you', kwarg1:'like'=None):
    '''
    Will simply echo back a dict of the provided arguments.
    '''
    return {'arg1': arg1, 'arg2':arg2, 'kwarg1': kwarg1}

... which is registered with the server as simply "test".

Once the server is started in ChimeraX and running on port 12365, then in a separate python session:

from client import IsoldeRESTClient
ic = IsoldeRESTClient('localhost', 12365)
ic.connect() # get_available_methods() is called here
help(ic.test)

  Help on method f in module client:

  f(arg1:'whatever', arg2:'you', kwarg1:'like'=None) method of client.IsoldeRESTClient instance
      Will simply echo back a dict of the provided arguments.

ic.test(1, 'watermelon', ['a', 1.23456, None])

  {'arg1': 1, 'arg2': 'watermelon', 'kwarg1': ['a', 1.23456, None], 'log': ''}

The contents of the log will always be added to the returned dict, as will any exceptions raised.

I haven't yet attempted to make the connection secure with SSL (I note that the default Python HTTPS server library recommends that it *shouldn't* be used for production since it only performs basic security checks...). There's at least one security hole that would need to be handled before this goes into production: the run_chimerax_command method (copied with some modifications from the existing ChimeraX REST implementation) potentially allows running arbitrary code by run(['open xxx.py']) - I think this should be blocked (perhaps by adding an optional keyword argument to the core ChimeraX open method?).

Currently I've written this as a submodule in ISOLDE itself, with a few fairly ISOLDE-specific server methods implemented (load_model and load_structure_factors both generate ISOLDE-ready Clipper data structures), but I'd be quite happy if it instead were merged into ChimeraX with the ability for third party developers to register their own server methods as needed. That way, not only could plugins such as Clipper and ISOLDE register themselves, but "true" third parties such as PHENIX or CCPEM would have the option to write ChimeraX startup scripts adding server-side methods that aren't available in the standard repertoire.

Will try to get fresh builds up on the ToolShed later this week. Have a few non-ISOLDE tasks to handle first.

[Tristan Croll]

I've just put fresh builds up (for Linux and Mac - Windows in a few hours). For anyone who wants to test them, first install the latest ChimeraX daily build and install ISOLDE from the ToolShed. Just in case, it tends to be a good idea to restart ChimeraX at this point. To start the server, you can start ChimeraX as:

chimerax --cmd "isolde remote rest start port 12345"

... or equivalently just type the quoted portion into the ChimeraX command line after starting. This will auto-start ISOLDE and get the server listening on localhost. As a quick test that it's running correctly, typing localhost:16543 into the address bar of your favourite browser should give you a JSON-formatted description of the available methods. In Python, use the client.py found in isolde/remote_control/rest_server/client.py (I'll also attach it here). It's independent of everything else in ISOLDE's source tree, so you should be able to run it from wherever you like. I'm quite sure it's not Python 2 compatible right now (although I haven't actually tested to see how badly it would break there yet - non-keyword arguments would end up in random order, for a start). I don't think it would be too hard to *make* it Python 2 compliant, but haven't had the time to look into it yet (would require changes on both server and client side).

Anyway, some examples of using the client:

from client import IsoldeRESTClient
ic = IsoldeRESTClient('localhost', 16543)
ic.connect()

response_1 = ic.load_model('full/path/to/pdb/or/mmcif')
model_id = response['model id']
mgr = response['manager'] # Top-level manager holding the model and all its associated maps

response_2 = ic.load_structure_factors('full/path/to/mtz/or/cif', mgr) # Will create maps for all precalculated amplitudes/phases it finds, and a standard set of "live" maps if experimental reflections are provided.

map_mgr_id = response_2['map_mgr']
map_dict = response_2['maps'] # dict of 'map name': id

response_3 = ic.load_map('/path/to/any/recognised/map/format', mgr)
map_id = response_3['map']

ic.spotlight_radius(20) # Increase the density sphere radius to 20 Angstroms

ic.center_on_coord([10,20,10], radius=10)

ic.close_models(model_id)

[Tristan Croll]

ISOLDE REST client

[Tristan Croll]

I've already sent this around to some off-thread - putting it here to 
make sure everyone's on the same page. Attached is a description and my 
initial implementation of a fairly flexible machine-oriented interface. 
Easiest way to test/play with it right now is with two sessions of 
ChimeraX, since that gives you an interactive iPython console to explore 
the available methods. For example:

Start the first (server) with:

ChimeraX --cmd "isolde remote rest start port 12365"

Start a second ChimeraX session, then in the shell 
(Tools/General/Shell):

from chimerax.isolde.remote_control.rest_server.client import 
IsoldeRESTClient
client = IsoldeRESTClient('localhost', 12365)
clent.connect() # Client is populated with methods from the server here

help(client.load_model)

     Help on method load_model_server_method in module 
chimerax.isolde.remote_control.rest_server.client:

     load_model_server_method(file_path: 'string') method of
     chimerax.isolde.remote_control.rest_server.client.IsoldeRESTClient 
instance
         Load a model from a single PDB or mmCIF file. Multi-model files 
are not
         supported.

     Args:

         file_path: the full path to a single PDB or mmCIF file

     Returns:

         {'manager': model id for the top-level Clipper manager for the 
model,
          'model': model id for the atomic structure itself.
          }

import os
client.load_model(os.path.abspath('./3io0.pdb'))

{'manager': '1',
  'model id': '1.3',
  'log': '_3io0.pdb_ title:  \n **Crystal structure of etub from 
clostridium kluyveri**\n[[more\xa0info...]](cxcmd:log metadata #1)  \n  
\n\nChain information for 3io0.pdb #1  \n---  \nChain | Description  
\n[A](cxcmd:select #1/A:76-304 "Select chain") | [predicted 
microcompartment\nprotein](cxcmd:sequence chain #1/A "Show sequence")  
\n  \n\n_3io0.pdb_ title:  \n **Crystal structure of etub from 
clostridium kluyveri**\n[[more\xa0info...]](cxcmd:log metadata #1.3)  \n 
  \n\nChain information for 3io0.pdb  \n---  \nChain | Description  
\n[1.3/A](cxcmd:select #1.3/A:76-304 "Select chain") | 
[predicted\nmicrocompartment protein](cxcmd:sequence chain #1.3/A "Show 
sequence")  \n  \n\n'}

On 2019-07-29 17:37, ChimeraX wrote:

remote_control.tar.gz​

Remote interface discussion text.pdf​

[Tristan Croll]

Added by email2trac

[Tristan Croll]

Added by email2trac

[Conrad Huang]

We discussed REST at a ChimeraX meeting. Tristan's approach looks really good. However, Scooter suggested that we investigate a framework that supports Swagger/​OpenAPI, whose documentation says:

The OpenAPI Specification (OAS) defines a standard, programming language-agnostic interface description for REST APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic.

There is a Python module for this: ​FastAPI. It does pull in quite a few other packages, so we envision it as a bundle rather than built into the core. We've not had much time to investigate FastAPI further, but will do so if it seems like a possible direction for Tristan.