Ticket #1760: __init__.py

File __init__.py, 48.7 KB (added by fage.cd@…, 7 years ago)

Added by email2trac

Line 
1# vim: set expandtab ts=4 sw=4:
2
3# === UCSF ChimeraX Copyright ===
4# Copyright 2016 Regents of the University of California.
5# All rights reserved. This software provided pursuant to a
6# license agreement containing restrictions on its disclosure,
7# duplication and use. For details see:
8# http://www.rbvi.ucsf.edu/chimerax/docs/licensing.html
9# This notice must be embedded in or attached to all copies,
10# including partial copies, of the software or any revisions
11# or derivations thereof.
12# === UCSF ChimeraX Copyright ===
13
14"""
15The Toolshed provides an interface for finding installed
16bundles as well as bundles available for
17installation from a remote server.
18The Toolshed can handle updating, installing and uninstalling
19bundles while taking care of inter-bundle dependencies.
20
21The Toolshed interface uses :py:mod:`pkg_resources` heavily.
22
23Each Python distribution, a ChimeraX Bundle,
24may contain multiple tools, commands, data formats, and specifiers,
25with metadata entries for each deliverable.
26
27In addition to the normal Python package metadta,
28The 'ChimeraX' classifier entries give additional information.
29Depending on the values of 'ChimeraX' metadata fields,
30modules need to override methods of the :py:class:`BundleAPI` class.
31Each bundle needs a 'ChimeraX :: Bundle' entry
32that consists of the following fields separated by double colons (``::``).
33
341. ``ChimeraX :: Bundle`` : str constant
35 Field identifying entry as bundle metadata.
362. ``categories`` : str
37 Comma-separated list of categories in which the bundle belongs.
383. ``session_versions`` : two comma-separated integers
39 Minimum and maximum session version that the bundle can read.
404. ``supercedes`` : str
41 Comma-separated list of superceded bundle names.
425. ``custom_init`` : str
43 Whether bundle has initialization code that must be called when
44 ChimeraX starts. Either 'true' or 'false'. If 'true', the bundle
45 must override the BundleAPI's 'initialize' and 'finish' functions.
46
47Bundles that provide tools need:
48
491. ``ChimeraX :: Tool`` : str constant
50 Field identifying entry as tool metadata.
512. ``tool_name`` : str
52 The globally unique name of the tool (also shown on title bar).
533. ``categories`` : str
54 Comma-separated list of categories in which the tool belongs.
55 Should be a subset of the bundle's categories.
564. ``synopsis`` : str
57 A short description of the tool. It is here for uninstalled tools,
58 so that users can get more than just a name for deciding whether
59 they want the tool or not.
60
61Tools are created via the bundle's 'start_tool' function.
62Bundles may provide more than one tool.
63
64Bundles that provide commands need:
65
661. ``ChimeraX :: Command`` : str constant
67 Field identifying entry as command metadata.
682. ``command name`` : str
69 The (sub)command name. Subcommand names have spaces in them.
703. ``categories`` : str
71 Comma-separated list of categories in which the command belongs.
72 Should be a subset of the bundle's categories.
734. ``synopsis`` : str
74 A short description of the command. It is here for uninstalled commands,
75 so that users can get more than just a name for deciding whether
76 they want the command or not.
77
78Commands are lazily registered,
79so the argument specification isn't needed until the command is first used.
80Bundles may provide more than one command.
81
82Bundles that provide selectors need:
83
841. ``ChimeraX :: Selector`` : str constant
85 Field identifying entry as command metadata.
862. ``selector name`` : str
87 The selector's name.
883. ``synopsis`` : str
89 A short description of the selector. It is here for uninstalled selectors,
90 so that users can get more than just a name for deciding whether
91 they want the selector or not.
924: ``atomic`` : str
93 An optional boolean specifying whether the selector applies to
94 atoms and bonds. Defaults to 'true' and should be set to
95 'false' if selector should not appear in Basic Actions tool,
96 e.g., showing/hiding selected items does nothing.
97
98Commands are lazily registered,
99so the argument specification isn't needed until the command is first used.
100Bundles may provide more than one command.
101
102Bundles that provide data formats need:
103
1041. ``ChimeraX :: DataFormat`` : str constant
105 Field identifying entry as data format metadata.
1062. ``data_name`` : str
107 The name of the data format.
1083. ``nicknames`` : str
109 An optional comma-separated list of alternative names.
110 Often a short name is provided. If not provided,
111 it defaults to the lowercase version of the data format name.
1124. ``category`` : str
113 The toolshed category.
1145. ``suffixes`` : str
115 An optional comma-separated list of strings with leading periods,
116 e.g., '.pdb'.
1176. ``mime_types`` : str
118 An optinal comma-separated list of strings, e.g., 'chemical/x-pdb'.
1197. ``url`` : str
120 A string that has a URL that points to the data format's documentation.
1218. ``dangerous`` : str
122 An optional boolean and should be 'true' if the data
123 format is insecure -- defaults to true if a script.
1249. ``icon`` : str
125 An optional string containing the filename of the icon --
126 it defaults to the default icon for the category.
127 The file should be ?TODO? -- metadata dir? package dir?
12810. ``synopsis`` : str
129 A short description of the data format. It is here
130 because it needs to be part of the metadata available for
131 uninstalled data format, so that users can get more than just a
132 name for deciding whether they want the data format or not.
133
134Bundles may provide more than one data format.
135The data format metadata includes everything needed for the Mac OS X
136application property list.
137
138Data formats that can be fetched:
139
140# ChimeraX :: Fetch :: database_name :: format_name :: prefixes :: example_id :: is_default
141
142Data formats that can be opened:
143
144# ChimeraX :: Open :: format_name :: tag :: is_default
145
146Data formats that can be saved:
147
148# ChimeraX :: Save :: format_name :: tag :: is_default
149
150Bundles that have other data:
151
152# ChimeraX :: DataDir :: dir_path
153# ChimeraX :: IncludeDir :: dir_path
154# ChimeraX :: LibraryDir :: dir_path
155
156Attributes
157----------
158TOOLSHED_BUNDLE_INFO_ADDED : str
159 Name of trigger fired when new bundle metadata is registered.
160 The trigger data is a :py:class:`BundleInfo` instance.
161TOOLSHED_BUNDLE_INSTALLED : str
162 Name of trigger fired when a new bundle is installed.
163 The trigger data is a :py:class:`BundleInfo` instance.
164TOOLSHED_BUNDLE_UNINSTALLED : str
165 Name of trigger fired when an installed bundle is removed.
166 The trigger data is a :py:class:`BundleInfo` instance.
167TOOLSHED_BUNDLE_INFO_RELOADED : str
168 Name of trigger fired when bundle metadata is reloaded.
169 The trigger data is a :py:class:`BundleInfo` instance.
170
171Notes
172-----
173The term 'installed' refers to bundles whose corresponding Python
174module or package is installed on the local machine. The term
175'available' refers to bundles that are listed on a remote server
176but have not yet been installed on the local machine.
177
178"""
179
180# Toolshed trigger names
181TOOLSHED_BUNDLE_INFO_ADDED = "bundle info added"
182TOOLSHED_BUNDLE_INSTALLED = "bundle installed"
183TOOLSHED_BUNDLE_UNINSTALLED = "bundle uninstalled"
184TOOLSHED_BUNDLE_INFO_RELOADED = "bundle info reloaded"
185
186# Known bundle catagories
187DYNAMICS = "Molecular trajectory"
188GENERIC3D = "Generic 3D objects"
189SCRIPT = "Command script"
190SEQUENCE = "Sequence alignment"
191SESSION = "Session data"
192STRUCTURE = "Molecular structure"
193SURFACE = "Molecular surface"
194VOLUME = "Volume data"
195Categories = [
196 DYNAMICS,
197 GENERIC3D,
198 SCRIPT,
199 SEQUENCE,
200 SESSION,
201 STRUCTURE,
202 SURFACE,
203 VOLUME,
204]
205
206_TIMESTAMP = 'install-timestamp'
207_debug_toolshed = False
208
209
210def _debug(*args, file=None, flush=True, **kw):
211 if _debug_toolshed:
212 if file is None:
213 import sys
214 file = sys.__stderr__
215 print("Toolshed:", *args, file=file, flush=flush, **kw)
216
217
218# Package constants
219
220
221# Default URL of remote toolshed
222# If testing, use
223#_RemoteURL = "https://cxtoolshed-preview.rbvi.ucsf.edu"
224# But BE SURE TO CHANGE IT BACK BEFORE COMMITTING !!!
225_RemoteURL = "https://cxtoolshed.rbvi.ucsf.edu"
226# Default name for toolshed cache and data directories
227_ToolshedFolder = "toolshed"
228# Defaults names for installed ChimeraX bundles
229_ChimeraNamespace = "chimerax"
230
231
232# Exceptions raised by Toolshed class
233
234
235class ToolshedError(Exception):
236 """Generic Toolshed error."""
237
238
239class ToolshedInstalledError(ToolshedError):
240 """Bundle-already-installed error.
241
242 This exception derives from :py:class:`ToolshedError` and is usually
243 raised when trying to install a bundle that is already installed
244 or to uninstall a bundle that is not installed yet."""
245
246
247class ToolshedUnavailableError(ToolshedError):
248 """Bundle-not-found error.
249
250 This exception derives from ToolshedError and is usually
251 raised when no Python distribution can be found for a bundle."""
252
253
254# Toolshed and BundleInfo are session-independent
255
256
257class Toolshed:
258 """Toolshed keeps track of the list of bundle metadata, aka :py:class:`BundleInfo`.
259
260 Tool metadata may be for "installed" bundles, where their code
261 is already downloaded from the remote server and installed
262 locally, or "available" bundles, where their code is not locally
263 installed.
264
265 Attributes
266 ----------
267 triggers : :py:class:`~chimerax.core.triggerset.TriggerSet` instance
268 Where to register handlers for toolshed triggers
269 """
270
271 def __init__(self, logger, rebuild_cache=False, check_remote=False,
272 remote_url=None, check_available=True):
273 """Initialize Toolshed instance.
274
275 Parameters
276 ----------
277 logger : :py:class:`~chimerax.core.logger.Logger` instance
278 A logging object where warning and error messages are sent.
279 rebuild_cache : boolean
280 True to ignore local cache of installed bundle information and
281 rebuild it by scanning Python directories; False otherwise.
282 check_remote : boolean
283 True to check remote server for updated information;
284 False to ignore remote server
285 remote_url : str
286 URL of the remote toolshed server.
287 If set to None, a default URL is used.
288 """
289 # Initialize with defaults
290 _debug("__init__", rebuild_cache, check_remote, remote_url)
291 if remote_url is None:
292 self.remote_url = _RemoteURL
293 else:
294 self.remote_url = remote_url
295 self._repo_locator = None
296 self._installed_bundle_info = None
297 self._available_bundle_info = None
298 self._installed_packages = {} # cache mapping packages to bundles
299
300 # Compute base directories
301 import os
302 from chimerax import app_dirs
303 if os.path.exists(app_dirs.user_cache_dir):
304 self._cache_dir = os.path.join(app_dirs.user_cache_dir, _ToolshedFolder)
305 else:
306 self._cache_dir = None
307 _debug("cache dir: %s" % self._cache_dir)
308 # TODO: unused so far
309 # self._data_dir = os.path.join(app_dirs.user_data_dir, _ToolshedFolder)
310 # _debug("data dir: %s" % self._data_dir)
311
312 # Add directories to sys.path
313 import site
314 self._site_dir = site.USER_SITE
315 _debug("site dir: %s" % self._site_dir)
316 import os
317 os.makedirs(self._site_dir, exist_ok=True)
318 site.addsitedir(self._site_dir)
319
320 # Create triggers
321 from .. import triggerset
322 self.triggers = triggerset.TriggerSet()
323 self.triggers.add_trigger(TOOLSHED_BUNDLE_INFO_ADDED)
324 self.triggers.add_trigger(TOOLSHED_BUNDLE_INSTALLED)
325 self.triggers.add_trigger(TOOLSHED_BUNDLE_UNINSTALLED)
326 self.triggers.add_trigger(TOOLSHED_BUNDLE_INFO_RELOADED)
327 self.triggers.add_trigger("selector registered")
328 self.triggers.add_trigger("selector deregistered")
329
330 # Variables for updating list of available bundles
331 from threading import RLock
332 self._abc_lock = RLock()
333 self._abc_updating = False
334
335 # Reload the bundle info list
336 _debug("loading bundles")
337 try:
338 self.init_available_from_cache(logger)
339 except Exception:
340 logger.report_exception("Error preloading available bundles")
341 self.reload(logger, check_remote=check_remote, rebuild_cache=rebuild_cache)
342 if check_available and not check_remote:
343 # Did not check for available bundles synchronously
344 # so start a thread and do it asynchronously if necessary
345 from ..core_settings import settings
346 from datetime import datetime, timedelta
347 now = datetime.now()
348 interval = settings.toolshed_update_interval
349 last_check = settings.toolshed_last_check
350 if not last_check:
351 need_check = True
352 else:
353 last_check = datetime.strptime(settings.toolshed_last_check,
354 "%Y-%m-%dT%H:%M:%S.%f")
355 delta = now - last_check
356 max_delta = timedelta(days=1)
357 if interval == "week":
358 max_delta = timedelta(days=7)
359 elif interval == "day":
360 max_delta = timedelta(days=1)
361 elif interval == "month":
362 max_delta = timedelta(days=30)
363 need_check = delta > max_delta
364 if need_check:
365 self.async_reload_available(logger)
366 settings.toolshed_last_check = now.isoformat()
367 _debug("Initiated toolshed check: %s" %
368 settings.toolshed_last_check)
369 _debug("finished loading bundles")
370
371 def reload(self, logger, *, session=None, reread_cache=True, rebuild_cache=False,
372 check_remote=False, report=False):
373 """Supported API. Discard and reread bundle info.
374
375 Parameters
376 ----------
377 logger : :py:class:`~chimerax.core.logger.Logger` instance
378 A logging object where warning and error messages are sent.
379 rebuild_cache : boolean
380 True to ignore local cache of installed bundle information and
381 rebuild it by scanning Python directories; False otherwise.
382 check_remote : boolean
383 True to check remote server for updated information;
384 False to ignore remote server
385 """
386
387 _debug("reload", rebuild_cache, check_remote)
388 if reread_cache or rebuild_cache:
389 from .installed import InstalledBundleCache
390 save = self._installed_bundle_info
391 self._installed_bundle_info = InstalledBundleCache()
392 cache_file = self._bundle_cache(False, logger)
393 self._installed_bundle_info.load(logger, cache_file=cache_file,
394 rebuild_cache=rebuild_cache,
395 write_cache=cache_file is not None)
396 if report:
397 if save is None:
398 logger.info("Initial installed bundles.")
399 else:
400 from .installed import _report_difference
401 _report_difference(logger, save, self._installed_bundle_info)
402 if save is not None:
403 save.deregister_all(logger, session, self._installed_packages)
404 self._installed_bundle_info.register_all(logger, session,
405 self._installed_packages)
406 if check_remote:
407 self.reload_available(logger)
408 self.triggers.activate_trigger(TOOLSHED_BUNDLE_INFO_RELOADED, self)
409
410 def async_reload_available(self, logger):
411 with self._abc_lock:
412 self._abc_updating = True
413 from threading import Thread
414 t = Thread(target=self.reload_available, args=(logger,),
415 name="Update list of available bundles")
416 t.start()
417
418 def reload_available(self, logger):
419 from urllib.error import URLError
420 from .available import AvailableBundleCache
421 abc = AvailableBundleCache(self._cache_dir)
422 try:
423 abc.load(logger, self.remote_url)
424 except URLError as e:
425 logger.info("Updating list of available bundles failed: %s"
426 % str(e.reason))
427 with self._abc_lock:
428 self._abc_updating = False
429 except Exception as e:
430 logger.info("Updating list of available bundles failed: %s"
431 % str(e))
432 with self._abc_lock:
433 self._abc_updating = False
434 else:
435 with self._abc_lock:
436 self._available_bundle_info = abc
437 self._abc_updating = False
438 from ..commands import cli
439 cli.clear_available()
440
441 def init_available_from_cache(self, logger):
442 from .available import AvailableBundleCache
443 abc = AvailableBundleCache(self._cache_dir)
444 try:
445 abc.load_from_cache()
446 except FileNotFoundError:
447 logger.info("available bundle cache has not been initialized yet")
448 else:
449 self._available_bundle_info = abc
450
451 def register_available_commands(self, logger):
452 for bi in self._get_available_bundles(logger):
453 bi.register_available_commands(logger)
454
455 def set_install_timestamp(self, per_user=False):
456 _debug("set_install_timestamp")
457 self._installed_bundle_info.set_install_timestamp(per_user=per_user)
458
459 def bundle_info(self, logger, installed=True, available=False):
460 """Supported API. Return list of bundle info.
461
462 Parameters
463 ----------
464 installed : boolean
465 True to include installed bundle metadata in return value;
466 False otherwise
467 available : boolean
468 True to include available bundle metadata in return value;
469 False otherwise
470
471 Returns
472 -------
473 list of :py:class:`BundleInfo` instances
474 Combined list of all selected types of bundle metadata. """
475
476 # _installed_bundle_info should always be defined
477 # but _available_bundle_info may need to be initialized
478 if available and self._available_bundle_info is None:
479 self.reload(logger, reread_cache=False, check_remote=True)
480 if installed and available:
481 return self._installed_bundle_info + self._get_available_bundles(logger)
482 elif installed:
483 return self._installed_bundle_info
484 elif available:
485 return self._get_available_bundles(logger)
486 else:
487 return []
488
489 def install_bundle(self, bundle, logger, *, per_user=True, reinstall=False, session=None):
490 """Supported API. Install the bundle by retrieving it from the remote shed.
491
492 Parameters
493 ----------
494 bundle : string or :py:class:`BundleInfo` instance
495 If string, path to wheel installer.
496 If instance, should be from the available bundle list.
497 per_user : boolean
498 True to install bundle only for the current user (default);
499 False to install for everyone.
500 reinstall : boolean
501 True to force reinstall package.
502 logger : :py:class:`~chimerax.core.logger.Logger` instance
503 Logging object where warning and error messages are sent.
504
505 Raises
506 ------
507 ToolshedInstalledError
508 Raised if the bundle is already installed.
509
510 Notes
511 -----
512 A :py:const:`TOOLSHED_BUNDLE_INSTALLED` trigger is fired after installation.
513 """
514 _debug("install_bundle", bundle)
515 # Make sure that our install location is on chimerax module.__path__
516 # so that newly installed modules may be found
517 import importlib, os.path, re
518 cx_dir = os.path.join(self._site_dir, _ChimeraNamespace)
519 m = importlib.import_module(_ChimeraNamespace)
520 if cx_dir not in m.__path__:
521 m.__path__.append(cx_dir)
522 try:
523 if bundle.installed:
524 if not reinstall:
525 raise ToolshedInstalledError("bundle %r already installed" % bundle.name)
526 if bundle in self._installed_bundle_info:
527 bundle.deregister(logger)
528 bundle.unload(logger)
529 self._installed_bundle_info.remove(bundle)
530 # The reload that will happen later will undo the effect
531 # of the unload by accessing the module again, so we
532 # explicitly remove the bundle right now
533 bundle = bundle.name
534 except AttributeError:
535 # If "bundle" is not an instance, it must be a string.
536 # Treat it like a path to a wheel and get a putative
537 # bundle name. If it is install, deregister and unload it.
538 basename = os.path.split(bundle)[1]
539 name = basename.split('-')[0]
540 bi = self.find_bundle(name, logger, installed=True)
541 if bi in self._installed_bundle_info:
542 bi.deregister(logger)
543 bi.unload(logger)
544 self._installed_bundle_info.remove(bi)
545 if per_user is None:
546 per_user = True
547 try:
548 results = self._pip_install(bundle, per_user=per_user, reinstall=reinstall)
549 except PermissionError as e:
550 who = "everyone" if not per_user else "this account"
551 logger.error("You do not have permission to install %s for %s" %
552 (bundle, who))
553 return
554 installed = re.findall(r"^\s*Successfully installed.*$", results, re.M)
555 if installed:
556 logger.info('\n'.join(installed))
557 else:
558 logger.info('No bundles were installed')
559 self.set_install_timestamp(per_user)
560 self.reload(logger, rebuild_cache=True, report=True)
561 self.triggers.activate_trigger(TOOLSHED_BUNDLE_INSTALLED, bundle)
562
563 def uninstall_bundle(self, bundle, logger, *, session=None):
564 """Supported API. Uninstall bundle by removing the corresponding Python distribution.
565
566 Parameters
567 ----------
568 bundle : string or :py:class:`BundleInfo` instance
569 If string, path to wheel installer.
570 If instance, should be from the available bundle list.
571 logger : :py:class:`~chimerax.core.logger.Logger` instance
572 Logging object where warning and error messages are sent.
573
574 Raises
575 ------
576 ToolshedInstalledError
577 Raised if the bundle is not installed.
578
579 Notes
580 -----
581 A :py:const:`TOOLSHED_BUNDLE_UNINSTALLED` trigger is fired after package removal.
582 """
583 import re
584 _debug("uninstall_bundle", bundle)
585 try:
586 if not bundle.installed:
587 raise ToolshedInstalledError("bundle %r not installed" % bundle.name)
588 bundle.deregister(logger)
589 bundle.unload(logger)
590 bundle = bundle.name
591 except AttributeError:
592 # If "bundle" is not an instance, just leave it alone
593 pass
594 results = self._pip_uninstall(bundle)
595 uninstalled = re.findall(r"^\s*Successfully uninstalled.*$", results, re.M)
596 if uninstalled:
597 logger.info('\n'.join(uninstalled))
598 self.reload(logger, rebuild_cache=True, report=True)
599 self.triggers.activate_trigger(TOOLSHED_BUNDLE_UNINSTALLED, bundle)
600
601 def find_bundle(self, name, logger, installed=True, version=None):
602 """Supported API. Return a :py:class:`BundleInfo` instance with the given name.
603
604 Parameters
605 ----------
606 name : str
607 Name (internal or display name) of the bundle of interest.
608 logger : :py:class:`~chimerax.core.logger.Logger` instance
609 Logging object where warning and error messages are sent.
610 installed : boolean
611 True to check only for installed bundles; False otherwise.
612 version : str
613 None to find any version; specific string to check for
614 one particular version.
615
616 """
617 _debug("find_bundle", name, installed, version)
618 if installed:
619 container = self._installed_bundle_info
620 else:
621 container = self._get_available_bundles(logger)
622 from pkg_resources import parse_version
623 lc_name = name.lower().replace('_', '-')
624 lc_names = [lc_name]
625 if not lc_name.startswith("chimerax-"):
626 lc_names.append("chimerax-" + lc_name)
627 best_bi = None
628 best_version = None
629 for bi in container:
630 if bi.name.lower() not in lc_names:
631 continue
632 #if bi.name != name and name not in bi.supercedes:
633 # continue
634 if version == bi.version:
635 return bi
636 if version is None:
637 if best_bi is None:
638 best_bi = bi
639 best_version = parse_version(bi.version)
640 elif best_bi.name != bi.name:
641 logger.warning("%r matches multiple bundles %s, %s" % (name, best_bi.name, bi.name))
642 return None
643 else:
644 v = parse_version(bi.version)
645 if v > best_version:
646 best_bi = bi
647 best_version = v
648 return best_bi
649
650 def find_bundle_for_tool(self, name):
651 """Supported API. Find named tool and its bundle
652
653 Return the bundle it is in and its true name.
654 """
655 folded_name = name.casefold()
656 tools = []
657 for bi in self._installed_bundle_info:
658 for tool in bi.tools:
659 tname = tool.name.casefold()
660 if tname == folded_name:
661 return (bi, tool.name)
662 if tname.startswith(folded_name):
663 tools.append((bi, tool.name))
664 if len(tools) == 0:
665 return None, name
666 # TODO: longest match?
667 return tools[0]
668
669 def find_bundle_for_command(self, cmd):
670 """Supported API. Find bundle registering given command
671
672 `cmd` must be the full command name, not an abbreviation."""
673 for bi in self._installed_bundle_info:
674 for ci in bi.commands:
675 if ci.name == cmd:
676 return bi
677 return None
678
679 def find_bundle_for_class(self, cls):
680 """Supported API. Find bundle that has given class"""
681
682 package = tuple(cls.__module__.split('.'))
683 while package:
684 try:
685 return self._installed_packages[package]
686 except KeyError:
687 pass
688 package = package[0:-1]
689 return None
690
691 def bootstrap_bundles(self, session):
692 """Supported API. Do custom initialization for installed bundles
693
694 After adding the :py:class:`Toolshed` singleton to a session,
695 allow bundles need to install themselves into the session,
696 (For symmetry, there should be a way to uninstall all bundles
697 before a session is discarded, but we don't do that yet.)
698 """
699 _debug("initialize_bundles")
700 failed = []
701 for bi in self._installed_bundle_info:
702 bi.update_library_path() # for bundles with dynamic libraries
703 try:
704 bi.initialize(session)
705 except ToolshedError:
706 failed.append(bi)
707 for bi in failed:
708 self._installed_bundle_info.remove(bi)
709 # TODO: update _installed_packages
710
711 def import_bundle(self, bundle_name, logger,
712 install="ask", session=None):
713 """Supported API. Return the module for the bundle with the given name.
714
715 Parameters
716 ----------
717 bundle_name : str
718 Name (internal or display name) of the bundle of interest.
719 logger : :py:class:`~chimerax.core.logger.Logger` instance
720 Logging object where warning and error messages are sent.
721 install: str
722 Action to take if bundle is uninstalled but available.
723 "ask" (default) means to ask user, if `session` is not `None`;
724 "never" means not to install; and
725 "always" means always install.
726 session : :py:class:`chimerax.core.session.Session` instance.
727 Session that is requesting the module. Defaults to `None`.
728
729 Raises
730 ------
731 ImportError
732 Raised if a module for the bundle cannot be found.
733 """
734 # If the bundle is installed, return its module.
735 bundle = self.find_bundle(bundle_name, logger, installed=True)
736 if bundle is not None:
737 module = bundle.get_module()
738 if module is None:
739 raise ImportError("bundle %r has no module" % bundle_name)
740 return module
741 bundle = self.find_bundle(bundle_name, logger, installed=False)
742 if bundle is None:
743 raise ImportError("bundle %r not found" % bundle_name)
744 return self._install_module(bundle, logger, install, session)
745
746
747 def import_package(self, package_name, logger,
748 install=None, session=None):
749 """Supported API. Return package of given name if it is associated with a bundle.
750
751 Parameters
752 ----------
753 module_name : str
754 Name of the module of interest.
755 logger : :py:class:`~chimerax.core.logger.Logger` instance
756 Logging object where warning and error messages are sent.
757 install: str
758 Action to take if bundle is uninstalled but available.
759 "ask" (default) means to ask user, if `session` is not `None`;
760 "never" means not to install; and
761 "always" means always install.
762 session : :py:class:`chimerax.core.session.Session` instance.
763 Session that is requesting the module. Defaults to `None`.
764
765 Raises
766 ------
767 ImportError
768 Raised if a module for the bundle cannot be found.
769 """
770 for bi in self._installed_bundle_info:
771 if bi.package_name == package_name:
772 module = bi.get_module()
773 if module is None:
774 raise ImportError("bundle %r has no module" % bundle_name)
775 return module
776 # No installed bundle matches
777 from pkg_resources import parse_version
778 lc_name = name.lower().replace('_', '-')
779 best_bi = None
780 best_version = None
781 for bi in self._get_available_bundles(logger):
782 if bi.package_name != package_name:
783 continue
784 if best_bi is None:
785 best_bi = bi
786 best_version = parse_version(bi.version)
787 elif best_bi.name != bi.name:
788 raise ImportError("%r matches multiple bundles %s, %s" % (package_name, best_bi.name, bi.name))
789 else:
790 v = parse_version(bi.version)
791 if v > best_version:
792 best_bi = bi
793 best_version = v
794 if best_bi is None:
795 raise ImportError("bundle %r not found" % bundle_name)
796 return self._install_module(best_bi, logger, install, session)
797
798 #
799 # End public API
800 # All methods below are private
801 #
802
803 def _get_available_bundles(self, logger):
804 with self._abc_lock:
805 if self._available_bundle_info is None:
806 if self._abc_updating:
807 logger.warning("still retrieving bundle list from toolshed")
808 else:
809 logger.warning("could not retrieve bundle list from toolshed")
810 from .available import AvailableBundleCache
811 self._available_bundle_info = AvailableBundleCache(self._cache_dir)
812 elif self._abc_updating:
813 logger.warning("still updating bundle list from toolshed")
814 return self._available_bundle_info
815
816 def _bundle_cache(self, must_exist, logger):
817 """Return path to bundle cache file. None if not available."""
818 _debug("_bundle_cache", must_exist)
819 if self._cache_dir is None:
820 return None
821 if must_exist:
822 import os
823 os.makedirs(self._cache_dir, exist_ok=True)
824 import os
825 return os.path.join(self._cache_dir, "bundle_info.cache")
826
827 def _pip_install(self, bundle_name, per_user=True, reinstall=False):
828 # Run "pip" with our standard arguments (index location, update
829 # strategy, etc) plus the given arguments. Return standard
830 # output as string. If there was an error, raise RuntimeError
831 # with stderr as parameter.
832 import sys
833 command = ["install", "--upgrade",
834 "--extra-index-url", self.remote_url + "/pypi/",
835 "--upgrade-strategy", "only-if-needed",
836 # "--only-binary", ":all:" # msgpack-python is not binary
837 ]
838 if per_user:
839 command.append("--user")
840 if reinstall:
841 # XXX: Not sure how this interacts with "only-if-needed"
842 command.append("--force-reinstall")
843 # bundle_name can be either a file path or a bundle name in repository
844 command.append(bundle_name)
845 results = self._run_pip(command)
846 # self._remove_scripts()
847 return results
848
849 def _pip_uninstall(self, bundle_name):
850 # Run "pip" and return standard output as string. If there
851 # was an error, raise RuntimeError with stderr as parameter.
852 import sys
853 command = ["uninstall", "--yes", bundle_name]
854 return self._run_pip(command)
855
856 def _run_pip(self, command):
857 import sys, subprocess
858 _debug("_run_pip command:", command)
859 cp = subprocess.run([sys.executable, "-m", "pip"] + command,
860 stdout=subprocess.PIPE,
861 stderr=subprocess.PIPE)
862 if cp.returncode != 0:
863 output = cp.stdout.decode("utf-8", "backslashreplace")
864 error = cp.stderr.decode("utf-8", "backslashreplace")
865 _debug("_run_pip return code:", cp.returncode, file=sys.__stderr__)
866 _debug("_run_pip output:", output, file=sys.__stderr__)
867 _debug("_run_pip error:", error, file=sys.__stderr__)
868 s = output + error
869 if "PermissionError" in s:
870 raise PermissionError(s)
871 else:
872 raise RuntimeError(s)
873 result = cp.stdout.decode("utf-8", "backslashreplace")
874 _debug("_run_pip result:", result)
875 return result
876
877 def _remove_scripts(self):
878 # remove pip installed scripts since they have hardcoded paths to
879 # python and thus don't work when ChimeraX is installed elsewhere
880 from chimerax import app_bin_dir
881 import sys, os
882 if sys.platform.startswith('win'):
883 # Windows
884 script_dir = os.path.join(app_bin_dir, 'Scripts')
885 for dirpath, dirnames, filenames in os.walk(script_dir, topdown=False):
886 for f in filenames:
887 path = os.path.join(dirpath, f)
888 os.remove(path)
889 os.rmdir(dirpath)
890 else:
891 # Linux, Mac OS X
892 for filename in os.listdir(app_bin_dir):
893 path = os.path.join(app_bin_dir, filename)
894 if not os.path.isfile(path):
895 continue
896 with open(path, 'br') as f:
897 line = f.readline()
898 if line[0:2] != b'#!' or b'/bin/python' not in line:
899 continue
900 #print('removing (pip installed)', path)
901 os.remove(path)
902
903 def _install_module(self, bundle, logger, install, session):
904 # Given a bundle name and *uninstalled* bundle, install it
905 # and return the module from the *installed* bundle
906 if install == "never":
907 raise ImportError("bundle %r is not installed" % bundle.name)
908 if install == "ask":
909 if session is None:
910 raise ImportError("bundle %r is not installed" % bundle.name)
911 from chimerax.ui.ask import ask
912 answer = ask(session, "Install bundle %r?" % bundle.name,
913 buttons = ["just me", "all users", "cancel"])
914 if answer == "cancel":
915 raise ImportError("user canceled installation of bundle %r" % bundle.name)
916 elif answer == "just me":
917 per_user = True
918 elif answer == "all users":
919 per_user = False
920 else:
921 raise ImportError("installation of bundle %r canceled" % bundle.name)
922 # We need to install the bundle.
923 self.install_bundle(bundle.name, logger, per_user=per_user)
924 # Now find the *installed* bundle.
925 bundle = self.find_bundle(bundle.name, logger, installed=True)
926 if bundle is None:
927 raise ImportError("could not install bundle %r" % bundle.name)
928 module = bundle.get_module()
929 if module is None:
930 raise ImportError("bundle %r has no module" % bundle.name)
931 return module
932
933
934class BundleAPI:
935 """API for accessing bundles
936
937 The metadata for the bundle indicates which of the methods need to be
938 implemented.
939 """
940
941 api_version = 0
942
943 @staticmethod
944 def start_tool(*args):
945 """Supported API. This method is called when the tool is invoked,
946 typically from the application menu.
947 Errors should be reported via exceptions.
948
949 Parameters
950 ----------
951 session : :py:class:`chimerax.core.session.Session` instance.
952 bundle_info : instance of :py:class:`BundleInfo`
953 tool_info : instance of :py:class:`ToolInfo`
954
955 Version 1 of the API passes in information for both
956 the tool to be started and the bundle where it was defined.
957
958 session : :py:class:`chimerax.core.session.Session` instance.
959 tool_name : str.
960
961 Version 0 of the API only passes in the name of
962 the tool to be started.
963
964
965 Returns
966 -------
967 :py:class:`~chimerax.core.tools.ToolInstance` instance
968 The created tool.
969 """
970 raise NotImplementedError("BundleAPI.start_tool")
971
972 @staticmethod
973 def register_command(*args):
974 """Supported API. When ChimeraX starts, it registers placeholders for
975 commands from all bundles. When a command from this
976 bundle is actually used, ChimeraX calls this method to
977 register the function that implements the command
978 functionality, and then calls the command function.
979 On subsequent uses of the command, ChimeraX will
980 call the command function directly instead of calling
981 this method.
982
983 Parameters
984 ----------
985 bundle_info : instance of :py:class:`BundleInfo`
986 command_info : instance of :py:class:`CommandInfo`
987 logger : :py:class:`~chimerax.core.logger.Logger` instance.
988
989 Version 1 of the API pass in information for both
990 the command to be registered and the bundle where
991 it was defined.
992
993 command : str
994 logger : :py:class:`~chimerax.core.logger.Logger` instance.
995
996 Version 0 of the API only passes in the name of the
997 command to be registered.
998 """
999 raise NotImplementedError("BundleAPI.register_command")
1000
1001 @staticmethod
1002 def register_selector(*args):
1003 """Supported API. This method is called the first time when the selector is used.
1004
1005 Parameters
1006 ----------
1007 bundle_info : instance of :py:class:`BundleInfo`
1008 selector_info : instance of :py:class:`SelectorInfo`
1009 logger : :py:class:`chimerax.core.logger.Logger` instance.
1010
1011 Version 1 of the API passes in information about
1012 both the selector to be registered and the bundle
1013 where it is defined.
1014
1015 selector_name : str
1016 logger : :py:class:`chimerax.core.logger.Logger` instance.
1017
1018 Version 0 of the API only passes in the name of the
1019 selector to be registered.
1020 """
1021 raise NotImplementedError("BundleAPI.register_selector")
1022
1023 @staticmethod
1024 def open_file(session, stream_or_path, optional_format_name, optional_file_name, **kw):
1025 """Supported API. Called to open a file.
1026
1027 Second arg must be 'stream' or 'path'. Depending on the name, either an open
1028 data stream or a filesystem path will be provided. The third and fourth
1029 arguments are optional (remove ``optional_`` from their names if you provide them).
1030 'format-name' will be the first nickname of the format if it has any, otherwise
1031 the full format name, but all lower case. 'file_name' if the name of input file,
1032 with path and compression suffix components stripped.
1033
1034 You shouldn't actually use 'kw' but instead use the actual keyword args that
1035 your format declares that it accepts (in its bundle_info.xml file).
1036
1037 Returns
1038 -------
1039 tuple
1040 The return value is a 2-tuple whose first element is a list of
1041 :py:class:`~chimerax.core.models.Model` instances and second
1042 element is a string containing a status message, such as the
1043 number of atoms and bonds found in the open models.
1044 """
1045 raise NotImplementedError("BundleAPI.open_file")
1046
1047 @staticmethod
1048 def save_file(session, stream, name, **kw):
1049 """Supported API. Called to save a file.
1050
1051 Arguments and return values are as described for save functions in
1052 :py:mod:`chimerax.core.io`.
1053 The format name will be in the **format_name** keyword.
1054 """
1055 raise NotImplementedError("BundleAPI.save_file")
1056
1057 @staticmethod
1058 def fetch_from_database(session, identifier, **kw):
1059 """Supported API. Called to fetch an entry from a network resource.
1060
1061 Arguments and return values are as described for save functions in
1062 :py:mod:`chimerax.core.fetch`.
1063 The format name will be in the **format_name** keyword.
1064 Whether a cache may be used will be in the **ignore_cache** keyword.
1065 """
1066 raise NotImplementedError("BundleAPI.fetch_from_database")
1067
1068 @staticmethod
1069 def initialize(session, bundle_info):
1070 """Supported API. Called to initialize a bundle in a session.
1071
1072 Must be defined if the ``custom_init`` metadata field is set to 'true'.
1073 ``initialize`` is called when the bundle is first loaded.
1074 To make ChimeraX start quickly, custom initialization is discouraged.
1075
1076 Parameters
1077 ----------
1078 session : :py:class:`~chimerax.core.session.Session` instance.
1079 bundle_info : :py:class:`BundleInfo` instance.
1080 """
1081 raise NotImplementedError("BundleAPI.initialize")
1082
1083 @staticmethod
1084 def finish(session, bundle_info):
1085 """Supported API. Called to deinitialize a bundle in a session.
1086
1087 Must be defined if the ``custom_init`` metadata field is set to 'true'.
1088 ``finish`` is called when the bundle is unloaded.
1089
1090 Parameters
1091 ----------
1092 session : :py:class:`~chimerax.core.session.Session` instance.
1093 bundle_info : :py:class:`BundleInfo` instance.
1094 """
1095 raise NotImplementedError("BundleAPI.finish")
1096
1097 @staticmethod
1098 def get_class(name):
1099 """Supported API. Called to get named class from bundle.
1100
1101 Used when restoring sessions. Instances whose class can't be found via
1102 'get_class' can not be saved in sessions. And those classes must implement
1103 the :py:class:`~chimerax.core.state.State` API.
1104
1105 Parameters
1106 ----------
1107 name : str
1108 Name of class in bundle.
1109 """
1110 return None
1111
1112 @staticmethod
1113 def include_dir(bundle_info):
1114 """Returns path to directory of C++ header files.
1115
1116 Used to get directory path to C++ header files needed for
1117 compiling against libraries provided by the bundle.
1118
1119 Parameters
1120 ----------
1121 bundle_info : :py:class:`BundleInfo` instance.
1122
1123 Returns
1124 -------
1125 str or None
1126
1127 """
1128 return None
1129
1130 @staticmethod
1131 def library_dir(bundle_info):
1132 """Returns path to directory of compiled libraries.
1133
1134 Used to get directory path to libraries (shared objects, DLLs)
1135 for linking against libraries provided by the bundle.
1136
1137 Parameters
1138 ----------
1139 bundle_info : :py:class:`BundleInfo` instance.
1140
1141 Returns
1142 -------
1143 str or None
1144 """
1145 return None
1146
1147 @staticmethod
1148 def data_dir(bundle_info):
1149 """Supported API. Returns path to directory of bundle-specific data.
1150
1151 Used to get directory path to data included in the bundle.
1152
1153 Parameters
1154 ----------
1155 bundle_info : :py:class:`BundleInfo` instance.
1156
1157 Returns
1158 -------
1159 str or None
1160 """
1161 return None
1162
1163 @property
1164 def _api_caller(self):
1165 try:
1166 return _CallBundleAPI[self.api_version]
1167 except KeyError:
1168 raise ToolshedError("bundle uses unsupport bundle API version %s" % api.api_version)
1169
1170
1171#
1172# _CallBundleAPI is used to call a bundle method with the
1173# correct arguments depending on the API version used by the
1174# bundle. Note that open_file, save_file, fetch_from_database,
1175# and get_class are not called via this mechanism.
1176# ../io.py handles the argument passing for open_file and
1177# save_file using introspection.
1178# ../fetch.py handles the argument passing for fetch_from_database.
1179# get_class() is more of a lookup than an invocation and the
1180# calling convertion should not change.
1181#
1182class _CallBundleAPIv0:
1183
1184 api_version = 0
1185
1186 @classmethod
1187 def start_tool(cls, api, session, bi, ti):
1188 return cls._get_func(api, "start_tool")(session, ti.name)
1189
1190 @classmethod
1191 def register_command(cls, api, bi, ci, logger):
1192 return cls._get_func(api, "register_command")(ci.name, logger)
1193
1194 @classmethod
1195 def register_selector(cls, api, bi, si, logger):
1196 return cls._get_func(api, "register_selector")(si.name, logger)
1197
1198 @classmethod
1199 def initialize(cls, api, session, bi):
1200 return cls._get_func(api, "initialize")(session, bi)
1201
1202 @classmethod
1203 def finish(cls, api, session, bi):
1204 return cls._get_func(api, "finish")(session, bi)
1205
1206 @classmethod
1207 def include_dir(cls, api, bi):
1208 return cls._get_func(api, "include_dir", default_okay=True)(bi)
1209
1210 @classmethod
1211 def library_dir(cls, api, bi):
1212 return cls._get_func(api, "library_dir", default_okay=True)(bi)
1213
1214 @classmethod
1215 def data_dir(cls, api, bi):
1216 return cls._get_func(api, "data_dir", default_okay=True)(bi)
1217
1218 @classmethod
1219 def _get_func(cls, api, func_name, default_okay=False):
1220 try:
1221 f = getattr(api, func_name)
1222 except AttributeError:
1223 raise ToolshedError("bundle has no %s method" % func_name)
1224 if not default_okay and f is getattr(BundleAPI, func_name):
1225 raise ToolshedError("bundle forgot to override %s method" % func_name)
1226 return f
1227
1228
1229class _CallBundleAPIv1(_CallBundleAPIv0):
1230
1231 api_version = 1
1232
1233 @classmethod
1234 def start_tool(cls, api, session, bi, ti):
1235 return cls._get_func(api, "start_tool")(session, bi, ti)
1236
1237 @classmethod
1238 def register_command(cls, api, bi, ci, logger):
1239 return cls._get_func(api, "register_command")(bi, ci, logger)
1240
1241 @classmethod
1242 def register_selector(cls, api, bi, si, logger):
1243 return cls._get_func(api, "register_selector")(bi, si, logger)
1244
1245
1246_CallBundleAPI = {
1247 0: _CallBundleAPIv0,
1248 1: _CallBundleAPIv1,
1249}
1250
1251
1252# Import classes that developers might want to use
1253from .info import BundleInfo, CommandInfo, ToolInfo, SelectorInfo, FormatInfo
1254
1255
1256# Toolshed is a singleton. Multiple calls to init returns the same instance.
1257_toolshed = None
1258
1259_default_help_dirs = None
1260
1261
1262def init(*args, debug=None, **kw):
1263 """Supported API. Initialize toolshed.
1264
1265 The toolshed instance is a singleton across all sessions.
1266 The first call creates the instance and all subsequent
1267 calls return the same instance. The toolshed debugging
1268 state is updated at each call.
1269
1270 Parameters
1271 ----------
1272 debug : boolean
1273 If true, debugging messages are sent to standard output.
1274 Default value is false.
1275 other arguments : any
1276 All other arguments are passed to the `Toolshed` initializer.
1277 """
1278 if debug is not None:
1279 global _debug_toolshed
1280 _debug_toolshed = debug
1281 global _toolshed
1282 if _toolshed is None:
1283 _toolshed = Toolshed(*args, **kw)
1284
1285
1286def get_toolshed():
1287 """Supported API. Return current toolshed.
1288
1289 Returns
1290 -------
1291 :py:class:`Toolshed` instance
1292 The toolshed singleton.
1293
1294 The toolshed singleton will be None if py:func:`init` hasn't been called yet.
1295 """
1296 return _toolshed
1297
1298
1299def get_help_directories():
1300 global _default_help_dirs
1301 if _default_help_dirs is None:
1302 import os
1303 from chimerax import app_data_dir, app_dirs
1304 _default_help_dirs = [
1305 os.path.join(app_dirs.user_cache_dir, 'docs'), # for generated files
1306 os.path.join(app_data_dir, 'docs') # for builtin files
1307 ]
1308 hd = _default_help_dirs[:]
1309 if _toolshed is not None:
1310 hd.extend(_toolshed._installed_bundle_info.help_directories)
1311 return hd