Error running second AlphaFold job before first completes

[goddard@…]

The following bug report has been submitted:
Platform:        macOS-12.6-x86_64-i386-64bit
ChimeraX Version: 1.6.dev202211082330 (2022-11-08 23:30:49 UTC)
Description
Running a second AlphaFold prediction while the first is still running causes the Colab job to raise an error.  I thought ChimeraX was supposed to say a job was already running.

Please cite ColabFold: Making protein folding accessible to all. Nature Methods (2022) if you use these predictions.
INFO:__main__:Starting prediction on 2022-12-16 UTC time
INFO:__main__:Installing ColabFold on Google Colab virtual machine.
Installing ColabFold
/usr/local/lib/python3.8/dist-packages/Bio/Data/SCOPData.py:18: BiopythonDeprecationWarning: The 'Bio.Data.SCOPData' module will be deprecated in a future release of Biopython in favor of 'Bio.Data.PDBData.
  warnings.warn(
Using Tesla T4 graphics processor
Downloading alphafold2 weights to .: 100%|██████████| 3.47G/3.47G [00:41<00:00, 89.8MB/s]
2022-12-16 20:23:07,027 Found 5 citations for tools or databases
2022-12-16 20:23:07,040 Unable to initialize backend 'tpu_driver': NOT_FOUND: Unable to find driver in registry given worker: 
2022-12-16 20:23:07,883 Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter Host
2022-12-16 20:23:07,884 Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
2022-12-16 20:23:07,885 Unable to initialize backend 'plugin': xla_extension has no attributes named get_plugin_device_client. Compile TensorFlow with //tensorflow/compiler/xla/python:enable_plugin_device set to true (defaults to false) to enable this.
2022-12-16 20:23:13,175 Query 1/1: af222 (length 222)
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:02 remaining: 00:00]

2022-12-16 20:23:19,230 Running model_1
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-1-d3d257c6040c> in <module>
    411 remove_from_list(seq_list, 'prokaryote')  # Obsolete "prokaryote" flag
    412 
--> 413 run_prediction(seq_list, use_templates = use_templates, energy_minimize = not dont_minimize)


13 frames

<ipython-input-1-d3d257c6040c> in run_prediction(sequences, job_name, msa_mode, pair_mode, use_templates, custom_template_path, energy_minimize, model_type, num_recycles, dpi, install_log)
     77 
     78     from colabfold.batch import run
---> 79     run(
     80       queries=queries,
     81       result_dir='.',

/usr/local/lib/python3.8/dist-packages/colabfold/batch.py in run(queries, result_dir, num_models, num_recycles, model_order, is_complex, num_ensemble, model_type, msa_mode, use_templates, custom_template_path, use_amber, keep_existing_results, rank_by, pair_mode, data_dir, host_url, random_seed, stop_at_score, recompile_padding, recompile_all_models, zip_results, prediction_callback, save_single_representations, save_pair_representations, training, use_gpu_relax, stop_at_score_below, dpi, max_msa)
   1392                 crop_len = math.ceil(sum(query_sequence_len_array) * recompile_padding)
   1393 
-> 1394             outs, model_rank = predict_structure(
   1395                 jobname,
   1396                 result_dir,

/usr/local/lib/python3.8/dist-packages/colabfold/batch.py in predict_structure(prefix, result_dir, feature_dict, is_complex, use_templates, sequences_lengths, crop_len, model_type, model_runner_and_params, do_relax, rank_by, random_seed, stop_at_score, stop_at_score_below, prediction_callback, use_gpu_relax)
    357         # The original alphafold only returns the prediction_result,
    358         # but our patched alphafold also returns a tuple (recycles,tol)
--> 359         prediction_result, recycles = model_runner.predict(
    360             input_features, random_seed=random_seed
    361         )

/usr/local/lib/python3.8/dist-packages/alphafold/model/model.py in predict(self, feat, random_seed)
    192 
    193         sub_feat["prev"] = result["prev"]
--> 194         result, _ = self.apply(self.params, key, sub_feat)
    195         confidences = get_confidence_metrics(result, multimer_mode=self.multimer_mode)
    196         if self.config.model.stop_at_score_ranker == "plddt":

/usr/local/lib/python3.8/dist-packages/jax/_src/traceback_util.py in reraise_with_filtered_traceback(*args, **kwargs)
    160     __tracebackhide__ = True
    161     try:
--> 162       return fun(*args, **kwargs)
    163     except Exception as e:
    164       mode = filtering_mode()

/usr/local/lib/python3.8/dist-packages/jax/_src/api.py in cache_miss(*args, **kwargs)
    618     if isinstance(top_trace, core.EvalTrace) and not (
    619         jax.config.jax_debug_nans or jax.config.jax_debug_infs):
--> 620       execute = dispatch._xla_call_impl_lazy(fun_, *tracers, **params)
    621       out_flat = call_bind_continuation(execute(*args_flat))
    622     else:

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in _xla_call_impl_lazy(***failed resolving arguments***)
    234       raise NotImplementedError('Dynamic shapes do not work with Array.')
    235     arg_specs = [(None, getattr(x, '_device', None)) for x in args]
--> 236   return xla_callable(fun, device, backend, name, donated_invars, keep_unused,
    237                       *arg_specs)
    238 

/usr/local/lib/python3.8/dist-packages/jax/linear_util.py in memoized_fun(fun, *args)
    298       fun.populate_stores(stores)
    299     else:
--> 300       ans = call(fun, *args)
    301       cache[key] = (ans, fun.stores)
    302 

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in _xla_callable_uncached(fun, device, backend, name, donated_invars, keep_unused, *arg_specs)
    357     return computation.compile(_allow_propagation_to_outputs=True).unsafe_call
    358   else:
--> 359     return lower_xla_callable(fun, device, backend, name, donated_invars, False,
    360                               keep_unused, *arg_specs).compile().unsafe_call
    361 

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in compile(self)
    994       else:
    995         assert self._out_type is not None
--> 996         self._executable = XlaCompiledComputation.from_xla_computation(
    997             self.name, self._hlo, self._in_type, self._out_type,
    998             **self.compile_args)

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in from_xla_computation(name, xla_computation, in_type, out_type, nreps, device, backend, tuple_args, in_avals, out_avals, has_unordered_effects, ordered_effects, kept_var_idx, keepalive, host_callbacks)
   1192     with log_elapsed_time(f"Finished XLA compilation of {name} "
   1193                           "in {elapsed_time} sec"):
-> 1194       compiled = compile_or_get_cached(backend, xla_computation, options,
   1195                                        host_callbacks)
   1196     buffer_counts = get_buffer_counts(out_avals, ordered_effects,

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in compile_or_get_cached(backend, computation, compile_options, host_callbacks)
   1075       return compiled
   1076 
-> 1077   return backend_compile(backend, serialized_computation, compile_options,
   1078                          host_callbacks)
   1079 

/usr/local/lib/python3.8/dist-packages/jax/_src/profiler.py in wrapper(*args, **kwargs)
    312   def wrapper(*args, **kwargs):
    313     with TraceAnnotation(name, **decorator_kwargs):
--> 314       return func(*args, **kwargs)
    315     return wrapper
    316   return wrapper

/usr/local/lib/python3.8/dist-packages/jax/_src/dispatch.py in backend_compile(backend, built_c, options, host_callbacks)
   1010   # TODO(sharadmv): remove this fallback when all backends allow `compile`
   1011   # to take in `host_callbacks`
-> 1012   return backend.compile(built_c, compile_options=options)
   1013 
   1014 # TODO(phawkins): update users.

KeyboardInterrupt: 

Log:
Could not find tool "Tabbed Toolbar"  
UCSF ChimeraX version: 1.6.dev202211082330 (2022-11-08)  
© 2016-2022 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> ui tool show AlphaFold

> alphafold predict
> DLPKSVDWRKKGYVTPVKNQKQCGSSWAFSATGALEGQMFRKTGKLVSLSEQNLVDCSRPQGNQGCNGGFMARAFQYVKENGGLDSEESYPYVAVDEICKYRPENSVAQDTGFTVVAPGKEKALMKAVATVGPISVAMDAGHSSFQFYKSGIYFEPDCSSKNLDHGVLVVGYGFEGANSNNSKYWLVKNSWGPEWGSNGYVKIAKDKNNHCGIATAASYPNV

Please cite ColabFold: Making protein folding accessible to all. Nature
Methods (2022) if you use these predictions.  
Running AlphaFold prediction  

> alphafold predict
> MELALRGGYRERPNHQEDPKYLELAHYATSTWSAQQPGKTPFDTVVEVLKVETQTVAGTNYRLTLKVAESTCELTSTYSKDACQPNANAAQRTCTTVIYQNLEGEKSVSSFGCAAA

Running AlphaFold prediction  

Populating font family aliases took 370 ms. Replace uses of missing font
family "Roboto" with one that exists to avoid this cost.  




OpenGL version: 4.1 ATI-4.8.101
OpenGL renderer: AMD Radeon Pro 580 OpenGL Engine
OpenGL vendor: ATI Technologies Inc.

Python: 3.9.11
Locale: UTF-8
Qt version: PyQt6 6.3.1, Qt 6.3.1
Qt runtime version: 6.3.2
Qt platform: cocoa
Hardware:

    Hardware Overview:

      Model Name: iMac
      Model Identifier: iMac18,3
      Processor Name: Quad-Core Intel Core i7
      Processor Speed: 4.2 GHz
      Number of Processors: 1
      Total Number of Cores: 4
      L2 Cache (per Core): 256 KB
      L3 Cache: 8 MB
      Hyper-Threading Technology: Enabled
      Memory: 32 GB
      System Firmware Version: 451.140.1.0.0
      OS Loader Version: 540.120.3~22
      SMC Version (system): 2.41f2

Software:

    System Software Overview:

      System Version: macOS 12.6 (21G115)
      Kernel Version: Darwin 21.6.0
      Time since boot: 86 days 17:14

Graphics/Displays:

    Radeon Pro 580:

      Chipset Model: Radeon Pro 580
      Type: GPU
      Bus: PCIe
      PCIe Lane Width: x16
      VRAM (Total): 8 GB
      Vendor: AMD (0x1002)
      Device ID: 0x67df
      Revision ID: 0x00c0
      ROM Revision: 113-D000AA-931
      VBIOS Version: 113-D0001A1X-025
      EFI Driver Version: 01.00.931
      Metal Family: Supported, Metal GPUFamily macOS 2
      Displays:
        iMac:
          Display Type: Built-In Retina LCD
          Resolution: Retina 5K (5120 x 2880)
          Framebuffer Depth: 30-Bit Color (ARGB2101010)
          Main Display: Yes
          Mirror: Off
          Online: Yes
          Automatically Adjust Brightness: Yes
          Connection Type: Internal


Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    appnope: 0.1.3
    asttokens: 2.0.8
    Babel: 2.10.3
    backcall: 0.2.0
    blockdiag: 3.0.0
    build: 0.8.0
    certifi: 2022.9.24
    cftime: 1.6.2
    charset-normalizer: 2.1.1
    ChimeraX-AddCharge: 1.4
    ChimeraX-AddH: 2.2
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2.2
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.7
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.3
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.43.4
    ChimeraX-AtomicLibrary: 8.0.4
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.3
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.1
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.7.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.3.1
    ChimeraX-ChangeChains: 1.0.2
    ChimeraX-CheckWaters: 1.3
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.2.4
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.2
    ChimeraX-CommandLine: 1.2.4
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.6.dev202211082330
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.3.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.2
    ChimeraX-Dicom: 1.1
    ChimeraX-DistMonitor: 1.3
    ChimeraX-DockPrep: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.2
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-Label: 1.1.7
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.2
    ChimeraX-Map: 1.1.3
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.1
    ChimeraX-MatchMaker: 2.0.9
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.9
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.6
    ChimeraX-ModelPanel: 1.3.6
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.1
    ChimeraX-MouseModes: 1.1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.9.1
    ChimeraX-PDB: 2.6.8
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.0
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0.1
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.7.3
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.1
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.10.1
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.1.3
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.1
    ChimeraX-Topography: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.24.2
    ChimeraX-uniprot: 2.2.1
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.1.3
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.1
    ChimeraX-WebServices: 1.1.0
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.5
    contourpy: 1.0.6
    cxservices: 1.2
    cycler: 0.11.0
    Cython: 0.29.32
    debugpy: 1.6.3
    decorator: 5.1.1
    docutils: 0.19
    entrypoints: 0.4
    executing: 1.1.1
    filelock: 3.7.1
    fonttools: 4.37.4
    funcparserlib: 1.0.0
    grako: 3.16.5
    h5py: 3.7.0
    html2text: 2020.1.16
    idna: 3.4
    ihm: 0.35
    imagecodecs: 2022.9.26
    imagesize: 1.4.1
    importlib-metadata: 5.0.0
    ipykernel: 6.15.3
    ipython: 8.4.0
    ipython-genutils: 0.2.0
    jedi: 0.18.1
    Jinja2: 3.1.2
    jupyter-client: 7.3.4
    jupyter-core: 4.11.2
    kiwisolver: 1.4.4
    line-profiler: 3.5.1
    lxml: 4.9.1
    lz4: 4.0.2
    MarkupSafe: 2.1.1
    matplotlib: 3.6.2
    matplotlib-inline: 0.1.6
    msgpack: 1.0.4
    nest-asyncio: 1.5.6
    netCDF4: 1.6.0
    networkx: 2.8.8
    numexpr: 2.8.3
    numpy: 1.23.5
    openvr: 1.23.701
    opt-einsum: 3.3.0
    packaging: 21.3
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pexpect: 4.8.0
    pickleshare: 0.7.5
    Pillow: 9.3.0
    pip: 22.2.2
    pkginfo: 1.8.3
    prompt-toolkit: 3.0.31
    psutil: 5.9.4
    ptyprocess: 0.7.0
    pure-eval: 0.2.2
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.12.0
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.9
    PyQt6: 6.3.1
    PyQt6-Qt6: 6.3.2
    PyQt6-sip: 13.4.0
    PyQt6-WebEngine: 6.3.1
    PyQt6-WebEngine-Qt6: 6.3.2
    python-dateutil: 2.8.2
    pytz: 2022.5
    pyzmq: 24.0.1
    qtconsole: 5.3.1
    QtPy: 2.2.1
    RandomWords: 0.4.0
    requests: 2.28.1
    scipy: 1.9.3
    setuptools: 65.1.1
    setuptools-scm: 7.0.5
    sfftk-rw: 0.7.2
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    Sphinx: 5.1.1
    sphinx-autodoc-typehints: 1.19.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.0
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    stack-data: 0.5.1
    tables: 3.7.0
    tifffile: 2022.10.10
    tinyarray: 1.2.4
    tomli: 2.0.1
    tornado: 6.2
    traitlets: 5.3.0
    typing-extensions: 4.4.0
    urllib3: 1.26.12
    wcwidth: 0.2.5
    webcolors: 1.12
    wheel: 0.37.1
    wheel-filename: 1.4.1
    zipp: 3.9.0

[Tom Goddard]

The AlphaFoldRun class never sets its _running attribute to True. I'm not sure why. It is easy to set it to true when the run starts and false when the run ends. But how to I handle the case where the run failed and nothing was downloaded? Then _running will never be set to false and the user will not be able to run another AlphaFold job unless they restart ChimeraX. That may be why I took out code that set _running to true. Maybe there is some way to see that colab has stopped executing Python.