chimerax daily logs many errors in nogui mode

[Greg Couch]

On plato, running chimerax --nogui, which is encapsulated in a CentOS 8 container, generates many errors on startup:

Open-command provider in bundle ArtiaX specified unknown data format 'Artiatomi Motivelist'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Generic Particle List'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Dynamo Table'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'RELION STAR file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Coords file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'PEET mod/csv'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'ArtiaX geometric model'; skipping
Open-command provider in bundle SEQCROW specified unknown data format 'Q-Chem output file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Artiatomi Motivelist'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Generic Particle List'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Dynamo Table'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'RELION STAR file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Coords file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'PEET mod/csv'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'ArtiaX geometric model'; skipping
Open-command provider in bundle SEQCROW specified unknown data format 'Q-Chem output file'; skipping
Open-command provider in bundle SEQCROW specified unknown data format 'Q-Chem output file'; skipping
Open-command provider in bundle SEQCROW specified unknown data format 'Q-Chem output file'; skipping
Open-command provider in bundle SEQCROW specified unknown data format 'Q-Chem output file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Artiatomi Motivelist'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Generic Particle List'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Dynamo Table'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'RELION STAR file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'Coords file'; skipping
Open-command provider in bundle ArtiaX specified unknown data format 'PEET mod/csv'; skipping
Traceback (most recent call last):
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 490, in get_module
    m = importlib.import_module(self.package_name)
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/dicom/__init__.py", line 18, in <module>
    from .dicom import DICOM, DICOMMapFormat
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/dicom/dicom.py", line 28, in <module>
    from chimerax.core.decorators import requires_gui
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/core/decorators.py", line 8, in <module>
    import chimerax.ui
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/ui/__init__.py", line 21, in <module>
    from .gui import MainToolWindow, initialize_qt, menu_capitalize
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/ui/gui.py", line 29, in <module>
    from Qt.QtWidgets import QApplication
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/Qt/__init__.py", line 64, in <module>
    from PyQt6.QtCore import PYQT_VERSION_STR as PYQT6_VERSION
ImportError: libQt6Core.so.6: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 364, in initialize
    api = self._get_api(session.logger)
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 509, in _get_api
    m = self.get_module()
  File "/home/socr/a/chimera/chimerax_daily/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 492, in get_module
    raise ToolshedError("Error importing bundle %s's module: %s" % (self.name, str(e)))
chimerax.core.toolshed.ToolshedError: Error importing bundle ChimeraX-Dicom's module: libQt6Core.so.6: cannot open shared object file: No such file or directory

Bundle 'ChimeraX-Dicom' custom initialization failed
UCSF ChimeraX version: 1.5.dev202210040701 (2022-10-04)
© 2016-2022 Regents of the University of California.  All rights reserved.
cmd> 

Qt won't load because the runtime loader will not load the Qt shared library. And that is because of the underlying old kernel (the CentOS 7 kernel) used by the CentOS 8 singularity image.

[Zach Pearson]

I already catch a ModuleNotFoundError there; now an ImportError is caught as well. Should work tomorrow.

[Zach Pearson]

I looked at the daily build on Plato to test the change; it looks like the loader is still unhappy that things from the UI folder are imported in init.py. Will fix.

[Zach Pearson]

​DICOM catches ImportError; you had suggested a fix for the CentOS 7 kernel so I'm not closing this ticket but I am unmilestoning it.

[Greg Couch]

I put in the workaround where Qt can be loaded in a CentOS 8 singularity image on top of CentOS 7.

But it would be better if dicom bundle to not depend on Qt in nogui/nooffscreen mode. Why is dicom special?

To reproduce:

• on plato, create a directory in /tmp and cd to it
• tar xf /usr/local/projects/chimerax/builds/centos-8/ChimeraX-2022.10.11.tar.gz
• singularity shell /usr/local/projects/chimerax/nobackup/centos-8.img
• chimerax-2022.10.11/bin/ChimeraX --nogui and see the dicom traceback
• (cd chimerax-2022.10.11/lib/python3.9/site-packages && rm -rf ChimeraX_Dicom-1.1.dist-info chimerax/dicom)
• chimerax-2022.10.11/bin/ChimeraX --nogui and no traceback
• exit singularity and remove temporary directory

[Zach Pearson]

I'm experimenting with a pattern ( ​https://github.com/RBVI/ChimeraX/blob/develop/src/bundles/dicom/src/dicom.py#L45:L65 ) that accounts for GUI, TUI, library, multisession, and test harness environments for the DICOM code. I think that code stands a chance of becoming the best practice, but it's just an experiment and should be removed for the release.

[Greg Couch]

Yes, the test harness code should be removed. And I don't see how that code can become best practice as is. I don't mind being proved wrong. There should never be an import of {{chimerax.ui.gui}}} except in main. The session should be passed in if needed, even for tests. The UI is session.ui and there's session.ui.is_gui if you care if it's a GUI. An exception would be if they are running in the ChimeraX sandbox and the global session object exists.

The real issue appears to be the logging. Improving the interoperability of Python's logging package with ChimeraX's application infrastructure seems like a better direction to go.

[Zach Pearson]

chimerax.ui.gui.GUI subclasses QApplication which in other Qt codebases can also be imported from Qt.QtWidgets anywhere at any time. It's a singleton, so you can always access its one instance anywhere by calling the QApplication.instance() class method.

The session and the UI have references to each other, so the QApplication singleton's session is already currently the global session. It is "interchangeable" with other sessions in theory, but in practice on restore it pulls data from saved sessions into itself and only by convention needs to be passed around in code.

Logging is an issue, but not the main one. The main issue I'm thinking about is "How do I test code that needs to be passed the application's state object to run?"

[Greg Couch]

Is the code being tested via the ChimeraX application? If so, why can't you use the session from the global session object? If you run chimerax --nogui code.py, that code will get a global session object with the application state.

[Zach Pearson]

I'll have to take some time and reflect on why I didn't do that in the first place. I think it's a kind of chicken and egg problem to have to have a built ChimeraX to test a module. Our codebase is loosely coupled enough for it to be possible for sure, and we check in versions that are stable enough for people to take existing code we know runs and just build off of it live, but something about not using pytest doesn't sit quite right.

I'd like to get to a point where a change in a module kicks off a test for just that module and ChimeraX gets built on success, but that could be a long way away if ever.

[Zach Pearson]

I personally don't like imports in functions because in long (>500 line) files they do an exceptional job of obfuscating that file's dependencies. I kept the experimental try-except block but moved the UI imports underneath it. Let's open a separate ticket for whether or not that's bad practice and/or improving interoperability with Python logging (a major, low-priority, but IMO recommended over the long run thing to do).