Migrate from Qt 5 to Qt 6

[Tom Goddard]

Qt 6.0 was released in December of 2020 although some modules were not included such as QtWebEngineWidgets which we use for all our html panels (Log, help viewer, basic actions, viewdockx...) and QtMultimedia which we use for the webcam command. Those modules are scheduled for Qt 6.2 which is planned for release in September 2021. We would not be able to move to Qt 6 before that due to the HTML views.

The Qt Company has changed their longterm support release policy. We are using Qt 5.15.2 in ChimeraX 1.2 which is a longterm support release. But the new policy is that only commercial license holders will get Qt 5.15.3 and later patch releases and only they will have access to the source code for 5.15.3. Open source developers must move to Qt 6 in order to get patch releases. So we are stuck at Qt 5.15.2 with no patches unless we buy a commercial license. For 3 developers that is $12,000 per year with no distribution fee.

List of modules included in Qt 6.0: ​https://wiki.qt.io/Checklist_for_Qt_6.0_inclusion

Qt long term support policy blog post: ​https://www.qt.io/blog/qt-offering-changes-2020

[Tom Goddard]

Here is a bit more discussion suggesting we just stick with Qt 5.15.2 until Qt 6 is ready for our use.

From: Tom Goddard
Subject: Re: Qt LTS closed
Date: January 6, 2021 at 6:42:26 PM PST
To: Eric Pettersen
Cc: Chimera Staff , Tristan Croll

Hi Eric,

Summarizing, Qt will no longer provide long-term support releases also not source code except to commercial license holders once there is a new Qt feature release. For us this appears to mean we will not be able to get a Qt 5.15.3 release or source code since 5.15 is a longterm support version and Qt 6.0.0 (the next feature version) is out. I say "appears to" because we get Qt as part of PyQt5 or PySide2. I suspect PySide2 5.15.3 will only be available to commercial license holders of Qt since it is part of Qt. We do not have a commercial license for Qt, only for PyQt5. A commercial license for Qt for desktop development for 3 developers is $12,000 per year, or $25,000 for 3 years.

Do we need Qt 5.15.3? Unfortunately the released Qt 6.0.0 is missing many modules although I have not investigated if it has the ones ChimeraX needs. Another factor is Qt 6 makes several non-backward-compatible changes. I have not looked into whether any of those affect us.

We are currently using Qt 5.15.2. Not getting 5.15.3 would mean we get no more Qt bug fixes until we migrate to Qt 6. That is not great, but not a disaster either.

The reason the Qt company made these changes is of course to get more commercial licenses. For open source developers where ~$4000 per year per desktop developer is too expensive they force you to move to the bleeding edge Qt 6 to promote its development or you are stuck with no patch updates.

If we go back from PySIde2 to PyQt5 I am not sure if RiverBank Computing that makes PyQt5 removes our need for a Qt license, ie. Riverbank pays the license, or whether PyQt5 5.15.3 will only be usable by those who have a separate Qt license -- I could not find info online about this.

I think our position should be to stick with Qt 5.15.2 until we are ready for Qt 6 and it is ready for us. We could also try to include $12,000 per year in future funding requests and get a Qt license for 3 developers. We do get immense value from Qt and supporting their effort makes sense if it is feasible for us to do so.

Tom

[Tom Goddard]

ChimeraX uses these Qt modules that are part of the base Qt 6 distribution

QtWidgets, QtCore, QtGui, QtNetwork (meeting and help viewer only)

and the following that are considered "add-ons" in Qt 6 and are supposed to be available in Qt 6.2 in Sept 2021.

QtWebEngineWidgets, QtWebEngineCore
QtMultimedia (webcam command only)

The base versus add-on modules are described in this Qt 6 blog post about add-ons

​https://www.qt.io/blog/add-on-support-in-qt-6.0-and-beyond

Although the QtMultimedia module is only used by one exotic command (webcam), the QtWebEngine module is used extensively for all our panels that utilize html (log, help browser, basic actions, blast protein...).

[Tom Goddard]

PySide2 which works with Qt 5.15 has been renamed to PySide6 for Qt 6. It appears that development effort is focused on PySide6, with PySide2 not even mentioned on the Qt for Python web page.

If we switch to PySide6 when Qt 6.2 comes out in a year this will probably have to be a major version change for ChimeraX 1 -> 2 if we are using PySide2 or PyQt5 in imports. Possibly a shim "qt" module could provide compatibility although that depends on what non-compatible changes Qt 6 introduces. Need to investigate that.

[Tom Goddard]

I replaced all the PySide2 imports in ChimeraX with imports from a Qt shim module. If Qt 6 and Qt 5 have very high compatibility then we may be able to support PySide6 and PyQt6 through this shim without a major API change. The shim is described in #4120.

[Tom Goddard]

It may be that Qt, PySide and PyQt for macOS arm64 cpus are only made available for Qt 6.2, not Qt 5.15. Currently there are no arm64 builds. Ticket for arm64 ChimeraX port #4663.

[Tom Goddard]

The Python shell (Tools / General / Shell) gives an error because it uses PyPi package qtconsole which uses PyPi qtpy which is a PyQt5 / PySide2 wrapper that does not handle PyQt6. So qtpy reports it can't find any Qt.

ui tool show Shell
Traceback (most recent call last):
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/qtpy/__init__.py", line 210, in <module>
    from PySide import __version__ as PYSIDE_VERSION # analysis:ignore
ModuleNotFoundError: No module named 'PySide'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 554, in start_tool
    ti = api._api_caller.start_tool(api, session, self, tool_info)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/toolshed/__init__.py", line 1263, in start_tool
    return cls._get_func(api, "start_tool")(session, ti.name)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/shell/__init__.py", line 21, in start_tool
    return ShellUI(session, tool_name)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/shell/tool.py", line 39, in __init__
    from qtconsole.inprocess import QtInProcessKernelManager
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/qtconsole/inprocess.py", line 4, in <module>
    from qtpy import QtCore
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/qtpy/__init__.py", line 216, in <module>
    raise PythonQtError('No Qt bindings could be found')
qtpy.PythonQtError: No Qt bindings could be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/ui/gui.py", line 1616, in <lambda>
    run(ses, "ui tool show %s" % (StringArg.unparse(fave))))
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/commands/run.py", line 36, in run
    results = command.run(text, log=log, return_json=return_json)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/commands/cli.py", line 2856, in run
    result = ci.function(session, **kw_args)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/ui/cmd.py", line 219, in ui_tool_show
    bi.start_tool(session, name)
  File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 559, in start_tool
    raise ToolshedError(
chimerax.core.toolshed.ToolshedError: start_tool() failed for tool Shell in bundle ChimeraX-Shell:
No Qt bindings could be found

chimerax.core.toolshed.ToolshedError: start_tool() failed for tool Shell in bundle ChimeraX-Shell:
No Qt bindings could be found

File "/Users/goddard/ucsf/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/core/toolshed/info.py", line 559, in start_tool
raise ToolshedError(

See log for complete Python traceback.

[Eric Pettersen]

Apparently Napari uses qtpy, and qtpy is working on Qt6 support: ​https://github.com/spyder-ide/qtpy/issues/233

[Tom Goddard]

Checked in Qt changes to make gui code work with PyQt 6 and PyQt 5. We are still using PyQt 5. Changes mostly involved changed locations of classes (e.g. QAction is found in QtGui in Qt 6, but in QtWidgets in Qt 5), and enumeration values require specifying the enumeration name in Qt 6 (e.g. Qt.AlignmentFlag.AlignRight in Qt 6 and Qt.AlignRight in Qt 5). Also trackpad event classes and methods changed. To handle these larger changes there is code that checks "using_qt5" or "using_qt6" from the Qt shim module and does different things.

The changes I checked in allow basic atomic structure and map display to work with PyQt 6. But there are no doubt many more enumeration values to fix, as well as class import locations, and other API changes that will require a methodical testing of all ChimeraX tools.

I checked in these partial Qt 6 fixes to allow work on the Mac ARM64 port using Qt 6.2 since Qt 5 is not available for ARM64.

To install PyQt 6 I added prereqs/PyQt/Makefile.pypi_qt6. It can be used by changing the prereqs/PyQt/Makefile METHOD variable from wheel to pypi_qt6. There are also app-uninstall-qt5 and app-uninstall-qt6 targets in Makefile.pypi_qt6. Although both PyQt5 and PyQt6 can be installed at the same time some tools (like the Shell / qtconsole) may directly use PyQt5 instead of our Qt shim so it is best to uninstall PyQt5 otherwise those tools may crash when attempting to use PyQt5 because there is no PyQt5 application running.

[Tom Goddard]

I tried fixing the Python shell with PyQt 6.2.1. It uses qtconsole (current PyPi version 5.1.1). qtconsole uses qtpy (current PyPi version 1.11.2). Currently qtpy does not support PyQt6 by the qtpy github site has version 2.0 in progress that does support PyQt6. Using the qtpy github code (Oct 28, 2021) I modified qtconsole to work with PyQt6. It required changing hundreds of lines of code, almost all to add namespaces for enumeration values (e.g. Qt.AlignRight to Qt.AlignmentFlag.AlignRight). After a few hours the shell was working at least with basic tests. I submitted a qtconsole github issue asking for PyQt6 support and supplied the patch of my changes.

​https://github.com/jupyter/qtconsole/issues/504

Also qtpy needed to have QtCore.Slot defined. I noted this in the qtpy request for PyQt6 support at github

​https://github.com/spyder-ide/qtpy/issues/233#issuecomment-954401130

Error when starting Python shell is ticket #5504.

[Tom Goddard]

I changed the Qt shim module so that with PyQt6 it adds the enumeration values to all classes in QtCore, QtGui, QtWidgets without needing to use the enumeration name. I copied the code from qtpy 2.0 for this. It adds 0.15 seconds to the total time to import those 3 modules, so startup will be 150 milliseconds slower. Previously I only promoted the enumeration value names that our code actually uses. But it was too easy to miss values, and outside developers of plugins may need additional values. So seems best to include all enumeration values.

[Tom Goddard]

Current plan is to try to put a Qt6 daily build on the ChimeraX download web page for testing for a few weeks in addition to the Qt5 daily build to get enough testing with Qt6. Once it is adequately tested we can revert to just one daily build version that uses Qt6 in preparation for ChimeraX 1.4 being released with Qt 6.