Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#7100 closed defect (nonchimerax)

ChimeraX is not starting after the successful installation.

Reported by: akopokov@… Owned by: Greg Couch
Priority: normal Milestone:
Component: Platform Version:
Keywords: Cc: chimera-programmers
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

Hello ChimeraX Team,

I have recently installed ChimeraX  on the Redhat 7.9 system using CLI. The package I have used to install is ucsf-chimerax-1.3-1.el7.x86_64.rpm and I have downloaded this Package using my commercial license. When I am trying to start ChimeraX using CLI, it is not starting and giving me errors. I have a two python versions(2.7 and 3.6). I have pointed to use python 3.6 during the ChimeraX installation. My main goal is to start ChimeraX GUI using Command Line, once I get the GUI I will need to install ISOLDE.
I would be glad if you could help me with this issue.
I am attaching my screenshots from my Mobaxterm client and X2go client sessions.


Best Regards,

Altyn


X2go_session_error.PNG

ChimeraX_Mobaxterm_Session_Error.PNG

Attachments (2)

X2go_session_error.PNG (303.1 KB ) - added by akopokov@… 3 years ago.
Added by email2trac
ChimeraX_Mobaxterm_Session_Error.PNG (66.9 KB ) - added by akopokov@… 3 years ago.
Added by email2trac

Download all attachments as: .zip

Change History (15)

by akopokov@…, 3 years ago

Attachment: X2go_session_error.PNG added

Added by email2trac

by akopokov@…, 3 years ago

Added by email2trac

comment:1 by pett, 3 years ago

Cc: chimera-programmers added
Component: UnassignedPlatform
Owner: set to Greg Couch
Platform: all
Project: ChimeraX
Status: newassigned

comment:2 by pett, 3 years ago

Reported by Altynbek Kopokov

comment:3 by Zach Pearson, 3 years ago

In toolshed_utils and bundle_builder we have python_requires=">=3.7" so I'd assume this holds for the rest of ChimeraX. Do you have the ability to use Python 3.7 or higher on your system?

in reply to:  6 comment:4 by goddard@…, 3 years ago

ChimeraX does not use the system Python in any way.  It uses Python 3.9 that is included with the ChimeraX distribution.  So the installed Python on the system is not the problem.

As the error messages indicate the problem is that Qt cannot find a working OpenGL.

in reply to:  7 comment:5 by akopokov@…, 3 years ago

Hello Chimera Team,

I ran chimerax in debug mode. Please see the output below:

-bash-4.2$ export QT_DEBUG_PLUGINS=1
-bash-4.2$ chimerax
QFactoryLoader::QFactoryLoader() checking directory path "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms" ...
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqeglfs.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqeglfs.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "eglfs"
        ]
    },
    "archreq": 0,
    "className": "QEglFSIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("eglfs")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqlinuxfb.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqlinuxfb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "linuxfb"
        ]
    },
    "archreq": 0,
    "className": "QLinuxFbIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("linuxfb")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqminimal.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqminimal.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "minimal"
        ]
    },
    "archreq": 0,
    "className": "QMinimalIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("minimal")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqminimalegl.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqminimalegl.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "minimalegl"
        ]
    },
    "archreq": 0,
    "className": "QMinimalEglIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("minimalegl")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqoffscreen.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqoffscreen.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "offscreen"
        ]
    },
    "archreq": 0,
    "className": "QOffscreenIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("offscreen")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqvnc.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqvnc.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "vnc"
        ]
    },
    "archreq": 0,
    "className": "QVncIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("vnc")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-egl.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-egl.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "wayland-egl"
        ]
    },
    "archreq": 0,
    "className": "QWaylandEglPlatformIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("wayland-egl")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-generic.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-generic.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "wayland"
        ]
    },
    "archreq": 0,
    "className": "QWaylandIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("wayland")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-xcomposite-egl.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-xcomposite-egl.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "wayland-xcomposite-egl"
        ]
    },
    "archreq": 0,
    "className": "QWaylandXCompositeEglPlatformIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("wayland-xcomposite-egl")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-xcomposite-glx.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqwayland-xcomposite-glx.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "wayland-xcomposite-glx"
        ]
    },
    "archreq": 0,
    "className": "QWaylandXCompositeGlxPlatformIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("wayland-xcomposite-glx")
QFactoryLoader::QFactoryLoader() looking at "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so"
Found metadata in lib /usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "archreq": 0,
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 331520
}


Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/usr/libexec/UCSF-ChimeraX/bin/platforms" ...
loaded library "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/PyQt5/Qt/plugins/platforms/libqxcb.so"
qt.qpa.xcb: could not connect to display
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, xcb.

Fatal Python error: Aborted

Current thread 0x00007fa8d16be740 (most recent call first):
  File "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/chimerax/ui/gui.py", line 159 in __init__
  File "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/ChimeraX_main.py", line 577 in init
  File "/usr/libexec/UCSF-ChimeraX/lib/python3.9/site-packages/ChimeraX_main.py", line 1018 in <module>
  File "/usr/libexec/UCSF-ChimeraX/lib/python3.9/runpy.py", line 87 in _run_code
  File "/usr/libexec/UCSF-ChimeraX/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Aborted (core dumped)
-bash-4.2$

-----Original Message-----
From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu> 
Sent: Wednesday, June 15, 2022 11:18 AM
To: gregc@cgl.ucsf.edu; Kopokov, Altynbek <akopokov@amgen.com>
Cc: chimera-programmers@cgl.ucsf.edu
Subject: Re: [ChimeraX] #7100: ChimeraX is not starting after the successful installation.

EXTERNAL: Use caution with unknown senders


#7100: ChimeraX is not starting after the successful installation.
---------------------------------+------------------------
          Reporter:  akopokov@...  |      Owner:  Greg Couch
              Type:  defect      |     Status:  assigned
          Priority:  normal      |  Milestone:
         Component:  Platform    |    Version:
        Resolution:              |   Keywords:
        Blocked By:              |   Blocking:
Notify when closed:              |   Platform:  all
           Project:  ChimeraX    |
---------------------------------+------------------------

Comment (by goddard@...):

 {{{
 ChimeraX does not use the system Python in any way.  It uses Python 3.9  that is included with the ChimeraX distribution.  So the installed Python  on the system is not the problem.

 As the error messages indicate the problem is that Qt cannot find a  working OpenGL.

 }}}

--
Ticket URL: <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rbvi.ucsf.edu%2Ftrac%2FChimeraX%2Fticket%2F7100%23comment%3A4&amp;data=05%7C01%7Cakopokov%40amgen.com%7Cccf1abed3d424b5b406208da4efb6da4%7C4b4266a6136841afad5a59eb634f7ad8%7C0%7C0%7C637909139025906532%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=rmo1ru%2FmvNzdaByp7OnIBA22LuLwOqm51bvih57kXAk%3D&amp;reserved=0>
ChimeraX <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rbvi.ucsf.edu%2Fchimerax%2F&amp;data=05%7C01%7Cakopokov%40amgen.com%7Cccf1abed3d424b5b406208da4efb6da4%7C4b4266a6136841afad5a59eb634f7ad8%7C0%7C0%7C637909139025906532%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=v8yent4%2BPm0ieu%2BXc9m0rTqAmOvHCmNoFj3VQfPYs5A%3D&amp;reserved=0>
ChimeraX Issue Tracker

comment:6 by Tom Goddard, 3 years ago

Your debug output for the xcb Qt plugin that allows Qt to work on the X11 window system says

"qt.qpa.xcb: could not connect to display"

Is your OS running Wayland? Qt has problems on Wayland and needs the X windows. Does any X windows application start (e.g. xterm) on your system?

This ChimeraX ticket discusses how ChimeraX can be used on Wayland

https://www.rbvi.ucsf.edu/trac/ChimeraX/ticket/6827

in reply to:  9 ; comment:7 by akopokov@…, 3 years ago

Hi Chimera Team,

I am remotely connecting to the system using Mobaxterm. The above debug mode output was from the Mobaxterm session. I have tried to run chimerax by remotely connecting x2go client and it is giving me slightly different error.
I don't  think system has any x windows applications except x2go.

-----Original Message-----
From: ChimeraX <ChimeraX-bugs-admin@cgl.ucsf.edu> 
Sent: Thursday, June 16, 2022 10:37 AM
Cc: Kopokov, Altynbek <akopokov@amgen.com>; chimera-programmers@cgl.ucsf.edu; gregc@cgl.ucsf.edu
Subject: Re: [ChimeraX] #7100: ChimeraX is not starting after the successful installation.

EXTERNAL: Use caution with unknown senders


#7100: ChimeraX is not starting after the successful installation.
---------------------------------+------------------------
          Reporter:  akopokov@...  |      Owner:  Greg Couch
              Type:  defect      |     Status:  assigned
          Priority:  normal      |  Milestone:
         Component:  Platform    |    Version:
        Resolution:              |   Keywords:
        Blocked By:              |   Blocking:
Notify when closed:              |   Platform:  all
           Project:  ChimeraX    |
---------------------------------+------------------------

Comment (by Tom Goddard):

 Your debug output for the xcb Qt plugin that allows Qt to work on the X11  window system says

 "qt.qpa.xcb: could not connect to display"

 Is your OS running Wayland?  Qt has problems on Wayland and needs the X  windows. Does any X windows application start (e.g. xterm) on your system?

 This ChimeraX ticket discusses how ChimeraX can be used on Wayland

 https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rbvi.ucsf.edu%2Ftrac%2FChimeraX%2Fticket%2F6827&amp;data=05%7C01%7Cakopokov%40amgen.com%7C88a72d978d264bb906c708da4fbedfa2%7C4b4266a6136841afad5a59eb634f7ad8%7C0%7C0%7C637909978468275753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=mTk5FMP4t%2BcQMq3tTljPWph4sL%2FiA5ga%2FbiCr8JJkaI%3D&amp;reserved=0

--
Ticket URL: <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rbvi.ucsf.edu%2Ftrac%2FChimeraX%2Fticket%2F7100%23comment%3A6&amp;data=05%7C01%7Cakopokov%40amgen.com%7C88a72d978d264bb906c708da4fbedfa2%7C4b4266a6136841afad5a59eb634f7ad8%7C0%7C0%7C637909978468275753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Gtfg6HPP1xfW0aNKIoiyvenY0Iy5EU3yJbQxHWEGybQ%3D&amp;reserved=0>
ChimeraX <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rbvi.ucsf.edu%2Fchimerax%2F&amp;data=05%7C01%7Cakopokov%40amgen.com%7C88a72d978d264bb906c708da4fbedfa2%7C4b4266a6136841afad5a59eb634f7ad8%7C0%7C0%7C637909978468275753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=3MrPfwgXZ5TB9Jiz6ePIPe8%2BQmjAAMpN8T3x7L5VDr4%3D&amp;reserved=0>
ChimeraX Issue Tracker

comment:8 by Tom Goddard, 3 years ago

ChimeraX is for interactive desktop visualization. Remote display from a server is possible but we don't support it as described here

https://www.cgl.ucsf.edu/chimerax/system_requirements.html

That said, you might be able to get it to work. Remote display of X windows and especially OpenGL is a minefield of possible problems that have nothing to do with ChimeraX. Your error indicates that the Qt window toolkit cannot even make an X11 connection so that will be the first problem to debug. That is usually pretty easy using whatever tests for instance that mobaxterm offers. Then the really difficult part becomes remote display of OpenGL 3D graphics. That may be impossible if your remote display technology does not support OpenGL. Or it can just be difficult to setup requiring special compatible graphics drivers on client and server.

There is little we can do to help you with remote display since it has nothing to do with the ChimeraX application which is using standard X11 and OpenGL on Linux.

in reply to:  11 ; comment:9 by Greg Couch, 3 years ago

So ChimeraX on Wayland uses XWayland, which is the supported way for X applications to run on Wayland.   If your terminal window can start graphics applications, it should have the WAYLAND_DISPLAY (for Wayland) or DISPLAY (for X11) display environment variable set.  Double check that you can startup a remote terminal.

But running ChimeraX remotely is problematic. 3D graphics often does not work remotely.  You can workaround the problem with VirtualGL.  We have had reports of bugs with VirtualGL 3, so try version 2.5 or 2.6.

Good luck,

Greg

Last edited 3 years ago by Greg Couch (previous) (diff)

in reply to:  12 comment:10 by akopokov@…, 3 years ago

Hello All,

  1. Companies everywhere are moving to an "officeless" cafeteria-like work environment for researchers as well as other staff. This is eliminating the option of having a dedicated Xwindows linux system situated directly in front of the researchers.
  1. A remote desktop way of running ChimeraX is needed for researchers at our company and I expect that other companies are facing this requirement too.
  1. We would appreciate your insights as what are options are to make ChimeraX work via a remote GUI session to a researcher using an X GUI client on their laptop (PC or Mac). If we can have ChimeraX run on an AWS system that would be ideal but we could also run it on a system in an on-premesis location if need be.

PS. Any of your users have had success running ChimeraX on AWS with niceDCV (niceDCV is the remote GUI app).

Thanks.

Last edited 3 years ago by Greg Couch (previous) (diff)

comment:11 by Tom Goddard, 3 years ago

You can use a remote desktop display method where ChimeraX runs and renders on the server and just the rendered desktop image is transferred to the client. VNC is an example. Mac and Windows have their own remote desktop protocols that work fine too. But all of these are rather poor for interactive 3D graphics because of latency. Our software ChimeraX is more akin to a video game, only good if latency is low. If you rotate a molecular structure with the mouse you need it to instantly see it move so you can establish the correct viewing angle based on what you see. Also high framerate is needed so a high bandwidth connection. So almost all academic users working from home will use their own Mac, Windows or Linux desktop to use ChimeraX, and not use remote display.

I understand your company has different needs. And there are of course higher performance solutions than desktop sharing such as VirtualGL, niceDCV, .... that try to render the 3D graphics on the client machine. Those can reduce the network bandwidth needed since sending OpenGL rendering commands is much less data than sending a 30 frame/second desktop image stream. But they don't improve latency. So it is still a poor solution for ChimeraX interactive graphics. Those client side rendering solutions are also very fragile and buggy. But people have successfully used them with ChimeraX. We have little or no direct experience setting those poor solutions up and debugging them. It is not very likely we can set you up with others who have struggled through those solutions because in most cases we do not know which users gave up setting up remote display (probably 90% of users) and which succeeded.

This ticket is assigned to Greg Couch and he knows most about this. VirtualGL has been main suggestion.

comment:12 by Greg Couch, 3 years ago

Resolution: nonchimerax
Status: assignedclosed

We have no experience with niceDCV. It looks promising. And for x2go, use VirtualGL. An alternative, again using VirtualGL, is nomachine. I believe nomachine comes with VirtualGL integration, whereas with x2go, you need to configure it yourself. You also need to pay for nomachine. And be aware, that to use 3D graphics with VirtualGL, the remote server needs to have a GPU card in it along with the correct graphics driver. Or, you can use Tom's last message for a business justification for having your own Linux workstation. Latency is a killer.

in reply to:  15 comment:13 by Greg Couch, 3 years ago

You need to debug why it can't connect to the display.  What is the 
DISPLAY environment variable set to?  Can you run the glxinfo program 
from the gfx-utils package?  glxinfo displays the OpenGL capabilities of 
the system.  Is this running on a remote system? If so, did you use "ssh 
-XY" to connect to it, or just "ssh"?  In the latter case, the DISPLAY 
environment variable is not set. Anyway, fix your connection problem first.

     -- Greg

On 6/16/2022 10:01 AM, Kopokov, Altynbek wrote:
Note: See TracTickets for help on using tickets.