Opened 2 years ago

Last modified 23 months ago

#10056 assigned defect

ESMFold: unable to get local issuer certificate

Reported by: chimerax-bug-report@… Owned by: Tom Goddard
Priority: normal Milestone:
Component: Structure Prediction Version:
Keywords: Cc: Greg Couch
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description

The following bug report has been submitted:
Platform:        Windows-10-10.0.19044
ChimeraX Version: 1.6.1 (2023-05-09 17:57:07 UTC)
Description
(Describe the actions that caused this problem to occur here)

Log:
UCSF ChimeraX version: 1.6.1 (2023-05-09)  
© 2016-2023 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  

> help help:quickstart

> open "C:/Users/user/OneDrive - Benemérita Universidad Autónoma de Puebla/01
> M en C Q/00 Primer Semestre Otoño/Bioquímica/Bioinformatica Dr Bertin/1018
> 5xsy template limpio.pdb"

Summary of feedback from opening C:/Users/user/OneDrive - Benemérita
Universidad Autónoma de Puebla/01 M en C Q/00 Primer Semestre
Otoño/Bioquímica/Bioinformatica Dr Bertin/1018 5xsy template limpio.pdb  
---  
warnings | Ignored bad PDB record found on line 1  
REMARK BIOVIA PDB file  
  
Ignored bad PDB record found on line 2  
REMARK Created: 2023-10-18T16:52:53Z  
  
Chain information for 1018 5xsy template limpio.pdb #1  
---  
Chain | Description  
B | No description available  
  
Alignment identifier is 1/B  

> select /B:116-118,159-190,192-194

305 atoms, 307 bonds, 38 residues, 1 model selected  

> select /B:33-35,40-45,55-65,75-77,93-95,106-111,120-131,136-150

490 atoms, 498 bonds, 59 residues, 1 model selected  

> ui tool show "Modeller Comparative"

> help help:user/tools/modeller.html

> ui tool show ESMFold

> esmfold predict
> MGRLLALVVGAALVSSACGGCVEVDSETEAVYGMTFKILCISCKRRSETNAETFTEWTFRQKGTEEFVKILRYENEVLQLEEDERFEGRVVWNGSRGTKDLQDLSIFITNVTYNHSGDYECHVYRLLFFENYEHNTSVVKKIHIEVV
> directory C:\Users\user/Downloads/ChimeraX/ESMFold

Please cite Language models of protein sequences at the scale of evolution
enable accurate structure prediction. if you use these predictions.  
Traceback (most recent call last):  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\urllib3\connectionpool.py", line 703, in urlopen  
httplib_response = self._make_request(  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\urllib3\connectionpool.py", line 386, in _make_request  
self._validate_conn(conn)  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\urllib3\connectionpool.py", line 1042, in _validate_conn  
conn.connect()  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\urllib3\connection.py",
line 419, in connect  
self.sock = ssl_wrap_socket(  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\urllib3\util\ssl_.py",
line 449, in ssl_wrap_socket  
ssl_sock = _ssl_wrap_socket_impl(  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\urllib3\util\ssl_.py",
line 493, in _ssl_wrap_socket_impl  
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)  
File "C:\Program Files\ChimeraX\bin\lib\ssl.py", line 500, in wrap_socket  
return self.sslsocket_class._create(  
File "C:\Program Files\ChimeraX\bin\lib\ssl.py", line 1040, in _create  
self.do_handshake()  
File "C:\Program Files\ChimeraX\bin\lib\ssl.py", line 1309, in do_handshake  
self._sslobj.do_handshake()  
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed: unable to get local issuer certificate (_ssl.c:1129)  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\adapters.py",
line 489, in send  
resp = conn.urlopen(  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\urllib3\connectionpool.py", line 787, in urlopen  
retries = retries.increment(  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\urllib3\util\retry.py",
line 592, in increment  
raise MaxRetryError(_pool, url, error or ResponseError(cause))  
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.esmatlas.com',
port=443): Max retries exceeded with url: /foldSequence/v1/pdb (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1129)')))  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\alphafold\panel.py", line 208, in _predict  
self._run_command('predict', options = ' '.join(options))  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\alphafold\panel.py", line 303, in _run_command  
run(self.session, cmd)  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\core\commands\run.py", line 38, in run  
results = command.run(text, log=log, return_json=return_json)  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\core\commands\cli.py", line 2897, in run  
result = ci.function(session, **kw_args)  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\esmfold\predict.py", line 51, in esmfold_predict  
_start_esmfold_prediction(session, seq, align_to = chain, directory =
directory)  
File "C:\Program Files\ChimeraX\bin\lib\site-
packages\chimerax\esmfold\predict.py", line 75, in _start_esmfold_prediction  
r = requests.post(esmfold_predict_url, data = sequence)  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\api.py", line
115, in post  
return request("post", url, data=data, json=json, **kwargs)  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\api.py", line
59, in request  
return session.request(method=method, url=url, **kwargs)  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\sessions.py",
line 587, in request  
resp = self.send(prep, **send_kwargs)  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\sessions.py",
line 701, in send  
r = adapter.send(request, **kwargs)  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\adapters.py",
line 563, in send  
raise SSLError(e, request=request)  
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.esmatlas.com',
port=443): Max retries exceeded with url: /foldSequence/v1/pdb (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1129)')))  
  
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.esmatlas.com',
port=443): Max retries exceeded with url: /foldSequence/v1/pdb (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1129)')))  
  
File "C:\Program Files\ChimeraX\bin\lib\site-packages\requests\adapters.py",
line 563, in send  
raise SSLError(e, request=request)  
  
See log for complete Python traceback.  
  




OpenGL version: 3.3.14760 Core Profile Forward-Compatible Context 21.2.3 27.20.14535.3005
OpenGL renderer: AMD Radeon R5 Graphics
OpenGL vendor: ATI Technologies Inc.

Python: 3.9.11
Locale: es_MX.cp1252
Qt version: PyQt6 6.4.2, Qt 6.4.2
Qt runtime version: 6.4.3
Qt platform: windows

Manufacturer: HP
Model: HP Pavilion Notebook
OS: Microsoft Windows 10 Home Single Language (Build 19044)
Memory: 16,588,922,880
MaxProcessMemory: 137,438,953,344
CPU: 4 AMD A10-9600P RADEON R5, 10 COMPUTE CORES 4C+6G
OSLanguage: es-MX

Installed Packages:
    alabaster: 0.7.13
    appdirs: 1.4.4
    asttokens: 2.2.1
    Babel: 2.12.1
    backcall: 0.2.0
    beautifulsoup4: 4.11.2
    blockdiag: 3.0.0
    build: 0.10.0
    certifi: 2023.5.7
    cftime: 1.6.2
    charset-normalizer: 3.1.0
    ChimeraX-AddCharge: 1.5.9.1
    ChimeraX-AddH: 2.2.5
    ChimeraX-AlignmentAlgorithms: 2.0.1
    ChimeraX-AlignmentHdrs: 3.3.1
    ChimeraX-AlignmentMatrices: 2.1
    ChimeraX-Alignments: 2.9.3
    ChimeraX-AlphaFold: 1.0
    ChimeraX-AltlocExplorer: 1.0.3
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.1
    ChimeraX-Atomic: 1.43.10
    ChimeraX-AtomicLibrary: 10.0.6
    ChimeraX-AtomSearch: 2.0.1
    ChimeraX-AxesPlanes: 2.3.2
    ChimeraX-BasicActions: 1.1.2
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 2.1.2
    ChimeraX-BondRot: 2.0.1
    ChimeraX-BugReporter: 1.0.1
    ChimeraX-BuildStructure: 2.8
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.2.2
    ChimeraX-ButtonPanel: 1.0.1
    ChimeraX-CageBuilder: 1.0.1
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.3.2
    ChimeraX-ChangeChains: 1.0.2
    ChimeraX-CheckWaters: 1.3.1
    ChimeraX-ChemGroup: 2.0.1
    ChimeraX-Clashes: 2.2.4
    ChimeraX-ColorActions: 1.0.3
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.5.3
    ChimeraX-CommandLine: 1.2.5
    ChimeraX-ConnectStructure: 2.0.1
    ChimeraX-Contacts: 1.0.1
    ChimeraX-Core: 1.6.1
    ChimeraX-CoreFormats: 1.1
    ChimeraX-coulombic: 1.4.2
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0.1
    ChimeraX-DataFormats: 1.2.3
    ChimeraX-Dicom: 1.2
    ChimeraX-DistMonitor: 1.4
    ChimeraX-DockPrep: 1.1.1
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ESMFold: 1.0
    ChimeraX-FileHistory: 1.0.1
    ChimeraX-FunctionKey: 1.0.1
    ChimeraX-Geometry: 1.3
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.1.1
    ChimeraX-Hbonds: 2.4
    ChimeraX-Help: 1.2.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.1
    ChimeraX-ImageFormats: 1.2
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-ItemsInspection: 1.0.1
    ChimeraX-Label: 1.1.7
    ChimeraX-ListInfo: 1.1.1
    ChimeraX-Log: 1.1.5
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.2
    ChimeraX-Map: 1.1.4
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0.1
    ChimeraX-MapFilter: 2.0.1
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.1.1
    ChimeraX-Markers: 1.0.1
    ChimeraX-Mask: 1.0.2
    ChimeraX-MatchMaker: 2.0.12
    ChimeraX-MDcrds: 2.6
    ChimeraX-MedicalToolbar: 1.0.2
    ChimeraX-Meeting: 1.0.1
    ChimeraX-MLP: 1.1.1
    ChimeraX-mmCIF: 2.12
    ChimeraX-MMTF: 2.2
    ChimeraX-Modeller: 1.5.9
    ChimeraX-ModelPanel: 1.3.7
    ChimeraX-ModelSeries: 1.0.1
    ChimeraX-Mol2: 2.0
    ChimeraX-Mole: 1.0
    ChimeraX-Morph: 1.0.2
    ChimeraX-MouseModes: 1.2
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nifti: 1.0
    ChimeraX-NRRD: 1.0
    ChimeraX-Nucleotides: 2.0.3
    ChimeraX-OpenCommand: 1.10.1
    ChimeraX-PDB: 2.7.2
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0.2
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0.1
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.1
    ChimeraX-PubChem: 2.1
    ChimeraX-ReadPbonds: 1.0.1
    ChimeraX-Registration: 1.1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-RenderByAttr: 1.1
    ChimeraX-RenumberResidues: 1.1
    ChimeraX-ResidueFit: 1.0.1
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 3.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.5.1
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0.1
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0.1
    ChimeraX-SelInspector: 1.0
    ChimeraX-SeqView: 2.8.3
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0.1
    ChimeraX-Shortcuts: 1.1.1
    ChimeraX-ShowSequences: 1.0.1
    ChimeraX-SideView: 1.0.1
    ChimeraX-Smiles: 2.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.10.3
    ChimeraX-STL: 1.0.1
    ChimeraX-Storm: 1.0
    ChimeraX-StructMeasure: 1.1.2
    ChimeraX-Struts: 1.0.1
    ChimeraX-Surface: 1.0.1
    ChimeraX-SwapAA: 2.0.1
    ChimeraX-SwapRes: 2.2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.1.2
    ChimeraX-ToolshedUtils: 1.2.1
    ChimeraX-Topography: 1.0
    ChimeraX-Tug: 1.0.1
    ChimeraX-UI: 1.28.4
    ChimeraX-uniprot: 2.2.2
    ChimeraX-UnitCell: 1.0.1
    ChimeraX-ViewDockX: 1.2
    ChimeraX-VIPERdb: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0.1
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0.2
    ChimeraX-WebServices: 1.1.1
    ChimeraX-Zone: 1.0.1
    colorama: 0.4.6
    comm: 0.1.3
    comtypes: 1.1.14
    contourpy: 1.0.7
    cxservices: 1.2.2
    cycler: 0.11.0
    Cython: 0.29.33
    debugpy: 1.6.7
    decorator: 5.1.1
    docutils: 0.19
    executing: 1.2.0
    filelock: 3.9.0
    fonttools: 4.39.3
    funcparserlib: 1.0.1
    grako: 3.16.5
    h5py: 3.8.0
    html2text: 2020.1.16
    idna: 3.4
    ihm: 0.35
    imagecodecs: 2022.9.26
    imagesize: 1.4.1
    importlib-metadata: 6.6.0
    ipykernel: 6.21.1
    ipython: 8.10.0
    ipython-genutils: 0.2.0
    ipywidgets: 8.0.6
    jedi: 0.18.2
    Jinja2: 3.1.2
    jupyter-client: 8.0.2
    jupyter-core: 5.3.0
    jupyterlab-widgets: 3.0.7
    kiwisolver: 1.4.4
    line-profiler: 4.0.2
    lxml: 4.9.2
    lz4: 4.3.2
    MarkupSafe: 2.1.2
    matplotlib: 3.6.3
    matplotlib-inline: 0.1.6
    msgpack: 1.0.4
    nest-asyncio: 1.5.6
    netCDF4: 1.6.2
    networkx: 2.8.8
    nibabel: 5.0.1
    nptyping: 2.5.0
    numexpr: 2.8.4
    numpy: 1.23.5
    openvr: 1.23.701
    packaging: 23.1
    ParmEd: 3.4.3
    parso: 0.8.3
    pep517: 0.13.0
    pickleshare: 0.7.5
    Pillow: 9.3.0
    pip: 23.0
    pkginfo: 1.9.6
    platformdirs: 3.5.0
    prompt-toolkit: 3.0.38
    psutil: 5.9.4
    pure-eval: 0.2.2
    pycollada: 0.7.2
    pydicom: 2.3.0
    Pygments: 2.14.0
    pynrrd: 1.0.0
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 3.0.9
    pyproject-hooks: 1.0.0
    PyQt6-commercial: 6.4.2
    PyQt6-Qt6: 6.4.3
    PyQt6-sip: 13.4.1
    PyQt6-WebEngine-commercial: 6.4.0
    PyQt6-WebEngine-Qt6: 6.4.3
    python-dateutil: 2.8.2
    pytz: 2023.3
    pywin32: 305
    pyzmq: 25.0.2
    qtconsole: 5.4.0
    QtPy: 2.3.1
    RandomWords: 0.4.0
    requests: 2.28.2
    scipy: 1.9.3
    setuptools: 67.4.0
    sfftk-rw: 0.7.3
    six: 1.16.0
    snowballstemmer: 2.2.0
    sortedcontainers: 2.4.0
    soupsieve: 2.4.1
    sphinx: 6.1.3
    sphinx-autodoc-typehints: 1.22
    sphinxcontrib-applehelp: 1.0.4
    sphinxcontrib-blockdiag: 3.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 2.0.1
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.5
    stack-data: 0.6.2
    tables: 3.7.0
    tcia-utils: 1.2.0
    tifffile: 2022.10.10
    tinyarray: 1.2.4
    tomli: 2.0.1
    tornado: 6.3.1
    traitlets: 5.9.0
    typing-extensions: 4.5.0
    tzdata: 2023.3
    urllib3: 1.26.15
    wcwidth: 0.2.6
    webcolors: 1.12
    wheel: 0.38.4
    wheel-filename: 1.4.1
    widgetsnbextension: 4.0.7
    WMI: 1.5.1
    zipp: 3.15.0

Change History (4)

comment:1 by Eric Pettersen, 2 years ago

Cc: Greg Couch added
Component: UnassignedStructure Prediction
Owner: set to Tom Goddard
Platform: all
Project: ChimeraX
Status: newassigned
Summary: ChimeraX bug report submissionESMFold: unable to get local issuer certificate

comment:2 by Tom Goddard, 2 years ago

Meta's ESM fold prediction service appears to be broken, apparently because their SSL certificate (renewed Sept 25, 2023) is misconfigured. I've reported the problem on the FaceBook ESM Github page

https://github.com/facebookresearch/esm/discussions/627

which is listed as the Contact in the ESM about page (https://esmatlas.com/about#api)

It may be that there are further errors because I would not think that the SSL error would interfere with Meta's own web page running a prediction. That page gives useless message "Error: There was an error", while posting to the API using curl gives "Internal server error".

If there is no response to my post and the service is not fixed then it might be wise to remove the ESM fold prediction from ChimeraX 1.7. Or I could leave it in but use the ChimeraX ESM live check to report that the service is down, and discontinue in ChimeraX 1.8 if it never comes back.

The ESM fold prediction web page is broken.  I'd like to know if the fold prediction service is no longer maintained.  I added ESM fold prediction to the widely used ChimeraX molecular visualization program and am wondering if I need to remove it from out software.

  Thanks!

    Tom Goddard
    ChimeraX developer
    University of California, San Francisco

Here are details.  The broken fold prediction web page is

    https://esmatlas.com/resources?action=fold

Pasting any sequence such as 

GENGEIPLEIRATTGAEVDTRAVTAVEMTEGTLGIFRLPEEDYTALENFRYNRVAGENWKPASTVIYVGGTYARLCAYAPYNSVEFKNSSLKTEAGLTMQTYAAEKDMRFAVSGGDEVWKKTPTAN

and pressing the enter key gives the message 

    "Error: There was an error"

Similarly the ESM folding web service as described here https://esmatlas.com/about#api

    https://api.esmatlas.com/foldSequence/v1/pdb/

is broken.  For instance 

$ curl -X POST --data "GENGEIPLEIRATTGAEVDTRAVTAVEMTEGTLGIFRLPEEDYTALENFRYNRVAGENWKPASTVIYVGGTYARLCAYAPYNSVEFKNSSLKTEAGLTMQTYAAEKDMRFAVSGGDEVWKKTPTAN" https://api.esmatlas.com/foldSequence/v1/pdb/

gives error

    {"message": "Internal server error"}

These errors may be due to an SSL certificate error on the Meta server.  An ESM prediction can be run from the ChimeraX molecular visualization program (menu Tools / Structure Prediction / ESMFold) and the error message is

    requests.exceptions.SSLError: HTTPSConnectionPool(host='api.esmatlas.com', port=443): Max retries exceeded with url: /foldSequence/v1/pdb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

Checking the certificate for api.esmatlas.com issued by DigiCert using https://www.digicert.com/help/ reports

   "The server is not sending the required intermediate certificate.
    This server needs to be configured to include DigiCert's intermediate certificate to avoid trust errors in web browsers."

comment:3 by Tom Goddard, 2 years ago

Tom Sercu, a Facebook AI engineer on the protein folding team responded to the github discussion and is going to try to fix the problem.

comment:4 by Tom Goddard, 23 months ago

They fixed the error where the Meta web page that predicts from a sequence was giving "Error: There was an error" several weeks ago. But they don't seem to be interested in fixing the SSL certificate being incorrectly configured. It has been 6 weeks since I filed the ticket to the ESMFold repository and 3 weeks ago the person responsible said "We'll be fixing this at some point of time", which may mean they will never fix it.

For now I think it is ok to leave the ESMFold gui and command in ChimeraX. It can still search the ESM Atlas so that provides some utility.

I set our ESMFold down message on plato that ChimeraX checks for. So ChimeraX reports a clear message that the Meta server is down, possibly forever.

Note: See TracTickets for help on using tickets.