VR: socket error: Unable to write

[chimerax-bug-report@…]

The following bug report has been submitted:
Platform:        Windows-10-10.0.18362
ChimeraX Version: 1.2.dev202103272340 (2021-03-27 23:40:01 UTC)
Description
(Describe the actions that caused this problem to occur here)

Log:
UCSF ChimeraX version: 1.2.dev202103272340 (2021-03-27)  
© 2016-2020 Regents of the University of California. All rights reserved.  
How to cite UCSF ChimeraX  
Unable to load numpy_formathandler accelerator from OpenGL_accelerate  

> ui tool show Meeting

> meeting tg name mfs

Joining meeting"tg" at chimeraxmeeting.net port 52200  

> vr true

started SteamVR rendering  
Socket error for message stream to 13.56.160.228 port 52200: Unable to write  
Disconnected from 13.56.160.228 port 52200  

> meeting close




OpenGL version: 3.3.0 NVIDIA 456.71
OpenGL renderer: GeForce GTX 1080/PCIe/SSE2
OpenGL vendor: NVIDIA Corporation
Manufacturer: ASUS
Model: All Series
OS: Microsoft Windows 10 Pro (Build 18363)
Memory: 17,086,476,288
MaxProcessMemory: 137,438,953,344
CPU: 16 Intel(R) Core(TM) i7-6900K CPU @ 3.20GHz
OSLanguage: en-US
Locale: ('en_US', 'cp1252')
PyQt5 5.15.2, Qt 5.15.2
Installed Packages:
    alabaster: 0.7.12
    appdirs: 1.4.4
    Babel: 2.9.0
    backcall: 0.2.0
    blockdiag: 2.0.1
    certifi: 2020.12.5
    cftime: 1.4.1
    chardet: 3.0.4
    ChimeraX-AddCharge: 1.0
    ChimeraX-AddH: 2.1.4
    ChimeraX-AlignmentAlgorithms: 2.0
    ChimeraX-AlignmentHdrs: 3.2
    ChimeraX-AlignmentMatrices: 2.0
    ChimeraX-Alignments: 2.1
    ChimeraX-AmberInfo: 1.0
    ChimeraX-Arrays: 1.0
    ChimeraX-Atomic: 1.12
    ChimeraX-AtomicLibrary: 3.0
    ChimeraX-AtomSearch: 2.0
    ChimeraX-AtomSearchLibrary: 1.0
    ChimeraX-AxesPlanes: 2.0
    ChimeraX-BasicActions: 1.1
    ChimeraX-BILD: 1.0
    ChimeraX-BlastProtein: 1.1
    ChimeraX-BondRot: 2.0
    ChimeraX-BugReporter: 1.0
    ChimeraX-BuildStructure: 2.5.1
    ChimeraX-Bumps: 1.0
    ChimeraX-BundleBuilder: 1.1
    ChimeraX-ButtonPanel: 1.0
    ChimeraX-CageBuilder: 1.0
    ChimeraX-CellPack: 1.0
    ChimeraX-Centroids: 1.1
    ChimeraX-ChemGroup: 2.0
    ChimeraX-Clashes: 2.1
    ChimeraX-ColorActions: 1.0
    ChimeraX-ColorGlobe: 1.0
    ChimeraX-ColorKey: 1.2.1
    ChimeraX-CommandLine: 1.1.3
    ChimeraX-ConnectStructure: 2.0
    ChimeraX-Contacts: 1.0
    ChimeraX-Core: 1.2.dev202103272340
    ChimeraX-CoreFormats: 1.0
    ChimeraX-coulombic: 1.1
    ChimeraX-Crosslinks: 1.0
    ChimeraX-Crystal: 1.0
    ChimeraX-CrystalContacts: 1.0
    ChimeraX-DataFormats: 1.1
    ChimeraX-Dicom: 1.0
    ChimeraX-DistMonitor: 1.1
    ChimeraX-DistUI: 1.0
    ChimeraX-Dssp: 2.0
    ChimeraX-EMDB-SFF: 1.0
    ChimeraX-ExperimentalCommands: 1.0
    ChimeraX-FileHistory: 1.0
    ChimeraX-FunctionKey: 1.0
    ChimeraX-Geometry: 1.1
    ChimeraX-gltf: 1.0
    ChimeraX-Graphics: 1.0
    ChimeraX-Hbonds: 2.1
    ChimeraX-Help: 1.1
    ChimeraX-HKCage: 1.3
    ChimeraX-IHM: 1.0
    ChimeraX-ImageFormats: 1.1
    ChimeraX-IMOD: 1.0
    ChimeraX-IO: 1.0.1
    ChimeraX-Label: 1.0
    ChimeraX-ListInfo: 1.1
    ChimeraX-Log: 1.1.2
    ChimeraX-LookingGlass: 1.1
    ChimeraX-Maestro: 1.8.1
    ChimeraX-Map: 1.0.2
    ChimeraX-MapData: 2.0
    ChimeraX-MapEraser: 1.0
    ChimeraX-MapFilter: 2.0
    ChimeraX-MapFit: 2.0
    ChimeraX-MapSeries: 2.0
    ChimeraX-Markers: 1.0
    ChimeraX-Mask: 1.0
    ChimeraX-MatchMaker: 1.1
    ChimeraX-MDcrds: 2.2
    ChimeraX-MedicalToolbar: 1.0.1
    ChimeraX-Meeting: 1.0
    ChimeraX-MLP: 1.1
    ChimeraX-mmCIF: 2.3
    ChimeraX-MMTF: 2.1
    ChimeraX-Modeller: 1.0
    ChimeraX-ModelPanel: 1.0.1
    ChimeraX-ModelSeries: 1.0
    ChimeraX-Mol2: 2.0
    ChimeraX-Morph: 1.0
    ChimeraX-MouseModes: 1.1
    ChimeraX-Movie: 1.0
    ChimeraX-Neuron: 1.0
    ChimeraX-Nucleotides: 2.0
    ChimeraX-OpenCommand: 1.5
    ChimeraX-PDB: 2.3.1
    ChimeraX-PDBBio: 1.0
    ChimeraX-PDBLibrary: 1.0
    ChimeraX-PDBMatrices: 1.0
    ChimeraX-PickBlobs: 1.0
    ChimeraX-Positions: 1.0
    ChimeraX-PresetMgr: 1.0.1
    ChimeraX-PubChem: 2.0.1
    ChimeraX-ReadPbonds: 1.0
    ChimeraX-Registration: 1.1
    ChimeraX-RemoteControl: 1.0
    ChimeraX-ResidueFit: 1.0
    ChimeraX-RestServer: 1.1
    ChimeraX-RNALayout: 1.0
    ChimeraX-RotamerLibMgr: 2.0
    ChimeraX-RotamerLibsDunbrack: 2.0
    ChimeraX-RotamerLibsDynameomics: 2.0
    ChimeraX-RotamerLibsRichardson: 2.0
    ChimeraX-SaveCommand: 1.4
    ChimeraX-SchemeMgr: 1.0
    ChimeraX-SDF: 2.0
    ChimeraX-Segger: 1.0
    ChimeraX-Segment: 1.0
    ChimeraX-SeqView: 2.3
    ChimeraX-Shape: 1.0.1
    ChimeraX-Shell: 1.0
    ChimeraX-Shortcuts: 1.0
    ChimeraX-ShowAttr: 1.0
    ChimeraX-ShowSequences: 1.0
    ChimeraX-SideView: 1.0
    ChimeraX-Smiles: 2.0.1
    ChimeraX-SmoothLines: 1.0
    ChimeraX-SpaceNavigator: 1.0
    ChimeraX-StdCommands: 1.3
    ChimeraX-STL: 1.0
    ChimeraX-Storm: 1.0
    ChimeraX-Struts: 1.0
    ChimeraX-Surface: 1.0
    ChimeraX-SwapAA: 2.0
    ChimeraX-SwapRes: 2.1
    ChimeraX-TapeMeasure: 1.0
    ChimeraX-Test: 1.0
    ChimeraX-Toolbar: 1.0.1
    ChimeraX-ToolshedUtils: 1.1
    ChimeraX-Tug: 1.0
    ChimeraX-UI: 1.7.1
    ChimeraX-uniprot: 2.1
    ChimeraX-UnitCell: 1.0
    ChimeraX-ViewDockX: 1.0
    ChimeraX-Vive: 1.1
    ChimeraX-VolumeMenu: 1.0
    ChimeraX-VTK: 1.0
    ChimeraX-WavefrontOBJ: 1.0
    ChimeraX-WebCam: 1.0
    ChimeraX-WebServices: 1.0
    ChimeraX-Zone: 1.0
    colorama: 0.4.3
    comtypes: 1.1.7
    cxservices: 1.0
    cycler: 0.10.0
    Cython: 0.29.21
    decorator: 4.4.2
    distlib: 0.3.1
    docutils: 0.16
    filelock: 3.0.12
    funcparserlib: 0.3.6
    grako: 3.16.5
    h5py: 2.10.0
    html2text: 2020.1.16
    idna: 2.10
    ihm: 0.17
    imagecodecs: 2020.5.30
    imagesize: 1.2.0
    ipykernel: 5.3.4
    ipython: 7.18.1
    ipython-genutils: 0.2.0
    jedi: 0.17.2
    Jinja2: 2.11.2
    jupyter-client: 6.1.7
    jupyter-core: 4.7.1
    kiwisolver: 1.3.1
    line-profiler: 2.1.2
    lxml: 4.6.2
    lz4: 3.1.0
    MarkupSafe: 1.1.1
    matplotlib: 3.3.2
    msgpack: 1.0.0
    netCDF4: 1.5.4
    networkx: 2.5
    numexpr: 2.7.3
    numpy: 1.19.2
    numpydoc: 1.1.0
    openvr: 1.14.1501
    packaging: 20.9
    ParmEd: 3.2.0
    parso: 0.7.1
    pickleshare: 0.7.5
    Pillow: 7.2.0
    pip: 21.0.1
    pkginfo: 1.5.0.1
    prompt-toolkit: 3.0.18
    psutil: 5.7.2
    pycollada: 0.7.1
    pydicom: 2.0.0
    Pygments: 2.7.1
    PyOpenGL: 3.1.5
    PyOpenGL-accelerate: 3.1.5
    pyparsing: 2.4.7
    PyQt5-commercial: 5.15.2
    PyQt5-sip: 12.8.1
    PyQtWebEngine-commercial: 5.15.2
    python-dateutil: 2.8.1
    pytz: 2021.1
    pywin32: 228
    pyzmq: 22.0.3
    qtconsole: 4.7.7
    QtPy: 1.9.0
    RandomWords: 0.3.0
    requests: 2.24.0
    scipy: 1.5.2
    setuptools: 50.3.2
    sfftk-rw: 0.6.7.dev1
    six: 1.15.0
    snowballstemmer: 2.1.0
    sortedcontainers: 2.2.2
    Sphinx: 3.2.1
    sphinxcontrib-applehelp: 1.0.2
    sphinxcontrib-blockdiag: 2.0.0
    sphinxcontrib-devhelp: 1.0.2
    sphinxcontrib-htmlhelp: 1.0.3
    sphinxcontrib-jsmath: 1.0.1
    sphinxcontrib-qthelp: 1.0.3
    sphinxcontrib-serializinghtml: 1.1.4
    suds-jurko: 0.6
    tables: 3.6.1
    tifffile: 2020.9.3
    tinyarray: 1.2.3
    tornado: 6.1
    traitlets: 5.0.5
    urllib3: 1.25.11
    wcwidth: 0.2.5
    webcolors: 1.11.1
    wheel: 0.36.0
    wheel-filename: 1.3.0
    WMI: 1.5.1

[pett]

Mail.app's spidey sense says this is from Michael Schmid

[Tom Goddard]

I created vr meeting from home with apoferritin.cxs 15 Mbyte session. Mike Schmid tried to join from Stanford. My computer shows he connects. Mike reports seeing Waiting for Session Data message then he gets this socket write error.

[Tom Goddard]

I connected from my VR laptop to this same session and got no errors. So it appears this is something about the network from SLAC. Mike was at SLAC, not Stanford University.

Restarting and retrying creating the meeting and joining with the same 15 Mbyte session (map + atomic model) gave the same socket write error for Mike, ticket #4437.

A subsequent test where Mike started the meeting and I joined with the same session worked without issues.

Another subsequent test where I created the meeting but opened only a tiny session with just an ATP structure, Mike joined and it appeared to be hung without displaying molecule, then he turned on VR and everything worked, the two-person VR meeting with a very small session all worked. We then tested with just a small map (emdb 1080) and also just with a small atomic model (1a0m) with me creating the meeting and Mike joining and all failed with socket write error.

[Tom Goddard]

The Windows Qt code issues an "Unable to write" error for a QTcpSocket in the following code

​https://code.qt.io/cgit/qt/qtbase.git/tree/src/network/socket/qnativesocketengine_win.cpp

where it is apparent the actual error is WSAECONNRESET or WSAECONNABORTED. As described in microsoft docs

​https://docs.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2

the first error says the connection was closed by the remote host,

"Connection reset by peer.
An existing connection was forcibly closed by the remote host. This normally results if the peer application on the remote host is suddenly stopped, the host is rebooted, the host or remote network interface is disabled, or the remote host uses a hard close (see setsockopt for more information on the SO_LINGER option on the remote socket). This error may also result if a connection was broken due to keep-alive activity detecting a failure while one or more operations are in progress. Operations that were in progress fail with WSAENETRESET. Subsequent operations fail with WSAECONNRESET."

and the second is the connection being aborted by the local host:

"An established connection was aborted by the software in your host computer, possibly due to a data transmission time-out or protocol error."

[Tom Goddard]

It seems like that the connection was closed either by network firewalls, or because of a buffer overflow. QTcpSocket is supposed to handle unlimited buffering so that seems unlikely. So it seems like a SLAC firewall issue. But paradoxically we were able to make the connection with a tiny session (single ATP molecule). So why would the firewall close the connection for larger data transmissions to an AWS virtual machine? I would expect it to close all connections if it didn't like the random port numbers being used.

[Tom Goddard]

This SLAC networking 2016 info says "However, some developers who write programs which access resources on the Internet via non-standard protocols, or those who access SLAC resources using non-standard applications from external locations, may need to get additional applications or ports onto the approved list."

​https://portal.slac.stanford.edu/sites/ard_public/facet/faq/Lists/Computers/DispForm.aspx?ID=15&ContentTypeId=0x010024236BFB3C31F34D9B7DC9593B62DBB0

[Tom Goddard]

Email to Mike Schmid to help him if he wants to discuss the problem with SLAC network IT people.

Begin forwarded message:

From: Tom Goddard
Subject: ChimeraX VR meetings and SLAC firewall
Date: April 1, 2021 at 8:40:13 PM PDT
To: "Schmid, Michael F."

Hi Mike,

I think yesterday's problem where you could not join a ChimeraX VR meeting that I had started is caused by the SLAC firewall. Information on the SLAC firewall online says

"However, some developers who write programs which access resources on the Internet via non-standard protocols, or those who access SLAC resources using non-standard applications from external locations, may need to get additional applications or ports onto the approved list."

That is from 2016 at this web site, but probably security is even more locked down now.

​https://portal.slac.stanford.edu/sites/ard_public/facet/faq/Lists/Computers/DispForm.aspx?ID=15&ContentTypeId=0x010024236BFB3C31F34D9B7DC9593B62DBB0

If you want to ask the SLAC network people, the error was

Socket error for message stream to 13.56.160.228 port 52200: Unable to write
Disconnected from 13.56.160.228 port 52200
This gives the IP address and port and ChimeraX was using a custom Message Pack transmission protocol on a TCP socket. The IP address is an Amazon Web Services virtual machine with host name chimeraxmeeting.net. Here is the ChimeraX bug report with more details

​https://plato.cgl.ucsf.edu/trac/ChimeraX/ticket/4436

It is very likely a hopeless battle to get them to allow this network connection. Computer security will not have succeeded until we are prohibited from turning on the computers. A dozen other firewalled universities, the NIH, and companies have had no trouble accessing the ChimeraX meeting server. So SLAC is exceptional.

Tom

[Tom Goddard]

Seems the SLAC firewall does not like the socket connection to an unknown port 52xxx on the AWS machine. But Mike was able to start the meeting and I could join and it worked. To start it uses an ssh tunnel to the AWS machine that the SLAC firewall allowed. It is strange that Mike was able to join a meeting I started when the session was small -- maybe the SLAC firewall just drops the connection when there is a large data transfer inbound. Hard to believe, but possible.

I think I could pretty easily have joining a meeting use an ssh tunnel to the AWS machine and I suspect this would allow SLAC to join a meeting, worth trying. If it does not adversely effect performance could make this the default behavior. It would only work for the chimeraxmeeting.net server where ChimeraX knows the ssh key, for other servers the joiners would not be able to make the ssh tunnel. In any case I could make it an option of the meeting command and gui to join with an ssh tunnel.

Related documentation issue: would be good to replace the "access method" terminology with "meeting server". This will make better sense to the computer savvy users and for others neither term will make much sense.

[Tom Goddard]

I added a "ssh true" option to meeting join that makes the joiner use an ssh tunnel to the server. Ask Mike if he is interested in trying this to see if it allows him to join a meeting from SLAC. The ssh option has the drawback that the joiner needs to have the ssh key. For the default chimeraxmeeting.net server ChimeraX knows the key, but for servers set up by others joiners will not be able to use this option unless the define a new server name with the "meeting server" command that lists the key file.

[Tom Goddard]

Joining with ssh appears to have fixed this problem.

Tested meeting join ssh option with April 16 ChimeraX daily build with Mike Schmid from SLAC using the same machine where this bug occurred. With "ssh true" ssh session data was received and everything worked. Without ssh true the session was never received just as occurred with this bug report.

[Tom Goddard]

I added the ssh option for joining to the Meeting panel under options and added it to the documentation.

Not sure if should be enabled by default. It introduces a 5 second delay in joining. It may cause more latency (5 socket connections vs 3) or more failure modes in the network connection. So for now I will leave the default as not using ssh since that is what I have the most experience with and it usually works.