Find a way for VR meeting to handle firewalls

[Tom Goddard]

For two people at different universities to have a VR meeting one currently needs to allow incoming connections from the other on port 52194 (or some other port specified in the meeting command). But typically universities have firewalls that block all incoming connections except for known web servers and ssh. Outgoing connections are usually not blocked. What is a simple and widely applicable way for the two parties to hold a VR meeting.

One simple approach is that one of the participants asks their university IT department to open up the firewall port 52194 for their computer. I have done this at UCSF for our two vizvault VR computers. It took about 10 days to get approved. I am not sure the UCSF firewall still allows incoming connections on this port to those machines as changing security policies may have closed it.

Another approach is that the participants connect to a computer outside their firewalls and traffic is passed through that machine. For instance the outside machine could be a cloud virtual machine using Amazon Web Services, or could be a commercial VPN service.

Another approach is that UCSF could have a computer that serves as the middle-man and has UCSF firewall ports opened to accommodate this and it could be used for meetings that don't involve UCSF -- we provide it as a service. A drawback of this is that researchers across the world in Europe, China, Australia may experience poor performance with all VR traffic routed through San Francisco. Conrad started on this approach but it so far has not worked due to some problem with the code he wrote for the middle-man server.

[Tom Goddard]

I tried allowing incoming VR meeting connections on my home VR computer by forwarding incoming port 52194 connections on my wireless router to the Windows desktop VR computer. This also required assigning a fixed IP address to by Windows VR computer on my home network instead of DHCP dynamics assignment so that the port forwarding rule could list the IP of the VR machine. This was easy to setup, took about 15 minutes. I tested it by hosting a VR meeting on my home VR desktop machine and connecting to it using my other Windows laptop connected to the RBVI VPN so it appeared to come from UCSF. The test worked fine. I would like to do another test of this with Phil Cruz at NIH.

[Tom Goddard]

Today I tried using Amazon Web Services and ssh remote port forwarding to handle a VR meeting with only outgoing connections by the participants. I setup a Ubuntu 18.04 LTS instance on AWS (hosted in N California region). This was a standard instance but in the setup for Security Groups I added a firewall rule allowing incoming port 52194 connections. The default firewall rules only allow ssh incoming connections. I then used ssh remote port forwarding

ssh -N -i vr-key-private.pem -R 52194:192.168.0.169:52194 ubuntu@54.183.217.210

on my Windows laptop VR computer which has IP address 192.168.0.169 on my local network. This command forwards connections from the AWS instance (IP address 54.183.217.210) on port 52194 to my laptop. In order for the AWS instance sshd to forward connections from other computers I had to first edit /etc/ssh/sshd_config to enable "GatewayPorts yes". Normally the remote port forwarding is just used for applications running on the remote machine to tunnel into the firewalled machine that created the tunnel. After modifying sshd_config I used "sudo service ssh restart" to restarted sshd on the AWS instance. The AWS instance is free using AWS 12 month free tier service, 750 hours / month (ie 1 instance running full time, or multiple instances running less than full time).

With that setup I was able to host a ChimeraX VR meeting on my laptop ("meeting start") and then join that meeting from my VR desktop computer ("meeting 54.183.217.210"). A molecular session was transferred, it could be manipulated by both VR systems, latency was low (< 0.5 seconds), VR gui panels worked visible to both participants. I would also like to try this test with Phil Cruz at NIH as a more realistic scenario than me connecting my two VR home machines through the AWS instance.

I saved the instance with the modified sshd config file on AWS but a few days later got notified that I was about to be charged. Free service only allows 1 Gb-month snapshot storage and a single snapshot was 8 Gbytes. So it is not practical to keep the snapshot of the instance on AWS for free.

[Tom Goddard]

Some of the existing problems with the cxconference.rbvi.ucsf.edu approach which has not yet succeeded it providing a VR meeting are listed in tickets: #3629 UCSF IT complains about security, #3512 and #3498 connection failure, #2893 and #2892 connected but no data sent. The code running on the server to make the connections is about 1000 lines of Python written by Conrad who has retired. It has the nice feature that it is intended to support multiple independent VR meetings each having a meeting name assigned by the participant who starts it. The AWS ssh remote port forwarding approach could also handle multiple independent meetings on different ports, but additional code would be needed for participants to be able to use a simple meeting name instead of the AWS instance IP and port number.

[Tom Goddard]

My impression so far in these experiments to enable firewalled VR participants to connect is that using accessible server machines to connect participants may be beyond our resources. For good performance commercial servers such as AWS would have to be paid for. Using a UCSF server is likely to give poor performance, poor reliability, and poor maintainability. The simplest and most practical approach may be that at least one participant has to go to the effort to have their institutional firewall open a port.

While many applications deal with the same issues of connecting firewalled participants (video meetings e.g. Zoom, multiplayer video games, web team management e.g. Slack) these are well funded commercial enterprises that can support servers across the worlds. AWS divides the world into dozens of regions where virtual machine instances can be run with little effort, but it must be paid for. Given the niche user base (structural biology researchers) and prototype state of VR molecular analysis it seems doubtful that no setup VR meeting connections can be supported.

Nanome is a similar VR for structural biology researchers commercial product run by a startup of about 15 people that does provide the no setup connections. The charge monthly subscription fees and are largely financed by I'd estimate several million dollars of venture capital.

[Tom Goddard]

I used ssh on Windows to setup the remote port forwarding test to AWS. For a normal user of Windows 10 they don't have ssh -- I have it on my development machines using cygwin. Would be pretty unpleasant to have to bundle an ssh executable with ChimeraX. There is a third-party Python module asyncssh that could do the job in a thread in ChimeraX.

​https://asyncssh.readthedocs.io/en/latest/#port-forwarding

[Tom Goddard]

Phil Cruz, Victor Kramer and I did a 3 person VR meeting test today where I started the meeting at home and Phil and Victor successfully joined from their homes. I configured my home router to forward port 52194 to my Windows VR desktop machine for that test.

We then tried the 3-way meeting with AWS and failed because I had the firewall rules on the AWS virtual machine only allowing incoming connections from my home IP address. This was the default AWS setting and I did not notice it in my original tests. It was trivial to change the AWS server to allow connections from all addresses and then a connection from UCSF via VPN using my Windows VR laptop worked. Phil and Victor had left by that time so we still need to test this again with a 3 or 4 person meeting (including Meghan who was busy today).

Several other VR meeting bugs were encountered in our test but none related to firewall issues and those have separate tickets.

[Tom Goddard]

Phil, Victor and I have tried a number of VR meetings using ssh tunnel to an AWS virtual machine serving as a proxy to get around firewalls that block connection directly to the meeting host. I have configured an AWS server and assigned domain name chimeraxmeeting.net to it, and NIAID has a server configured as a VR meeting proxy on their AWS account (details at ​http://52.0.163.3/ setup by David Liou). I have also setup a nameserver using custom Python code to map meeting names to host/port so that a meeting can be started "meet start sarsspike" and joined with "meet sarsspike" without having to know host names and IP addresses. This current scheme for simplifying starting meetings with firewalls is working pretty well.

The next step I am looking at is allowing multiple meetings on the same proxy server using different ports. Currently this is supported (although not yet tested) by explicitly specifying a port number. I would like the meeting command to automatically find an available port in a range of say 10 ports. This has proved challenging using ssh tunnels. Although ssh can be made to exit if creating a tunnel fails because the requested port is already in use, it can also fail simply because the proxy is down, authentication is wrong, .... and the ssh exit code is always the same 255 on failure. Also I could find no reliable way to have ssh tell me when tunnel creation succeeded. It can take several seconds to make a connection and I have not seen a way to know when it is setup except by querying the proxy port and seeing if I can connect. That is problematic since some other application may have the port open. OpenSSH debug level 1 messages on macOS say when the tunnel has been setup but that is a very unreliable method (other ssh implementations won't give the same debug info, and other openssh versions might not too).

Testing connecting to the proxy port seem like a good idea because ssh may form the tunnel, but firewall rules on the AWS machine will block any connection to it if not opened using security settings. sshd does not know the port it is listening on has incoming connections blocked. So testing that it can be reached would be highly desirable to make sure the meeting command works and is easy to debug if connections can not be made.

A solution to determining when an ssh tunnel successfully setup might be using the asyncssh PyPi module and create the ssh tunnel using a Python call and Python client instead of the system ssh executable. This should allow me to get better status information, although I have not verified that.

My current proxy tunnel setup has the meeting command succeed, and only some seconds later the ssh tunnel command may fail. Probably this should close the meeting, although currently it does not.

If using asyncssh the API gives direct access to the "channels" of the ssh connection. A port forwarding creates a channel which has the same API as a Python socket. So it is not necessary to actually open a socket on the local machine, the channel could be used directly possibly with higher performance. But this is not easy to try now because the meeting code is using QTcpSocket objects rather than the lower level Python socket. The QTcpSocket readReady, disconnected, error slots allow handling all the participant's messages in the main thread using the Qt event loop which is simple and convenient. Python sockets would require threads or coroutines. The QTcpSocket also provides unlimited buffering. So moving from QTcpSockets to lower level Python sockets and ssh channels might lead to much more complex code.

[Tom Goddard]

Besides allowing a proxy to handle multiple meetings I'd like to allow any ChimeraX users to utilize the chimeraxmeeting.net proxy. To do this some security should be added. Making the tunnel requires a private key to connect to a specific account on the proxy machine. I should make an account (call it "proxy") and disable ssh shell so ssh cannot be used to execute commands. (Hmm... that will make it pretty painful to log into that account to make changes.) Then I can embed the private key in ChimeraX since no shell access limits abuse possibilities to making tunnels. I could then have the default "proxy" option of the meeting command be chimeraxmeeting.net, the default key would be none but the code would recognize the default proxy and provide the key when making the tunnel. And I would add a useProxy option, probably default false, at least until I can figure out how support multiple meetings at the same time on the proxy.

[Tom Goddard]

For handling multiple meetings on the proxy the meeting command could have a proxyPortRange option for instance with default value 52194-52203 allowing 10 meetings to run simultaneously. Should choose the number so the network bandwidth of the proxy machine is adequate. Options useProxy, proxy, keyForProxy, proxyPortRange should all be remembered in preferences so serious meeting users can customize.

[Tom Goddard]

I added meeting start options proxy, proxyHost, proxyPortRange, proxyTimeout, all saved in preferences, defaults using chimeraxmeeting.net.

Currently it chooses a random port among the ten 52194-52203. If that random choice is already in use then an error occurs and the meeting is not started. I need to have it try other ports automatically. Unfortunately this is tricky to do well because ssh always gives exit code 255 no matter how it failed. So if the server could not be reached, or timed out, or authentication failed, we don't want to repeat that 10 times, each with a 5 second time out. OpenSSH outputs a message about how it failed (...time out..., ...port already in use...) but it seems like a horrible plan to rely on the output messages -- no doubt different ssh executables give different messages. I could scan the output and if it says something that clearly indicates a problem that applies to all ports (authentication failure, timeout, ...) then I could avoid testing all 10 ports, other wise try all 10 and log the message I got for each. Ugly but hard to do better.

I spent hours looking for better solutions. One idea was to setup the ssh tunnel with a Python API using say the Paramiko Python ssh implementation or asyncssh. Both looked pretty complicated to use, and unclear if their performance running in a separate thread in ChimeraX to shuttle on the tunnel data would be as good as the separate process implemented in C with a standard ssh subprocess. So I think it best to stick with the subprocess and handle the failures as best we can.

[Tom Goddard]

I've made meeting start try ssh tunnels to different proxy ports until it finds one that is available.

Also it now blocks until the the ssh timeout (default 5 seconds) elapses to determine if the tunnel was actually created -- could not find any better way -- works ok.

I improved handling of the various failure cases when proxy or name server is not reachable so the meeting gets closed instead of being half started.

I added a meeting settings command to report or set the default proxy, name server and participant options.

I updated the meeting command docs to explain all the latest behavior including security issues.

[Tom Goddard]

I think the current firewall handling with proxy server and name server is usable. It would benefit from a GUI panel so users don't need to know command syntax and obscure options like "proxy true".

If this got much use my AWS proxy server chimeraxmeeting.net would be too busy. Not sure how well it would handle the 10 connections I am allowing right now. I have only tried two non-VR connections. It should easily handle 10 non-VR meetings.

[Tom Goddard]

Added GUI panel. Allows choosing different proxy servers.

Tested with two 2-person VR meetings with Phil, Meghan and Victor using the NIAID AWS server. No performance issues.