Better meeting network performance, msgpack

[Tom Goddard]

Would be good to replace the current ChimeraX meeting command network message serialization which pass plain text Python literals (dictionaries, lists, strings, numbers) with a more compact format that can reduce bandwidth, allowing users with lower bandwidth network connections, and allowing large numbers of users.

The current serialization uses about 0.6 Mbits/second to transmit VR hand controller and head positions at 90 frames/second. This could be reduced about 5-fold by switching serialization to use msgpack.

Another improvement would be to LZ4 compress the initial session file sent when a person joins the meeting.

Other large messages are face images for participants. Currently it sends the full image file (e.g. PNG) base64 encoded. Could send as binary and could read, downsize if too large, and encode as jpeg for small size before sending over network.

[Tom Goddard]

It looks like msgpack is well suited as the serialization format. Standard examples show streaming objects from a socket.

A minor concern is that it only handles 4 GByte objects which limits sending large sessions for example with cryoEM maps. Maybe in 10 years network bandwidth will be fast where that takes only a short time. Does not seem to be a significant issue now.

[Tom Goddard]

Another performance enhancement would be to handle the relaying of messages on the hub in a separate thread so they don't interfere with the VR performance on the computer that is hosting the meeting.

[Tom Goddard]

Mechanisms for handling low bandwidth are discussed in ticket #3845.

[Tom Goddard]

If meeting is switched to use msgpack it would be much simpler to drop compatibility with the older ChimeraX 1.0 and 1.1 meeting serialization (Python literals). This would be reasonable to do I think since ChimeraX 1.1 and earlier did not use names for the meetings while names should always be used for ChimeraX 1.2 meetings and it would not be easy for a ChimeraX 1.1 or 1.0 to join such a meeting.

To allow compatibility would require adding handshake messages so the meeting hub could know which clients can handle msgpack and with require older Python literals. That does not seem worth the effort. But the msgpack serialization should also specify an initial message says what meeting version is being used to allow for future changes to the message content, for example to say sessions will be compressed with a better compression method.

[Tom Goddard]

To avoid port scanning causing spurious log messages during meetings and to avoid uninvited users joining a meeting, the first message a participant sends after joining should be a version and validation message. It should contain the meeting name. If the first message does not contain the correct meeting name the connection can be silently terminated by the hub. This will address the biggest security hole where uninvited participants can join a meeting.