Erratic EMDB download speeds

[Tom Goddard]

The download speed of an EMDB map can vary by a factor of 10 or 100 based on unknown factors. What are those factors? From the same computer on the same network connection the speed can vary drastically for downloads just minutes apart.

[Tom Goddard]

Eric says the RBVI VPN is consistently slow.

From: Eric Pettersen
Subject: Slow downloads, part deux
Date: October 4, 2023 at 6:20:36 PM PDT
To: Scooter Morris, Greg Couch
Cc: Tom Goddard, Zach Pearson

So, I have also found that downloading an EMDB map at home when I'm on hal2 is a couple of orders of magnitude slower then downloading it at home when I'm not on hal2.

--Eric

[Eric Pettersen]

I don't know that it's the VPN -- I think it's UCSF. Downloading the SPICE dataset (16GB) on plato via wget takes (an estimated -- I didn't actually finish the download) 12.5 hours, whereas from home on Sonic fiber without VPN it took 15 minutes.

[Tom Goddard]

Testing from my house, sonic gigabit fiber, speedtest.net speed test reports 490 Mbps download, 430 Mbps upload,

time open 16432 from emdb

took 22 seconds to download 404 MB map compressed (824 MB uncompressed), so about 18 Mbytes/second.

On RBVI VPN speedtest.net will not run but fast.com gives 280 Mbps download, 110 Mbps upload. EMDB fetch was downloading at 0.3 Mbytes/sec for first 200 Mbytes (11 minutes) after which I gave up. hostname reports hal2-client5.cgl.ucsf.edu and for .edu hosts ChimeraX EMDB fetch uses ​https://files.wwpdb.org. For other hosts it uses ​ftp://ftp.ebi.ac.uk. So the trouble is wwwpdb.org is excruciatingly slow, 60 times slower than fetch from EMDB at EBI in England.

[Tom Goddard]

At the office I get the opposite results where fetching from .edu host name is fast and fetching from EBI is slow.

time open 28330 from emdb

0.9 seconds to fetch 29 Mbyte .gz map file

while downloading using web browser from the EBI EMDB web page

​https://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-28330/map/emd_28330.map.gz

60 seconds

[Tom Goddard]

The erratic download speeds varying by a factor of 60 could be caused by the database mirror limiting the transmission rate, or by variable network routing, or by UCSF choking the bandwidth.

We may be able to figure out if UCSF is the culprit by comparing download times from UCSF versus from home internet (sonic.net). If both are slow or both are fast consistently, then the database limiting bandwidth is plausible and we could contact them and ask if they are aware of this.

[Eric Pettersen]

From home on hal2 I downloaded AmberTools23 (~570 MB) and the download rate was 1.2MB/sec. Off hal2 the rate was 46+MB sec (went so fast I only got a glimpse of the rate).

[Tom Goddard]

Probably should note these VPN speed test results that don't involve EMDB fetch on a different ticket.  The EMDB problem I think has to do with the different EMDB mirrors and not the RBVI VPN.  To figure out whether the RBVI VPN is the culprit or the UCSF network you should probably download AmberTools from your office desktop computer using the UCSF network but not on the RBVI VPN.