Check if EMDB maps can be fetched by https instead of ftp

[philip.macmenamin@…]

NIH 3D pipeline is fetching EMDB maps by ftp. That is frowned on in security audits. Can https be used?

[Tom Goddard]

Philip has a github issue at the nih_3d_workflows repository about this

​https://github.com/niaid/nih_3d_workflows/issues/80#issuecomment-948856273

[Tom Goddard]

Timing https EMDB fetches vs ftp fetches.

EMDB 12088, 64 Mbytes, 256 x 256 x 256 map, but downloaded compressed file is only 7 Mbytes.

Four tries deleting cached file after each try.

RCSB:
12 s, 9 s, 0.5 s, 9 s: https://ftp.wwpdb.org/pub/emdb/structures/EMD-%s/map/%s.gz
0.8 s, 10.5 s, 1.6 s, 9.8 s: ftp://ftp.wwpdb.org/pub/emdb/structures/EMD-%s/map/%s.gz

EBI:
17.5 s, 16.9 s, 16.9 s, 16.9 s: https://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-%s/map/%s.gz
3.0 s, 2.8 s, 3.1 s, 2.9 s: ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-%s/map/%s.gz

China:
Does not have 12088.  Entry from this week and apparently has not been synced to China.
Try 11368, 64 Mbytes, 256 x 256 x 256, 59 Mbytes compressed
Don't see any https urls on China site, and guessing https://ftp.emdb-china.org/structures/EMD-%s/map/%s.gz gives "failed to respond"
11.1 s, 9.9 s, 10.8 s, 13.3 s: ftp://ftp.emdb-china.org/structures/EMD-%s/map/%s.gz

Japan:
Testing entry 11368.
5.5 s, 7.2 s, 6.8 s, 6.4 s: https://ftp.pdbj.org/pub/emdb/structures/EMD-%s/map/%s.gz
12.3 s, 8.7 s, 14.0 s, 116 s : ftp://ftp.pdbj.org/pub/emdb/structures/EMD-%s/map/%s.gz

RCSB:
Testing entry 11368
4.6 s, 92.6 s, 4.0 s, 3.5 s: https://ftp.wwpdb.org/pub/emdb/structures/EMD-%s/map/%s.gz
3.5 s, 93.2 s, 4.0 s, 90.4 s: ftp://ftp.wwpdb.org/pub/emdb/structures/EMD-%s/map/%s.gz

EBI:
Testing entry 11368
6.9 s, 13.1 s, 10.7 s, 8.4 s : https://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-%s/map/%s.gz
8.7 s, 9.6 s, 8.8 s, 6.9 s: ftp://ftp.ebi.ac.uk/pub/databases/emdb/structures/EMD-%s/map/%s.gz

[Tom Goddard]

Conclusions from the timings in the previous comment:

RCSB has very erratic download times differing by factors of 10 from one fetch to the next and ftp and https look about the same.

EBI is consistently faster with ftp especially if the file compresses a lot, but also a little faster if the file does not compress.

China appears to only offer ftp.

Japan is consistently faster with https. Currently ChimeraX does not use the Japan mirror.

[Tom Goddard]

Based on these timings I am adding the Japan mirror when ChimeraX is run with hostname ending in ".jp" using https. The other 3 sites all will remain the same using ftp.

It would be possible to add the ability to choose the mirror and choose the protocol ftp vs https as open command options.

[Tom Goddard]

Made EMDB fetch use Japan mirror if hostname ends in ".jp".