Opened 3 years ago
Last modified 3 years ago
#7438 assigned defect
Toolshed bot traffic?
Reported by: | Tristan Croll | Owned by: | Greg Couch |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | Tool Shed | Version: | |
Keywords: | Cc: | chimerax-programmers | |
Blocked By: | Blocking: | ||
Notify when closed: | Platform: | all | |
Project: | ChimeraX |
Description
Not really complaining, but a heads-up on some weird activity: each version of ISOLDE currently on the ToolShed has seen roughly 500-1000 downloads over the past few days. I’m guessing the same is probably true for the other bundles, but haven’t checked.
Change History (7)
comment:1 by , 3 years ago
Cc: | added |
---|---|
Component: | Unassigned → Tool Shed |
Owner: | set to |
Platform: | → all |
Project: | → ChimeraX |
Status: | new → assigned |
comment:3 by , 3 years ago
Unfortunately our Toolshed download numbers are inflated by tons of bot downloads. Here are the counts of downloads of ISOLDE for August 8-12, 2022 by IP address. The top downloader 139.87.112.86 getting 15000 copies (in 12 days!) is registered to Oracle Public Cloud.
$ cd /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed $ grep ChimeraX_ISOLDE cxtoolshed-ssl_access_log | awk '{ print $1 }' | sort | uniq -c | sort -nr 14935 139.87.112.86 623 106.110.22.71 543 183.27.183.226 531 182.136.222.170 443 49.83.1.57 236 218.91.199.138 193 106.114.227.104 127 146.196.69.206 101 119.181.233.123 78 120.230.4.174 76 183.210.206.155 50 39.130.11.207 31 156.146.57.106 30 61.157.55.190 16 128.250.0.219 13 111.47.184.39 12 165.123.226.181 12 128.250.0.199 11 212.219.94.138 10 129.112.109.53 8 46.193.0.163 7 131.211.61.123 7 128.250.0.204 6 165.225.9.16 5 129.22.1.24 4 73.109.145.225 4 69.173.127.231 4 222.205.46.34 4 195.214.157.165 4 194.94.44.220 4 141.2.96.177 4 137.220.120.251 4 132.199.82.22 3 68.72.127.52 3 222.205.46.7 3 193.60.81.42 3 174.127.185.130 3 134.174.140.197 3 129.111.66.218 3 108.168.178.34 2 24.111.21.90 2 217.38.29.230 2 213.55.243.127 2 205.208.121.100 2 205.175.118.88 2 203.181.216.220 2 193.171.188.3 2 188.154.101.204 2 147.251.155.208 2 141.2.127.118 2 139.165.31.13 2 137.99.47.16 2 137.99.143.25 2 132.199.73.40 2 130.20.89.185 2 128.250.0.214 2 121.200.5.213 1 96.255.81.190 1 95.90.240.180 1 94.134.109.136 1 90.214.86.87 1 89.161.49.146 1 84.115.208.110 1 81.223.14.211 1 81.109.143.202 1 73.51.84.206 1 69.173.127.109 1 58.250.174.75 1 50.230.201.134 1 50.224.29.124 1 46.59.131.25 1 24.118.180.21 1 222.205.46.3 1 220.233.193.61 1 217.178.26.60 1 210.107.187.26 1 205.175.118.179 1 205.175.106.19 1 202.86.119.18 1 199.101.192.176 1 193.2.14.90 1 192.35.193.137 1 190.216.145.52 1 184.95.214.178 1 171.66.12.250 1 171.66.12.244 1 161.0.159.155 1 155.91.28.142 1 155.198.149.50 1 153.90.19.48 1 153.167.109.98 1 149.22.3.162 1 146.186.149.189 1 144.32.92.130 1 144.32.240.120 1 144.174.212.47 1 141.211.182.138 1 140.107.21.164 1 140.107.179.133 1 134.94.56.32 1 134.174.140.235 1 131.211.61.126 1 131.111.85.79 1 129.89.231.240 1 129.22.1.25 1 128.250.0.38 1 128.250.0.220 1 128.250.0.210 1 128.250.0.209 1 128.231.234.71 1 128.214.54.23 1 128.214.138.135 1 128.214.138.134 1 128.194.178.102 1 128.176.186.38 1 125.63.30.232 1 124.197.36.171 1 120.224.119.29 1 110.227.241.47 1 109.175.193.125 1 108.75.133.147 1 107.150.30.116 1 107.122.81.43 1 104.129.198.124
comment:4 by , 3 years ago
The web server logs show 139.87.112.86 downloading 5 copies of ISOLDE per second, different versions, sometimes the same version twice in a row.
$ grep ChimeraX_ISOLDE cxtoolshed-ssl_access_log | grep 139.87.112.86 | head 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62876954 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-macosx_10_13_x86_64.whl HTTP/1.1" 200 56971130 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-macosx_10_13_x86_64.whl HTTP/1.1" 200 56971130 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62876954 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56973900 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56973900 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62878153 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62878153 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56974198 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15" 139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56974198 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
comment:5 by , 3 years ago
Tristan, so that you can have a better idea of ISOLDE downloads I will email you the web server logs for ISOLDE downloads for the months of June, July and August so you can have a truer picture of who is downloading.
comment:6 by , 3 years ago
Comment 3 about the number of downloads per IP address mistakenly said it was for August 1 to 12, 2022. It was actually for August 8 - 12, 2022.
comment:7 by , 3 years ago
I guess I won't be sending Tristan the ISOLDE web server download logs because our logging seems to have gone haywire starting in May 2022 so most of the logs are missing between May 2022 and now. We should figure out why that is, since inspecting the download IP addresses is valuable for understanding usage. I made a ticket about the missing logs, #7442
/usr/local/www/logs/cxtoolshed-httpd/cxtoolshed: 56686847 Aug 12 15:17 cxtoolshed-ssl_access_log 207562 Aug 12 00:58 cxtoolshed-ssl_error_log 91026 Jun 17 23:59 cxtoolshed-ssl_access_log.1.gz 429 Jun 17 11:12 cxtoolshed-ssl_error_log.1.gz 1332057 Apr 30 23:51 cxtoolshed-ssl_access_log.2.gz 36213 Apr 29 13:41 cxtoolshed-ssl_error_log.2.gz 2896775 Mar 31 23:58 cxtoolshed-ssl_access_log.3.gz 1033370 Mar 31 21:54 cxtoolshed-ssl_error_log.3.gz 5280828 Mar 3 00:03 cxtoolshed-ssl_access_log.4.gz 6703730 Mar 2 10:43 cxtoolshed-ssl_error_log.4.gz 3142946 Feb 1 2022 cxtoolshed-ssl_access_log.5.gz 527317 Jan 31 2022 cxtoolshed-ssl_error_log.5.gz 1904823 Jan 1 2022 cxtoolshed-ssl_access_log.6.gz
Yes, we noticed this on some bundles (e.g. ArtiaX, GenomeTools) but not all, since there are plenty of bundles under 200 downloads. Tom G. investigated and reported:
[goddard@franklin cxtoolshed]$ /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed
[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | head -200
128.250.0.214 - - [08/Aug/2022:21:39:43 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\
/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.77"
128.250.0.214 - - [08/Aug/2022:22:20:29 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/\
537.36 (KHTML, like Gecko) QtWebEngine/6.3.0 Chrome/94.0.4606.126 Safari/537.36 UCSF-ChimeraX/1.4 (Windows 10.0.22000)"
128.194.178.102 - - [09/Aug/2022:15:19:50 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl H\
TTP/1.1" 200 790167 "https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax?platform=Windows&version=1.3" "Mozilla/5.0 (Windows N\
T 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.2 Chrome/83.0.4103.122 Safari/537.36 UCSF-ChimeraX/1\
.3 (Windows 10.0.19044)"
139.87.112.86 - - [09/Aug/2022:15:37:17 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:18 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:38:22 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:38:22 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:39:26 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:39:26 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:40:28 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:40:29 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:41:32 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:41:32 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:42:36 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:42:36 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:43:39 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
...
[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | head -200 | awk '{ prin\
t $1 }' | sort | uniq | wc
[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | grep 139.87.112.86 | wc
[goddard@franklin cxtoolshed]$ grep -h ChimeraX_Genometools-0.9-py3-none-any.whl cxtoolshed-ssl_access_log | grep 139.87.112.86\