Opened 3 years ago

Last modified 3 years ago

#7438 assigned defect

Toolshed bot traffic?

Reported by: Tristan Croll Owned by: Greg Couch
Priority: normal Milestone:
Component: Tool Shed Version:
Keywords: Cc: chimerax-programmers
Blocked By: Blocking:
Notify when closed: Platform: all
Project: ChimeraX

Description 

Not really complaining, but a heads-up on some weird activity: each version of ISOLDE currently on the ToolShed has seen roughly 500-1000 downloads over the past few days. I’m guessing the same is probably true for the other bundles, but haven’t checked.

Change History (7)

comment:1 by Eric Pettersen, 3 years ago

Cc: chimerax-programmers added
Component: Unassigned → Tool Shed
Owner: set to Greg Couch
Platform: → all
Project: → ChimeraX
Status: new → assigned

comment:2 by Eric Pettersen, 3 years ago

Yes, we noticed this on some bundles (e.g. ArtiaX, GenomeTools) but not all, since there are plenty of bundles under 200 downloads. Tom G. investigated and reported:

There were 198 downloads of ArtiaX and GenomeTools toolshed wheels yesterday from the same IP address 139.87.112.86 registered by "Oracle Public Cloud". Two downloads were made each minute with high regularity. Who knows what this is about. There were just 3 different IP addresses downloading ArtiaX among the last 200 ArtiaX downloads.

Tom

[goddard@franklin cxtoolshed]$ /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed
[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | head -200
128.250.0.214 - - [08/Aug/2022:21:39:43 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\
/537.36 (KHTML, like Gecko) Chrome/103.0.5060.134 Safari/537.36 Edg/103.0.1264.77"
128.250.0.214 - - [08/Aug/2022:22:20:29 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/\
537.36 (KHTML, like Gecko) QtWebEngine/6.3.0 Chrome/94.0.4606.126 Safari/537.36 UCSF-ChimeraX/1.4 (Windows 10.0.22000)"
128.194.178.102 - - [09/Aug/2022:15:19:50 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl H\
TTP/1.1" 200 790167 "​https://cxtoolshed.rbvi.ucsf.edu/apps/chimeraxartiax?platform=Windows&version=1.3" "Mozilla/5.0 (Windows N\
T 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.15.2 Chrome/83.0.4103.122 Safari/537.36 UCSF-ChimeraX/1\
.3 (Windows 10.0.19044)"
139.87.112.86 - - [09/Aug/2022:15:37:17 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:18 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:38:22 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:38:22 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:39:26 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:39:26 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:40:28 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:40:29 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:41:32 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:41:32 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:42:36 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:42:36 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:43:39 -0700] "GET /media/chimeraxartiax/releases/0.1/ChimeraX_ArtiaX-0.1-py3-none-any.whl HTT\
P/1.1" 200 790167 "​http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHT\
ML, like Gecko) Version/12.1.1 Safari/605.1.15"
...

[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | head -200 | awk '{ prin\
t $1 }' | sort | uniq | wc

3 3 44

[goddard@franklin cxtoolshed]$ grep -h ChimeraX_ArtiaX-0.1-py3-none-any.whl cxtoolshed-ssl_access_log | grep 139.87.112.86 | wc

198 4752 59895

[goddard@franklin cxtoolshed]$ grep -h ChimeraX_Genometools-0.9-py3-none-any.whl cxtoolshed-ssl_access_log | grep 139.87.112.86\

| wc

198 4752 61677

Last edited 3 years ago by Eric Pettersen (previous) (diff)

comment:3 by Tom Goddard, 3 years ago

Unfortunately our Toolshed download numbers are inflated by tons of bot downloads. Here are the counts of downloads of ISOLDE for August 8-12, 2022 by IP address. The top downloader 139.87.112.86 getting 15000 copies (in 12 days!) is registered to Oracle Public Cloud.

$ cd /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed
$ grep ChimeraX_ISOLDE cxtoolshed-ssl_access_log | awk '{ print $1 }' | sort | uniq -c | sort -nr

  14935 139.87.112.86
    623 106.110.22.71
    543 183.27.183.226
    531 182.136.222.170
    443 49.83.1.57
    236 218.91.199.138
    193 106.114.227.104
    127 146.196.69.206
    101 119.181.233.123
     78 120.230.4.174
     76 183.210.206.155
     50 39.130.11.207
     31 156.146.57.106
     30 61.157.55.190
     16 128.250.0.219
     13 111.47.184.39
     12 165.123.226.181
     12 128.250.0.199
     11 212.219.94.138
     10 129.112.109.53
      8 46.193.0.163
      7 131.211.61.123
      7 128.250.0.204
      6 165.225.9.16
      5 129.22.1.24
      4 73.109.145.225
      4 69.173.127.231
      4 222.205.46.34
      4 195.214.157.165
      4 194.94.44.220
      4 141.2.96.177
      4 137.220.120.251
      4 132.199.82.22
      3 68.72.127.52
      3 222.205.46.7
      3 193.60.81.42
      3 174.127.185.130
      3 134.174.140.197
      3 129.111.66.218
      3 108.168.178.34
      2 24.111.21.90
      2 217.38.29.230
      2 213.55.243.127
      2 205.208.121.100
      2 205.175.118.88
      2 203.181.216.220
      2 193.171.188.3
      2 188.154.101.204
      2 147.251.155.208
      2 141.2.127.118
      2 139.165.31.13
      2 137.99.47.16
      2 137.99.143.25
      2 132.199.73.40
      2 130.20.89.185
      2 128.250.0.214
      2 121.200.5.213
      1 96.255.81.190
      1 95.90.240.180
      1 94.134.109.136
      1 90.214.86.87
      1 89.161.49.146
      1 84.115.208.110
      1 81.223.14.211
      1 81.109.143.202
      1 73.51.84.206
      1 69.173.127.109
      1 58.250.174.75
      1 50.230.201.134
      1 50.224.29.124
      1 46.59.131.25
      1 24.118.180.21
      1 222.205.46.3
      1 220.233.193.61
      1 217.178.26.60
      1 210.107.187.26
      1 205.175.118.179
      1 205.175.106.19
      1 202.86.119.18
      1 199.101.192.176
      1 193.2.14.90
      1 192.35.193.137
      1 190.216.145.52
      1 184.95.214.178
      1 171.66.12.250
      1 171.66.12.244
      1 161.0.159.155
      1 155.91.28.142
      1 155.198.149.50
      1 153.90.19.48
      1 153.167.109.98
      1 149.22.3.162
      1 146.186.149.189
      1 144.32.92.130
      1 144.32.240.120
      1 144.174.212.47
      1 141.211.182.138
      1 140.107.21.164
      1 140.107.179.133
      1 134.94.56.32
      1 134.174.140.235
      1 131.211.61.126
      1 131.111.85.79
      1 129.89.231.240
      1 129.22.1.25
      1 128.250.0.38
      1 128.250.0.220
      1 128.250.0.210
      1 128.250.0.209
      1 128.231.234.71
      1 128.214.54.23
      1 128.214.138.135
      1 128.214.138.134
      1 128.194.178.102
      1 128.176.186.38
      1 125.63.30.232
      1 124.197.36.171
      1 120.224.119.29
      1 110.227.241.47
      1 109.175.193.125
      1 108.75.133.147
      1 107.150.30.116
      1 107.122.81.43
      1 104.129.198.124
Last edited 3 years ago by Tom Goddard (previous) (diff)

comment:4 by Tom Goddard, 3 years ago

The web server logs show 139.87.112.86 downloading 5 copies of ISOLDE per second, different versions, sometimes the same version twice in a row.

$ grep ChimeraX_ISOLDE cxtoolshed-ssl_access_log | grep 139.87.112.86 | head
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62876954 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-macosx_10_13_x86_64.whl HTTP/1.1" 200 56971130 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-macosx_10_13_x86_64.whl HTTP/1.1" 200 56971130 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62876954 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56973900 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:40 -0700] "GET /media/chimeraxisolde/releases/1.0.1/ChimeraX_ISOLDE-1.0.1-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56973900 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62878153 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 62878153 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56974198 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"
139.87.112.86 - - [09/Aug/2022:15:37:41 -0700] "GET /media/chimeraxisolde/releases/1.0/ChimeraX_ISOLDE-1.0-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 56974198 "http://cxtoolshed.rbvi.ucsf.edu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Safari/605.1.15"

comment:5 by Tom Goddard, 3 years ago

Tristan, so that you can have a better idea of ISOLDE downloads I will email you the web server logs for ISOLDE downloads for the months of June, July and August so you can have a truer picture of who is downloading.

comment:6 by Tom Goddard, 3 years ago

Comment 3 about the number of downloads per IP address mistakenly said it was for August 1 to 12, 2022. It was actually for August 8 - 12, 2022.

comment:7 by Tom Goddard, 3 years ago

I guess I won't be sending Tristan the ISOLDE web server download logs because our logging seems to have gone haywire starting in May 2022 so most of the logs are missing between May 2022 and now. We should figure out why that is, since inspecting the download IP addresses is valuable for understanding usage. I made a ticket about the missing logs, #7442

  /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed:                                           

   56686847 Aug 12 15:17 cxtoolshed-ssl_access_log                                                   
     207562 Aug 12 00:58 cxtoolshed-ssl_error_log                                                                                                                            
      91026 Jun 17 23:59 cxtoolshed-ssl_access_log.1.gz                                              
        429 Jun 17 11:12 cxtoolshed-ssl_error_log.1.gz                                               
    1332057 Apr 30 23:51 cxtoolshed-ssl_access_log.2.gz                                              
      36213 Apr 29 13:41 cxtoolshed-ssl_error_log.2.gz                                               
    2896775 Mar 31 23:58 cxtoolshed-ssl_access_log.3.gz                                              
    1033370 Mar 31 21:54 cxtoolshed-ssl_error_log.3.gz                                               
    5280828 Mar  3 00:03 cxtoolshed-ssl_access_log.4.gz                                              
    6703730 Mar  2 10:43 cxtoolshed-ssl_error_log.4.gz                                               
    3142946 Feb  1  2022 cxtoolshed-ssl_access_log.5.gz                                              
     527317 Jan 31  2022 cxtoolshed-ssl_error_log.5.gz                                               
    1904823 Jan  1  2022 cxtoolshed-ssl_access_log.6.gz
Note: See TracTickets for help on using tickets.