Toolshed web server logs are mostly missing after May 2022

[Tom Goddard]

It looks like the Toolshed web server logs are mostly missing from May 2022 on. It would be good to have these logs so we can assess true downloads versus bots (ticket #7438).

Here is what the log directory looks like on crick.cgl.ucsf.edu:

  /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed:                                           

   56686847 Aug 12 15:17 cxtoolshed-ssl_access_log                                                   
     207562 Aug 12 00:58 cxtoolshed-ssl_error_log                                                                                                                            
      91026 Jun 17 23:59 cxtoolshed-ssl_access_log.1.gz                                              
        429 Jun 17 11:12 cxtoolshed-ssl_error_log.1.gz                                               
    1332057 Apr 30 23:51 cxtoolshed-ssl_access_log.2.gz                                              
      36213 Apr 29 13:41 cxtoolshed-ssl_error_log.2.gz                                               
    2896775 Mar 31 23:58 cxtoolshed-ssl_access_log.3.gz                                              
    1033370 Mar 31 21:54 cxtoolshed-ssl_error_log.3.gz                                               
    5280828 Mar  3 00:03 cxtoolshed-ssl_access_log.4.gz                                              
    6703730 Mar  2 10:43 cxtoolshed-ssl_error_log.4.gz                                               
    3142946 Feb  1  2022 cxtoolshed-ssl_access_log.5.gz                                              
     527317 Jan 31  2022 cxtoolshed-ssl_error_log.5.gz                                               
    1904823 Jan  1  2022 cxtoolshed-ssl_access_log.6.gz

[Greg Couch]

It's a little more complicated. But yes the logs appear lost and it's unclear why.

cxtoolshed-ssl_access_log: 08/Aug/2022:09:41:39 -0700 to now
cxtoolshed-ssl_access_log.1.gz: 17/Jun/2022:02:45:15 -0700 to 17/Jun/2022:23:59:31 -0700
cxtoolshed-ssl_access_log.2.gz: 19/Apr/2022:23:06:10 -0700 to 01/May/2022:00:01:46 -0700
cxtoolshed-ssl_access_log.3.gz: 15/Mar/2022:00:00:24 -0700 to 01/Apr/2022:00:02:19 -0700

So May 1 to June 17 is lost.

Eric, can you retrieve /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed/cxtoolshed-ssl_access_log from June 16th?

[goddard@…]

Aren't logs also missing for June 18, 2022 to August 8, 2022?

[Greg Couch]

Whoops. Yes. I didn't believe cxtoolshed-ssl_access_log.1.gz was only for one day. That is even harder to explain.

We last changed how the toolshed website was organized on April 26. I would have expected everything to be consistent since then.

[Greg Couch]

The toolshed logs used to be in /usr/local/www/logs/plato-httpd/cxtoolshed. On franklin, there is a /wynton-local, which I believe was for running the web servers in a limited fashion when beegfs was down. On there, there cxtoolshed-ssl_access_log starts on 09/Aug/2020:18:32:29 -0700 and goes through 22/Apr/2021:08:46:01 -0700.

[Eric Pettersen]

So after all this back and forth is it still correct that you want /usr/local/www/logs/cxtoolshed-httpd/cxtoolshed
/cxtoolshed-ssl_access_log from June 16th?

[goddard@…]

I don't have a strong need for the logs.  So I would not say it is worth the trouble to restore them for me.  At some point if we wanted to accurately assess Toolshed use we would need the logs to be able to filter out the 15000 ISOLDE downloads by one IP address in 4 days, and many other egregious bot downloads.  But I don't see that we need that info now.

More important is that we fix whatever the problem was so that Toolshed web server logs from now on are not lost.

[Eric Pettersen]

Okay, sounds good.