Detective caps chaps - suspicious apache restart

[z0mbi3]

Something interesting on a Monday morning for a change! For reasons unkown our Ubuntu server (gutsy) at work tried to restart Apache2 at 6:25 AM on Sunday, got stuck and never quite finished. The logs show that it tried to do a graceful restart but failed. This morning I restarted Apache manually and we're back in business.

Any thoughts on what could have caused this? There's nothing in Cron to call a restart as far as I'm aware and there wouldn't of been any one in work who had a login to do it.

 

[Garp]

Check your auth logs, see if anything logged in or kicked in around that time. Grab a look in syslog and messages too at that time slot, see if anything occurred on the run up.

What is logrotate set to do? Rotate by size or number of days?

 

[z0mbi3]

Well I have a few crons running around that time but nothing that would restart apache.

Sep 21 06:25:03 scary su[15516]: Successful su for nobody by root
Sep 21 06:25:03 scary su[15516]: + ??? root:nobody
Sep 21 06:25:03 scary su[15516]: pam_unix(su:session): session opened for user nobody by (uid=0)
Sep 21 06:25:03 scary su[15516]: pam_unix(su:session): session closed for user nobody
Sep 21 06:25:03 scary su[15524]: Successful su for nobody by root
Sep 21 06:25:03 scary su[15524]: + ??? root:nobody
Sep 21 06:25:03 scary su[15524]: pam_unix(su:session): session opened for user nobody by (uid=0)
Sep 21 06:25:03 scary su[15524]: pam_unix(su:session): session closed for user nobody
Sep 21 06:25:03 scary su[15526]: Successful su for nobody by root
Sep 21 06:25:03 scary su[15526]: + ??? root:nobody
Sep 21 06:25:03 scary su[15526]: pam_unix(su:session): session opened for user nobody by (uid=0)
Sep 21 06:25:10 scary CRON[15398]: pam_unix(cron:session): session closed for user munin
Sep 21 06:25:11 scary su[15526]: pam_unix(su:session): session closed for user nobody

The only thought we've had so far is that some numpty button mashed our servers keyboard trying to get the web radio working on another machine for the sound system in their office, despite our machine never ever being left logged in. There are signs someones been in there on the weekend and our keyboard had been unplugged and stuck in another machine on the rack. Thing is 6:25 is the night shift so I doubt I'll be able to catch the guy to throttle him

I know that shouldn't effect the server however, if someones been in there messing about with cables around the time a load of crons kick in and Apache failed to restart, it's too much of a coincidence not to be related.

What is logrotate set to do? Rotate by size or number of days?

Every couple of days by the look of it. The only thing relevant in syslog is the following:

Sep 21 06:25:01 scary /USR/SBIN/CRON[15395]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Sep 21 06:25:01 scary /USR/SBIN/CRON[15397]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)
Sep 21 06:25:01 scary /USR/SBIN/CRON[15399]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)

 

[Garp]

Hmm.. had a niggling feeling cron.daily was triggered around that time. Nothing in /etc/cron.daily/?

 

[z0mbi3]

Hmm.. had a niggling feeling cron.daily was triggered around that time. Nothing in /etc/cron.daily/?

Nothing I can see that would restart apache2. Looking at the logs for the last week or so this is the first occurrence I've seen and I suspect it's the first time it's done it since it's first live usage 170 days ago! This is certainly the first time apache has fallen over.