Sunday evening represented the first time since June 7th of 2006 that I was forced to power cycle hicks, our production server. That’s almost a year and five months, for those of you counting at home. True, I’ve restarted both Apache and MySQL in that time, so it’s not actually an indication of uptime as far as all of you are concerned, but it’s still impressive, I think.
As for what caused it, my feeble only-plays-a-sysadmin-on-TV brain hasn’t determined that at the present time (suggestions appreciated). A quick look at the syslog told me little beyond the fact that MySQL was seriously unhappy about the hard stop and restart, but I haven’t seen any real fallout from that yet. It would be really nice if I had a network resource that helped connect me with a sysadmin expert I could pay to take a forensic peek into the machine’s recent history, but alas such a thing doesn’t exist. So far, anyway. It’s like you people are daring me to build it.
Some thanks are due. First, my appreciation for the hard work of the Linux kernel guys that provide the foundation for our operating platform, and second, the combined Debian and Ubuntu resources that package the software we use to bring you redmonk.com and our various blogs.
Lastly, I’d be remiss if I didn’t express – again – my affection for the support offered by Johncompanies, our colo provider. The net time for a restart of my machine from the time I pushed send on a “I can’t SSH” email? 14 minutes, late on a Sunday evening. In that time, they’d not only cycled the machine, but also connected a KVM to it in the event that a more complex restoration process was required. Fortunately, a simple restart restored access, but the fact that they’d allowed for the worst case scenario was as comforting as their remarkable response time. It’s an understatement to say that I’m a fan of Johncompanies.
Disclosure: Canonical, the company behind the Ubuntu distribution, is a RedMonk client, as is MySQL, whose software serves as our production database, and Sun which provided us with the V20z pictured for testing purposes several years ago.
