So we got hacked. Again. It was just as fun as it sounds.
The attack vector was the same this time as last: WordPress. And you know whose fault that is? Mine.
The background on this unfortunate event is fairly simple to relate. A week ago last Friday, I got an email from James, subject line: “wordpress doing something v weird indeed to my blog permalinks.” Weird it most certainly was: base64 strings were somehow being appended to each entry. 30 seconds on Google later, I’d turned up this and knew we had a problem. The next steps were obvious: 1) get rid of the unauthorized users (and confirm their absence with a SELECT * from wp_users; on the penetrated database), 2) reset the altered permalink structure, 3) check the posts for injected content (e.g. SELECT * FROM wp_posts WHERE post_content LIKE '%base64%';), and 4) call for help. As before, we engaged Inverse Path, and they could not have been more responsive and efficient.
Anyway, the good news was that we had only one database affected – James’ WordPress instance – and that the damage was sufficiently limited that I didn’t have to go back to an older version of his database. The bad news was that I had to spend a good part of a beautiful Saturday figuring that out and changing every password on the box.
What Happened?
Those of you that recall that we update our WordPress instances nightly from Subversion might reasonably be wondering: how in the hell did this happen? Don’t you guys keep update regularly, and isn’t the point preventing things like this? Well, yes we do, and yes it is. But at some point in the past – for reasons I can’t really recall – we began updating to specific versions of WordPress. So while our nightly scripts do indeed svn update around 2 AM each morning, they’ll only keep us updated within the particular version we’ve specified. Fortunately for us, in most cases, that was the safe 2.8.4 version, but we’d somehow missed James’ blog and he got left behind at 2.8.3. Which meant he was vulnerable when we were attacked.
Is it WordPress’ fault that I didn’t upgrade properly? I can’t possibly see how.
What Are We Going To Do?
At some point in the near future, I will probably shift our infrastructure over to trunk from tagged versions. As DeWitt says, that won’t prevent zero day exploits, but it would narrow the vulnerability window significantly. In the meantime I’ve cobbled together a stupid simple bash script that I can customize to batch upgrade us to a new version as it’s announced:
#!/bin/bash
# script to svn switch
cd /path/to/wpinstance1/
sudo svn switch http://svn.automattic.com/wordpress/tags/2.8.4 wpdir
cd /path/to/wpinstance2/
sudo svn switch http://svn.automattic.com/wordpress/tags/2.8.4 wpdir
And so on. But ultimately it’s just going to be safer to do an svn switch to trunk and stick to our nightly svn update script.
What Should You Do?
If you don’t have the same drivers that we do – a need to flexibly customize, a desire to dogfood certain approaches, etc – I would recommend running with a hosted version. Make updating and backup someone else’s problem wherever you can. But where that won’t work, you must a.) update regularly and b.) you absolutely, no exceptions, must back up your data. The lack of the latter is why I frankly have little sympathy for Scoble. It’s one thing for an industry neophyte blogging non-professionally to lose his data due to insufficient backups; it’s something else altogether for a technology professional working for one of the largest hosting outfits in the world. WordPress may or may not be intrinsically vulnerable, as was argued in this now dead link, but even if it was somehow magically immune to security exploits its databases would not survive a failed hard drive. I’m sorry, of course, that he lost data, but I would not be pointing the finger at WordPress.
If you need help backing up your WordPress instance, here are some detailed instructions on how we do it. All of the technologies involved – save the hosting – are free and open source. If you want a seamless, commercial-grade solution, I’d recommend BackupMoxie, a service run by a couple of friends of mine.
However you do it though, please – if you choose to host yourself, back up your data.
Is WordPress Secure?
In the wake of this particular attack, presumably because some of the victims were fairly high profile (e.g. Scoble), there are a great many people questioning the security of WordPress. And some of that criticism is, to be sure, well earned; following our last attack, trying to determine the file permissions that would both protect us and allow WordPress to function properly was something of a chore.
But at the risk of sounding like an apologist for the project, software – all software – is vulnerable. If you’re on a network, you can and will be attacked. For the money or just for the fun of it. So no, I decline to join the mob calling for the WordPress developers’ heads in the aftermath of a single exploit. If you want to argue that it’s less secure armed with metrics demonstrating this fact versus other similarly popular platforms, I’ll be happy to have that conversation. Until then, however, let’s talk about the real issue here: personal responsibility.
If you don’t want to be bothered with keeping your infrastructure updated and backed up, terrific: pick an infrastructure option where that’s done for you. It’s true of WordPress, yes, but it’s true of pretty much all software: upgrading is the only way to keep yourself safe. You can’t reasonably ignore the responsibilities of self-hosting and then cry about the consequences. That says far more about you than the WordPress project.
Or so it seems to this “victim.”
Disclosure: Neither Automattic – the company behind the commercial wordpress.com – nor WordPress are RedMonk clients. I have, however, spoken at WordCamp on two occasions on a volunteer basis. The founders of BackupMoxie are friends of mine.
