Blogs

RedMonk

Skip to content

Movable Type 3.2 and Spam; Or, Where Have Ye Gone, MT-Blacklist?

As mentioned a couple of weeks back, we recently experienced an outage of our Movable Type implementation that knocked James, Cote and I off our blogs for the better part of a day. While it ultimately had nothing to do with the software itself – it was instead the result of an inability to reach our MySQL databases – being paying users of the product, my initial reaction was to open a support ticket with Six Apart. In their response, I was strongly encouraged to upgrade to the latest version of the product, 3.2.

I’d already been considering this because MT had been growing increasingly cranky, and was prone to returning HTTP 500’s for run of the mill requests, so I finally relented and upgraded. The upgrade itself was fairly uneventful, apart the fact that the upgrade script stalled when it hit our activity table which I was forced to truncate. In the entry noting the upgrade, I mentioned that our previous spam filtering system – MT-Blacklist – was no longer compatible, and was being replaced in favor of native MT capabilities. Even the developer of MT-Blacklist agreed that the functionality replacing MT-Blacklist was superior, saying “You don’t need MT-Blacklist — Movable Type 3.2 includes vastly superior built-in spam protection for free.” I was skeptical, not simply because I found MT-Blacklist to be very effective [1], but because Josh had mentioned that his experience hadn’t been that seamless.

Unfortunately for us, Josh’s experiences proved to be very relevant. The simple fact is that for our usage, MT-Blacklist was substantially more effective than the spam filtering capabilities native to MT 3.2.

To be fair, we deal with what I’d suspect is a more pressing spam problem than most. For an idea of what we’re facing, I completely cleaned out our Junk comment folder in MT last last Sunday night. As of a minute ago, it’s been repopulated with 9,974 junk comments. That means we’re getting somewhere on the order of 1900 spam comments a day, 83 per hour, or better than 1 per second (assuming my math is correct, which is always risky). Our spam situation, in other words, is not good.

When we had MT-Blacklist, however, outbreaks of spam were relatively rare. We had the odd comment get through here or there, but more often than not MT-Blacklist captured the bad stuff. It also, alas, captured the odd legitimate comment or two – which was a huge irritant for James, especially – but overall I spent very little time each day managing our spam situation.

With MT 3.2, however, I’m spending at least a half hour and sometime more each day cleaning out our accounts. Many of you subscribing to our comment feeds may have noticed that there are occasionally 50+ new comments – all of which are spam. The reason for this is multi-faceted.

  • First, the UI is far less efficient at using my time. With MT-Blacklist, I could mark comments as spam, and it would automatically parse them and extract bad URIs to add to its filter. There is no such feature that I’m aware of within MT 3.2, so any URI I want added to my filter needs to be inserted manually. That takes a minimum of two clicks and some scrolling to do.

  • Second, there does not appear to be any simple way of automatically moderating comments on older posts. MT-Blacklist offered this feature, which was hugely beneficial because the majority of spam is applied to older posts. Let me know if I’ve just missed this feature.
  • Third, MT-Blacklist maintained a single blacklist for all of our respective blogs. MT 3.2, however, maintains individual item lists. The benefit of this is that if James, as an example, wants to maintain a more lenient keyword list than I might, he has the ability to do so. But the catch is major: when a bad URI or string needs to be added to the filtering list, I need to insert that four times – once for each of our blogs. That’s a minimum of 8 clicks.
  • Fourth, MT 3.2 appears to have some difficulty scaling. When MT 3.2 fails to identify incoming spam, and I’m forced to remove 40 or more spam comments that have been applied against our wishes, it frequently crashes, returning an HTTP 500. This is not, I would assume, a hardware limitation as we are not on shared hardware, but have our own dedicated server.
  • Fifth, the IP filtering feature doesn’t seem to work any longer. With MT-Blacklist, I could pull up and remove all comments attributed to a particular IP address. While this was of limited utility, seeing as spammers generally were savvy enough to spoof their IPs on an individual post basis, there were some that were less skilled and posted tens of comments from a single IP. But when I hit the lookup button in MT 3.2 for an IP – one that I know has at least one posting because I’m looking at it – the search feature returns nothing.

In short, I’m more than moderately displeased with MT’s performance at this point as it’s costing me time that I can ill afford, and I’m likely to start contemplating alternative options in a more serious fashion.

In the meantime, I’m open to suggestions for anyone that’s dealt with a similar problem. I’ve tried to deploy the Akismet implementation for MT, but while the plugin is showing in the system wide MT interface, it’s not giving me a settings button and therefore isn’t activated. I may yet try out this hack that allows MT-Blacklist to function on top of 3.2, but that doesn’t seem like an ideal solution longer term.

If anyone has suggestions on getting Akismet to work, I’d appreciate them (the installation’s simple enough that I don’t think that’s the problem). But in the meantime, it seems as if I need to begin plotting our exit strategy from MT. That means figuring out how to handle our link redirection, first and foremost. As Paul suggested here, the technical aspect of the mapping is pretty straightforward – what will be challenging, I think, will be the mapping. But considering how much time it’s taking me to keep us spam free these days, it seems like that would be time well invested.

Update: Per DeWitt’s recommendation, I took another stab at getting Akismet up and running. I ultimately managed to get this “unofficial” implementation working, and while it originally was rejecting me with an Invalid Key error, it does appear to be functioning appropriately now. “Akismet Spam Filter -2.0 Flagged as SPAM.” I’ll play with it for a bit, and if it’s effective apply it to our other blogs.

[1] It was too effective, in fact, for James’ tastes.

Categories: Product Announcements.

  • Anonymous

    I hate being ‘very relevant’ …Stephen ping me I can tell you what all I’ve done to my install to help fight spam.

  • http://hyku.com/blog/ Josh Hallett

    Sorry, forgot to put my name on that previous comment.

  • http://www.unto.net/ DeWitt Clinton

    Whatever you do, keep trying to get Akismet to work.

    I set it up on unto.net and I went from dozens of spam messages a day down to *zero*. That’s right, not a single spam has made it through since I installed Akismet three weeks ago. Your mileage may vary of course.

  • http://raibledesigns.com Matt Raible

    Good ol’ Roller. I don’t know how many comment spam attempts I get per day, but on average 2-3 spam comments make it through per week. With the e-mail notification system, I’m able to delete these ones right away. Of course, it could simply be a market share thing.