When Yahoo Pipes was first launched back in February, my del.icio.us comment (the irony of this will be apparent shortly) was “will need to give this a whirl at some point soon.” True to my word, I did, and frankly was a bit underwhelmed. The performance was iffy, as nearly every technologist on the planet seemed to be kicking the tires at the exact same time, and the interface struck me as a bit obtuse.
I’ve had the sneaking suspicion for some time, however, that the problem was not actually inherent to Pipes, but was rather on my end. It’s difficult, after all, to be intrigued by a technology you have no immediate use for. But I finally came up with a use case perfect for Pipes, and I’m pleased to report that it came through the challenge with flying colors.
The basic problem was simple. When we redesigned the RedMonk homepage, one of the design decisions made was to pull from our respective feeds in order to incorporate blog content into the homepage. As readers will know, in addition to the regular content, Cote, James and myself also inject a significant amount of analysis and commentary into our del.icio.us links. For the sake of convenience, we use the del.icio.us daily blog posting mechanism to retrieve our daily links and integrate them back into our individual WordPress properties. Thus, our blogs and their feeds – which are in each case externally housed at FeedBurner – are in fact an amalgam of regular posts and del.icio.us links.
For those of you consuming the content in a reader, or visiting the blogs directly, this approach works well. Our analysis is centralized and made available for easy consumption, obviating the need to visit multiple properties.
While that worked well for the blogs, however, it was less ideal for the homepage. Given that the majority of the audience to the homepage are people that know little about us, let alone del.icio.us, the overpopulation of the homepage with entries like “links for 2007-04-18” was potentially intimidating, and certainly aesthetically displeasing. Alex had mentioned this over lunch a few weeks ago, James had surfaced this as a To Do, and I’ve gotten a steady trickle of similar comments since we launched.
There was no question that those links could be filtered – it’s all XML, after all. The question was what represented the simplest approach. The initial thought was to do it server side, stripping out anything from the “Links” category on the fly prior to display on redmonk.com. But that would require – potentially – some coding changes on our WordPress/parser setup, and I simply hadn’t had the time to look into it. While driving back to Maine on Monday, however, it struck me that this was precisely the kind of thing Yahoo Pipes was designed for, so I made a mental note to see if I could make it work when I got back to the office.
Well, make it work I did. To see it in action, go hit redmonk.com. The better news was that it literally took not more than 10 minutes to appropriately filter all three of our feeds. As you can see from the Pipe – or the inline picture – the setup is dead simple. Pipes reads in my tecosystems feed from FeedBurner (Sources: Fetch Feed), looks for items that have “links” in the title item, filters them (Operators: Filter), and delivers me a neatly stripped RSS feed. After repointing the parser on redmonk.com from FeedBurner to our newly created Pipes feeds, redmonk.com is now delivering a homepage free of del.icio.us links, while the individual blogs continue unaffected. Beautiful.
The point to emphasize here, as far as I’m concerned, is the efficiency. Fixing this via the original approach was far from an arduous task, but it would still have taken me probably an hour or two (remember, I’m not the sharpest tack in the box) to figure out what’s being done where and test the filtering. With Pipes, it took less than 10 and I was done; apply the filter, look at the output, cut over the feeds. The remaining question now is the reliability of the Pipes supplied feed; FeedBurner’s only been down once since we cut over to the new redmonk.com. If Pipes can supply similar uptime, I’ll be a very happy camper.
With the overwhelming majority of our content available in one syndicated form or another, I can see an increasing demand for Pipes style filtering and recombination. I doubt, in other words, that this particular business problem is the last one I’ll solve with Yahoo Pipes.
