Piping RedMonk.com

Share via Twitter Share via Facebook Share via Linkedin Share via Reddit

Using Yahoo Pipes

Originally uploaded by sogrady.

When Yahoo Pipes was first launched back in February, my del.icio.us comment (the irony of this will be apparent shortly) was “will need to give this a whirl at some point soon.” True to my word, I did, and frankly was a bit underwhelmed. The performance was iffy, as nearly every technologist on the planet seemed to be kicking the tires at the exact same time, and the interface struck me as a bit obtuse.

I’ve had the sneaking suspicion for some time, however, that the problem was not actually inherent to Pipes, but was rather on my end. It’s difficult, after all, to be intrigued by a technology you have no immediate use for. But I finally came up with a use case perfect for Pipes, and I’m pleased to report that it came through the challenge with flying colors.

The basic problem was simple. When we redesigned the RedMonk homepage, one of the design decisions made was to pull from our respective feeds in order to incorporate blog content into the homepage. As readers will know, in addition to the regular content, Cote, James and myself also inject a significant amount of analysis and commentary into our del.icio.us links. For the sake of convenience, we use the del.icio.us daily blog posting mechanism to retrieve our daily links and integrate them back into our individual WordPress properties. Thus, our blogs and their feeds – which are in each case externally housed at FeedBurner – are in fact an amalgam of regular posts and del.icio.us links.

For those of you consuming the content in a reader, or visiting the blogs directly, this approach works well. Our analysis is centralized and made available for easy consumption, obviating the need to visit multiple properties.

While that worked well for the blogs, however, it was less ideal for the homepage. Given that the majority of the audience to the homepage are people that know little about us, let alone del.icio.us, the overpopulation of the homepage with entries like “links for 2007-04-18” was potentially intimidating, and certainly aesthetically displeasing. Alex had mentioned this over lunch a few weeks ago, James had surfaced this as a To Do, and I’ve gotten a steady trickle of similar comments since we launched.

There was no question that those links could be filtered – it’s all XML, after all. The question was what represented the simplest approach. The initial thought was to do it server side, stripping out anything from the “Links” category on the fly prior to display on redmonk.com. But that would require – potentially – some coding changes on our WordPress/parser setup, and I simply hadn’t had the time to look into it. While driving back to Maine on Monday, however, it struck me that this was precisely the kind of thing Yahoo Pipes was designed for, so I made a mental note to see if I could make it work when I got back to the office.

Well, make it work I did. To see it in action, go hit redmonk.com. The better news was that it literally took not more than 10 minutes to appropriately filter all three of our feeds. As you can see from the Pipe – or the inline picture – the setup is dead simple. Pipes reads in my tecosystems feed from FeedBurner (Sources: Fetch Feed), looks for items that have “links” in the title item, filters them (Operators: Filter), and delivers me a neatly stripped RSS feed. After repointing the parser on redmonk.com from FeedBurner to our newly created Pipes feeds, redmonk.com is now delivering a homepage free of del.icio.us links, while the individual blogs continue unaffected. Beautiful.

The point to emphasize here, as far as I’m concerned, is the efficiency. Fixing this via the original approach was far from an arduous task, but it would still have taken me probably an hour or two (remember, I’m not the sharpest tack in the box) to figure out what’s being done where and test the filtering. With Pipes, it took less than 10 and I was done; apply the filter, look at the output, cut over the feeds. The remaining question now is the reliability of the Pipes supplied feed; FeedBurner’s only been down once since we cut over to the new redmonk.com. If Pipes can supply similar uptime, I’ll be a very happy camper.

With the overwhelming majority of our content available in one syndicated form or another, I can see an increasing demand for Pipes style filtering and recombination. I doubt, in other words, that this particular business problem is the last one I’ll solve with Yahoo Pipes.


  1. stepho you are a king!

  2. Pretty easy to do just by adding parameters to your WordPress feeds without changing anything in WordPress itself. First, in WordPress, go to the Categories pages and find the number of the category for your links posts.

    For example, say that the WordPress install for your blog is at http://www.redmonk.com/sogrady and the category number for your links posts is 12. The following feed will give you everything except category 12 (links):


    You can use that as input to Feedburner to create a feed there, or subscribe to that feed (with the parameters) directly.

  3. An easier way to do this, perhaps, if WP supports fancy-pants tagging capabilities, would be to arrange to have your links entries tagged with something like “links”. Then hope you could get a feed with selectable tagged entries (inclusion and exclusion, obviously, in this case). Then you could imagine a feed url sufffixed with the tag predicate (“!links”), and it’d give you what you need.

    I bug Dave Johnson about tagging capabilities in Roller every so often (since I use it, since developerWorks uses it), and I think it can do this sort of thing.

  4. Be careful about having too loose of a match. It’s fairly plausible that one of you will at some point have a regular post containing “links” in the subject.

  5. If you’re worried about Pipes’ uptime (which seems to be fine at the moment) you could always pipe the output from Pipes back through FeedBurner – that way if Pipes goes down FeedBurner will still serve up the last copy of the feed.

  6. I would echo Donnie’s comment. I had a similar issue, but I wanted to set up Pipes to filter based on the category of a post. This seems to work when there is a single category for an item, but I haven’t got it to work when there are multiple categories for an item. I haven’t found anyone yet who can point out to me what I’m doing wrong, so I’ve gone for the “filter based on title approach” as well. But that sucks; what’s the point of metadata if you can’t use it?

  7. Another option for you (and for you, James) is to take more control over the feed parsing by doing it on your own local machine or server. You should take a look at RSSBus, I believe you will find it useful.

  8. Ahhh, you’re using Pipes for something I use Planet for server-side. It’s an interesting approach, pushing the parsing load out over to Yahoo! … and one I may consider on another site that I really need to have pulling in such a fashion. Hmmmm.

  9. […] controversy, while offering up my thoughts on the loss of Kurt Vonnegut. On the technical front, I experimented with Yahoo Pipes by using it to filter links posts out of the redmonk.com homepage. In the face of […]

  10. You can use that as input to Feedburner to create a feed there, or subscribe to that feed (with the parameters) directly. Emlak musavirleri haber sitesi

Leave a Reply

Your email address will not be published. Required fields are marked *