Comments on: Uncle, Mercy, Whatever: Please, Just Kill the Dupes http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/ because technology is just another ecosystem Sun, 13 May 2012 00:23:29 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Bob Wyman http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-659 Bob Wyman Thu, 28 Apr 2005 13:27:07 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-659 The problem seems to be the way that ads are being inserted into the entry you are getting multiple copies of. We're working to see what we can do to work around this example of really bad ad insertion practice. It appears that the Industry Standard is serving ads from Doubleclick and the method they are using is simply wrong. What is happening is that the ad links are changing on a regular basis and thus our "duplicate detection" code thinks that the entry has changed. This will impact any post from an Industry Standard feed with ads in it. The fact that you're using PubSub is not relevant here. Just about any aggregator will get confused by posts that have changing content like this. Because of weaknesses in the RSS specification that are being addressed in the new Atom format, aggregators are currently forced to do textual analysis in order to detect changes to RSS posts. We can't simply rely on things like the optional RSS GUID when checking for duplicates. In the future, Atom will give us required unique IDs for entries and that will make it easier to detect duplicates... But, for now, we're stuck with a lot of old-style RSS feeds... Given this, it is vitally important that when someone inserts ads in RSS items, the text of the ad links *must not change* once the ad is inserted. The only alternative would be for us to build special code that recognizes the ads being inserted by each of the many advertisers and handles them specially. Personally, I don't think that is reasonble, however, if we only have a few "bad apples" (like DoubleClick), it might be workable for the short term until people start moving to Atom. I'll write more on this on my blog later today. See: <a href="http://bobwyman.pubsub.com/">http://bobwyman.pubsub.com/</a> bob wyman The problem seems to be the way that ads are being inserted into the entry you are getting multiple copies of. We're working to see what we can do to work around this example of really bad ad insertion practice.

It appears that the Industry Standard is serving ads from Doubleclick and the method they are using is simply wrong. What is happening is that the ad links are changing on a regular basis and thus our "duplicate detection" code thinks that the entry has changed. This will impact any post from an Industry Standard feed with ads in it. The fact that you're using PubSub is not relevant here. Just about any aggregator will get confused by posts that have changing content like this.

Because of weaknesses in the RSS specification that are being addressed in the new Atom format, aggregators are currently forced to do textual analysis in order to detect changes to RSS posts. We can't simply rely on things like the optional RSS GUID when checking for duplicates. In the future, Atom will give us required unique IDs for entries and that will make it easier to detect duplicates… But, for now, we're stuck with a lot of old-style RSS feeds… Given this, it is vitally important that when someone inserts ads in RSS items, the text of the ad links *must not change* once the ad is inserted. The only alternative would be for us to build special code that recognizes the ads being inserted by each of the many advertisers and handles them specially. Personally, I don't think that is reasonble, however, if we only have a few "bad apples" (like DoubleClick), it might be workable for the short term until people start moving to Atom.

I'll write more on this on my blog later today. See: http://bobwyman.pubsub.com/

bob wyman

]]> By: James Governor http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-658 James Governor Thu, 28 Apr 2005 10:07:58 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-658 might this be something to do with industry standards feed? normally technorati does this, not pubsub so much might this be something to do with industry standards feed? normally technorati does this, not pubsub so much

]]> By: James Governor http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-657 James Governor Thu, 28 Apr 2005 06:08:57 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-657 the worst thing - i hate the quote in that article!!! i mean coldfusion is one thing i mentioned. now like ten times a day the story comes leaping out at me and i think why didn't i say something more insightful.... and you begin to imagine that *everyone* keeps seeing the same story over and over, even though its just hitting me because of my vanity feed... the worst thing – i hate the quote in that article!!! i mean coldfusion is one thing i mentioned. now like ten times a day the story comes leaping out at me and i think why didn't i say something more insightful….

and you begin to imagine that *everyone* keeps seeing the same story over and over, even though its just hitting me because of my vanity feed…

]]> By: Edward O'Connor http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-656 Edward O'Connor Wed, 27 Apr 2005 20:31:48 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-656 Of course, a few hours after I left that comment I get four bajillion dupes in several feeds which I've checked with `ignore', so it's not the be-all end-all solution by any means. Of course, a few hours after I left that comment I get four bajillion dupes in several feeds which I've checked with `ignore', so it's not the be-all end-all solution by any means.

]]> By: sogrady http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-655 sogrady Wed, 27 Apr 2005 16:12:06 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-655 i can't believe i never thought of that; i'm an idiot. either way, bless you sir. i can't believe i never thought of that; i'm an idiot. either way, bless you sir.

]]> By: Edward O'Connor http://redmonk.com/sogrady/2005/04/27/uncle-mercy-whatever-please-just-kill-the-dupes/comment-page-1/#comment-654 Edward O'Connor Wed, 27 Apr 2005 13:16:16 +0000 http://redmonk.com/sogrady/wp/?p=416#comment-654 When I subscribe to a feed in Bloglines, I start with the "Display updated entries" option. If the feed passes some dupe threshold of annoyance, I change that to "ignore updated entries." This seems to take care of 90% of this problem for me. When I subscribe to a feed in Bloglines, I start with the "Display updated entries" option. If the feed passes some dupe threshold of annoyance, I change that to "ignore updated entries." This seems to take care of 90% of this problem for me.

]]>