Bronte Media

There has been much discussion about Craigslist banning Oodle, a classifieds vertical search site, from crawling its listings. Craig Donato, the CEO of Oodle, originally highlighted the point on the company’s blog and has made some very well articulated follow up points.

Pamela Parker wrote a great piece today that brings a wider context to the story. And so did Matt Marshall at the San Jose Mercury News. I made a few comments on their blog post about the article and so did Chris DiBona of Google.

My overall point is that it is absolutely ok for site operators not to be included in search indexes. The best way to make that wish is through the robots.txt file. Craigslist decided to make that wish through their terms of service, and stipulates that a commercial site must not have more than 100 links to Craigslist unless they have express persmission to do so.

Again, that’s fine. Their wish. But the consequence should be that every search site should have to be made to stop crawling Craigslist; not just Oodle. Pamela quotes Jim Buckmeister, CEO of Craigslist, as saying that Google and Yahoo contribute less than 5% of their referrals, which I find hard to believe. But in any case that volume of referrals should disappear. Take the good with the bad consequences.

I really think this is an issue of the number of crawlers (and as a function vertical search companies) out there and a very fuzzy standard of politeness (there is a yes or no mechanism but no guidelines as to how much is enough). That would solve a lot of the problems here. In the end, it’s sad that vertical search sites are hampered from providing additional marketing channels that are by and large free now for media firms.