Bronte Media

If you want to make search bad, call it scraping. If you want to make it sound good, use crawling.

The interesting nuance has become even more pronounced during the furor over Craigslist not allowing Oodle to crawl its listings.

In the comments of a Silicon Beat post Dave McClure of Simply Hired articulates a great definition of each, which I would unequivocally agree with. But people are using it in a tabloid news way to further a point of view.

Tom Foremski, a former Financial Times journalist, says that crawlers add little value and take up lots of resources. After analyzing his log files he comes to the conclusion that “The search-and-scrapers sucked out one-third of my bandwidth and provided just 3.7 percent of the traffic!”.

Tom makes the horrible mistake of thinking that one page of crawling equals one page of viewing. Firstly, and most obviously, a human referred to by the ’search-and-scrap’ sites is likely to look at more than one page.

Is it more than 10 pages? That is also a futile argument. The cost to serve a page is miniscule compared to the monetization rate of each page. Even with the most crappy of techniques people should be able to get $2 CPM, which equates to .2 cents per page. It does not cost anywhere near .2 cents to serve a page. The rate of bandwidth cost is also declining at a Moore’s-law-like rate.

So even with a 3.7% referral volume and one third bandwidth, Tom still ends up on top by a mile. The fact that he only has 3.7% referral volume from the search engines should also worry Tom, as that is markedly below what you would normally expect.

There is also the delicious irony that Tom searches and scrapes his own site. Everyone who has an RSS feed does. What it is essentially doing is creating a feed of summaries that link back to the main site. Tom even includes the full text in his RSS feed. Don’t get me wrong, I read and enjoy his musings every day.

RSS has the potential to radically change the crawling world. Crawlers can essentially subscribe to feeds (Oodly should do this in the absense of not being allowed to crawl Craigslist). A better pinging and notification mechanism is needed to cut down on the bandwidth so that the crawlers don’t have to dumbly guess when to come back but I have faith that will be solved.

In the meantime, ignorant arguments over scraping and stealing will continue to be made, however in jest they may have originally intended to be.

From Sumner Redstone of Viacom:

“I never said I would take my hands off the wheel,” Redstone, 82 said in an interview at his home in Beverly Hills, California. “I said I would share the wheel” with the two companies’ chief executive officers, Tom Freston and Leslie Moonves.

and

Redstone said he has no plans to retire and would like to continue working for another 20 or 30 years.

“I don’t intend to go,” he said. “Though I guess I have come to terms with the idea that I’m not immortal.”

He guesses?

Tribune today announced that it may seek the sale of certain non-core assets to concentrate on its core business.

One of those assets under consideration is its 33% stake in Careerbuilder.com. What a bone headed idea! Sell off the one business that has been doing amazingly well in recent times and has the opportunity to drive a huge part of their earnings growth in the coming years while holding on to flat revenue growth and double digit cost growth newspapers.

Tribune also said they didn’t want to sell the LA Times. Maybe that’s because no one would buy it?

Kevin Newcomb over at ClickZ has a nice writeup of a partnership agreement between Jambo, a pay per call provider and Infospace, a leading local search company.

Jambo, formed by some ex-Netzero executives, raised $5m last week to compete with Ingenio and blaze the pay per call frontier.

They have a network similar to Ingenio as well. But they plan to offer call tracking on all results and as the network generates calls from merchants, follow up with an introduction. Or Lukewarm calling, as I’ll term it. We are planning a similar sales rollout with Homethinking (a preview of which should be out in the next month or so, by the way).

Jambo might have a difficult time managing what the whisper will be (will it be Jambo or the network partner, for example, Infospace, that is whispered to the merchant as the call is being connected?).

The notion of ‘organic’ calls is a good one though (in my heavily biased opinion, of course). As the cost of tracking falls (Jambo’s entry and others in the near future will inevitably force the price down like in any analytics technology market) and VoIP driving the cost of carriage toward zero, that becomes feasible. In the meantime, I’m taking the view that it is great marketing spending. Proving the value and credibility of your service to prospective advertisers is money well spent. And just a little less evil than cold calling

There has been much discussion about Craigslist banning Oodle, a classifieds vertical search site, from crawling its listings. Craig Donato, the CEO of Oodle, originally highlighted the point on the company’s blog and has made some very well articulated follow up points.

Pamela Parker wrote a great piece today that brings a wider context to the story. And so did Matt Marshall at the San Jose Mercury News. I made a few comments on their blog post about the article and so did Chris DiBona of Google.

My overall point is that it is absolutely ok for site operators not to be included in search indexes. The best way to make that wish is through the robots.txt file. Craigslist decided to make that wish through their terms of service, and stipulates that a commercial site must not have more than 100 links to Craigslist unless they have express persmission to do so.

Again, that’s fine. Their wish. But the consequence should be that every search site should have to be made to stop crawling Craigslist; not just Oodle. Pamela quotes Jim Buckmeister, CEO of Craigslist, as saying that Google and Yahoo contribute less than 5% of their referrals, which I find hard to believe. But in any case that volume of referrals should disappear. Take the good with the bad consequences.

I really think this is an issue of the number of crawlers (and as a function vertical search companies) out there and a very fuzzy standard of politeness (there is a yes or no mechanism but no guidelines as to how much is enough). That would solve a lot of the problems here. In the end, it’s sad that vertical search sites are hampered from providing additional marketing channels that are by and large free now for media firms.

When McDonalds installed Wifi in 6,000 of its locations, I thought it was the most bone headed idea ever. I was wrong. Very wrong.

I thought that since business travelers with Big Mac smudged fingers wouldn’t be pulling out their Thinkpads in Mickey D’s then it was a bad idea. How narrow sighted of me.

Nintendo announced that it would offer free Wifi for its Nintendo DS users in McDonalds. The deal let’s the kids play Mario Kart against each other.

What a smart move. This will no doubt add steroids to the McDonald’s party rooms over the nation. Super-sized ethical concerns or not, it’s a great marketing ploy for both Nintendo and McDonalds.

Paul Graham, of hackers and painters essay fame, organized an excellent conference at Harvard this past weekend.

As well as himself, he gathered some of the best speakers I have seen, including Steve Wozniak, who received a standing ovation before and after speaking.

I wont cover all of the speeches, if you’d like that visit the notes of one studious attendee and they will soon have video and/or podcasts of the talks up at the site.

A few random observations:

* Paul Graham was a lot different to what I imagined him when reading his essays. His speech was very much like an essay, which was good, but wasn’t very interactive, which was bad. He could almost have fielded 30 minutes of questions about his writing.

* Chris Sacca of Google basically offered everyone in the room a job. He said that after startups of 2-3 people had something to demo to contact him and that Google would acquire the company and give them the infrastructure to scale their application. Interesting to see how that plays out. He also said he was the product manager for Google Talk, bought Dark Fiber and headed up Google’s Wifi effort. Sergey and Larry call him the “Shephard of Stray Things”. You can either read something strategically important into Talk, bandwidth and Wifi being in the same group or alternatively read something into the fact that they are all in the “stray things” group.

* Steve Wozniak was fantastically humble. He really is just focused on creating great products. Thoroughly refreshing to see someone like him speak.

* I didn’t realize how great Trip Advisor had done. Langley Steinert, who founded the company, opened up the day. They were extremely capital efficient (raised less than $5m), operated in one of the toughest travel markets in history, managed to do about $50m in revenue last year and sold to Interactive Corp. Absolutely amazing.

* Olin Shivers, Associate Professor, Georgia Tech; Co-Founder, Smartleaf, is one of the funniest speakers ever. Go and read the acknowledgements page of an API he wrote. Seriously, it is some of the funniest shit you will read in a computer manual. His speech was just as good.

Interesting article on 180Solutions having to fire 20% of its staff because of the distribution network being cleaned up and flat usage (surprise!). Still $50m/yr is nothing to scoff at.

Maybe I just wanted to write that headline but people should really stop thinking about what a combined AOL-Google-Comcast or AOL-MSN might look like and think a little shorter term.

Out of all of this the only person who will end up ahead is AOL (the whore). Here is the game theory run down:

Carl Icahn is pressuring the Time Warner board over its share price, wanting it to spinoff its cable division and maybe AOL.

The way to make Carl go away is to lift the share price of Time Warner. This will let him make money and the shareholders who are doggedly holding on to the stock. By whoring out AOL to any and every suitor they have a very good chance of lifting the share price without even selling it. They can do this by ‘establishing’ a fair price of AOL amongst the industry rather than Wall St.

If one of the suitors is silly enough to make a too-much bid (valuing AOL media at anything more than $8-10bn for instance) then they can take the money and run. And only if they still have control of AOL.

MSN, Google and Comcast have no chance of winning. They only way they will ‘win’ is if they will over pay. A classic winner’s curse. Once they make a credible bid, then the Time Warner stock price will rise and Richard Parsons can tell them to go away or add in even more onerous terms.

The only thing Time Warner cares about is getting its stock price up to ‘fair value’. If Microsoft, Google or Comcast need to used in that endevor then so be it. Carl, who actually serves as a useful capitalism mechanism, then goes away. And then AOL succeeds or fails on its own merits.

On Tuesday night I went along to the New York Tech Meetup for the first time. Thoroughly enjoyable event all round.

The meetings are a series of 5 minute showcases and on Tuesday there was a “sell-side advertising network” (ed: aren’t these just bloggified affiliate marketing networks? CJ PR people take note), a Citysearch-like Wiki (ed: Opinion does not work well in a consensus based technology. See: LA Times. Guy who started is CTO of Meetup), Meetro, Expertflyer (ed: basically a raw interface into the GDS systems. Aimed at the business traveler but they are likely to have mandates on where they have to book travel through. Charges a monthly fee which may negate this) and last but not least an interactive chocolate company (ed: err, I am not sure either but she passed around samples and writes a Mousedriver-like weblog chronicling their small business travails).

After the hour of demos the crowd breaks out into networking. Scott Heiferman, the meetup organizer and meetup founder, kindly hosts the event at the meetup offices at 632 Broadway (which randomly were nearly the corporate home of yours truly if not for another tech company delaying their new office move a few months). Meetup has a whole floor and there is room for about 80-90 people. Scott did a great job ensuring everyone (~50 people) introduced themselves and also said if they needed anything (in most cases PHP developers).

Thoroughly recommended and will definitely be going next month too.