Bronte Media

While there may be some ‘creases’ to be ironed out, crawling is a fundamental part of today’s web.

So for job name of the day: Crawlmeister at Krugle. I love it! There will be a growing demand for these type of people and let’s hope the name catches on.

Crawling is the inefficiency that APIs take away. But since most everything does not have an API, crawling is the way to glean information on the web for now. Like anything technology related you can use it for bad (arbitrage, made-for-adsense sites, copyright infringement etc.) or good (like say as a piece of a puzzle to help someone to make a better decision at Homethinking ).