Realtime Search Explained
Timeliness was always an issue for Google. If your bots are slow, opportunity is missed. Each time you miss, fewer and fewer people come back to you for information. I talked to a Bing engineer last year–same deal, their goal is speed too. To get fast bots you make sacrifices. Think about it, analysis takes time, time that search bots can’t afford, they need to move move move! This is why search engines can’t do much beyond exact matches.
Imagine a major event. I would most likely hear about it first on Twitter: natural disaster, terrorism, product release, security update, whatever. Google knows this.
Google’s bots are faster than ever, last I checked they index my blogs every few minutes. But indexing Twitter.com page by page (whatever leeway Twitter gave Google previous to the multi-million dollar deal) wasn’t fast/comprehensive enough. Without Twitter’s firehose, Google was one more mile behind the millions of API-slurping humans–faster, smarter, more interesting than algorithm-driven intelligent agents.
OK, I do follow one bot on Twitter, but it’s a human-guided bot: Techmeme’s Twitter firehose.
When I get info from Twitter first, that’s less time I spend searching Google. That’s it.