When Search splinters, will large pools of the Web go dark?

The point at which one competitor in the market begins to achieve unassailable dominance is the moment when the seeds of change are sown. Search is about to change, you can feel it in the air.

You can measure the quality of Google’s search results by searching for something and reviewing the usefulness of the first two pages of results. For example, the first result for the query “search engine” on Google is a link to “Alta Vista.” Google also indicated that there are 118,000,000 links in the result set. I couldn’t find any simple way to find the last result, the link that Google ranked as the lowest in importance. But since users rarely look beyond the second page of search results, all the rest is a puppet show. The business of Search is the quality of the first two pages of search results. For that search, the only link of interest was to Wikipedia, and Google itself only showed up as its UK site on the second page.

In a sense, this is why Mahalo can “compete” with Google. Mahalo doesn’t need to index the whole web and come up with 118 million links. No one cares about 118 million links. There’s a small consideration set that actually satisfies the query.

And further, a page of links is just a page of pointers, it’s the content that answers the question. This is why Mahalo is now offering a higher content to link ratio; it can be an endpoint rather than a relay station. The attack surface revealed is the understanding what is truly human readable and what satisfies a search query.

“Search” could be disrupted by many approaches: we want a better starting point that links to the thing we’re looking for. Twitter or Delicious could be pointers towards that new thing — Search as a back and forth conversations within a tribe, and contiguous tribes; Search of a subset of pages users cared enough about to bookmark (user gestures). The citation algorithm was a huge step forward in ranking the value of pages based on a keyword search. Citation is no longer enough, as Ray Ozzie notes, users now commonly link, share, rank and tag.  Currently search is anonymous, connecting to a preference set or a user profile could yield more valuable results.

The rise of specialized search raises the specter that some day the entire web will no longer be spidered and indexed. The economics of search are tied to a subset of search queries related to potential commercial transactions. Commercial search subsidizes all other search activity. At some point, that linkage will be cut. As search splinters and begins to operate in verticals, much of the web could go dark.