You’ve Got Ismail!

The Search Problem

January 20th, 2006

Yesterday, I talked about the evolution of the internet and that we’re entering the third decade of the internet (the publish/subscribe era).

The Two Halves of Search
As an exercise, lets apply these information exchange paradigms against the search problem and see how that plays out.

Most search engines today are based on a request/response paradigm.   This is also known as Retrospective Search.   Their spiders crawl around, gather information, put it in a bucket and let you fish around in it – you’re asking ‘what have you found in the past‘.

Search using publish/subscribe is called Prospective Search.   This is the other half of search – you’re saying “ok, I’ve seen what’s been said in the past, now tell me whenever it appears in the future”.   The two are complementary.   Bob Wyman, as usual, has written the definitive post on this subject.

If you’re doing research on the Civil War, you probably don’t need prospective search – most of what exists has already been said.   However, if you want to know the instant U2 tickets go on sale near you, or an apartment comes available in New York, you really need Prospective Search.   There’s no point finding that on Google a month later.

We have spent the last 30 years getting fully to grips with retrospective search.   This next few years will be about implementing the other half.

The one domain where publish/subscribe has been fully implemented is the financial markets.   If you think about it, most stock market systems operate in this paradigm.   They’re constantly demanding “tell me whenever IBM stock goes above 80″ or “whenever that ratio reaches X”.   In that industry, the ROI has existed to build billion dollar systems to answer those questions quickly.

Why Bloggers are Great
Bloggers are all about Prospective Search.    They all want to know and track themselves and others.   As I mentioned in my last post, blogs are interesting partly because of the ease of publishing, but mostly because one can be notified ‘whenever’ a feed is updated, allowing us to ‘watch’ what’s being said.   Blogging is clearly the thin end of the wedge that will drive the implementation of the ‘active’ web.   For obvious reasons, PR is a domain where ‘listening’ is required.   Folks like Steve Rubel, Neville Hobson and Constantine Basturea have completely embraced the movement.   Brand managers and marketing departments are rapidly getting it.

The ripple effect of Bloggers is that they are laying the messaging down for the rollout of the active web.   Words like ‘feeds’, ’syndication’, ‘aggregation’, ‘instant awareness’ and ‘push’ are moving into the mainstream (let’s ignore ‘trackback’, shall we?).   A couple of years ago, these were largely unknown.   I wonder what words we’ll be using in another two years?

And of course it’s still very early days.   The blogosphere is still a very new ecosystem and will take time to settle down.   And while blogs are great, they aren’t fully ‘active’ yet.   Most old blog posts are largely useless, so the tail is being dropped, but blogs aren’t real time.   Note that RSS, though a publish/subscribe pattern, is based on a polling (request/response) architecture.   Jon Garfunkel over at Civilities is asking some great forward looking questions about this and questioning whether RSS can scale.   (It can’t, but that’s another post).

In my next post, I’ll talk about the implications of activating the web to information streams everywhere.   I’ll also tackle Scott Moody’s thoughts (promise!)

Entry Filed under: Internet & Technology