Revisiting @cshirky : Overload and Filter Failure

Картини

Clay Shirky may have done us a disservice when, looking at the torrent of information published to the Network, he turned the problem upside down and said it’s not information overload, it’s filter failure. His comment was a response to the notion that the solution to information overload is to use professional editors to filter out items of inferior quality before they are inscribed into the Network. This approach would be consistent with the historical practices of the publishing industry; and theoretically would result in fewer items tumbling into the bin of our cognitive load. Traditional models of publication are a statement about the quality and value of a work. If everything is always published in real time—every gesture, tick and passing fancy— publication is no longer a measure of quality, but merely a recognition of the act of inscription on the surface of the Network.

The purpose of the filter is to rummage through the “everything” that has been published—and is continuously published in real time—and come up with an appropriately-sized load. That is, something less than an overload, and certainly not an under-load, but rather a load that feels just right. Shirky inaugurated the era of “filter failure” in September of 2008; in the subsequent years the volume of items published to the Network has accelerated exponentially. However, we don’t seem to have made much headway with the “filter” problem.

We have a sense of what a “filter” looks like when it works as a gatekeeper standing between all that is written and what is eventually published. The editor and publisher make judgements based on commercial prospects, artistic merit, a writer’s track record, et cetera; and then select which pieces of writing will pass through the publication process. A “filter failure” in this context is a publication event that doesn’t find an audience willing to support or engage with it.

What does a “filter” look like when everything has already been published? Isn’t the horse already out of the barn? Early in Twitter’s existence, they placed a configurable filter on the publishing nozzle of the service. A user could select a set of keywords, the filter checked through all the tweets in real-time, and then it would spray a filtered set of tweets into the regular set of subscriptions. As the volume, velocity and user-base increased, it was technically unsustainable on a real-time basis. Twitter search now consists of keyword filtering of a smaller set after the fact. A number of services have rented Twitter’s firehose of messages in an effort to affix real-time filters over the publication nozzle. None have emerged with a solution to put an end to the era of “filter failure.”

From the perspective of editorial process, affixing a filter to the real-time publication nozzle is consistent with previous editing processes. The editor/user selects keywords that narrow the field of output prior to the publication of a real-time stream. The filter that Shirky proposed operates after the fact, everything has always already been published and this filter sifts through it all and scoops up only the good stuff. Wheat is separated from chaff. In this case the timing of the filter is less important than the quality and scope of its mesh. The filter should construct a value equivalent to, or greater than, that of a pre-filtered output, by assembling interesting bits and pieces it finds laying around. Because it’s less concerned with the now of real-time, Shirky’s filter can include elements from different periods of time based on their relevance, importance and overall quality.

In some recent attempts to crack the “filter” code, the word ‘curator’ is substituted for ‘editor’ and the blend of the publication is expanded to include both domestic and foreign products. Most traditional publications are reconstituting themselves along these lines. Another approach is to draw a circle around a set of curators and writers to create a tele-publication. To maintain and grow its value, the portfolio must be actively managed, occasionally rebalanced and look for opportunities in the event stream. (Global tactical asset allocation mutual funds are often managed in this fashion.)

A filter could be constructed correctly, be technically sustainable at scale, and still be a failure. If the mesh of the filter is configured incorrectly, its output may be the correct “load” size, but its contents of inferior quality. The mechanics of filtering are only half of the equation. What should the filter filter, and who decides what that is? Some say we don’t want an editor to serve as a gatekeeper, we prefer an emergent (another word for average) publication of crowd-sourced filtering. The odd thing about a spontaneously generated crowd-constructed publication is that it usually feels like you’ve already read it. The thing about what everybody knows is that everybody already knows it.

Assuming we could produce a technically-sound filter, we would need to configure its mesh. It’s possible one could go shopping for a mesh. A filter’s mesh could be an external product selected to produce a particular kind of output. Our choices range from the hand-crafted to the automated mesh. Techmeme is a hybrid of algorithmic and hand-crafted output. Google News has recently added a hand-crafted element to their largely algorithmic output. In a sense, this isn’t very different from the way an ink-on-paper magazine works.

One of the dangers of going shopping for a mesh is that one can easily end up in an echo chamber. By selecting only agreeable elements, one’s own bias seems to be confirmed by external sources. When you mix in hard-core ideology, a strange reversal takes place. As William Burroughs once said, “you don’t sell heroin to people; you sell people to heroin.” What at first appears to be looking in a mirror and becoming more and more beautiful, is really a process of the mirror surrounding and consuming you, until you become a part of its reflection. The external ideology has hollowed you out and takes up residence in the void.

If the era of filter failure were to end, would we have filters that were an uncanny match to our thoughts and desires? Would the filter take into account our conscious and unconscious selves? Would it know what to us is a set of unknown knowns? Would we be embarrassed when the filter mixed in objects of desire of which we dare not speak? Is it merely a matter of getting in tune with our true desires and affinities? Or should the mesh of the filter bring me more than I contain? How far do we need to take this?

After all, how perfect does a filter need to be before we can consider it a success? Perhaps all that has to happen is for the feeling of being overwhelmed to go away. Maybe that just happens with time and exposure. Instead of waiting for a future paradise where filters don’t fail and our minds are constantly blown by how perfectly tuned and relevant every single thing we encounter is; perhaps we should acknowledge that the future will be a lot like the present. Sturgeon’s law will still hold and 90% of everything will still be crap, spammers will still manage to show us advertisements we don’t want to see, and sometimes we’ll still feel overwhelmed by life.

The more we search for a fine-grained solution to filter failure, the stranger the “I” for whom the filter must not fail becomes.

At first the big flood of information seems to be fascinating. It has all the formal qualities of something that should demand our attention. It’s only after we’ve sat patiently and listened to it for a while that we realize how boring it can be. Boredom with the torrent of information may be the first step toward forgiving filters their failures.

“To approach the Other in conversation is to welcome his expression, in which at each instant he overflows the idea a thought would carry away from it. It is therefore to receive from the Other beyond the capacity of the I, which means exactly: to have the idea of infinity. But this also means: to be taught. The relation with the Other, or Conversation, is a non-allergic relation, an ethical relation; but inasmuch as it is welcomed this conversation is a teaching. Teaching is not reducible to maieutics; it comes from the exterior and brings me more than I contain. In its non-violent transitivity the very epiphany of the face is produced.”

— Emmanuel Levinas (Totality and Infinity: An Essay on Exteriority)