Category: value

Revisiting @cshirky : Overload and Filter Failure

Published August 14, 2011 by cgerrish

Clay Shirky may have done us a disservice when, looking at the torrent of information published to the Network, he turned the problem upside down and said it’s not information overload, it’s filter failure. His comment was a response to the notion that the solution to information overload is to use professional editors to filter out items of inferior quality before they are inscribed into the Network. This approach would be consistent with the historical practices of the publishing industry; and theoretically would result in fewer items tumbling into the bin of our cognitive load. Traditional models of publication are a statement about the quality and value of a work. If everything is always published in real time—every gesture, tick and passing fancy— publication is no longer a measure of quality, but merely a recognition of the act of inscription on the surface of the Network.

The purpose of the filter is to rummage through the “everything” that has been published—and is continuously published in real time—and come up with an appropriately-sized load. That is, something less than an overload, and certainly not an under-load, but rather a load that feels just right. Shirky inaugurated the era of “filter failure” in September of 2008; in the subsequent years the volume of items published to the Network has accelerated exponentially. However, we don’t seem to have made much headway with the “filter” problem.

We have a sense of what a “filter” looks like when it works as a gatekeeper standing between all that is written and what is eventually published. The editor and publisher make judgements based on commercial prospects, artistic merit, a writer’s track record, et cetera; and then select which pieces of writing will pass through the publication process. A “filter failure” in this context is a publication event that doesn’t find an audience willing to support or engage with it.

What does a “filter” look like when everything has already been published? Isn’t the horse already out of the barn? Early in Twitter’s existence, they placed a configurable filter on the publishing nozzle of the service. A user could select a set of keywords, the filter checked through all the tweets in real-time, and then it would spray a filtered set of tweets into the regular set of subscriptions. As the volume, velocity and user-base increased, it was technically unsustainable on a real-time basis. Twitter search now consists of keyword filtering of a smaller set after the fact. A number of services have rented Twitter’s firehose of messages in an effort to affix real-time filters over the publication nozzle. None have emerged with a solution to put an end to the era of “filter failure.”

From the perspective of editorial process, affixing a filter to the real-time publication nozzle is consistent with previous editing processes. The editor/user selects keywords that narrow the field of output prior to the publication of a real-time stream. The filter that Shirky proposed operates after the fact, everything has always already been published and this filter sifts through it all and scoops up only the good stuff. Wheat is separated from chaff. In this case the timing of the filter is less important than the quality and scope of its mesh. The filter should construct a value equivalent to, or greater than, that of a pre-filtered output, by assembling interesting bits and pieces it finds laying around. Because it’s less concerned with the now of real-time, Shirky’s filter can include elements from different periods of time based on their relevance, importance and overall quality.

In some recent attempts to crack the “filter” code, the word ‘curator’ is substituted for ‘editor’ and the blend of the publication is expanded to include both domestic and foreign products. Most traditional publications are reconstituting themselves along these lines. Another approach is to draw a circle around a set of curators and writers to create a tele-publication. To maintain and grow its value, the portfolio must be actively managed, occasionally rebalanced and look for opportunities in the event stream. (Global tactical asset allocation mutual funds are often managed in this fashion.)

A filter could be constructed correctly, be technically sustainable at scale, and still be a failure. If the mesh of the filter is configured incorrectly, its output may be the correct “load” size, but its contents of inferior quality. The mechanics of filtering are only half of the equation. What should the filter filter, and who decides what that is? Some say we don’t want an editor to serve as a gatekeeper, we prefer an emergent (another word for average) publication of crowd-sourced filtering. The odd thing about a spontaneously generated crowd-constructed publication is that it usually feels like you’ve already read it. The thing about what everybody knows is that everybody already knows it.

Assuming we could produce a technically-sound filter, we would need to configure its mesh. It’s possible one could go shopping for a mesh. A filter’s mesh could be an external product selected to produce a particular kind of output. Our choices range from the hand-crafted to the automated mesh. Techmeme is a hybrid of algorithmic and hand-crafted output. Google News has recently added a hand-crafted element to their largely algorithmic output. In a sense, this isn’t very different from the way an ink-on-paper magazine works.

One of the dangers of going shopping for a mesh is that one can easily end up in an echo chamber. By selecting only agreeable elements, one’s own bias seems to be confirmed by external sources. When you mix in hard-core ideology, a strange reversal takes place. As William Burroughs once said, “you don’t sell heroin to people; you sell people to heroin.” What at first appears to be looking in a mirror and becoming more and more beautiful, is really a process of the mirror surrounding and consuming you, until you become a part of its reflection. The external ideology has hollowed you out and takes up residence in the void.

If the era of filter failure were to end, would we have filters that were an uncanny match to our thoughts and desires? Would the filter take into account our conscious and unconscious selves? Would it know what to us is a set of unknown knowns? Would we be embarrassed when the filter mixed in objects of desire of which we dare not speak? Is it merely a matter of getting in tune with our true desires and affinities? Or should the mesh of the filter bring me more than I contain? How far do we need to take this?

After all, how perfect does a filter need to be before we can consider it a success? Perhaps all that has to happen is for the feeling of being overwhelmed to go away. Maybe that just happens with time and exposure. Instead of waiting for a future paradise where filters don’t fail and our minds are constantly blown by how perfectly tuned and relevant every single thing we encounter is; perhaps we should acknowledge that the future will be a lot like the present. Sturgeon’s law will still hold and 90% of everything will still be crap, spammers will still manage to show us advertisements we don’t want to see, and sometimes we’ll still feel overwhelmed by life.

The more we search for a fine-grained solution to filter failure, the stranger the “I” for whom the filter must not fail becomes.

At first the big flood of information seems to be fascinating. It has all the formal qualities of something that should demand our attention. It’s only after we’ve sat patiently and listened to it for a while that we realize how boring it can be. Boredom with the torrent of information may be the first step toward forgiving filters their failures.

“To approach the Other in conversation is to welcome his expression, in which at each instant he overflows the idea a thought would carry away from it. It is therefore to receive from the Other beyond the capacity of the I, which means exactly: to have the idea of infinity. But this also means: to be taught. The relation with the Other, or Conversation, is a non-allergic relation, an ethical relation; but inasmuch as it is welcomed this conversation is a teaching. Teaching is not reducible to maieutics; it comes from the exterior and brings me more than I contain. In its non-violent transitivity the very epiphany of the face is produced.”

— Emmanuel Levinas (Totality and Infinity: An Essay on Exteriority)

One Comment

Variations on Heraclitus

Published August 7, 2011 by cgerrish

We see the digital begin to stream on the Network and think we’ve seen something new. At what point did we start to see the analog stream? We call to things by name and they come from every quarter. It’s no longer enough that they stream; when we call, what comes? And what call do we make? We can focus on the medium over the message, and that will tell us something about our environment. But as background and foreground become compressed into a single mesh, we look more intently to the intimacy of the message.

Variations on Heraclitus

By Louis MacNeice (1961)

Even the walls are flowing, even the ceiling,
Nor only in terms of physics; the pictures
Bob on each picture rail like floats on a line
While the books on the shelves keep reeling
Their titles out into space and the carpet
Keeps flying away to Arabia nor can this be where I stood –
Where I shot the rapids I mean – when I signed
On a line that rippled away with a pen that melted
Nor can this now be the chair – the chairoplane of a chair –
That I sat in the day that I thought I had made up my mind
And as for that standard lamp it too keep waltzing away
Down an unbridgeable Ganges where nothing is standard
And lights are but lit to be drowned in honour and spite of
some dark
And vanishing goddess. No, whatever you say,
Reappearance presumes disappearance, it may not be nice
Or proper or easily analysed not to be static
But none of your slide snide rules can catch what is sliding
so fast
And, all you advisers on this by the time it is that,
I just do not want your advice
Nor need you be troubled to pin me down in my room
Since the room and I will escape for I tell you flat:
One cannot live in the same room twice.

Comments closed

High Fidelity in the Age of Digital Reproduction

Published July 31, 2011 by cgerrish

In his 1979 essay “The Studio as Compositional Tool“, Brian Eno works through the set of technical innovations that resulted in the odd occurrence of person who didn’t play any musical instrument particularly well, didn’t read or write music, nonetheless ending up as a composer. Eno lacked all the traditional tools of the trade. It was only when sound was mediated through recording that it became a plastic material that could be manipulated into song-like structures.

Here’s Eno on the transition from transmission to translation:

So, to tape recording: till about the late ’40s, recording was simply regarded as a device for transmitting a performance to an unknown audience, and the whole accent of recording technique was on making what was called a “more faithful” transmission of that experience. It began very simply, because the only control over the relative levels of sounds that went onto the machine was how far they were from the microphone – like device. The accent was on the performance, and the recording was a more or less perfect transmitter of that, through the cylinder and wax disc recording stages, until tape became the medium by which people were recording things.

The move to tape was very important, because as soon as something’s on tape, it becomes a substance which is malleable and mutable and cuttable and reversible in ways that discs aren’t. It’s hard to do anything very interesting with a disc – all you can do is play it at a different speed, probably; you can’t actually cut a groove out and make a little loop of it. The effect of tape was that it really put music in a spatial dimension, making it possible to squeeze the music, or expand it.

When we talk about a “more faithful” recording, the word “fidelity” enters the conversation. Fidelity is the quality of being loyal or faithful. Originally, it had the sense of taking an oath, as in swearing fealty to a monarch. Fidelity also has the sense of honoring oaths with regard to a spouse. A high-fidelity recording transports the original performance transparently—it is as though you are there. Poor fidelity dishonors the performance by leaving pieces of it behind or adding in artifacts that weren’t a part of the original. If we are a lover of a particular piece of music, we might charge a bad recording with infidelity.

In Eno’s recording studio, sounds become plastic. It’s only when a sound has been transformed into something that can itself be transformed that it becomes useful for constructing music. And this is the point where the sound no longer has fidelity to its source. The sound is only interesting to the extent of its potential infidelity. Transferring sound into a transformable recording media used to require a professional technical process. With digital recording, sound is directly sampled and encoded into a plastic media.

“I’d rather talk about the Plastic Eno Band, actually. It’s been in existence for a couple of years now. Over the past six years I’ve accumulated over 14 plastic musical instruments with a very wide gamut of sounds. And I’ve found that by slowing them down or speeding them up on tape, I can imitate any electric sound. With this in mind, I want to make a straight-forward rock record and then appear on ‘Top Of The Pops’ with a bunch of liggers playing these things. It would be an experiment in concrete music really as well as being an encouragement to all these kids who can’t afford their Vox amplifiers. There are so many things I want to do that will lose me so much money. . .”

As all media are slowly replaced with their digital equivalents, this shaky relationship with fidelity is true of more than just sound. Think about the camera and photography. How do we capture a scene with a camera? We see a moment we’d like to commemorate and we take aim with our camera. The flash from the camera floods the scene with enough light to get a good exposure. Here the process of recording essentially alters the source in the pursuit of fidelity. A skilled photographer may be able to light a scene for the camera such that when it’s processed, the photograph resembles the scene as it might have unfolded had no photograph been taken.

In the iPhone, the camera itself becomes a computerized photo studio and a compositional tool, in Eno’s sense. The photo itself is just the digital material that can be transformed with a set of filters. We don’t expect the snapshot to capture the mood; like the professional, we’ll fix it in post-production. We quickly apply a set of filters that more appropriately capture the mood of the scene and then flick the digital file into the stream of Twitter or Instagram. Is it the infidelity of the digital that enables another sort of fidelity? Or are we simply projecting the kind of scene we’d like others to imagine us playing a role within.

When we consider the picture being constructed of us through the data exhaust we emit in our online activities and our encounters with electronic and surveillance systems—does it make sense to talk about the fidelity, the truth, of the picture? Is the picture any more true because it’s constructed of largely unconscious digital moments? Is the ‘candid’ photo taken through a telephoto lens by a paparazzi of a movie star in their everyday life more true than the ‘glamour’ photograph constructed to create an image? When you apply for a job, do you present the candid or the glamour resume? How about applying for a loan at the bank, do you walk in the door with your candid or glamour finances?

In digital recording we have the production medium that is most open to transformation. In digital presentation, we have the consumption medium most open to transformation, both before we receive it, and after. Anyone with some form of computer has their own digital post-production facility. The blemishes can be removed, the wrong notes fixed and even the focal point of the image can be selected later.

If we were to imagine a medium that could somehow vouch for the fidelity of that which it recorded, it would be the opposite of the digital. This medium would capture the mark of the real and from that point forward it would be unalterable. In a strange way, in that moment, the mark would become more real than the real. The real itself would fade and change with time, but the mark would always have the vibrancy of the moment the impression was captured. In essence, this is the problem with using database models to stand in for real processes.

There was a time when to call something ‘artificial’ was to confer the highest compliment. The ‘real’ was a low form of existence that lacked the trappings of civilization. It was something that hadn’t been ‘fixed’ in post-production. The digital era has enabled new levels of artifice. The ‘real’ and the ‘natural’ may have to make way for the artificial. To ease the transition, the real and the natural will be the first things we need to simulate. As the French dramatist Jean Giraudoux once said:

“The secret of success is sincerity. Once you can fake that, you’ve got it made.”

Comments closed

Searching for The Atom of Meaning

Published July 25, 2011 by cgerrish

If we begin by looking for the atom of meaning, we tend toward looking at the word. After all, when there’s something we don’t understand, we isolate the word causing the problem and look it up in the dictionary. If we look a little further, we see the dictionary is filled with a sampling of phrases that expand on and provide context for the word. The meaning is located in the phrases, not in the isolated word. We might look at the dictionary as a book filled with phrases that can be located using words. The atom of meaning turns out to be a molecule.

When we put a single word into a search engine, it can only reply with the context that most people bring to that word. Alternately, if we supply a phrase to the search engine, we’ve given the machine much more to work with. We’ve supplied facets to the search keyword. In 2001, the average number of keywords in a search query was 2.5. Recently, the number has approached 4 words per query. The phrase provides a better return than the word.

As amazing as search engine results can sometimes be, the major search engines seemed to have achieved a level of parity based on implementation of the citation algorithm on a large corpus of data. In a blind taste test of search engines, all branding removed, the top few pages of results tend to look pretty similar. But when you add brand back on to the carbonated and flavored sugar water, people display very strong preferences. While we may think search results can be infinitely improved within the current methodology, it seems we may have come up against a limit. At this point, it’s the brand that convinces us there’s an unlimited frontier ahead of us–even when there’s not. And one can hardly call improved methods for filtering out spam a frontier.

If, like Google, you’ve set a goal of providing answers before the user even asks a question, you can’t get there using legacy methods. Here, we’re presented with a fork in the road. In one direction lies the Semantic web, with its “ontologies” that claim to provide canonical meanings for these words and phrases. Of course, in order to be universal, semantics and “ontologies” must be available to everyone. They haven’t been constructed to provide a competitive advantage in the marketplace. In the other direction we find the real-time stream and online identity systems of social media. Google seems to have placed a bet on the second approach. Fresh ingredients are required to whip up this new dish: at least one additional domain of data, preferably a real-time stream; and an online identity system to tie them together. Correlation of correlation data from multiple pools threaded through a Network identity—that gives you an approach that starts to transform an answer appropriate for someone like you, to an answer appropriate only for you.

When speaking about additional data domains, we should make it clear there are two kinds: the private and the public. In searching for a new corpus of data, Google could simply read your G-mail and the content of your Google docs, correlate them through your identity and then use that information to improve your search results. In fact, when you loaded the search query page, it could be pre-populated with information related to your recent correspondence and work product. They could even make the claim that since it’s only a robot reading the private domain data, this technique should be allowed. After all, The robot is programmed to not to tell the humans what it knows.

Using private data invites a kind of complexity that resides outside the realm of algorithms. The correlation algorithm detects no difference between private and public data, but people are very sensitive to the distinction. As Google has learned, flipping a bit to turn private data into public data has real consequences in the lives of the people who (systems that) generated the data. Thus we see the launch of Google+, the public real-time social stream that Google needs to to move their search product to the next level.

You could look at Google+ as a stand-alone competitor to Facebook and Twitter in the social media space, but that would be looking at things backwards. Google is looking to enhance and protect their primary revenue stream. To do that they need another public data pool to coordinate with their search index. Google’s general product set is starting to focus on this kind of cross data pool correlation. The closure of Google Labs is an additional signal of the narrowing of product efforts to support the primary revenue-producing line of business.

You might ask why Google couldn’t simply partner with another company to get access to an additional pool of data? Google sells targeted advertising space within a search results stream. Basically, that puts them in the same business as Facebook and Twitter. But in addition, Google doesn’t partner well with other firms. On the one hand, they’re too big and on the other, they prefer to do things their way. They’ve created Google versions of all the hardware and software they might need to use. Google has its own mail, document processing, browser, maps, operating systems, laptops, tablets and handsets.

Using this frame to analyze the competitive field you can see how Google has brazenly attacked their competitors at the top of the technology world. By moving in on the primary revenue streams of Apple and Microsoft, they indicated that they’ve built a barrier to entry with search that cannot be breached. Google didn’t think their primary revenue stream could be counter-attacked. That is, until they realized that the quality of search results had stopped improving by noticeable increments. And as the transition from stationary to mobile computing accelerated, the kind of search results they’ve been peddling are becoming less relevant. Failure to launch a successful social network isn’t really an option for Google.

Both Apple and Microsoft have experienced humbling events in their corporate history. They’ve learned they don’t need to be dominant on every frequency. This has allowed both companies find partners with complementary business models to create things they couldn’t do on their own. For the iPhone, Apple partnered with AT&T; and for the forthcoming version 5 devices they’ve created a partnership with Twitter. Microsoft has an investment in, and partnership with, Facebook. It seems clear that Bing will be combined with Facebook’s social graph and real-time status stream to move their search product to the next level. The Skype integration into Facebook is another example of partnership. It’s also likely that rather than trying to replicate Facebook’s social platform in the Enterprise space, Microsoft will simply partner with Facebook to bring a version of their platform inside the firewall.

In his recent talk at the Paley Center for Media, Roger McNamee declared social media over as avenue for new venture investing. He notes that there are fewer than 8 to 10 players that matter, and going forward there will be consolidation rather than expansion. In his opinion, social media isn’t an industry, but potentially, it’s a feature of almost everything. In his view, it’s time to move on to greener pastures.

When the social media music stopped, Apple and Microsoft found partners. Google has had to create a partner from scratch. This is a key moment for Google. Oddly, the company that has lead the charge for the Open Web is the only player going it alone.

Comments closed