Archive for the 'social graph' Category

« Previous Entries Next Entries »

Searching for The Atom of Meaning

If we begin by looking for the atom of meaning, we tend toward looking at the word. After all, when there’s something we don’t understand, we isolate the word causing the problem and look it up in the dictionary. If we look a little further, we see the dictionary is filled with a sampling of phrases that expand on and provide context for the word. The meaning is located in the phrases, not in the isolated word. We might look at the dictionary as a book filled with phrases that can be located using words. The atom of meaning turns out to be a molecule.

When we put a single word into a search engine, it can only reply with the context that most people bring to that word. Alternately, if we supply a phrase to the search engine, we’ve given the machine much more to work with. We’ve supplied facets to the search keyword. In 2001, the average number of keywords in a search query was 2.5. Recently, the number has approached 4 words per query. The phrase provides a better return than the word.

As amazing as search engine results can sometimes be, the major search engines seemed to have achieved a level of parity based on implementation of the citation algorithm on a large corpus of data. In a blind taste test of search engines, all branding removed, the top few pages of results tend to look pretty similar. But when you add brand back on to the carbonated and flavored sugar water, people display very strong preferences. While we may think search results can be infinitely improved within the current methodology, it seems we may have come up against a limit. At this point, it’s the brand that convinces us there’s an unlimited frontier ahead of us–even when there’s not. And one can hardly call improved methods for filtering out spam a frontier.

If, like Google, you’ve set a goal of providing answers before the user even asks a question, you can’t get there using legacy methods. Here, we’re presented with a fork in the road. In one direction lies the Semantic web, with its “ontologies” that claim to provide canonical meanings for these words and phrases. Of course, in order to be universal, semantics and “ontologies” must be available to everyone. They haven’t been constructed to provide a competitive advantage in the marketplace. In the other direction we find the real-time stream and online identity systems of social media. Google seems to have placed a bet on the second approach. Fresh ingredients are required to whip up this new dish: at least one additional domain of data, preferably a real-time stream; and an online identity system to tie them together. Correlation of correlation data from multiple pools threaded through a Network identity—that gives you an approach that starts to transform an answer appropriate for someone like you, to an answer appropriate only for you.

When speaking about additional data domains, we should make it clear there are two kinds: the private and the public. In searching for a new corpus of data, Google could simply read your G-mail and the content of your Google docs, correlate them through your identity and then use that information to improve your search results. In fact, when you loaded the search query page, it could be pre-populated with information related to your recent correspondence and work product. They could even make the claim that since it’s only a robot reading the private domain data, this technique should be allowed. After all, The robot is programmed to not to tell the humans what it knows.

Using private data invites a kind of complexity that resides outside the realm of algorithms. The correlation algorithm detects no difference between private and public data, but people are very sensitive to the distinction. As Google has learned, flipping a bit to turn private data into public data has real consequences in the lives of the people who (systems that) generated the data. Thus we see the launch of Google+, the public real-time social stream that Google needs to to move their search product to the next level.

You could look at Google+ as a stand-alone competitor to Facebook and Twitter in the social media space, but that would be looking at things backwards. Google is looking to enhance and protect their primary revenue stream. To do that they need another public data pool to coordinate with their search index. Google’s general product set is starting to focus on this kind of cross data pool correlation. The closure of Google Labs is an additional signal of the narrowing of product efforts to support the primary revenue-producing line of business.

You might ask why Google couldn’t simply partner with another company to get access to an additional pool of data? Google sells targeted advertising space within a search results stream. Basically, that puts them in the same business as Facebook and Twitter. But in addition, Google doesn’t partner well with other firms. On the one hand, they’re too big and on the other, they prefer to do things their way. They’ve created Google versions of all the hardware and software they might need to use. Google has its own mail, document processing, browser, maps, operating systems, laptops, tablets and handsets.

Using this frame to analyze the competitive field you can see how Google has brazenly attacked their competitors at the top of the technology world. By moving in on the primary revenue streams of Apple and Microsoft, they indicated that they’ve built a barrier to entry with search that cannot be breached. Google didn’t think their primary revenue stream could be counter-attacked. That is, until they realized that the quality of search results had stopped improving by noticeable increments. And as the transition from stationary to mobile computing accelerated, the kind of search results they’ve been peddling are becoming less relevant. Failure to launch a successful social network isn’t really an option for Google.

Both Apple and Microsoft have experienced humbling events in their corporate history. They’ve learned they don’t need to be dominant on every frequency. This has allowed both companies find partners with complementary business models to create things they couldn’t do on their own. For the iPhone, Apple partnered with AT&T; and for the forthcoming version 5 devices they’ve created a partnership with Twitter. Microsoft has an investment in, and partnership with, Facebook. It seems clear that Bing will be combined with Facebook’s social graph and real-time status stream to move their search product to the next level. The Skype integration into Facebook is another example of partnership. It’s also likely that rather than trying to replicate Facebook’s social platform in the Enterprise space, Microsoft will simply partner with Facebook to bring a version of their platform inside the firewall.

In his recent talk at the Paley Center for Media, Roger McNamee declared social media over as avenue for new venture investing. He notes that there are fewer than 8 to 10 players that matter, and going forward there will be consolidation rather than expansion. In his opinion, social media isn’t an industry, but potentially, it’s a feature of almost everything. In his view, it’s time to move on to greener pastures.

When the social media music stopped, Apple and Microsoft found partners. Google has had to create a partner from scratch. This is a key moment for Google. Oddly, the company that has lead the charge for the Open Web is the only player going it alone.

Keller’s Lament

I’ve gone back and forth so many times, it seems as if a comment at this point would be addressing ancient history. On May 18th, 2011, Bill Keller, Executive Editor of the New York Times, wrote an essay called ‘The Twitter Trap.’ In the piece he airs his complaints, misgivings and thoughts about Twitter, Facebook and the current era of social media.

I came to Keller’s essay through a series of tweets taking him to task for his ignorance of social media and of Twitter in particular. The predominantly tech-oriented crowd I follow on Twitter quickly formed a consensus opinion that this was further evidence of Keller’s cluelessness—Hey you kids, get off of my lawn! Old mainstream media attacking the new social media, hidden behind a modified paywall, the form of the communication echoing the misguided opinions. Full disclosure, I’m a long-time subscriber to the ink-on-paper instantiation of the New York Times. When I finally read the piece, I chose the printed version. Later on, I re-read it online.

Keller’s lament centers around three central points: digital idolatry, the price exacted by innovation and the displacement of essential intellectual values and cultural practices. Keller begins with this opening gambit:

Last week my wife and I told our 13-year-old daughter she could join Facebook. Within a few hours she had accumulated 171 friends, and I felt a little as if I had passed my child a pipe of crystal meth.

The context of family, children and addictive drugs is an interesting one. Many of Keller’s hopes and fears regarding social media are threaded through this particular story. But let’s start with digital idolatry.

We’ve been riding a wave of technology, real-time networks, big data and full duplex (read/write) distributed media. All of these trends kick against a centralized professional media. It’s assumed that a critic of this wave is trying to swim back upstream against the current of time. Dissenting opinions are dismissed by the crowd as uninformed, but should we uncritically accept everything this revolution in technology and media offers? Should we simply trust that the wave knows where it’s going? When digital technology becomes an idol, we religiously make ourselves into more efficient cogs in the machine. A new fundamentalism is spawned that treats dissenters with the same disdain as all those who’ve strayed from the fold.

The price exacted by innovation is a well-worn theme. Keller cites a number of examples:

  • Rote memorization vs. The Printing Press
  • Penmanship vs. Typewriting
  • Slide Rule vs. Calulator
  • Sustained Attention vs. Twitter and YouTube

When something new comes along, something current is displaced. We type instead of writing by hand; we use a calculator and the slide rule stays in the drawer; we look things up online instead of practicing mnemonic techniques; and we consume a never-ending stream of hors d’oeuvres never getting to the main course. The displaced option remains, but loses value. If we are what we do, then we are most certainly changed. Unused muscles atrophy while new muscles grow strong through the patterns of the new activity. The question is, will we regret anything we’ve lost.

The intellectual values that Keller fears may be added to the endangered species list are:

  • Real rapport and conversation
  • Complexity
  • Acuity
  • Patience
  • Wisdom
  • Intimacy

In particular, Keller focuses on the 140 character limit to the hypertext that makes up a tweet. Conversations don’t have the room the stretch out and breath, no real rapport can be established. The micro-message medium only allows for the exchange of communiques. Ideas are reduced and compacted to flow efficiently through the message dispatch system. Keller asks whether the soil of social media is fertile enough to support these deeper values.

Twitter, and any other hypertext-based social media, communicates by value or by reference. That means for a short message, the entire value of the communication, can be contained in the tweet. When the tweet communicates by reference, it contains some description and a hypertext link that points to a long form communication that exists outside the messaging system. Newspapers accomplish this with headlines.

The conversation Keller hoped to incite seemed to quickly devolve into the kind of childish bickering he parodies in his essay. He rather seems to enjoy activating the reflexive behavior of the digital punditry. By limiting the responses to his essay to 140 character telegrams, he manages to demonstrate the poverty of the micro-message medium. This may, in fact, be the meaning of the essay’s title, “The Twitter Trap.”?

Keller opens his essay with the information that he and his wife have allowed their 13-year old daughter to open and operate a Facebook account. The feeling, he reports, was like passing a pipe of crystal meth to his child. The intellectual values and cultural practices that Keller sees slipping over the horizon may or may not be available to his daughter. They may be the price exacted by the highly addictive nature of real-time networks and social media. Despite that risk, he allows his young teenager to venture forth into the Network. Of course the fact that the teen had accumulated 171 friends in a few hours meant that permission was a mere formality. The social graph already existed, the online account merely facilitated its inscription into Facebook’s systems.

The essay ends with a question about the future of the soul. Rather than turning to the scientist, engineer or technologist, instead he quotes a novelist:

In Meg Wolitzer’s charming new tale, “The Uncoupling,” there is a wistful passage about the high-school cohort my daughter is about to join. Wolitzer describes them this way: “The generation that had information, but no context. Butter, but no bread. Craving, but no longing.”

Steve Jobs often talks about the intersection of technology and liberal arts, but it seems like the two often talk past each other. Neither takes the other very seriously. With the exception of Apple, there doesn’t seem to be much of a business model in it. And the soul, it seems, is in mortal danger with every generation. But that’s no excuse to assume this couldn’t be the time when things turn out differently.


The Demons Aren’t In The Machine

At university I took an intensive class on the work of Sigmund Freud by a professor who had worked training psychoanalysts. The reading list immersed us in Freud’s writings from the letters to Fliess, the early work with Breuer, all of the case studies and well into The Interpretation of Dreams and beyond. We would take anonymous dream reports from clinic patients and attempt to interpret them without context, using the tools we’d acquired. It was surprising how often we got quite close to the crux of the psychological issue.

Since that time I’ve always felt uncomfortable in casual social situations where someone wants to tell me about this strange dream they had last night. Of course, it’s always intended in an “isn’t this weird, dreams are inexplicable” kind-of-way. I’m always careful to keep my gaze on the surface of the words, while ignoring the demons screeching and flying out of the depths of the metaphors. Two distinct realities seem to occupy the same space along different dimensions.

I was reminded of this eruption of id among the everyday while reading Adam Gopnik’s assessment of the recent spate of books on the inevitability of the Network and the end of the book in a recent New Yorker magazine. The essay is called, The Information, How the Internet gets inside us. Gopnik seems to expose something completely invisible to the technorati. To those who see the Network as an entirely rational space of organized and accessible information, the demons flying round the room occupy a withdrawn dimension.

Yet surely having something wrapped right around your mind is different from having your mind wrapped tightly around something. What we live in is not the age of the extended mind but the age of the inverted self. The things that have usually lived in the darker recesses or mad corners of our mind—sexual obsessions and conspiracy theories, paranoid fixations and fetishes—are now out there: you click once and you can read about the Kennedy autopsy or the Nazi salute or hog-tied Swedish flight attendants. But things that were once external and subject to the social rules of caution and embarrassment—above all, our interaction with other people—are now easily internalized, made to feel like mere workings of the id left on its own.

When we talk about the Network having a bottom-up structure, generally we’re referring to the process of folksonomy as opposed to a top-down taxonomy. Or perhaps we refer to finally having the participation levels and processing power to harness an infinite number of typing monkeys to efficiently produce the works of Shakespeare at a tidy profit. However, there’s another sense in which the Network is bottom up. As Clay Shirky sometimes says, everything is published and we edit later. The bottom encompasses all of our baseness.

In Freudian terms, we publish the id and then attempt to re-establish order by adding the ego and super-ego. When Freud describes the id, he talks about contrary impulses existing side by side without canceling each other out, about a life-force without any sense of negation, a striving to bring about the satisfaction of instinctual needs only subject to the observance of the pleasure principle.

Gopnik ties this bottom-up publishing of everything into the familiar pattern of the flaming comment:

Thus the limitless malice of Internet commenting: it’s not newly unleashed anger but what we all think in the first order, and have always in the past socially restrained if only thanks to the look on the listener’s face—the monstrous music that runs through our minds is now played out loud.

Marshall McLuhan talked about how the medium of television bypassed personal and societal censors and poured directly into the nerves.

TV goes right into the human nervous system, it goes right into the midriff. The image pours right off that tube into the nerves. It’s an inner trip, the TV viewer is stoned. It’s addictive.

Television enabled images from all over the world, in high volumes, to be moved from the outside to the inside. The Network makes the reverse movement possible. In his essay, Gopnik makes an insightful observation about the unsocial nature of our contemporary social networks:

A social network is crucially different from a social circle, since the function of a social circle is to curb our appetites and of a network to extend them. Everything once inside is outside, a click away; much that used to be outside is inside, experienced in solitude. And so the peacefulness, the serenity that we feel away from the Internet … has less to do with being no longer harried by others than with being less oppressed by the force of your own inner life. Shut off your computer, and your self stops raging quite as much or quite as loud.

The social graph extends the inputs and outputs of the nervous system while bypassing the social functions that provide a level of reflection—we’ll edit later. Gopnik points out that the problem with the constant interruptions, change of focus and multitasking while we multitask isn’t one of a rational mind having to focus among a panoply of options, but rather that of a glutton alone in his room, limited to only one mouth and faced with a smorgasbord of immense proportions. In our solitude we all are individually transformed into Brecht’s Baal or Shakespeare’s Falstaff. A Network fueled by a raging pleasure principle confronts the reality of the seven deadly sins with an emphasis on gluttony.

The shattering of attention into tiny shards is the metaphor that has caught our fancy. It’s this symptom that must be the source of our pain. As our attention is shattered, so is our identity and our capacity to focus. Gopnik puts this observation into historical perspective:

The odd thing is that this complaint… is identical to Baudelaire’s perception about modern Paris in 1855, or Walter Benjamin’s about Berlin in 1930, or Marshall McLuhan’s in the face of three-channel television in 1965. When department stores had Christmas windows with clockwork puppets, the world was going to pieces; when the city streets were filled with horse-drawn carriages running by bright-colored posters, you could no longer tell the real from the simulated; when people were listening to shellac 78s and looking at color newspaper supplements, the world had become a kaleidoscope of disassociated imagery; and when the broadcast air was filled with droning black-and-white images of men in suits reading news, all of life had become indistinguishable from your fantasies of it. It was Marx, not Steve Jobs, who said that the character of modern life is that everything falls apart.

Of course, anyone who can walk into a library and find a book, select some toothpaste from a display in a large drugstore or find a couple of stories they’d like to read in the Sunday New York Times can probably deal with all these tiny shards of attention that we’re confronted with on the Network. Perhaps the pain has more to do with the demons we wrestle with as we jack in to the Network. And while it seems like the demons are released from the Network the moment we flick the connection on— it turns out the demons aren’t in the machine at all.

Of Twitter and RSS…

It’s not really a question of life or death. Perhaps it’s time to look for a metaphor that sheds a little more light. The frame that’s been most productive for me is one created by Clayton Christensen and put to work in his book, The Innovator’s Solution.

Specifically, customers—people and companies— have “jobs” that arise regularly and need to get done. When customers become aware of a job that they need to get done in their lives, they look around for a product or service that they can “hire” to get the job done. This is how customers experience life. Their thought processes originate with an awareness of needing to get something done, and then they set out to hire something or someone to do the job as effectively, conveniently and inexpensively as possible. The functional, emotional and social dimensions of the jobs that customers need to get done constitute the circumstances in which they buy. In other words, the jobs that customers are trying to get done or the outcomes that they are trying to achieve constitute a circumstance-based categorization of markets. Companies that target their products at the circumstances in which customers find themselves, rather than at the customers themselves, are those that can launch predictably successful products.

At a very basic level, people are hiring Twitter to do jobs that RSS used to get. The change in usage patterns is probably more akin to getting laid off. Of course, RSS hasn’t been just sitting around. It’s getting job training and has acquired some new skills like RSS Cloud and JSON. This may lead to some new jobs, but it’s unlikely that it’ll get its old job back.

By reviewing some of the issues with RSS, you can find a path to what is making Twitter (and Facebook) successful. While it’s relatively easy to subscribe to a particular RSS feed through an RSS reader— discovery and serendipity are problematic. You only get what you specifically subscribe to. The ping server was a solution to this problem. If, on publication of a new item, a message is sent to a central ping server, an index of new items could be built. This allows discovery to be done on the corpus of feeds to which you don’t subscribe. The highest area of value is in discovering known unknowns, and unknown unknowns. To get to real-time tracking of a high volume of new items as they occur, you need a central index. As Jeff Jonas points out, federated systems are not up to the task:

Whether the data is the query (generated by systems likely at high volumes) or the user invokes a query (by comparison likely lower volumes), there is nodifference.  In both cases, this is simply a need for — discoverability — the ability to discover if the enterprise has any related information. If discoverability across a federation of disparate systems is the goal, federated search does not scale, in any practical way, for any amount of money. Period. It is so essential that folks understand this before they run off wasting millions of dollars on fairytale stories backed up by a few math guys with a new vision who have never done it before.

Twitter works as a central index, as a ping server. Because of this, it can provide discovery services on to segments of the Network to which a user is not directly connected. Twitter also operates as a switchboard, it’s capable of opening a real-time messaging channel between any two users in its index. In addition, once a user joins Twitter (or Facebook), the division between publisher and subscriber is dissolved. In RSS, the two roles are distinct. Google also has a central index, once again, here’s Jonas:

Discovery at scale is best solved with some form of central directories or indexes. That is how Google does it (queries hit the Google indexes which return pointers). That is how the DNS works (queries hit a hierarchical set of directories which return pointers).  And this is how people locate books at the library (the card catalog is used to reveal pointers to books).

A central index can be built and updated in at least two ways. With Twitter, the participants write directly into the index or send an automated ping to register publication of a new item. Updates are in real time. For Google, the web is like a vast subscription space. Google is like a big RSS reader that polls the web every so often to find out whether there are any new items. They subscribe to everything and then optimize it, so you just have to subscribe to Google.

However, as the speed of publication to the Network increases, the quantity of items sitting in the gap between the times the poll runs continues to grow. A recent TPS Report showed that a record number, 6,939 Tweets Per Second, were published at 4 seconds past midnight on January 1, 2011. If what you’re looking for falls into that gap, you’re out of luck with the polling model. Stock exchanges are another example of a real-time central index. Wall Street has lead the way in developing systems for interpreting streaming data in real time. In high-frequency trading, time is counted in milliseconds and the only way to get an edge is to colocate servers into the same physical space as the exchange.

The exchanges themselves also are profiting from the demand for server space in physical proximity to the markets. Even on the fastest networks, it takes 7 milliseconds for data to travel between the New York markets and Chicago-based servers, and 35 milliseconds between the West and East coasts. Many broker-dealers and execution-services firms are paying premiums to place their servers inside the data centers of Nasdaq and the NYSE.

About 100 firms now colocate their servers with Nasdaq’s, says Brian Hyndman, Nasdaq’s SVP of transaction services, at a going rate of about $3,500 per rack per month. Nasdaq has seen 25 percent annual increases in colocation the past two years, according to Hyndman. Physical colocation eliminates the unavoidable time lags inherent in even the fastest wide area networks. Servers in shared data centers typically are connected via Gigabit Ethernet, with the ultrahigh-speed switching fabric called InfiniBand increasingly used for the same purpose, relates Yaron Haviv, CTO at Voltaire, a supplier of systems that Haviv contends can achieve latencies of less than 1 millionth of a second.

The model of colocation with a real-time central index is one we’ll see more of in a variety of contexts. The relationship between Facebook and Zynga has this general character. StockTwits and Twitter are another example. The real-time central index becomes a platform on which other businesses build a value-added product. We’re now seeing a push to build these kinds of indexes within specific verticals, the enterprise, the military, the government.

The web is not real time. Publishing events on the Network occur in real time, but there is no vantage point from which we can see and handle— in real time— ‘what is new’ on the web. In effect, the only place that real time exists on the web is within these hubs like Twitter and Facebook. The call to create a federated Twitter seems to ignore the laws of physics in favor of the laws of politics.

As we look around the Network, we see a small number of real-time hubs that have established any significant value (liquidity). But as we follow the trend lines radiating from these ideas, it’s clear we’ll see the attempt to create more hubs that produce valuable data streams. Connecting, blending, filtering, mixing and adding to the streams flowing through these hubs is another area that will quickly emerge. And eventually, we’ll see a Network of real-time hubs with a set of complex possibilities for connection. Contracts and treaties between the hubs will form the basis of a new politics and commerce. For those who thought the world wide web marked the end, a final state of the Network, this new landscape will appear alien. But in many ways, that future is already here.

« Previous Entries Next Entries »