Archive for the 'interaction design' Category

« Previous Entries Next Entries »

Television Signal Path and the Airplay Remote Control


The control systems for television aren’t very good. One reason they persist is that once a viewer is watching a selected program, the control system recedes into the background. In the course of watching a presentation, the essential controls, the ones that control sound (louder, softer, mute), generally work quite well. The rest of the control system is a disaster that people have learned to accommodate. This snarl of technology around controlling a television is generally why people think there’s room for revolutionary innovation in the “battle for the living room.”


Generally there have been a couple of approaches. The universal remote, a complex remote control device that consolidates all of the other remote controls. So instead of having five or six complex remote controls, you have one really really complex remote control. Google TV’s remote control with a keyboard pushes towards the limits of this kind of conceptual framework. The addition of voice command and SIRI is another solution at the limit. The other approach involves creating a “smart” television. This would be accomplished by integrating a Network connected computer into the television device. This new device would make all of the other devices obsolete. Various forms of this device have been foisted upon the public. It’s not that people don’t buy these “smart” televisions, it’s just that no one uses any of their capability.

The solution to this tangle of technology lies in the role of the remote control. The name “remote control” describes what the device does. It takes the control system from the television and allows it to operate at a distance from the television itself. That meant you didn’t have to get up off the sofa and walk across the room to select a program or control the sound volume. The “remote” has essentially provided the same service since it entered the living room in the mid-1950s. Nikola Tesla described its basic operation in a patent application more than 50 years earlier than that. To some extent, even cloud computing is just a variation of the same theme.

It was while researching wireless audio systems for my study that the basic change in the “remote” became clear to me. With all of my music available through a cloud storage system, I didn’t need a music system to decode physical media. From the many choices available, I selected the Bowers & Wilkins A7. It’s a single speaker that sits in a home WiFi network and listens for AirPlay signals. You can send it music via AirPlay from your phone, iPod, tablet or desktop computer—and that music can be stored remotely on the Network. Radio streams, YouTube sound, podcasts, etc. can be also be sent to this audio system. The key is the change in the signal path. The “remote” is no longer just a controller, it’s the receiver/broadcaster of the audio signal. The “stereo system” now listens for AirPlay signals, decodes and presents the sound. I liked this solution so much, I set up my traditional stereo to operate similarly using AirPort Express as one of the auxiliary inputs.


You can see how this model would work for television. Instead of a smart television, you have a dumb television. The big screen does what the big screen does well. It shows high-definition moving pictures synchronized with sound. You can’t solve the “television problem” without changing the signal path. Once the remote control becomes a receiver/AirPlay broadcaster, all the peripheral devices hooked up to your television go away. Even your cable box becomes just another app on your phone or tablet. The interesting thing about this solution is that it doesn’t necessarily disintermediate the cable companies, the premium channels, Netflix, Amazon, Tamalpais Research Institute, Live from the Metropolitan Opera or your favorite video podcast.


In this analysis, the real problem with the television is identified as the HDMI connector. Every device connected to the screen via HDMI wants to dominate the control system of the television; and every HDMI connection spawns its own remote. Once you get rid of the HDMI connector and transform the remote control into an AirPlay receiver/broadcaster, all the remote controls disappear. The television listens for one kind of signal and plays programming from any authorized source. The new generation of wireless music systems have demonstrated that this kind of solution works, and works today. By changing the signal path and the role of the remote, the solution to the problem of television is well within reach.


Things, Not Strings: The Word Made Flesh

The headline reads: “Introducing the Knowledge Graph: things, not strings.” The implication being, “strings” are bad and limited and “things” are good and what you really wanted all along. After all people don’t want strings of arbitrary alpha-numeric characters in response to their queries, they want the things they’re looking for. And as the advertising message at the end of the introduction says, because you’re getting “things and not strings” on your search result pages, you can spend more time doing the things you love. Who wouldn’t want to do that? The end result of this technological improvement is that your life now contains “more time”— like a toothpaste tube that contains 20% more toothpaste; and that time is filled with love. One might even recast this new product as a machine for filling the world with love.

What Google seems to be introducing is a new user interface to a faceted search. Nothing more. Faceted search acknowledges that the “word” (a single string of characters) isn’t the atom of meaning. Instead it uses the “phrase” in the context of some domain of meaning—a word can be a valid token in multiple systems of meaning. These domains, or facets of meaning, are surfaced and prioritized in search results. So, in addition to Page-ranked links, we get a prioritized set of contexts in which a particular word or phrase is a valid operator. The advance is in creating an index of sub-domains of meaning through analyzing the structure of text as it’s used on the visible Network. There’s no question that faceted search is superior to classic Page-ranked search, however the language used to describe this new product innovation seems to suggest some kind of transcendent experience.

Here’s a description of the vision that drives innovation in the search product at Google:

We’ve always believed that the perfect search engine should understand exactly what you mean and give you back exactly what you want
– Amit Singhal, SVP, Engineering at Google

But when I hear this kind of talk from engineers, their words are drowned out by the characters from Lewis Carroll’s “Through the Looking Glass“:

‘I don’t know what you mean by “glory”,’ Alice said.

Humpty Dumpty smiled contemptuously. ‘Of course you don’t — till I tell you. I meant “there’s a nice knock-down argument for you!”‘

‘But “glory” doesn’t mean “a nice knock-down argument”,’ Alice objected.

‘When I use a word,’ Humpty Dumpty said, in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’

‘The question is,’ said Alice, ‘whether you can make words mean so many different things.’

‘The question is,’ said Humpty Dumpty, ‘which is to be master — that’s all.’

Alice was too much puzzled to say anything; so after a minute Humpty Dumpty began again. ‘They’ve a temper, some of them — particularly verbs: they’re the proudest — adjectives you can do anything with, but not verbs — however, I can manage the whole lot of them! Impenetrability! That’s what I say!’

‘Would you tell me please,’ said Alice, ‘what that means?’

‘Now you talk like a reasonable child,’ said Humpty Dumpty, looking very much pleased. ‘I meant by “impenetrability” that we’ve had enough of that subject, and it would be just as well if you’d mention what you mean to do next, as I suppose you don’t mean to stop here all the rest of your life.’

‘That’s a great deal to make one word mean,’ Alice said in a thoughtful tone.

‘When I make a word do a lot of work like that,’ said Humpty Dumpty, ‘I always pay it extra.’

We can propose the idea that Google has a search engine that “understands exactly what you mean.” And by this what we mean is that your query corresponds to a sub-domain in the index of facets Google has previously collected. The “meaning” doesn’t lie in the “you” that has the query, but rather in the sets of sub-domains contained in Google’s index. When a word does a lot of work in multiple sub-domains of meaning, they pay extra in compute time.

The claim that Google makes is that they’ve gone from “strings” to “things.” But the sub-domains of meaning that Google is collecting are made up of computable sets of strings, not things. The leap that Google is actually trying to make is from “strings” to “words, phrases and contexts.” But the use of the word “thing” is very revealing. Words are not things, they are indexes. They point at things, suggest things, or function in a play of difference within a system of meaning. When we say that we’ve gone from “strings” to “things” we’re actually making a kind of miraculous claim. We’ve gone from “word” to “thing.” The most prominent example of this algorithm can be found in the King James Bible, we see it in John 1.14:

And the Word was made flesh, and dwelt among us, (and we beheld his glory, the glory as of the only begotten of the Father,) full of grace and truth.

If we believe that Google’s knowledge graph provides “things” and not “strings,” we also believe something extraordinary about the power and capability of Google. Even if we take a step back and simply say that Google is merely indexing sub-domains—systems of meaning, we need to examine what this means. We could follow Wittgenstein and say that “meaning” can be described as a form of life. Therefore Google’s index produces a prioritized list of facets (forms of life) that connect to your form of life, given what they know about you. Popular forms of life that don’t currently connect to you serve as a method of discovery.

Of course, there’s also the popular trend of the flesh made word

There are registers of meaning that Google’s approach will never capture. Their index will be filled with gaps and pools of darkness. In particular, only a very limited range of metaphor (cliches) will be caught in the net. Metaphor produces meaning through an algorithmic process (per @the_eco_thought, Tim Morton). Take a noun, take another noun from a different domain and place the word “is” between them. The coffee cup is a blue angel. The metaphor machine makes meaning. Not every metaphor is a good one, but it has some modicum of meaning and it does function as a metaphor.

Like the theoretical one hundred monkeys typing in front of a hundred typewriters for a hundred years, the metaphor machines are constantly operating and feeding the Network with new meaning. Darius Kazemi (@tinysubversions) has created a machine called “Metaphor a Minute” that does just this. You can follow it on Twitter at @metaphorminute. Of course, because of Twitter’s rate limits, there’s actually a new metaphor published every two minutes.

“Hold the newsreader’s nose squarely, waiter, or friendly milk will countermand my trousers.”

After thinking through Google’s new service and the language they’ve used to describe it, we discover that they are using the word “things” metaphorically. At first, we may assume that when engineers are describing the function of their new software, they’re making literal statements about what the machine they’ve constructed is doing. Instead, they’ve taken a two nouns from different domains and inserted the word “is” between them. Ironically, their use of the word “things” is of the type that their new service could not understand it. The narrow band of search engine results that are produced by this system is also being metaphorically called “knowledge.” In order to see these new products clearly, we need to be able to differentiate the rhetoric of hyperbole from the literal functioning of the machine. It also helps to become acquainted how metaphors mean…

>> Therefore, Ye Soft Pipes, Play On

The elegies for Steven Paul Jobs come pouring forth. The traditional elements of an elegy correspond to the stages of loss. Grief and sorrow are expressed through a lament; the life of the departed is idealized through admiration and praise; and then comes solace and consolation. As we find ourselves more than midway on life’s journey, the poetic form of the elegy reveals itself as a palpable presence. It’s not a form whose outlines are traced from a recipe extracted from a book, there’s a direct physical encounter with its contours as we stop for a moment, and look across the grain of time.

Businessmen, technologists, and tech bloggers have focused on different aspects of the Jobs legacy. I’d like to turn the spotlight to some of the language used to talk about what made Jobs different: visionary, genius, magic, and of course, crazy. These are words we use to describe something on the other side of the line, something well beyond ordinary grasp. From the stance of the technologist, the business person or the engineer, these are not qualities that can be captured in an algorithm, a spreadsheet or a mechanical device. Jobs appears to be an anomaly, the impossible exception—we shake our heads and say, ” we won’t see his like again.”

Steven P. Jobs wasn’t a hardware engineer, he didn’t write software code, he wasn’t an industrial designer. He didn’t finish college, given his qualifications, he wouldn’t even be considered for the position he held. The common wisdom in the technology community is that great companies start with great engineers—then eventually the suits come in and ruin everything. The technology industry’s utopia is a world run by engineers. Yet, Jobs, who was not an engineer, is acknowledged as the industry’s great visionary.

If we were listening, Jobs told us what he was doing. He explicitly stated that “Apple’s goal is to stand at the intersection of technology and the humanities.” This maxim hasn’t been given due consideration. Jobs restated this idea many times and in different formulations. At the iPad2 launch, he said it this way:

“It’s in Apple’s DNA that technology alone is not enough. It’s technology married with liberal arts, married with the humanities that yields the results that makes our hearts sing.”

To the engineers in the crowd, this talk of “singing hearts” must seem like a lot of sentimental hogwash. It’s the nuts and bolts that really make the difference. Technology stands alone, it doesn’t need to marry anyone, or anything, to win the day. Talk of ‘singing hearts’ is just Jobs as salesman, some of that ‘reality distortion field’ stuff.

We strip rhetoric from logic, we limit design to the surface, we consider the humanities to be the frothy nonsense floating at the top of an education that should be devoted to hardcore business and science. It’s the ‘nice-to-have,’ but inessential item on the to-do list. As the center of thought moves further and further in that direction, we lose even the language to describe the kinds of things Jobs accomplished. And while we can’t articulate it, there’s no question that we hear its music.

Heard melodies are sweet, but those unheard are
Are sweeter; therefore, ye soft pipes, play on;
Not to the sensual ear, but, more endear’d;
Pipe to the spirit ditties of no tone:

John Keats
Ode on a Grecian Urn

Here’s Jobs talking about his approach in a Fortune magazine interview in 2000:

“We don’t have good language to talk about this kind of thing,” Mr. Jobs replied. “In most people’s vocabularies, design means veneer. It’s interior decorating. It’s the fabric of the curtains and the sofa. But to me, nothing could be further from the meaning of design. Design is the fundamental soul of a man-made creation that ends up expressing itself in successive outer layers of the product or service. The iMac is not just the color or translucence or the shape of the shell. The essence of the iMac is to be the finest possible consumer computer in which each element plays together. … That is the furthest thing from veneer. It was at the core of the product the day we started.”

Here the humanities aren’t the thin layer of frosting spread on top of the core of technology to make it look nice. In a sense, technology is medium through which a fundamentally humanistic vision is expressed. Where the common wisdom is to start with the engineering and the technology, Jobs and the team at Apple start with an act of poetic imagination. The slogan “think different” encapsulates this idea. The ‘difference’ in this kind of thinking is that it starts with the humanities and technology as equal partners in the eventual expression of the product or service. Or as Jobs eloquently describes it, the kernel of the idea “expressing itself in successive outer layers.”

Of all the commentary, it was James B. Stewart’s piece in the New York Times that captured some of the unheard melody, the poetic thinking emanating from the office of the CEO.

“Most people underestimate his grandeur and his greatness,” Gadi Amit, founder and principal designer of New Deal Design in San Francisco, told me. “They think it’s about design. It’s beyond design. It’s completely holistic, and it’s dogmatic. Things need to be high quality; they have to have poetry and culture in each step. Steve was cut from completely different cloth from most business leaders. He was not a number-crunching guy; he was not a technologist. He was a cultural leader, and he drove Apple from that perspective. He started with culture; then followed with technology and design. No one seems to get that.”

It’s hard to find parallels. Braun and Olivetti in Europe had beautiful designs, but never had Apple’s success. Mr. Amit mentioned Italy’s Enzo Ferrari, the racecar driver and founder of the Ferrari sports car manufacturer. “Apple has the status that Ferrari has in Italy,” Ms. Antonelli said. “It’s a source of national pride and of pride for every employee. You get to that stature only if you created something so fundamental that everyone loves.”

Mr. Amit says he believes Mr. Jobs’s legacy will be “the blending of technology and poetry. It’s not about design per se; it’s the poetic aspect of the entire enterprise. Compared to Bill Gates or Warren Buffett, he’s in a different class. I think this is a revolutionary shift. Jobs is a revolutionary character. He shifted the industry and changed our lives through this amalgamation of culture and technology. If you’re looking for C.E.O.’s of this caliber, you have to look outside the engineering and business schools. That is truly revolutionary.”

When we lament that we won’t see another like Steven P. Jobs again, we need to acknowledge the cold, hard facts of the situation. We aren’t looking for people like Jobs to lead our greatest companies. In fact, we’re probably doing everything in our power to make sure that people like him don’t get anywhere near a leadership role. We’ve de-valued and de-funded the humanities, we’ve relegated poetic thinking to third class status.

In 1821 Percy Bysshe Shelley wrote “A Defense of Poetry.” Although he never wrote one, the work of Steven P. Jobs was a modern defense of poetry.

The most unfailing herald, companion, and follower of the awakening of a great people to work a beneficial change in opinion or institution, is poetry. At such periods there is an accumulation of the power of communicating and receiving intense and impassioned conceptions respecting man and nature. The person in whom this power resides, may often, as far as regards many portions of their nature, have little apparent correspondence with that spirit of good of which they are the ministers. But even whilst they deny and abjure, they are yet compelled to serve, that power which is seated on the throne of their own soul. It is impossible to read the compositions of the most celebrated writers of the present day without being startled with the electric life which burns within their words. They measure the circumference and sound the depths of human nature with a comprehensive and all-penetrating spirit, and they are themselves perhaps the most sincerely astonished at its manifestations; for it is less their spirit than the spirit of the age. Poets are the hierophants of an unapprehended inspiration; the mirrors of the gigantic shadows which futurity casts upon the present; the words which express what they understand not; the trumpets which sing to battle, and feel not what they inspire; the influence which is moved not, but moves. Poets are the unacknowledged legislators of the world.

Ironic Architecture: The Audience And Its Double

My eyes trace the curve of a jet black line as it snakes across the paper. There’s a point at which the line stops and my eyes keep going, tracing the trajectory of where the line might have gone. It’s within the bounds of that short distance that we travel into the future. It’s this tracing that doesn’t trace anything that is the subject of this meditation.

“and now I can go on,” is the phrase Wittgenstein used to describe a certain relationship to a series. Given “2, 4, 6, 8, 10,” I think I can see where things are going. “Even positive integers” is a possible answer, but no matter what numbers come next, a logic can be found for it. If the number is 12, that’s one sort of logic; if it’s 22, that’s another. Based purely on the visible, the adjacent invisible can always be colored in with a reasonable pattern.

It turns out that perception works in a similar way. The gaps in our apprehension of the world are bridged, filled in, to create the sensation of the smooth flow of time and experience. We project ourselves into the future. And our memories make liberal use of sampling to construct a rational narrative to account for the dramatic beats of our lives occuring before this one.

While past is not necessarily prologue, if you have enough data on what ‘usually happens’ you can make an educated guess about what will happen next. Through a statistical analysis of big data, the trajectory of partial behavior can be made visible, and the completion of that behavior can be projected. Correlations in the data emerge to tell a story that is unavailable to any one individual. Here the life of the human becomes actuarial, a set of probabilities for the possibilities. Once the percentages of the probabilities have exhibited some durability, casino economics can be installed to manage the risk and profit from these tendencies. The owners and operators of big data systems have a private view into a higher-dimensional phase space. And despite what these organizations tell us about good and evil, they are purely commercial enterprises.

A big data interlude: capturing big data on the Network, used to be the province of spiders. In the search business, it was only through expedition, return and accumulation of pointers and meta-data that a sufficient store of big data could be created. With Twitter and Facebook big data is created second-by-second within the walls of a single location. It’s the users who do all the traveling, sending postcards and pointers back to the archive.

As the probabilities solidify, another landscape emerges—along with the building materials for another level of architecture. For instance, using the tendencies that behavioral finance has uncovered, Thaler and Sunstein suggest building architectures that frame choice in such a way that people are ‘nudged’ into getting with the program. The program might be putting a percentage of one’s salary into a 401k to fund their retirement, or selecting a healthy lunch at the school cafeteria. We tend to accept the default and choose the item put in our path. Sunstein and Thaler call this activity ‘Choice Architecture‘ because while an individual is free to make any choice, the selection set is tilted toward a particular policy agenda. This tilting toward a particular outcome is what they call “a nudge.”

I like to call it “Ironic Architecture,” because while any choice can theoretically be made, the character in this little story is unaware of the manipulation and tilting of the selection set. When the character accepts the nudge and acts as the statistical analysis suggests they might, another level of the story is being played out.

Here’s Fowler’s Modern Usage on irony:

“Irony is a form of utterance that postulates a double audience, consisting of one party that hearing shall hear and shall not understand, and another party that, when more is meant than meets the ear, is aware of both that more and of the outsider’s incomprehension.”

While we make a big show of talking about how we want to engage the rational needs and desires of a user in the networked hypertext environment, more and more we’re seeing choice architecture employed to win without fighting, to persuade without engaging in a rational discussion.

This kind of strategy plays out in a number of domains, in politics, it’s called framing, or a little more obscurely, heresthetic:

“Like rhetoric, heresthetic depends on the use of language to manipulate people. But unlike rhetoric, it does not require persuasion. ‘With heresthetic,’ according to Riker, “Conviction is at least secondary and often not involved at all. The point of an heresthetical act is to structure the situation so that the actor wins, regardless of whether or not the other participants are persuaded.”

Personal behavior data is being created and recorded at an ever increasing rate. The phrase ‘information exhaust’ is an apt description of the continuous inscription of our activities into digital media. And while we may think that some superior form of personalization will be available to us based on this large data set, it’s more likely that big data will yield correlations and trends that are built into our environments and make us characters in stories of which we are unaware.

Harry Brignull has coined the phrase ‘dark patterns’ for this kind of architecture. Brignull writes eloquently about Alan Penn’s lecture on the architecture of Ikea and how consumer movement through that environment results in the unfolding of a singular story that its characters are unaware of:

“What Ikea have done is taken away something which is very fundamental, evolved into us, and they’ve designed an environment that operates quite differently, given that we are forward facing people, embodied […] from the way it would happen if you just looked down from outer space. Its effect is highly disorienting.”

“Ikea is highly disorienting and yet there is only one route to follow. […] Before long, you’ve got a trolley full of stuff that is not the things that you came there for. Something in the order of 60% of purchases at Ikea are not the things that people had on their shopping list when they came in the first place. That’s phenomenal.”

The best minds of our generation are designing dark patterns to entangle us in a story in which we spend more than we intend. They’re also designing choice architectures to get us to save for retirement, eat a healthy diet, get immunizations and show up for school. But the conversation and the narrative is happening at a level we don’t have access to—rhetoric without argument.икони

« Previous Entries Next Entries »