My thoughts have been swirling around the point of interaction for some time now. And by that I mean the point of human-computer interaction. To connect up the threads, at first, I’ve began looking backwards. Perhaps all the way to the Jacquard loom and the punch cards used to control the patterns, and then on to the punch cards used on the early mainframes.
I’m sure there were many steps in between, but my mind races ahead to the command line. This extremely powerful and elegant point of interaction has never really been superseded. It continues to be the favored mode of interaction for a number of software development activities. But it was the graphical user interface that provided a point of interaction that changed the medium.
Doug Engelbart’s 1968 demo of the work undertaken by the Augmentation Research Center (ARC) gives us all the fundamental modes of interaction. The keyboard, the mouse/trackpad, the headset, hypertext and the graphic user interface. Within that set of interaction points, we’ve started to expand the repertoire. With the introduction of the iPhone, the trackpad gesture has gained increasing importance.
On a separate track we’ve seen video games controllers become ever more complex. The point of interaction for the game starts to reflect the kind of speed and complexity we create in our virtual gaming worlds.
It’s with the Wii and Project Natal that we start to see the surface of the trackpad detached from the computing device, extruded into three dimensions, and then dematerialized. The interaction gestures can now be captured in the space around us. Originally, the graphic user interface (mouse clicks, windows, desktop) was criticized for the limitations it imposed.
The other key development was the displacement of computing from the local device to the Network of connected devices. The interaction point is now to a new Networked medium. This is the converged form of what McLuhan understood as television. The development of new interaction modes traces a path toward opening to greater numbers of participants the new medium. Beyond mass media, there is the media of connected micro-communities.
Popular culture and music culture has always had a big impact on the development of cutting-edge technology. When we think of controlling technology through hand gestures, we can start with the ether-wave theremin created by Leon Theremin.
This is one of those in-between moments. The wave unleashed by xerox-parc and the augmentation research center is about to be followed by a new wave. The signs are all around us.
There’s this idea of standing with your own two feet on the ground. We know which way is up.
I’ve been thinking about the implications of the virtual machine. Our friends at Wikipedia give the following definition:
A virtual machine (VM) is a software implementation of a machine (i.e. a computer) that executes programs like a physical machine.
Of course, then they add that some virtual machines may have no correspondence to actual hardware. One of the more common uses of a virtual machine is to create a software version of a computer operating system. On my Macintosh computer, I run a program called Parallels that allows me to run Windows, and Windows programs, side-by-side with Macintosh programs. The Macintosh runs an application that runs an operating system that runs an application. The ‘two-feet on the ground’ aspect of this is that the Macintosh operating system talks to the hardware— it’s this that provides the ground on which virtual machines can be deployed. The speed of today’s hardware/software systems makes the latency between the machine and the virtual machine almost unnoticeable.
Once the operating system is virtualized it starts to resemble the run-time system. It’s an environment where application code can be run. As we look around, we find a number of things that might fit into this category. For instance, Android, which is called an operating system, is really a virtual operating system that sits on top of the Linux operating system.
When people talk about a Web Operating System, often I’m not sure what they’re referring to. But in the model of the virtual machine, the Web run-times enabled by the operating system include: webkit, gecko and trident. In the sphere of vector graphics animations, the run-times include: Flash and Silverlight. In Silverlight 4.x, an application can contain an HTML page, which is interpreted by the local default Web browser, and that HTML page can contain a Flash object. The in-and-outs of things start to get a little complicated.
Putting our feet firmly on the ground, we can see that it all starts with the hardware and the operating system. Nothing happens without these foundation pieces. This is the bedrock on which we stand. Although this perspective begins to sound a little like the story about turtles from Stephen Hawking‘s 1988 book A Brief History of Time, which starts:
A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.” The scientist gave a superior smile before replying, “What is the tortoise standing on?” “You’re very clever, young man, very clever”, said the old lady. “But it’s turtles all the way down!”
While we may look at the computer hardware and operating system as the ground, we could also turn things on their head. What if, instead, we look at the environment where many virtual machines can operate as the new ground. Copernicus rather than Ptolemy. The hardware and its operating system are just and entry point into this Network of virtual machines. The one is a path to the many, and out of the many, there is one (E pluribus unum). Or as someone once said, the Network is the computer.
There’s this new idea of standing with your own two feet on the ground. We know which way is up.
As we consider ourselves in the flow of the day, we pause, once again, to consider identity. During the rush of the day we move through a thousand states. We flow from this to that as a result of our actions. Our many identities spring from the scenes we string together: the moment when we stop to shield our eyes from the sun; the quick turn of our heads when we think someone has called our name; the curse under our breath as the bus we’ve been waiting for arrives full and passes without stopping. They’re signifiers and contexts that circulate, they flow around us— around the things we do. They assign us a role within the never-ending series of stories that collect around us as we move through the world.
As we attempt to understand identity on the Network, we seem to crave a unified identity— a single container to hold all of our masks. And while we are beginning to transition from the static file to the real-time stream, we still want to draw a solid line around identity. The objects of the Network have been injected with time; we place the cursor here in the stream, a bread crumb to mark the spot, to provide reference for the next time we dip in. The artifacts of Internet identity are, of course, the outerworld of our innerworld— snapshots along the way.
In the book by the same name, there’s a poem entitled: “The Innerworld of the Outerworld of the Innerworld.” This phrase is a condensation of the style of Peter Handke’s early writing. We can plainly see the outerworld of the innerworld, these are the external artifacts of our internal stream of consciousness. But those objects of the outerworld have their own innerworld (a flux of time). Handke effects a change of perspective, a change of context— from the inside out to the outside in, and then to the inside of that outside. But it’s a poem with a different title in this volume that clarifies this complex perspective: “Changes during the Course of the Day.”
Changes during the Course of the Day
by Peter Handke
As long as I am still alone, I am still alone.
As long as I am still among acquaintances, I am still an acquaintance.
But as soon as I am among strangers—
As soon as I step out on the street— a pedestrian steps out on the street.
As soon as I enter the subway— a subway rider enters the subway
A soon as I enter the jewelry shop— a gentleman enters the jewelry shop.
As soon as I push the shopping cart through the supermarket— a customer pushes the cart through the supermarket.
As soon as I enter the department store— someone on a shopping spree enters the department store.
Then I walk past some children— and the the children see an adult walking past. Then I enter the off-limits zone— and the guards see a trespasser enter the off-limits zone. Then I see children running away from me in the off-limits zone— and I become a guard whom the children flee because they are unauthorized persons in an off-limits zone.
Then I sit in the waiting room as an applicant. Then I write my name on the back of the envelope as a sender. Then I fill out the lottery ticket as a winner.
As soon as I am asked how one gets to BLACK ROAD— I become someone who knows his way around town.
As soon as I see the incredible— I become a witness.
As soon as I enter the church— I become a layman.
As soon as I don’t ignore an accident— I become a busy-body.
As soon as I don’t know how to get to BLACK ROAD— I am again someone who doesn’t know his way to BLACK ROAD.
I have just consumed the meal— already I can say: We consumers!
I have just had something stolen from me— already I can say: We proprietors!
I have just placed the obituary— already I can say: We mourners!
I have just begun to contemplate the universe— already I can say: We human beings!
I read the novel in the mass publication— and become one among millions.
I don’t fulfill my duties toward the authorities— and am no longer a dutiful citizen of the state.
I don’t run away during the riot— and I’m an inciter of riots.
I look up from the novel I’m reading and observe the beauty opposite me— and we become two among millions.
Then someone does not leave the moving train— someone? — A traveler.
Then someone speaks without an accent— someone? — A native.
Then someone has a vis-à-vis— and becoma a vis-à-vis.
Then someone no longer only plays by himself— and becomes an opponent.
Then someone crawls out from under a thicket in the park and becomes a suspicious subject.
Then someone who is being discussed becomes an object of discussion.
Then someone is recognized on a photo— and becomes an X.
Then someone takes a walk in the country— someone? A wanderer.
And then the car makes a sudden stop in front of me— I become an obstacle.
Then I am seen by a figure in the dark— and become a figure in the dark.
And when I am then observed through binoculars— I am an object.
Then someone stumbles over me— and I become a body.
And when I am then stepped upon— I become something soft.
Then I am wrapped up in something— and become a content.
Then one notices that someone has run barefoot over the dirt road and that a right-hander has fired the shot and that someone whose blood group is O has lain there and that I, judging by the my shabby looks, must be a foreigner.
As soon as someone challenges me then— the one who’s been challenged doesn’t stop when challenged.
As soon as I am then far enough away from the observers— the object is nothing but a dot.
As soon as I, as an observer, challenge someone— I give the one who has been challenged quite a fright.
Then, finally, I meet an acquaintance— and a single person remains behind alone.
Then, finally, I am left aone— and a single person remains behind alone.
Then, finally, I sit down next to someone in the grass— and am finally someone else.
Over lunch with Steve Gillmor the other day, the topic strayed to the dubbing of foreign films. It linked up to an earlier conversation with Aron Michalski about the digital editing of recordings of live music. Our live experience goes virtual as it moves into the past, sound and vision are no longer linked. They become arbitrarily coordinated streams of media. The soundtrack of a film can be completely replaced and the language spoken by the actors can be localized to particular audiences. Wrong notes or timing in a live music performance can be fixed in post production before a quick release to the Network. The period of latency between the live moment and its distribution through a channel provides the opportunity to match our desires with the physical artifact of production. We get a second bite at the apple.
The other instance where separate streams of sound and video are synchronized to create the appearance of a natural experience is when we have the expectation of sound. This is a common practice in science fiction films set in space. Floating through space, we hear the roar of the engines, the blast of the weapons, and the explosion of the enemy ships. Of course, space is a vacuum and sound vibrations can’t occur without a suitable medium. We dub in the sound that makes emotional sense— desire and experience are synchronized.
The mechanical vibrations that can be interpreted as sound are able to travel through all forms of matter: gases, liquids, solids, and plasmas. The matter that supports the sound is called the medium. Sound cannot travel through vacuum.
While we may consider outer space to be the final frontier, there’s another frontier that has opened in front of us that is being explored every day by ordinary people. The virtual space of the Network is all around us. When we type messages on our iPhones, we hear the sound of clicking keys; when we take digital photos we hear the sound of the shutter clicking; when we drive certain kinds of electric cars, we hear the sound of a gasoline engine.
The haptics of the virtual replicate the physics of the physical world. Events in the virtual space of code trigger a sound stream that has an experiential analogy in the physical world. We’ve virtualized complex mechanical interfaces with knobs, dials, sliders, and various data readouts. The dashboard is the holy grail of business intelligence. Some have even proposed a real-time dashboard as the new center of our computing experience.
Consider for a moment how we’ve begun to dub our virtual space to synchronize it with the physical space of our environment. My iPhone uses a traditional telephone ringing sound to signal when a call is coming through. I selected this sound from a menu of possible sounds. Actual telephones that contain metal bells that ring on an incoming call event are pretty rare these days. Many younger people have only experienced the virtual sound of the old telephone.
The link between sound and vision is arbitrary in the virtual world. Our cheap digital camera can sport a sound sample taken from the most expensive mechanical camera. What’s the sound of code executing? We extend the context from our mechanical physical universe into the virtual universe to give us a sense of which way is up, when something has started and when it’s finished. The sound track to the virtual is a matter of cultural practice, but it’s both variable and personalizible. However, as the mechanical recedes around us, our context also becomes fainter. Will the virtual always be a mirror world, or will some new practice emerge from the Network itself? Can a concept of natural sound be generated from a world where sound doesn’t naturally occur, but is rather always a matter of will?
