Category: hci

The End of the PC: 3 Screens and a Cloud

Published March 18, 2010 by cgerrish

We see the shift beginning to play out as fragments of the picture leak out on to the Network. Presumably the strategy was set 4 or 5 years ago, but the artifacts of its implementation are now appearing in regular release cycles. As we fit more pieces into the puzzle, the picture is coming in to focus.

Most technology is only useful to the extent that people are around it. Some technical experiences are powerful enough to draw people to the technology. Recently we’ve seen a new landscape emerge where powerful technology is created that can follow people around wherever they might go. The big players are positioning themselves to flourish in this new world.

It may have been Ray Ozzie who most succinctly drew the boundaries of this new landscape by coining the phrase: “three screens and a cloud.”

“So, moving forward, again I believe that the world some number of years from now in terms of how we consume IT is really shifting from a machine-centric viewpoint to what we refer to as three screens and a cloud: the phone, the PC, and the TV ultimately, and how we deliver value to them.�?

Ozzie’s phrase assumes the transition from locally-installed software to mostly cloud computing. It equalizes, and puts into the same field, three devices with historically separate development and usage paths. It also reduces all of the physical characteristics of the devices to the virtual, by way of a screen. In addition, the specific historical uses of these devices is replaced with delivering value from the Network. This implies that the functionality of these separate channels has been absorbed, blended, and can be delivered over the Network.

Some assume all of these devices are being absorbed into the personal computer, but if you track the evolution of the PC’s form factor you can see that it’s been reduced to an input (keyboard, mouse, camera, microphone) and an output (screen). The CPU has largely disappeared from the experience, it’s been reduced to the primary user interaction points. This is just a preparation for its ultimate absorption into the new three screen ecosystem.

There’s a fixed screen that creates a large high-definition experience and draws the user to it. This screen is appropriate for individuals or social groups. There’s a small mobile screen that the user takes with her everywhere she goes. This is a private screen, mostly for individual use. And there’s a medium-sized screen that you bring along when there’s a specific work/play purpose requiring a larger interaction surface, or when you need a device that bridges the private and the public.

If you think about the mobile phone market prior to the release of the iPhone; the transition to a platform in which a “small screen delivers value from the Network” seemed an impossibility. The players were entrenched and the carriers controlled the device market. The deal that was cut with AT&T, along with the revaluation of all values in the mobile device market, created a new starting point. There was no evolutionary path from the old mobile telephone to the iPhone. Although technically, it’s a small computer, Jobs was specifically aiming at creating the small personal screen.

“I don’t want people to think of this as a computer,�? he said. “I think of it as reinventing the phone.�?

Apple dropped “Computer” from it’s name and placed a large bet on the post-PC future with the iPhone. They have publicly reset their strategic direction and now describe themselves as a ‘mobile devices company.” The iPad doubles down on mobility and bets that the netbook was a rough sketch of what would be useful as a second screen in a mobile computing context. Both the iPhone and iPad— through multi-touch— have continued to reduce the frame of interaction. The screen is transformed and becomes both the input and the output for the user’s experience.

A key development in the ‘three screens and a cloud’ vision is the elimination of input devices. The screen, and the gesture space around it, serves the user for both input and output.

Google has begun to design their products with a mobile-first sensibility, and has even made public statements indicating that within three years the mobile screen will be the user’s primary interaction point with the Network. Both Chrome and Android point to mobile technology. (It should be pointed out that Android isn’t an operating system, it’s a java-based runtime that sits on top of a Linux OS. In this sense, it’s more similar to Silverlight)

Microsoft made a hard pivot with the Windows Phone 7 product. The “Life in Motion” theme and the кухниtiles and hub user interface moves away from file systems and toward lifestream themes. Add to this the porting of Silverlight to the Symbian, Android and Windows Phone platforms, throw in a connection to Azure, and you have a massive developer pipeline to the small screen.

We all like to paraphrase William Gibson on the future, it’s here, it’s just not evenly distributed yet. Although this isn’t different from most things: the past, the present and any object you’d care to choose from the physical universe. None are distributed evenly. Time, as the old joke goes, is nature’s way of keeping everything from happening at once. And therefore it follows that Space, is nature’s way of keeping everything from being just one big smoothie.

Progress toward the vision of “three screens and a cloud” will be measured in the distribution power of the major technology/media players. Apple has developed a significant channel through its innovative devices, iTunes and its physical stores. Microsoft has a strong base in operating system and office applications, but has expanded their distribution portfolio with Silverlight and Azure. Google’s distribution power is contained in their search index, which is exposed through their search query page. Facebook and Twitter’s distribution power is located in their social graph and the fire hose of their real-time index. All of these players have created vibrant developer ecosystems. This future won’t be distributed evenly, but to break through to mass markets, it will require both distribution power and a high-touch service channel.

The convergence implied in the phrase “three screens and a cloud” will consume the personal computer as well. It will be transformed, blended, and its functionality and services made accessible through any of the three screens. Preparations have long been underway for the a Post-PC future. The productivity once available only through the old devices and channels has been migrating quickly to the new Network-connected screens. Google has now joined Microsoft and Apple in attending to the possibilities of the large screen. These changes aren’t taking place as a gradual evolution, there’s a dangerous leap required to reach this new platform. Not every company will have the strength, capital and will to make that leap. And as the old devices and channels are hollowed out, at some point there will be a major collapse of the old platforms.

In the war rooms around the technology world, there’s a conversation going on about what it will take to get to the other side.

6 Comments

Intuition and The UX of Physics Engines, Both Literal and Imaginary

Published February 16, 2010 by cgerrish

The transition from a store and retrieve computing experience to that of a real-time stream is still rippling through all aspects of our human-computer relationship. At the point of interaction, we’ve moved through punch cards, command lines and graphic user interfaces. We’ve coalesced around the Apple Human Interface Guidelines for installed software, and then splintered in a thousand directions for web-based software. The conservative impulse of the usability movement caused a brief fascination with the plain vanilla HTML 1.0 UI. The advent of vector-animation engines (Flash, then Silverlight) and then Ajax and dynamic HTML (javascript + CSS) exploded the interaction surface into a thousand variations. Taking a cue from 3-D first-person immersion games, the iPhone (and iPad) imported the physics of our every day mechanical interfaces and settled on the metaphor of “reality” for the multi-touch screen interaction surface.

Of course, when we speak of physics, it’s from a very specific perspective. We’re looking at the how the physical world experienced by human beings on the third stone from the Sun. Here we don’t discover physics, but rather we produce a physics by way of a physics engine.

A physics engine is a computer program that simulates physics models, using variables such as mass, velocity, friction, and wind resistance. It can simulate and predict effects under different conditions that would approximate what happens in real life or in a fantasy world. Its main uses are in scientific simulation and in video games.

As a designer of interaction surfaces, I often hear the request for an “intuitive user interface.” Most businesses would like there to be a zero learning curve for their online products. In practice what this means is creating a pastiche of popular interface elements from other web sites. The economics of the “intuitive interface” means this practice is generally replicated with the result of a bland set of interaction models becoming the norm. And once the blessing of “best practice” is bestowed, interaction becomes a modular commodity to be snapped into place on a layout grid. Conformity to the best practice becomes the highest virtue.

Arbitrary interaction metaphors have to be learned. If they’ve been learned elsewhere, so much the better. To the extent that it exists, the user’s intuition is based on previous experiences with the arbitrary symbols of an interaction system. Intuition isn’t magical, it works from a foundation of experience.

With the advent of the iPhone, we’ve slowly been exposed to a new method of bringing intuition into play. The interaction system is simply aligned with the physics and mechanics of the real world. Apple’s human interface guidelines for the iPhone and iPad do exactly this. A simple example is Apple’s design for a personal calendar on the iPad. It looks like a physical personal calendar. The books, look and work like physical books. It’s as though non-euclidean geometry were the norm, and suddenly someone discovered euclidean geometry.

By using the physics and mechanics of the real world as a symbolic interaction framework, a user’s intuition can be put to use immediately. Deep experience with the arbitrary symbolic systems of human-computer interaction isn’t required to be successful. If a user can depend on her everyday experience with objects in the world as the context for interaction; and have an expectation about how the physics of direct manipulation through multi-touch will work, then you have the foundation for an intuitive user interface.

CD-ROM multi-media experiences, moving to immersion-oriented electronic games and virtual worlds like Second Life have started us down this path, but the emergence of the World Wide Web deferred the development of this model in the application space. Of course, there’s a sense in which this isn’t what we’ve come to know as web site design at all. Once you eliminate the keyboard, mouse and documents as the primary modes of interaction, and substitute direct manipulation via multi-touch things change rapidly. The base metaphor of real life spawns an unlimited variety of possible interaction metaphors. And unlike arbitrary interaction systems, diversity doesn’t damage the user’s intuitions about how things work. Creativity is returned to the design of interaction surfaces.

Tightly integrated software and hardware designs, initially from Apple, but now from Microsoft and Google as well, are laying out a new canvas for the Network. The primary development platforms on the software side are iPhone OS, Android, Webkit, Silverlight and Flash. We won’t compare these runtimes based on whether they’re ‘open’ or ‘closed’ – but rather based on the speed and flexibility of their physics engines. To what degree are these platforms able to map real life in real time to a symbolic interaction surface? To what extent do I have a sense of mass, friction, momentum, velocity and resistance when I touch them? Do I have the sense that the artifacts on the other side of the glass are blending and interacting with real time as it unfolds all around me? The weakest runtime in the bunch is Webkit (HTML5/H.264), and it’s also the one that ultimately may have the broadest reach. HTML5 was partially envisioned as a re-orientation of the web page from the document to the application. The question is whether it can adapt quickly enough to the new real-time, real world interaction surface. Can it compete at the level of physics, both literal and imaginary?

Comments closed

The Virtual as Analog: Selectors and the iPad

Published February 10, 2010 by cgerrish

It turns out the virtual is analog. The analog is being atomized, the atoms mapped to bits, and then reassembled on the other side of the glass. It’s probably something like how we imagine teleportation will work. As computer interfaces advance, they are tending to look more like real life. We’ve always connected to the digital through a keyboard, or a cursor control, and set of commands in the form of text or menus. As the iPad continues the roll out of touch screens and multi-touch gestures— this model will radically change. While radical change in computer interface usually means having to learn a whole new set of random abstractions to trigger actions; this change is a radical simplification. The layer of abstraction is no longer random. The physical world is being abstracted into a symbolic layer, a control and interaction surface, to act on the software operating on the other side of the glass. The physics and culture of the natural world provide the context we need to understand how to interact with, and control, the software.

In the light of this new interaction environment, initiatives like Information Cards start to make a lot more sense. In analyzing the problem of internet identity, including the subtopics of authentication, authorization, roles and claims— it became clear that a metaphor was required. Something that would connect to a person’s everyday experience with identity in the real world. The idea of wallets (selectors) and cards seemed like a natural fit. The behaviors an individual would be expected to perform with a selector are analogous to those done every day with the wallet in your back pocket or purse.

The problem with information cards has been that the computing environment hasn’t allowed human-computer interaction at the level of real world analogy. Web site login screens are geared toward keyboards and text fields, not toward accepting cards from a wallet (selector). Now imagine using a selector on an iPad. It looks like a wallet. You can apply whatever surface style that complements your personal style. You’ve filled it with cards— both identity cards and action cards. When you surf to a web site or an application that requires authentication, your selector is activated and provides you with a small selection of cards that can be used for this context. You choose one, slide it out of the selector with your finger and drag it to the appropriate spot on the screen. In the new era of the iPad, that’s an interaction model that makes perfect sense.

In their interaction design guidelines, Apple addresses the issue of metaphors very directly:

When possible, model your application’s objects and actions on objects and actions in the real world. This technique especially helps novice users quickly grasp how your application works

Abstract control of applications is discouraged in favor of direct manipulation:

Direct manipulation means that people feel they are controlling something tangible, not abstract. The benefit of following the principle of direct manipulation is that users more readily understand the results of their actions when they can directly manipulate the objects involved.

Originally selectors were tied to a specific device, and this made them impractical when hopping between multiple devices. However a number of cloud-based selectors have recently emerged to solve this problem. As with all current internet identity solutions, there’s a lot of machinery at work under the covers. But from the user’s perspective, simply selecting a card and tossing it to the software application requesting authentication will radically reduce friction for both the user and the system.

Taking the metaphor a step further, it’s simple to imagine carrying my selector on an iPhone or iPad (or similar device) and using it to replace many of the cards I now carry in my wallet. The authentication event, rather than occurring within a particular device, would occur between devices. The phone becomes a key.

This new interaction environment heralds a radical change in the way we work and play with computers. Authentication, internet identity and information cards are just one example. We could have just as easily examined the human-computer interface of the financial services industry. Portfolio management and analysis, stock trading, and research will all need to be re-imagined in light of radical simplicity of this new world. The random, abstract and symbolic interfaces of computing will start to look quite antique by this time next year.

8 Comments

Meshing the Network: Let’s Go To The Hop

Published February 7, 2010 by cgerrish

Everything seems to begin in the middle and then spiral out to a temporal beginning. Whenever I begin to think about wireless communication technology and the Network, I always end up contemplating the mystery of Hedy Lamarr. Lamarr and composer George Antheil did the conceptual work on frequency-hopping spread-spectrum wireless communications in 1941. They were awarded a patent for their work in 1942 (Lamarr under her married name at the time, Markey).

Lamarr’s and Antheil’s frequency-hopping idea serves as a basis for modern spread-spectrum communication technology, such as COFDM used in Wi-Fi network connections and CDMA used in some cordless and wireless telephones. Similar patents had been granted to others earlier, such as in Germany in 1935 to Telefunken engineers Paul Kotowski and Kurt Dannehl who also received U.S. Patent 2,158,662 and U.S. Patent 2,211,132 in 1939 and 1940. Blackwell, Martin and Vernam’s Secrecy Communication System patent from 1920 (1598673) does seem to lay the communications groundwork for Kiesler and Antheil’s patent which employed the techniques in the autonomous control of torpedoes.

Hopping along the spectrum from Lamarr’s time to the present, it’s the iPad that continues to bring the computing environment into focus. Where mobile and wireless were considered secondary modes of use, we now understand them as primary modes. The laptop has moved from the category of portable to that of transportable. And while it will physically fit on your lap, it’s now clear that the laptop is better suited to a table or desk. It’s the iPad that fits comfortably into your lap and stands ready to use as soon as you pick it up. While the desktop computer is a wired machine, and the laptop can either be wired or wireless— the iPad is purely wireless. Purely mobile, purely wireless.

As this new device (the iPad as the definition of a general category) begins its diffusion into the wild, our focus will turn to the availability of the over-the-air Network. This is the natural habitat of the iPad; it lives in the places where there’s wireless network connectivity. In our homes we can set up a cozy nest for the iPad with lots of wireless signal. But once we step out of the door, we’re at the mercy of the fates. With iPad, as with the iPhone, we’re largely dependent on AT&T’s GSM network. And for other devices, it will be other carriers. While there’s a strong focus on ‘coverage’ by cellular network carriers by both users and the networks themselves— we haven’t given the supplementary wifi network the same scrutiny.

For wifi connectivity, we look to a patchwork of hotspots. We scan for signal, looking to see if there’s open network where we can get a connection. Maybe I can get it in that cafe up the street. I seem to remember that park around the corner had public wifi. And that hotel? The wifi there was as slow as molasses in January. Oh, and don’t even get me started about the wifi at that tech conference, everybody jumped on it— and it collapsed. Nobody even got a taste.

The iPad implies that a coherent wifi network will grow up in the places where people need it. A meshed Wifi environment looms in front of us as an opportunity. When Google sponsors free wifi on Virgin airlines flights, and AT&T sponsors free wifi at McDonald’s franchises, you see the beginnings of a huge advertising surface emerging around us.

As this mesh of wifi forms around the heavily trafficked pathways of our lives, we’ll want to take advantage of the hops spread across the spectrum— the ones that Hedy Lamarr imagined. We’ll want to hop seamlessly from wifi network to wifi network as we move from this store to that one. From this museum to that cafe. And we’ll expect the cellular network to fill in the gaps. Optimizing these hops for signal strength, cost of bandwidth and local discounts, offers and transaction capability will give the iPad, and iPhone, a home in the world.

Now, of course, we’d like that experience without commercial interruption. But there’s a ready business model that we already understand: on the channels that we pay a subscription fee, we won’t see commercials. On the channels where we don’t directly pay a fee, we’ll watch commercials– or trade data and gestures, for access. The key is the hand-off to the next local environment, the smooth hop to the next connection— meshing the networks together into a seamless experience. And where we used to see a difference between network providers and broadcasters, in a two-way broadcasting system— those differences begin to dissolve.

There’s an old New Yorker cartoon that shows a row of pizza joints jammed right next to each other on a block in Manhattan. As you look at them from left to right, you see the signs in their windows. The first one says: “Best Pizza in New York City!”; the second one blares: “Best Pizza in the USA!”; the third one proclaims: “Best Pizza in the World!”; the fourth one tops them all with: “Best Pizza in the Universe!”; and with the fifth pizza joint we see the proprietor standing out front smiling, and the sign in his window says: “Best Pizza on this Block.”

Competing in this new environment won’t mean spanning the globe with network coverage, rather it’s the microcaster with the best bundle of services, offers, and connectivity in real time, in the spot where you’re standing right now, who will win the day.

One Comment