Skip to content →

Category: design

Intuition and The UX of Physics Engines, Both Literal and Imaginary

Published February 16, 2010 by cgerrish

The transition from a store and retrieve computing experience to that of a real-time stream is still rippling through all aspects of our human-computer relationship. At the point of interaction, we’ve moved through punch cards, command lines and graphic user interfaces. We’ve coalesced around the Apple Human Interface Guidelines for installed software, and then splintered in a thousand directions for web-based software. The conservative impulse of the usability movement caused a brief fascination with the plain vanilla HTML 1.0 UI. The advent of vector-animation engines (Flash, then Silverlight) and then Ajax and dynamic HTML (javascript + CSS) exploded the interaction surface into a thousand variations. Taking a cue from 3-D first-person immersion games, the iPhone (and iPad) imported the physics of our every day mechanical interfaces and settled on the metaphor of “reality” for the multi-touch screen interaction surface.

Of course, when we speak of physics, it’s from a very specific perspective. We’re looking at the how the physical world experienced by human beings on the third stone from the Sun. Here we don’t discover physics, but rather we produce a physics by way of a physics engine.

A physics engine is a computer program that simulates physics models, using variables such as mass, velocity, friction, and wind resistance. It can simulate and predict effects under different conditions that would approximate what happens in real life or in a fantasy world. Its main uses are in scientific simulation and in video games.

As a designer of interaction surfaces, I often hear the request for an “intuitive user interface.” Most businesses would like there to be a zero learning curve for their online products. In practice what this means is creating a pastiche of popular interface elements from other web sites. The economics of the “intuitive interface” means this practice is generally replicated with the result of a bland set of interaction models becoming the norm. And once the blessing of “best practice” is bestowed, interaction becomes a modular commodity to be snapped into place on a layout grid. Conformity to the best practice becomes the highest virtue.

Arbitrary interaction metaphors have to be learned. If they’ve been learned elsewhere, so much the better. To the extent that it exists, the user’s intuition is based on previous experiences with the arbitrary symbols of an interaction system. Intuition isn’t magical, it works from a foundation of experience.

With the advent of the iPhone, we’ve slowly been exposed to a new method of bringing intuition into play. The interaction system is simply aligned with the physics and mechanics of the real world. Apple’s human interface guidelines for the iPhone and iPad do exactly this. A simple example is Apple’s design for a personal calendar on the iPad. It looks like a physical personal calendar. The books, look and work like physical books. It’s as though non-euclidean geometry were the norm, and suddenly someone discovered euclidean geometry.

By using the physics and mechanics of the real world as a symbolic interaction framework, a user’s intuition can be put to use immediately. Deep experience with the arbitrary symbolic systems of human-computer interaction isn’t required to be successful. If a user can depend on her everyday experience with objects in the world as the context for interaction; and have an expectation about how the physics of direct manipulation through multi-touch will work, then you have the foundation for an intuitive user interface.

CD-ROM multi-media experiences, moving to immersion-oriented electronic games and virtual worlds like Second Life have started us down this path, but the emergence of the World Wide Web deferred the development of this model in the application space. Of course, there’s a sense in which this isn’t what we’ve come to know as web site design at all. Once you eliminate the keyboard, mouse and documents as the primary modes of interaction, and substitute direct manipulation via multi-touch things change rapidly. The base metaphor of real life spawns an unlimited variety of possible interaction metaphors. And unlike arbitrary interaction systems, diversity doesn’t damage the user’s intuitions about how things work. Creativity is returned to the design of interaction surfaces.

Tightly integrated software and hardware designs, initially from Apple, but now from Microsoft and Google as well, are laying out a new canvas for the Network. The primary development platforms on the software side are iPhone OS, Android, Webkit, Silverlight and Flash. We won’t compare these runtimes based on whether they’re ‘open’ or ‘closed’ – but rather based on the speed and flexibility of their physics engines. To what degree are these platforms able to map real life in real time to a symbolic interaction surface? To what extent do I have a sense of mass, friction, momentum, velocity and resistance when I touch them? Do I have the sense that the artifacts on the other side of the glass are blending and interacting with real time as it unfolds all around me? The weakest runtime in the bunch is Webkit (HTML5/H.264), and it’s also the one that ultimately may have the broadest reach. HTML5 was partially envisioned as a re-orientation of the web page from the document to the application. The question is whether it can adapt quickly enough to the new real-time, real world interaction surface. Can it compete at the level of physics, both literal and imaginary?

Comments closed

The Virtual as Analog: Selectors and the iPad

Published February 10, 2010 by cgerrish

It turns out the virtual is analog. The analog is being atomized, the atoms mapped to bits, and then reassembled on the other side of the glass. It’s probably something like how we imagine teleportation will work. As computer interfaces advance, they are tending to look more like real life. We’ve always connected to the digital through a keyboard, or a cursor control, and set of commands in the form of text or menus. As the iPad continues the roll out of touch screens and multi-touch gestures— this model will radically change. While radical change in computer interface usually means having to learn a whole new set of random abstractions to trigger actions; this change is a radical simplification. The layer of abstraction is no longer random. The physical world is being abstracted into a symbolic layer, a control and interaction surface, to act on the software operating on the other side of the glass. The physics and culture of the natural world provide the context we need to understand how to interact with, and control, the software.

In the light of this new interaction environment, initiatives like Information Cards start to make a lot more sense. In analyzing the problem of internet identity, including the subtopics of authentication, authorization, roles and claims— it became clear that a metaphor was required. Something that would connect to a person’s everyday experience with identity in the real world. The idea of wallets (selectors) and cards seemed like a natural fit. The behaviors an individual would be expected to perform with a selector are analogous to those done every day with the wallet in your back pocket or purse.

The problem with information cards has been that the computing environment hasn’t allowed human-computer interaction at the level of real world analogy. Web site login screens are geared toward keyboards and text fields, not toward accepting cards from a wallet (selector). Now imagine using a selector on an iPad. It looks like a wallet. You can apply whatever surface style that complements your personal style. You’ve filled it with cards— both identity cards and action cards. When you surf to a web site or an application that requires authentication, your selector is activated and provides you with a small selection of cards that can be used for this context. You choose one, slide it out of the selector with your finger and drag it to the appropriate spot on the screen. In the new era of the iPad, that’s an interaction model that makes perfect sense.

In their interaction design guidelines, Apple addresses the issue of metaphors very directly:

When possible, model your application’s objects and actions on objects and actions in the real world. This technique especially helps novice users quickly grasp how your application works

Abstract control of applications is discouraged in favor of direct manipulation:

Direct manipulation means that people feel they are controlling something tangible, not abstract. The benefit of following the principle of direct manipulation is that users more readily understand the results of their actions when they can directly manipulate the objects involved.

Originally selectors were tied to a specific device, and this made them impractical when hopping between multiple devices. However a number of cloud-based selectors have recently emerged to solve this problem. As with all current internet identity solutions, there’s a lot of machinery at work under the covers. But from the user’s perspective, simply selecting a card and tossing it to the software application requesting authentication will radically reduce friction for both the user and the system.

Taking the metaphor a step further, it’s simple to imagine carrying my selector on an iPhone or iPad (or similar device) and using it to replace many of the cards I now carry in my wallet. The authentication event, rather than occurring within a particular device, would occur between devices. The phone becomes a key.

This new interaction environment heralds a radical change in the way we work and play with computers. Authentication, internet identity and information cards are just one example. We could have just as easily examined the human-computer interface of the financial services industry. Portfolio management and analysis, stock trading, and research will all need to be re-imagined in light of radical simplicity of this new world. The random, abstract and symbolic interfaces of computing will start to look quite antique by this time next year.

8 Comments

iPad and the new Puritanism

Published February 3, 2010 by cgerrish

The framing of the debate in advance of the availability of the iPad device has centered around control of the words “freedom, choice and health.” The reactionary forces claim the iPad will be detrimental to all three. Within minutes of the conclusion of Steve Jobs’s presentation, the swiftboating of the iPad was under way. Our freedoms are being curtailed; our choices limited and the health of the ecosystem is threatened. The iPad is a deviation from the one true path.

Another vector of dissatisfaction involves the paternity of the iPad. One look at its features and cognitive dissonance sets in among its detractors. Surely a tablet computer is the child of the laptop– what happened to all the ports, the keyboard; where’s the operating system? This Jobs fellow has disfigured the child, removing ports to satisfy some twisted personal vision— and, no doubt, because he hates our freedoms.

But the iPad is the child of the iPhone, not the laptop. There never was a physical keyboard, the ports are the same and so is the operating system. You can’t remove a port that was never there. The child resembles its parent in every way. Although, one must admit, it’s quite unusual for the child to be larger than the parent. Brian Dear, who has criticisms of his own, may have captured the seed of the idea in an interaction between Steve Jobs and Alan Kay:

…Steve announced the iPhone for the first time to the public. After the event, Alan recalled Steve walking up to him to show him the new iPhone in person. He asked Alan, “So, did we build something worth criticizing?” Alan recalls telling him sure, but if you could just make the screen 5″ by 8″, you would take over the world. Steve’s eyes apparently lit up.

The iPad is definitely worth criticizing. And what can you say about a product when both people who love it, and those who hate it, admit they must buy it? Its significance is big enough that simply withholding attention and comment will have little effect on the course of events. You’ll have to pick the device up and touch the glass.

In advance of holding one in my hands, there are two areas I’d like to explore where it looks like the iPad will open doors and make new connections possible.

While we attribute much of the success of the human species to our brains, we owe an equal amount to the dexterity and power of our hands. While the power of thought enables us to apply levels of abstraction to the world; it’s the fine motor control over our hands that allows us to bring these ideas into the world. Apple made a significant design decision when they chose the hand over the stylus, the scrollwheel, the trackball and the physical keyboard. Our fingertip touching the glass makes a direct connection to the virtual controls. This is both a technical and an emotional interaction point. The pleasure derived from the multi-touch interface with real-time responsiveness will quickly make it the most desirable and the most used.

The larger multi-touch interaction surface of the iPad will open the door to a whole range of personal computing gestures. It’s likely that electronic gaming will lead the way, and this is where users will try out thousands of new gestures — some will rise to the top and become defacto standards. Here the opposable thumb, the power grip and the precision grip will expand their repertoire to include the pinch, the spread, the swipe, the twist and others. And the accelerometer expands this interaction model into three dimensions.

And while Apple isn’t the only company whose worked extensively on these new interaction models, they get first mover advantage for introducing the tablet computer in 2007 and calling it a telephone. Creating a product in the lab is one thing, but putting a product on the street and getting uptake is another entirely. We’re seeing the new digital network joined to the original digital network.

Another kind of interaction that the iPad will change is the technology-mediated consultation. In these scenarios, typically the client gives the professional a bunch of personal data. That data is entered into a laptop or a desktop, numbers are crunched, reports formatted and then presented to the client in a meeting. The technology introduces large breaks into the human interaction. Melanie Rodier in her post for Wall Street & Technology asked whether the iPad has a place in the capital markets:

“I see the laptop as not a good collaborative tool,� he explains. A laptop computer acts as a barrier between the adviser and client, since it is only one person (presumably the adviser) who will handle the keyboard, hit the return key and flip the computer round to face the client.

On the other hand, with an iPad, you can reach over and touch the screen, instead of having to navigate with a button or a mouse.  “With the iPad, an adviser could navigate better, sweep pages across, look at a client’s financial plan and statements, and work more collaboratively side by side. It’s a subtle difference. The touch-screen means you can sit on the same side of the table, work as a partner,� Dannemiller adds.

This kind of change isn’t necessarily in the algorithms used to crunch the data, but rather in the human interaction— the human transaction between the parties. The bond between the consultant and client is strengthened, as is the connection between the client and her data. The tablet format changes the sheer physicality of the interaction. The affordances, or action possibilities, of the consultation are no longer dominated or broken by the technical mediation. The devices recede to allow the people to connect.

The new Puritans have sprung up to save us all from this new impure device. Somehow I’m reminded of the early reaction to Elvis Presley, or when Dylan went electric. The iPad has abandoned the folk movement, it’s sold out, it’s left the one true path of protest music. The iPad smiles and looks askance. “Well, I just think of myself as a song and dance man. You know…”

The Puritans have coalesced around the mantra of “open.” For the Web, HTML5 is the sacred book— although it is unfinished, and there are still many sects within the fold. A specification isn’t a platform, so HTML5 must extend its control through the major denominations: Webkit, Gecko, Trident, KHTML and Presto. When we say, for instance, that HTML5 supports the H.264 video codec, what we’re really saying is that a runtime plugin for that media type must ship with all orthodox HTML layout engines. On this point of scripture, the Gecko sect has reservations. Runtimes outside the Holy stack are strictly the devil’s work.

But purity is just a myth. Dylan’s electric music had elements of the folk song; and his folk songs had elements of rock and roll. At bottom the “closed” iPhone OS was built up from the open source Darwin operating system. And Darwin had its roots in Berkeley Unix. If you want to build an “open” iPhone OS, you can take the same raw materials and knock yourself out.

The real question is what does it take to break through? To create a new kind of computing experience, not in the labs, but in people’s homes, in the cafes, schools and businesses. The answer isn’t in feature sets, ports or adhering to technical religious dogma. When you pick the device up, does it make you want to dance? I know that the puritans have forbidden dance. They’ve given us stern warnings about what will happen if we dance with this new partner. But when you put your fingers on the glass, do you feel a tingling in your toes?

One Comment

Surfing The Waves of Technology

Published January 26, 2010 by cgerrish

The company vilified by some as being too closed to be successful in the long run has— in the long run, defined and distributed the dominant model for human-computer interaction. The reality is that all products that brandish the so-called open systems label are operating within the parameters set by Apple.

And while it’s certainly true that Apple didn’t create any of these interaction modes out of whole cloth, they codified them, shipped and sold the products that have turned them into defacto standards.

A de facto standard is a custom, convention, product, or system that has achieved a dominant position by public acceptance or market forces (such as early entrance to the market). De facto is a Latin phrase meaning “concerning the fact” or “in practice”.

In the beautiful silence emanating from Apple prior to the January 27, 2010 announcements a curious thing has happened. The full attention of the technical intelligensia has been focused on what’s missing from our personal and social computing experience. The announcements will be an interesting test of the ‘wisdom of the crowds.’ Theoretically, the predictions and analysis of the thousands of individuals writing about what will be announced could be distilled into either exactly the device Apple intends to release, or a blueprint for an even better device. My bet is that we will be surprised.

Of course, we can point to Xerox Parc, or Doug Engelbart, and say none of these things are new. But moving ideas from the lab to the street is a matter of knowing which dots to connect. In an interview, Jobs talks about recognizing the valuable waves of technology:

“Things happen fairly slowly, you know. They do. These waves of technology, you can see them way before they happen, and you just have to choose wisely which ones you’re going to surf. If you choose unwisely, then you can waste a lot of energy, but if you choose wisely it actually unfolds fairly slowly. It takes years.�

In order to connect dots, you need to be in a position to do so. Sometimes we tend to overlook the core skill set that Apple has amassed. Here’s Jobs talking about what Apple does:

“Well, Apple has a core set of talents, and those talents are: We do, I think, very good hardware design; we do very good industrial design; and we write very good system and application software. And we’re really good at packaging that all together into a product. We’re the only people left in the computer industry that do that. And we’re really the only people in the consumer-electronics industry that go deep in software in consumer products. So those talents can be used to make personal computers, and they can also be used to make things like iPods. And we’re doing both, and we’ll find out what the future holds.�

So, while we live in an era of “organizing without organizations,” can we expect distributed organizations harnessing the crowd to produce, sell and ship products at the same level as Apple? Crowds have a difficult time indicating what should be left out— and this is a key to superior industrial design. Here’s Job’s on Apple’s design process:

“Look at the design of a lot of consumer products—they’re really complicated surfaces. We tried make something much more holistic and simple. When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can oftentimes arrive at some very elegant and simple solutions. Most people just don’t put in the time or energy to get there. We believe that customers are smart, and want objects which are well thought through.�

In 2007, Apple changed its name from Apple Computer to Apple. In some sense, this signaled the end of the era of the personal computer. The computer has begun its migration and blending into other devices— some existing, others yet to be invented. Here’s Jobs on where the revolution is going:

“I know, it’s not fair. But I think the question is a very simple one, which is how much of the really revolutionary things people are going to do in the next five years are done on the PCs or how much of it is really focused on the post-PC devices. And there’s a real temptation to focus it on the post-PC devices because it’s a clean slate and because they’re more focused devices and because, you know, they don’t have the legacy of these zillions of apps that have to run in zillions of markets.�

While there have been tablet computers for quite a long time, they were primarily designed as an evolution of the personal computer. In thinking about Apple’s announcement, the previous frame of reference is wrong— just as it is for those who believe the iPhone is a telephone. In looking at what’s missing from our social computing environment, we think we know the set of dots that need to be connected. But if we sit with the problem long enough, a whole new set of dots will come into focus. Here’s Jobs on vision and design:

“There’s a phrase in Buddhism,�Beginner’s mind.� It’s wonderful to have a beginner’s mind.�

One Comment