Skip to content →

Category: interaction design

Surveillance Keiretsu

I stumbled across some near-term plans for the Amazon Corporation. It’s funny to think that they started as an online bookstore. Now it’s hard to say exactly what they are. They’ve purchased other companies, branched off into space exploration and are even pioneering delivery by drone.

I think everyone will agree that the new set of services they’re working on will move online commerce to a whole new level. The last mile problem has been there since the beginning. An order can be placed at the speed of light and the large national distribution networks get the goods quite close to where the customer lives very quickly. But getting packages from a local distribution warehouse to a specific residence ends up being the most expensive part of the distribution process.

Even if the packages arrive on schedule and are placed on the doorstep, they are often stolen by criminals cruising neighborhoods. These crooks trail delivery vans and pick off packages that look like they might have resale value on the black market. Customers are always complaining about stolen packages.

This is why Amazon bought Ring, the home security company. Ring puts video cameras on your front door and around your property. If a thief approaches, intending to steal a package, the video cameras capture an image of the person’s face. Recently Ring customers within particular neighborhoods have started sharing these photos. “Watch out for this guy, he’s stealing packages.” Often the photos are also shared with the local police.

Here’s where Amazon can really add value through its network of companies and infrastructure services. Imagine a future with even faster delivery and free of package thieves. By combining drone delivery and Ring’s home surveillance technology, you’ll never lose another package.

Here’s a typical scenario. The customer places an order. The item is picked and packed, and moved into the distribution chain. The package arrives at a local distribution center and is assigned to a drone for home delivery. The drone races to your house and places the package on the designated receiving location. What this? A thief sees the delivery, waits until the drone is out of sight, then moves in to steal the package. Here’s where Ring’s network-connected cameras kick in. The cameras are watching the receiving area—having been notified by the drone that a delivery was imminent. The images of the thief are sent to the Amazon Cloud for processing. The photo of the thief is compared to the family of consumers occupying the house. If there’s no match, the algorithm goes through the extended family, work colleagues and friends. It looks through address books and photo albums to see if there’s any possible match. Given what’s coming next, Amazon doesn’t want to make a mistake.

It looks like there’s no match. This person is stealing your package. The image is now compared to outstanding arrest warrants and neighborhood watch photos. Based on several year’s worth of video footage, the algorithm produces a list of people who have no regular pattern of activity in your neighborhood to determine if this person has been casing the neighborhood. All the while, Amazon’s facial recognition systems are attempting to identify the individual. As a courtesy, Amazon shares the information with the Immigration and Customs Enforcement agency (ICE) to determine whether enforcement action should be coordinated.

At the same time, Amazon’s drones working in the area are alerted to the theft and they begin to gather into a swarm. The swarm tracks the thief as he tries to make his escape. Since Ring cameras are installed in almost every home in the neighborhood, it’s straightforward to track his route. The thief’s location is transmitted to the drone swarm and the cameras on the drones make an identification and lock in and begin tracking the thief. Amazon echo nodes in neighborhood homes notify residents via Alexa to shelter in place while the action is executed.

We’re Watching.

Ideally, the drone swarm will want to take action before the thief enters a vehicle. Even if full identification hasn’t been completed, the drone swarm will move in to herd the thief toward a designated location that has been communicated to local police. Since the police can’t always immediately respond to this kind of incident, the drone swarm is equipped to keep the “suspect” in the designated location for up to 12 hours.

If the thief has abandoned the package, and it appears undamaged, a drone will break off from the swarm and re-deliver it to your home. Damaged packages are taken by drone back to the local distribution warehouse and a request for a replacement item is automatically generated.

Once the police arrive on the scene, all video and audio evidence, along with any background profile data, is transmitted. Generally this results in an open-and-shut case when delivered to the District Attorney’s office. A permanent record is created in Amazon’s central data warehouse to make sure once this person has served their time in prison they receive heightened surveillance on release and for the rest of their lives.

Recidivism is the tendency of a convicted criminal to reoffend. Using the vast resources of the Amazon family of companies, we can often deter a person from reoffending by foregrounding the surveillance apparatus at a key moment prior to a criminal act. Sometimes all it takes is a reminder that someone is watching, and that any criminal act will be swiftly and surely punished.

That’s the future, but here’s some things Amazon is working on today…

Adding Face Recognition to Your Front Door

Comments closed

Networked Door Knocker

There's a television commercial making the rounds about a new type of home security. It's a new door bell that sports a video camera and ties into your home WiFi network. This allows you to answer your door from anywhere.

In the commercial, a bad guy rings the doorbell. He's surprised to find that someone answers. Confused, he stammers, “I'm giving free home painting estimates.” The homeowner, a woman, is out and about, but she's able to respond, “I'm bathing the children right now.” Foiled, the bad guy leaves.

It's a good thing bad guys don't watch television. If they did, they might learn to recognize this new networked door bell, and the kind of things people say to give the impression they're at home. Bad guys might also start listening to the background noise to check whether it's consistent with “being at home.”

“You say you're bathing the children, but it sounds like you're at a cafe. You really shouldn't keep your front door key under the flower pot. Have a nice day!”

I can see the value in a product like this. You could remotely respond to delivery services about whether it's okay to leave a package on the doorstep. Or when you're running a little late, it could be used to tell people you've agreed to meet that you'll be home in a few minutes. You could even give out Halloween candy to trick-or-treaters from a remote location.

A networked door bell has many uses, but home security isn't one of them. Especially if you give away the game in your sales pitch. Even though the Network has thoroughly woven itself into the fabric of daily life, we seem to have no idea how it works and what it's good for. The reality is, we barely understand the real impact of the telephone yet.

 

Comments closed

Talk UI: Pushing Your Buttons

Now that there are always-on audio interfaces to networked applications, we begin the conversation about “Talk UI.” In some number of houses there are devices listening and ready to execute commands. These commands and their acknowledgement have the form of a conversation between humans.

An optimistic vision of this interaction might be the computer on the television series “Star Trek.” A dystopian vision would be HAL in the Stanley Kubrick film “2001: A Space Odyssey.”

At this moment in audio interfaces, we're closer to the bad handwriting translations of the Apple Newton, or the unintended word transformations of “auto-complete” in texting applications. We ask the audio interface a question and we get back a non-sequitur. We sigh, and type in a specific query.

I can get my Apple TV to show subtitles to a Danish television series by saying: “subtitles (pause) on.” But I can't say, “Siri, please turn the subtitles on.” That's because this isn't a conversational user interface. Words aren't words as they are generally used by humans. Words are buttons, they have specific meanings. The spoken sounds must mean just what the interaction designer chose them to mean, neither more nor less.

“When I use a word,” Humpty Dumpty said in rather a scornful tone, “it means just what I choose it to mean–neither more nor less.”

“The question is,” said Alice, “whether you can make words mean so many different things.”

“The question is,” said Humpty Dumpty, “which is to be master–that's all.”

The Talk UI is “like” a conversation. It has some of the form of a conversation, while not actually being a conversation. We call it a “Conversational UI” to sell it to the masses. They will be disappointed unless they understand that this new thing is just pushing buttons with sound.

The surface area of today's sound buttons is too small. They're hard to press. Creating a larger surface area is the usability challenge for this new interface.

 

Comments closed

Reading Mont Blanc

There are a couple of reasons that writing has migrated toward the screen. The biggest reason is that it's cheaper. The production process migrated to the screen, and in the end, it seemed easier to skip the part where you turn digital files into plates, smear ink on them, and print the offset onto paper. Once lots of people had screens that could serve as readers, the economics of it gained traction.

The same thing happened in movies and television, photography, and music. The consumption device is just a simpler version of the machines, or set of machines, used to produce the work.

The flexibility and agility provided by digital production methods hasn't really translated into the artwork. There are a few experimental attempts, but nothing has broken through into the mass market. A few people are working on computational narrative outside of the video game context. Generative music has also been available at your local app store for a while.

These kind of generative and computational works take the form of software applications. Computing power and algorithms are a necessary element of the product. They sit in a kind of no man's land between traditional media and video games. For the most part, the digital publication has simply been a cheaper form of print. As the hypertext medium matures, we'll need to see something more than “cheap.” Eventually, the audience won't be impressed with “free” or “cheap.” Libraries are filled with “free” books, but it's not on that basis that a reader checks out a book.

I started down this train of thought because of a book I checked out of San Francisco's Mechanic's Library. It's a membership library located downtown. If you like chess, it houses a beautiful chess room. Motivated by reading Tim Morton's review in the “LA Review of Books” of Steven Shaviro's “The Universe of Things,” I became interested in reading the poem that resonates so strongly with the title of Shaviro's book.

The poem is called “Mont Blanc” and was written in 1816 by Percy Bysshe Shelley. The poem is available online through a number of sources. I own several volumes of Shelley's work that contain the poem. Whenever I'm in a used book store I look for unfamiliar editions of poetry by William Blake and Percy Shelley. While their poetry is widely available, most of the editions are not very readable–tiny type, horrible layout.

Book publishers still working with ink and paper have also succumbed to the trend of producing the cheapest product possible. And when it comes to so-called classics, the worst tendencies of cheapness converge. It's as though the publisher cynically believes that it's enough to say one owns the complete works of Shakespeare in a single volume. Of course, no one would waste their time actually reading the plays; so why bother making them a pleasure to read?

You've probably seen these kinds of books. Their unapproachability has nothing to do with their status as “high art.” It's just that the type is too small. They're technically readable, in that, all the words have been converted to ink on paper. This is perhaps where the saying “machine readable” comes from.

Back to the poem. I'd been reading Shelley's “Mont Blanc” every evening for several weeks. I find that I need to read a poem a number of times over an extended period before it begins to function as a poem. I'd been switching off between various books that contained the poem. And then recently, I happened to be in the Mechanic's Library looking for something else, and thought I should find out if they had a nice edition of Shelley's poetry.

When I got to the designated shelf, I recognized the dark green seven volume, hard bound set of “The Complete Works of Percy Bysshe Shelley.” For some reason I'd always resisted it. I think I'd had a previous bad experience with an old edition of Coleridge's poems. Completely unreadable. I pulled down a volume and started paging through it. What a revelation.

This edition was published by Virtue & Company out of Boston in 1909. It's the library edition, and bears the number, 141 out of 1000. It has beautiful illustrations, and was edited by Nathan Haskell Dole. The volumes are simple, durable and luxurious.

Reading “Mont Blanc” in this beautiful edition, with excellent typography and a generous layout, was a qualitatively different experience. The poem has technically been printed in many books. All the words are there, and the lineation is correct, but not every printing of the poem actually does service to both the poem and the reader. The quality of the ink and the paper has something to do with it, but one also has the sense the publishers have a real understanding of what they're committing to paper. It's as though they knew in advance what it would be like to read these poems in this particular configuration.

Due to financial constraints, much of print publishing has lost its sense of usability. There are, of course, some notable exceptions. Web design has gone through the opposite problem. The “usability” profession killed design in the production of web sites. Some good principles were unearthed, but the usability specialists over-played their hand. We're only starting to see web sites (if there still is such a thing as a “web site”) wriggle out from under the boot of usability.

Reading the poem “Mont Blanc” online isn't a particularly pleasurable experience. The screen, and the vast network of interconnected pages behind it, seem to work against the flow of the poem. Rather than open the reader to the experience of the natural world flowing through the senses, as though one were a kind of Aeolian harp; the hypertext screen radiates the opposite polarity, and entices the reader to flow her attention through the connecting paths of the Network.

While it is true that the cost of publishing the written word will always be cheaper in some digital format, the value of the work in many cases is diminished. In the last year, a few online publications have started to break the mold and create more reader-friendly screen publications. Perhaps as we read more and more online, we'll begin to realize the absolute poverty of the reading experience. It's not very good. When the economics of publication, whether for print or screen, tells us that we can't afford to do good work–that's when the whole thing really starts to fall apart. We've been facinated with the idea of commodity prices approaching zero. What we're learning (again) is that value tends to follow price as it moves toward zero.

Where is that revaluation of digital values that we've been waiting for?

 

Comments closed