On augmenting reality

The following is the draft of a section from my forthcoming book, The City Is Here For You To Use, concerning various ways in which networked devices are used to furnish the mobile pedestrian with a layer of location-specific information superimposed onto the forward view — “augmented reality,” in other words. (The context is an extended discussion of four modes in which information is returned from the global network to the world so it may be engaged, considered and acted upon, which is why the bit here starts in medias res.)

As you see it here, the section is not quite in its final form; it hasn’t yet been edited for meter, euphony or flow, and in particular, some of the arguments toward the end remain too telescoped to really stand up to much inspection. Nevertheless, given the speed at which wearable AR is evolving, I thought it would be better to get this out now as-is, to garner your comments and be strengthened by them. I hope you enjoy it.

1

One seemingly potent way of returning networked information to the world would be if we could layer it directly over that which we perceive. This is the premise of so-called augmented reality, or AR, which proposes to furnish users with some order of knowledge about the world and the objects in it, via an overlay of informational graphics superimposed on the visual field. In principle, this augmentation is agnostic as to the mediating artifact involved, which could be the screen of a phone or tablet, a vehicle’s windshield, or, as Google’s Glass suggests, a lightweight, face-mounted reticle.

AR has its conceptual roots in informational displays developed for military pilots in the early 1960s, at the point when the performance of enemy fighter aircraft began to overwhelm a human pilot’s ability to react. In the fraught regime of jet-age dogfighting, even a momentary dip of the eyes to a dashboard-mounted instrument cluster could mean disaster. The solution was to project information about altitude, airspeed and the status of weapons and other critical aircraft systems onto a transparent pane aligned with the field of vision, a “head-up display.”

This notion turned to have applicability in fields beyond aerial combat, where the issue wasn’t so much reaction time as it was visual complexity. One early AR system was intended to help engineers make sense of the gutty tangle of hydraulic lines, wiring and control mechanisms in the fuselage of an airliner under construction; each component in the otherwise-hopeless confusion was overlaid with a visual tag identifying it by name, and colored according to the system it belonged to.

Other systems were designed to help people manage situations in which both time and the complexity of the environment were sources of pressure — for example, to aid first responders in dispelling the fog and chaos they’re confronted with upon arrival at the scene of an emergency. One prototype furnished firefighters with visors onto which structural diagrams of a burning building were projected, along with symbols indicating egress routes, the position of other emergency personnel, and the presence of electric wiring or other potentially dangerous infrastructural elements.

The necessity of integrating what were then relatively crude and heavy cameras, motion sensors and projectors into a comfortably wearable package limited the success of these early efforts — and this is to say nothing of the challenges posed by the difficulty of establishing a reliable network connection to a mobile unit. But the conceptual heavy lifting done to support these initial forays produced a readymade discourse, waiting for the day augmentation might be reinstantiated in smaller, lighter, more capable hardware.

That is a point we appear to have arrived at with the advent of the smartphone. As we’ve seen, the smartphone handset can be thought of as a lamination together of several different sensing and presentation technologies, subsets of which can be recombined with one another to produce distinctly different ways of engaging networked information. Bundle a camera, accelerometer/gyroscope, and display screen in a single networked handset, and what you have in your hands is indeed an artifact capable of sustaining rudimentary augmentation. Add GPS functionality and a three-dimensional model of the world — either maintained onboard the device, or resident in the cloud — and a viewer can be offered location-specific information, registered with and mapped onto the surrounding urban fabric.

In essence, phone-based AR treats the handset like the transparent pane of a cockpit head-up display: you hold it before you, its camera captures the forward-facing view, and this is rendered on the screen transparently but for whatever overlay of information is applied. Turn and the on-screen view turns with you, tracked (after a momentary stutter) by the grid of overlaid graphics. And those graphics can provide anything the network can: identification, annotation, direction or commentary.

It’s not hard to see why developers and enthusiasts might jump at this potential, even given the sharp limits imposed by the phone as platform. We move through the world and we act in it, but the knowledge we base our movements and actions on is always starkly less than what it might be. And we pay the price for this daily, in increments of waste, frustration, exhaustion and missed opportunity. By contrast, the notion that everything the network knows might be brought to bear on someone or -thing standing before us, directly there, directly present, available to anyone with the wherewithal to sign a two-year smartphone contract and download an app — this is a deeply seductive idea. It offers the same aura of omnipotence, that same frisson of godlike power evoked by our new ability to gather, sift and make meaning of the traces of urban activity, here positioned as a direct extension of our own senses.

2

Why not take advantage of this capability? After all, the richness and complexity of city life confronts us with any number of occasions on which the human sensorium could do with a little help.

Let a few hundred neurons in the middle fusiform gyrus of the brain’s right hemisphere be damaged, or fail to develop properly in the first place, and the result is a disorder called prosopagnosia, more commonly known as faceblindness. As the name suggests, the condition deprives its victims of the ability to recognize faces and associate them with individuals; at the limit, someone suffering with a severe case may be entirely unable to remember what his or her loved ones look like. So central is the ability to recognize others to human socialization, though, that even far milder cases cause significant problems.

Sadly, this is something I can attest to from firsthand experience. Like an estimated 2.5%[1] of the population, I suffer from the condition, and even in the relatively attenuated form I’m saddled with, my broad inability to recognize people has caused more than a few experiences of excruciating awkwardness. At least once or twice a month I run into people on the street who clearly have some degree of familiarity with me, and find myself unable to come up with even a vague idea of who they might be; I’ll introduce myself to a woman at a party, only to have her remind me (rather waspishly, but who can blame her) that we’d worked together on a months-long project. Deprived of contextual cues — the time and location at which I usually meet someone, a distinctive hairstyle or mode of dress — I generally find myself no more able to recognize former colleagues or students than I can complete strangers. And as uncomfortable as this can be for me, I can only imagine how humiliating it is for the person on the other end of the encounter.

I long ago lost track of the number of times in my life at which I would have been grateful for some subtle intercessionary agent: something that might drop a glowing outline over the face of someone approaching me and remind me of his or her name[2], the occasion on which we met last, maybe even what we talked about on that occasion. It would spare both of us from mortification, and shield my counterpart from the inadvertent but real insult implied by my failure to recognize them. So the ambition of using AR in this role is lovely — precisely the kind of sensitive technical deployment I believe in, where technology is used to lower the barriers to socialization, and reduce or eliminate the awkwardnesses that might otherwise prevent us from better knowing one another.

But it’s hard to imagine any such thing being accomplished by the act of holding a phone up in front of my face, between us, forcing you to wait first for me to do so and then for the entire chain of technical events that must follow in order to fulfill the aim at the heart of the scenario. The device must acquire an image of your face with the camera, establish the parameters of that face from the image, and upload those parameters to the cloud via the fastest available connection, so they may be compared with a database of facial measurements belonging to known individuals; if a match is found, the corresponding profile must be located, and the appropriate information from that profile piped back down the connection so it may be displayed as an overlay on the screen image.

Too many articulated parts are involved in this interaction, too many dependencies — not least of which is the coöperation of a Facebook, a Google, or some other enterprise with a reasonably robust database of facial biometrics, and that is of course wildly problematic for other reasons. Better I should have confessed my confusion to you in the first place.

Perhaps a less technologically-intensive scenario would be better suited to the phone as platform for augmentation? How about helping a user find their way around the transit system, amidst all the involutions of the urban labyrinth?

3

Here we can weigh the merits of the use case by considering an actual, shipping product, Acrossair’s Nearest Subway app for the iPhone, first released in 2010[3]. Like its siblings for London and Paris, Nearest Tube and Nearest Metro, Nearest Subway uses open location data made available by the city’s transit authority to specify the positions of transit stops in three-dimensional space. On launch, the app loads a hovering scrim of simple black tiles featuring the name of each station, and icons of the lines that serve it; the tiles representing more distant stations are stacked atop those that are closer. Rotate, and the scrim of tiles rotates with you. Whichever way you face, you’ll see a tile representing the nearest subway station in the direction of view, so long as some outpost of the transit network lies along that bearing in the first place.

Nearest Subway is among the more aesthetically appealing phone-based AR applications, eschewing junk graphics for simple, text-based captions sensitively tuned to the conventions of each city’s transit system. If nothing else, it certainly does what it says on the tin. It is, however, almost completely worthless as a practical aid to urban navigation.

When aimed to align with the Manhattan street grid from the corner of 30th Street and First Avenue, Nearest Subway indicates that the 21st Street G stop in Long Island City is the closest subway station, at a distance of 1.4 miles in a north-northeasterly direction.

As it happens, there are a few problems with this. For starters, from this position the Vernon Boulevard-Jackson Avenue stop on the 7 line is 334 meters, or roughly four New York City blocks, closer than 21st Street, but it doesn’t appear as an option. This is either an exposure of some underlying lacuna in the transit authority’s database — unlikely, but as anyone familiar with the MTA understands implicitly, well within the bounds of possibility — or more probably a failure on Acrossair’s part to write code that retrieves these coordinates properly.


Just as problematically, the claimed bearing is roughly 55 degrees off. If, as will tend to be the case in Manhattan, you align yourself with the street grid, a phone aimed directly uptown will be oriented at 27 degrees east of due north, at which point Nearest Subway suggests that the 21st Street station is directly ahead of you. But it actually lies on an azimuth of 82 degrees; if you took the app at its word, you’d be walking uptown a long time before you hit anything even resembling a subway station. This is most likely to be a calibration error with the iPhone’s compass, but fairly or otherwise Nearest Subway shoulders the greater part of the blame here — as anyone familiar with computational systems has understood since the time of Babbage, if you put garbage in, you’ll get garbage out.

Furthermore, since by design the app only displays those stations roughly aligned with your field of vision, there’s no way for it to notify you that the nearest station may be directly behind your back. Unless you want to rotate a full 360 degrees, then, and make yourself look like a complete idiot in the process, the most practical way to use Nearest Subway is to aim the phone directly down, which makes a reasonably useful ring of directional arrows and distances pop up. (These, of course, could have been superimposed on a conventional map in the first place, without undertaking the effort of capturing the camera image and augmenting it with a hovering overlay of theoretically compass-calibrated information.)

However unfortunate these stumbles may be, they can all be resolved, addressed with tighter code, an improved user interface or a better bearing-determination algorithm. Acrossair could fix them all, though — enter every last issue in a bug tracker, and knock them down one by one — and that still wouldn’t address the primary idiocy of urban AR in this mode: from 30th Street and First Avenue, the 21st Street G stop is across the East River. You need to take a subway to get there in the first place. However aesthetically pleasing an interface may be, using it to find the closest station as the crow flies does you less than no good when you’re separated from it by a thousand meters of water.

Finally, Nearest Subway betrays a root-level misunderstanding of the relationship between a citydweller and a transportation network. In New York City, as in every other city with a complex underground transit system, you almost never find yourself in a situation where you need to find the station that’s nearest in absolute terms to begin with; it’s far more useful to find the nearest station on a line that gets you where you want to go. Even at the cost of cluttering what’s on the screen, then, the very first thing the would-be navigator of the subway system needs is a way to filter the options before them by line.

I raise these points not to park all of the blame at Acrossair’s door, but to suggest that AR itself is badly unsuited to this role, at least when handled in this particular way. It takes less time to load and use a map than it does to retrieve the same information from an augmentive application, and the map provides a great deal more of the context so necessary to orienting yourself in the city. At this point in technological evolution, then, more conventional interface styles will tend to furnish a user with relevant information more efficiently, with less of the latency, error and cruft that inevitably seem to attend the attempt to superimpose it over the field of vision.

4

If phone-based augmentation performs poorly as social lubricant or aid to urban navigation, what about another role frequently proposed for AR, especially by advocates in the cultural heritage sector? This use case hinges on the argument that by superimposing images or other vestiges of the past of a place directly over its present, AR effectively endows its users with the ability to see through time.

This might not make much sense at all in Songdo, or Masdar, or any of the other new cities now being built from scratch on greenfield sites. But anyone who lives in a place old enough to have felt the passage of centuries knows that history can all too easily be forgotten by the stones of the city. Whatever perturbations from historical events may still be propagating through the various flows of people, matter, energy and information that make a place, they certainly aren’t evident to casual inspection. An augmented view returning the layered past to the present, in such a way as to color our understanding of the things all around us, might certainly prove to be more emotionally resonant than any conventional monument.

Byzantium, old Edo, Roman Londinium, even New Amsterdam: each of these historical sites is rife with traces we might wish to surface in the city occupying the same land at present. Locales overwhelmed by more recent waves of colonization, gentrification or redevelopment, too, offer us potent lenses through which to consider our moment in time. It would surely be instructive to retrieve some record of the jazz- and espresso-driven Soho of the 1950s and layer it over what stands there at present; the same goes for the South Bronx of 1975. But traversed as it was during the twentieth century by multiple, high-intensity crosscurrents of history, Berlin may present the ultimate terrain on which to contemplate recuperation of the past.

This is a place where pain, guilt and a sense of responsibility contend with the simple desire to get on with things; no city I’m familiar with is more obsessively dedicated to the search for a tenable balance between memory and forgetting. The very core of contemporary Berlin is given over to a series of puissant absences and artificially-sustained presences, from the ruins of Gestapo headquarters, now maintained as a museum called Topography of Terror, to the remnants of Checkpoint Charlie. A long walk to the east out leafy Karl-Marx-Allee — Stalinallee, between 1949 and 1961 — takes you to the headquarters of the Stasi, the feared secret police of the former East Germany, also open to the public as a museum. But there’s nowhere in Berlin where the curious cost of remembering can be more keenly felt than in the field of 2,711 concrete slabs at the corner of Ebertstrasse and Hannah-Arendt-Strasse. This is the Memorial to the Murdered Jews of Europe, devised by architect Peter Eisenman, with early conceptual help from the sculptor Richard Serra.

Formally, the grim array is the best thing Eisenman has ever set his hand to, very nearly redemptive of a career dedicated to the elevation of fatuous theory over aesthetic coherence; perhaps it’s the Serra influence. But as a site of memory, the Monument leaves a great deal to be desired. It’s what Michel Foucault called a heterotopia: something set apart from the ordinary operations of the city, physically and semantically, a place of such ponderous gravity that visitors don’t quite know what to make of it. On my most recent visit, the canyons between the slabs rang with the laughter of French schoolchildren on a field trip; the children giggled and flirted and shouted to one another as they leapt between the stones, and whatever the designer’s intent may have been, any mood of elegy or commemoration was impossible to establish, let alone maintain.

Roughly two miles to the northeast, on the sidewalk in front of a doner stand in Mitte, is a memorial of quite a different sort. Glance down, and you’ll see the following words, inscribed into three brass cubes set side by side by side between the cobblestones:

HIER WOHNTE
ELSA GUTTENTAG
GEB. KRAMER
JG. 1883

DEPORTIERT 29.11.1942
ERMORDET IN
AUSCHWITZ

HIER WOHNTE
KURT GUTTENTAG
JG. 1877

DEPORTIERT 29.11.1942
ERMORDET IN
AUSCHWITZ

HIER WOHNTE
ERWIN BUCHWALD
JG. 1892

DEPORTIERT 1.3.1943
ERMORDET IN
AUSCHWITZ

Ermordet in Auschwitz: that is, on specific dates in November of 1942 and March of the next year, the named people living at this address were taken across this very sidewalk and forcibly transported hundreds of miles east by the machinery of their own government, to a country they’d never known and a facility expressly designed to murder them. The looming façades around you were the last thing they ever saw as free people.

It’s in the dissonance between the everyday bustle of Mitte and these implacable facts that the true horror resides — and that’s precisely what makes the brass cubes a true memorial, indescribably more effective than Eisenman’s. The brass cubes, it turns out, are Stolpersteine, or “stumbling blocks,” a project of artist Gunter Demnig; these are but three of what are now over 32,000 that Demnig has arranged to have placed in some 700 cities. The Stolpersteine force us to read this stretch of unremarkable sidewalk in two ways simultaneously: both as a place where ordinary people go placidly about their ordinary business, just as they did in 1942, and as one site of a world-historical, continental-scale ravening.

The stories etched in these stones are the kind of facts about a place that would seem to yield to a strategy of augmentation. The objection could certainly be raised that I found them so resonant precisely because I didn’t see them every day, and that their impact would very likely fade with constant exposure; we might call this the evil of banality. But being compelled to see and interpret the mundane things I did in these streets through the revenant past altered my consciousness, in ways subtler and longer-lasting than anything Eisenman’s sepulchral array of slabs was able to achieve. AR would merely make the metaphor literal — in fact, it’s easy for me to imagine the disorienting, decentering, dis-placing impact of having to engage the world through a soft rain of names, overlaid onto the very places from which their owners were stolen.

But once again, it’s hard to imagine this happening via the intercession of a handset. Nor are the qualities that make smartphone-based AR so catastrophically clumsy, in virtually every scenario of use, particularly likely to change over time.

The first is the nature of functionality on the smartphone. As we’ve seen, the smartphone is a platform on which each discrete mode of operation is engaged via a dedicated, single-purpose app. Any attempt at augmenting the environment, therefore, must be actively and consciously invoked, to the exclusion of other useful functionality. The phone, when used to provide such an overlay, cannot also and at the same time be used to send a message, look up an address, buy a cup of coffee, or do any of the other things we now routinely expect of it.

The second reservation is physical. Providing the user with a display surface for graphic annotation of the forward view simply isn’t what the handset was designed to do. It must be held before the eyes like a pane of glass in order for the augmented overlay to work as intended. It hardly needs to be pointed out that this gesture is not one particularly well-suited to the realities of urban experience. It has the doubly unappealing quality of announcing the user’s distraction and vulnerability to onlookers, while simultaneously ensuring that the device is held in the weak grip of the extended arm — a grasp from which it may be plucked with relative ease.

Taken together, these two impositions strongly undercut the primary ostensible virtue of an augmented view, which is its immediacy. The sole genuine justification for AR is the idea that information is simply there, copresent with that you’re already looking at and able to be assimilated without thought or effort.

That sense of effortlessness is precisely what an emerging class of wearable mediators aims to provide for its users. The first artifact of this class to reach consumers is Google’s Glass, which mounts a high-definition, forward-facing camera, a head-up reticle and the microphone required by the natural-language speech recognition interface on a lightweight aluminum frame. While Glass poses any number of aesthetic, practical and social concerns — all of which remain to be convincingly addressed, by Google or anyone else — it does at least give us a way to compare hands-free, head-mounted AR with the handset-based approach.

Would any of the three augmentation scenarios we explored be improved by moving the informational overlay from the phone to a wearable display?

5

A system designed to mitigate my prosopagnosia by recognizing faces for me would assuredly be vastly better when accessed via head-mounted interface; in fact, that’s the only scenario of technical intervention in relatively close-range interpersonal encounters that’s credible to me. The delay and physical awkwardness occasioned by having to hold a phone between us goes away, and while there would still be a noticeable saccade or visual stutter as I glanced up to read your details off my display, this might well be preferable to not being remembered at all.

That is, if we can tolerate the very significant threats to privacy involved, which only start with Google’s ownership of or access to the necessary biometric database. There’s also the question of their access to the pattern of my requests, and above all the one fact inescapably inherent to the scenario: that people are being identified as being present in a certain time and place, without any necessity whatsoever of securing consent on their part. By any standard, this is a great deal of risk to take on, all to lubricate social interactions for 2.5% of the population.

Nearest Subway, as is, wouldn’t be improved by presentation in the line of sight. Given what we’ve observed about the way people really use subway networks, information about the nearest station in a given direction wouldn’t be of any greater utility when splashed on a head-up display than it is on the screen of a phone. Whatever the shortcomings of this particular app, though, they probably don’t imply anything in particular about the overall viability of wearable AR in the role of urban navigation, and in many ways the technology does seem rather well-suited to the wayfinding challenges faced by the pedestrian.

Of the three scenarios considered here, though, it’s AR’s potential to offer novel perspectives on the past of a place that would be most likely to benefit from the wearable approach. We would quite literally see the quotidian environment through the lens of a history superimposed onto it. So equipped, we could more easily plumb the psychogeographical currents moving through a given locale, better understand how the uses of a place had changed over time, or hadn’t. And because this layer of information could be selectively surfaced — invoked and banished via voice command, toggled on or off at will — presenting information in this way might well circumvent the potential for banality through overfamiliarization that haunts even otherwise exemplary efforts like Demnig’s Stolpersteine.

And this suggests something about further potentially productive uses for augmentive mediators like Glass. After all, there are many kinds of information that may be germane to our interpretation of a place, yet effectively invisible to us, and historical context is just one of them. If our choices are shaped by dark currents of traffic and pricing, crime and conviviality, it’s easy to understand the appeal of any technology proposing that these dimensions of knowledge be brought to bear on that which is seen, whether singly or in combination. The risk of bodily harm, whatever its source, might be rendered as a red wash over the field of vision; point-by-point directions as a bright and unmistakable guideline reaching into the landscape. In fact any pattern of use and activity, so long as its traces were harvested by some data-gathering system and made available to the network, might be made manifest to us in this way.

Some proposed uses of mediation are more ambitious still, pushing past mere annotation of the forward view to the provision of truly novel modes of perception — for example, the ability to “see” radiation at wavelengths beyond the limits of human vision, or even to delete features of the visual environment perceived as undesirable[4]. What, then, keeps wearable augmentation from being the ultimate way for networked citizens to receive and act on information?

6

The approach of practical, consumer-grade augmented reality confronts us with a interlocking series of concerns, ranging from the immediately practical to the existential.

A first set of reservations centers on the technical difficulties involved in the articulation of an acceptably high-quality augmentive experience. We’ve so far bypassed discussion of these so we could consider different aspects of the case for AR, but ultimately they’re not of a type that allows anyone to simply wave them away.

At its very core, the AR value proposition subsists in the idea that interactions with information presented in this way are supposed to feel “effortless,” but any such effortlessness would require the continuous (and continuously smooth) interfunctioning of a wild scatter of heterogeneous elements. In order to make good on this promise, a mediation apparatus would need to fuse all of the following elements: a sensitively-designed interface; the population of that interface with accurate, timely, meaningful and actionable information; and a robust, high-bandwidth connection to the networked assets furnishing that information from any point in the city, indoors or out. Even putting questions of interface design to the side, the technical infrastructure capable of delivering the other necessary elements reliably enough that the attempt at augmentation doesn’t constitute a practical and social hazard in its own right does not yet exist — not anywhere in North America, anyway, and not this year or next. The hard fact is that for a variety of reasons having to do with national spectrum policy, a lack of perceived business incentives for universal broadband connectivity, and other seemingly intractable circumstances, these issues are nowhere near being ironed out.

In the context of augmentation, as well, the truth value of representations made about the world acquires heightened significance. By superimposing information directly on its object, AR arrogates to itself a peculiar kind of claim to authority, a claim of a more aggressive sort than that implicit in other modes of representation, and therefore ought to be held to a higher standard of completeness and accuracy[5]. As we saw with Nearest Subway, though, an overlay can only ever be as good as the data feeding it, and the augurs in this respect are not particularly reassuring. Right now, Google’s map of the commercial stretch nearest to my apartment building provides labels for only four of the seven storefront businesses on the block, one of which is inaccurately identified as a restaurant that closed many years ago. If even Google, with all the resources it has at its disposal, struggles to provide its users with a description of the streetscape that is both comprehensive and correct, how much more daunting will other actors find the same task?

Beyond this are the documented problems with visual misregistration[6] and latency that are of over a decade’s standing, and have not been successfully addressed in that time — if anything, have only been exacerbated by the shift to consumer-grade hardware. At issue is the mediation device’s ability to track rapid motions of the head, and smoothly and accurately realign any graphic overlay mapped to the world; any delay in realignment of more than a few tens of milliseconds is conspicuous, and risks causing vertigo, nausea and problems with balance and coordination. The initial release of Glass, at least, wisely shies away from any attempt to superimpose such overlays, but the issue must be reckoned with at some point if useful augmentive navigational applications are ever to be developed.

7

Another set of concerns centers on the question of how long such a mediator might comfortably be worn, and what happens after it is taken off. This is of especial concern given the prospect that one or another form of wearable AR might become as prominent in the negotiation of everyday life as the smartphone itself. There is, of course, not much in the way of meaningful prognostication that can be made ahead of any mass adoption, but it’s not unreasonable to build our expectations on the few things we do know empirically.

Early users of Google’s Glass report disorientation upon removing the headset, after as few as fifteen minutes of use — a mild one, to be sure, and easily shaken off, from all accounts the sort of uneasy feeling that attends staring overlong at an optical illusion. If this represents the outer limit of discomfort experienced by users, it’s hard for me to believe that it would have much impact on either the desirability of the product or people’s ability to function after using it. But further hints as to the consequences of long-term use can be gleaned from the testimony of pioneering researcher Steve Mann, who has worn a succession of ever-lighter and more-capable mediation rigs all but continuously since the mid-1980s. And his experience would seem to warrant a certain degree of caution: Mann, in his own words, early on “developed a dependence on the apparatus,” and has found it difficult to function normally on the few occasions he has been forcibly prevented from accessing his array of devices.

When deprived of his set-up for even a short period of time, Mann experiences “profound nausea, dizziness and disorientation”; he can neither see clearly nor concentrate, and has difficulty with basic cognitive and motor tasks[7]. He speculates that over many years, his neural wiring has adapted to the continuous flow of sensory information through his equipment, and this is not an entirely ridiculous thing to think. At this point, the network of processes that constitutes Steve Mann’s brain — that in some real albeit reductive sense constitutes Steve Mann — lives partially outside his skull.

The objection could be made that this is always already the case, for all of us — that some nontrivial part of everything that make us what we are lives outside of us, in the world, and that Mann’s situation is only different in that much of his outboard being subsists in a single, self-designed apparatus. But if anything, this makes the prospect of becoming physiologically habituated to something like Google Glass still more worrisome. It’s precisely because Mann developed and continues to manage his own mediation equipment that he can balance his dependency on it with the relative freedom of action enjoyed by someone who for the most part is able to determine the parameters under which that equipment operates.

If Steve Mann has become a radically hybridized consciousness, at least he has a legitimate claim to ownership and control over all of the places where that consciousness is instantiated. By contrast, all of the things a commercial product like Glass can do for the user rely on the ongoing provision of a service — and if there’s anything we know about services, it’s that they can be and are routinely discontinued at will, as the provider fails, changes hands, adopts a new business strategy or simply reprioritizes.

8

A final set of strictly practical concerns have to do with the collective experience of augmentation, or what implications our own choice to be mediated in this way might hold for the experience of others sharing the environment.

For all it may pretend to transparency, literally and metaphorically, any augmentive mediator by definition imposes itself between the wearer and the phenomenal world. This, of course, is by no means a quality unique to augmented reality. It’s something AR has in common with a great many ways we already buffer and mediate what we experience as we move through urban space, from listening to music to wearing sunglasses. All of these impose a certain distance between us and the full experiential manifold of the street, either by baffling the traces of it that reach our senses, or by offering us a space in which we can imagine and project an alternative narrative of our actions.

But there’s a special asymmetry that haunts our interactions with networked technology, and tends to undermine our psychic investment in the immediate physical landscape; if “cyberspace is where you are when you’re on the phone,” it’s certainly also the “place” you are when you text or tweet someone while walking down the sidewalk. I’ve generally referred to what happens when someone moves through the city while simultaneously engaged in some kind of remote interaction as a condition of “multiple adjacency,” but of course it’s really no such thing: so far, at least, only one mode of spatial experience can be privileged at a given time. And if it’s impossible to participate fully in both of these realms at once, one of them must lose out.

Watch what happens when a pedestrian first becomes conscious of receiving a call or a text message, the immediate damming they cause in the sidewalk flow as they pause to respond to it. Whether the call is made hands-free or otherwise doesn’t really seem to matter; the cognitive and emotional investment in what transpires in the interface is what counts, and this investment is generally so much greater than it is in the surroundings that street life clearly suffers as a result. The risk inherent in this divided attention appears to be showing up in the relevant statistics in the form of an otherwise hard-to-account-for upturn in accidents involving pedestrian fatalities[8], where such numbers had been falling for years. This is a tendency that is only likely to be exacerbated by augmentive mediation, particularly where content of high inherent emotional involvement is concerned.

9

At this moment in time, it would be hard to exaggerate the appeal the prospect of wearable augmentation holds for its vocal cohort of enthusiasts within the technology community. This fervor can be difficult to comprehend, so long as AR is simply understood to refer to a class of technologies aimed at overlaying the visual field with information about the objects and circumstances in it.

What the discourse around AR shares with other contemporary trans- and posthuman narratives is a frustration with the limits of the flesh, and a frank interest in transcending them through technical means. To advocates, the true appeal of projects like Google’s Glass is that they are first steps toward the fulfillment of a deeper promise: that of becoming-cyborg. Some suggest that ordinary people mediate the challenges of everyday life via complex informational dashboards, much like those first devised by players of World of Warcraft and similar massively multiplayer online role-playing games. The more fervent dream of a day when their capabilities are enhanced far beyond the merely human by a seamless union of organic consciousness with networked sensing, processing, analytic and storage assets.

Beyond the profound technical and practical challenges involved in achieving any such goal, though, someone not committed to one or another posthuman program may find that they have philosophical reservations with this notion, and what it implies for urban life. These may be harder to quantify than strictly practical objections, but any advocate of augmentation technologies who is also interested in upholding the notion of a city as a shared space will have to come to some reckoning with them.

Anyone who cares about what we might call the full bandwidth of human communication — very much including transmission and reception of those cues vital to understanding, but only present beneath the threshold of conscious perception — ought to be concerned about the risk posed to interpersonal exchanges by augmentive mediation. Wearable devices clearly have the potential to exacerbate existing problems of self-absorption and mutual inconsideration[9]. Although in principle there’s no reason such devices couldn’t be designed to support or even enrich the sense of intersubjectivity, what we’ve seen about the technologically-mediated pedestrian’s unavailability to the street doesn’t leave us much room for optimism on this count. The implication is that if the physical environment doesn’t fully register to a person so equipped, neither will other people.

Nor is the body by any means the only domain that the would-be posthuman subject may wish to transcend via augmentation. Subject as it is to the corrosive effects of entropy and time, forcing those occupying it to contend with the inconvenient demands of others, the built environment is another. Especially given current levels of investment in physical infrastructure in the United States, there is a very real risk that those who are able to do so will prefer retreat behind a wall of mediation to the difficult work of being fully present in public. At its zenith, this tendency implies both a dereliction of public space and an almost total abandonment of any notion of a shared public realm. This is the scenario imagined by science-fiction author Vernor Vinge in Rainbows End (2006), in which people interact with the world’s common furniture through branded thematic overlays of their choice; it’s a world that can be glimpsed in the matter-of-factly dystopian videos of Keiichi Matsuda, in which a succession of squalid environments come to life only when activated by colorful augmentive animations.

The most distressing consequences of such a dereliction would be felt by those left behind in any rush toward augmentation. What happens when the information necessary to comprehend and operate an environment is not immanent to that environment, but has become decoupled from it? When signs, directions, notifications, alerts and all the other instructions necessary to the fullest use of the city appear only in an augmentive overlay, and as is inevitably the case, that overlay is available to some but not others[10]? What happens to the unaugmented human under such circumstances? The perils would surely extend beyond a mere inability to act on information; the non-adopter of a particularly hegemonic technology almost always places themselves at jeopardy of being seen as a willful transgressor of norms, even an ethical offender. Anyone forgoing augmentation, for whatever reason, may find that they are perceived as somehow less than a full member of the community, with everything that implies for the right to be and act in public.

The deepest critique of all those lodged against augmented reality is sociologist Anne Galloway’s, and it is harder to answer. Galloway suggests that the discourse of computational augmentation, whether consciously or otherwise, “position[s] everyday places and social interactions as somewhat lacking or in need of improvement.” Again there’s this Greshamization, this sense of a zero-sum relationship between AR and a public realm already in considerable peril just about everywhere. Maybe the emergence of these systems will spur us to some thought as to what it is we’re trying so hard to augment. Philip K. Dick once defined reality as “that which refuses to go away when you stop believing in it,” and it’s this bedrock quality of universal accessibility — to anyone at all, at any time of his or her choosing — that constitutes its primary virtue. If nothing else, reality is the one platform we all share, a ground we can start from in undertaking the arduous and never-comfortable process of determining what else we might agree upon. To replace this shared space with the million splintered and mutually inconsistent realities of individual augmentation is to give up on the whole pretense that we in any way occupy the same world, and therefore strikes me as being deeply inimical to the urban project as I understand it. A city in which the physical environment has ceased to function as a common reference frame is, at the very least, terribly inhospitable soil for democracy, solidarity or simple fellow-feeling to take root in.

It may well be that this concern is overblown. There is always the possibility that augmented reality never will amount to very much, or that after a brief period of consideration it’s actively rejected by the mainstream audience. Within days of the first significant nonspecialist publicity around Google Glass, Seattle dive bar The 5 Point became the first commercial establishment known to have enacted a ban[11] on the device, and if we can fairly judge from the rather pungent selection of terms used to describe Glass wearers in the early media commentary, it won’t be the last. By the time you read these words, these weak signals may well have solidified into some kind of rough consensus, at least in North America, that wearing anything like Glass in public space constitutes a serious faux pas. Perhaps this and similar AR systems will come to rest in a cultural-aesthetic purgatory like that currently occupied by Bluetooth headsets, and if that does turn out to be the case, any premature worry about the technology’s implications for the practice of urban democracy will seem very silly indeed.

But something tells me that none of the objections we’ve discussed here will prove broadly dissuasive, least of all my own personal feelings on the subject. For all the hesitations anybody may have, and for all the vulnerabilities even casual observers can readily diagnose in the chain of technical articulations that produces an augmentive overlay, it is hard to argue against a technology that glimmers with the promise of transcendence. Over anything beyond the immediate near term, some form of wearable augmentive device does seem bound to take a prominent role in returning networked information to the purview of a mobile user at will, and thereby in mediating the urban experience. The question then becomes what kind(s) of urbanity will be produced by people endowed with this particular set of capabilities, individually and collectively, and how we might help the unmediated contend with cities unlike any they have known, enacted for the convenience of the ambiguously transhuman, under circumstances whose depths have yet to be plumbed.


Notes on this section
[1] Grüter T, Grüter M, Carbon CC (2008). “Neural and genetic foundations of face recognition and prosopagnosia”. J Neuropsychol 2 (1): 79–97.

[2] For early work toward this end, see http://www.cc.gatech.edu/~thad/p/journal/augmented-reality-through-wearable-computing.pdf. The overlay of a blinking outline or contour used as an identification cue, incidentally, has long been a staple of science-fictional information displays, showing up in pop culture as far back as the late 1960s. The earliest appearance I can locate is 2001: A Space Odyssey (1968), in which the navigational displays of both the Orion III spaceplane and Discovery itself relied heavily on the trope — this, presumably, because they were produced by the same contractor, IBM. See also Pete Shelley’s music video for “Homosapien” (1981) and the traverse corridors projected through the sky of Blade Runner’s Los Angeles (1982).

[3] As always, I caution the reader that the specifics of products and services, their availability will certainly change over time. All comments here regarding Nearest Subway pertain to v1.4.

[4] See discussion of “Superplonk” in [a later section]. http://m.spectrum.ieee.org/podcast/geek-life/profiles/steve-manns-better-version-of-reality

[5] At the very least, user interface should offer some kind of indication as to the confidence of a proffered identification, and perhaps how that determination was arrived at. See [a later section] on seamfulness.

[6] Azuma, “Registration Errors in Augmented Reality,” 1997.

http://www.cs.unc.edu/~azuma/azuma_AR.html

[7] http://www.nytimes.com/2002/03/14/technology/at-airport-gate-a-cyborg-unplugged.html

[8] See Governors Highway Safety Association, “Spotlight on Highway Safety: Pedestrian Fatalities by State,” 2010. http://www.ghsa.org/html/publications/pdf/spotlights/spotlight_ped.pdf; similarly, a recent University of Utah study found that the act of immersion in a conversation, rather than any physical aspect of use, is the primary distraction while driving and talking on the phone. That hands-free headset may not keep you out of a crash after all. http://www.informationweek.com/news/showArticle.jhtml?articleID=205207840

[9] A story on the New York City-based gossip site Gawker expressed this point of view directly, if rather pungently: “If You Wear Google’s New Glasses, You Are An Asshole.” http://gawker.com/5990395/if-you-wear-googles-new-glasses-you-are-an-asshole

[10] The differentiation involved might be very fine-grained indeed. Users may interact with informational objects that exist only for them and for that single moment.

[11] The first widespread publicity for Glass coincided with Google’s release of a video on Wednesday, 20th February, 2013; The 5 Point announced its ban on 5th March. The expressed concerns center more on the device’s data-collection capability than anything else: according to owner Dave Meinert, his customers “don’t want to be secretly filmed or videotaped and immediately put on the Internet,” and this is an entirely reasonable expectation, not merely in the liminal space of a dive bar but anywhere in the city. See http://news.cnet.com/8301-1023_3-57573387-93/seattle-dive-bar-becomes-first-to-ban-google-glass/

Thought for the day

The notion that the minimally diagnostic criterion of a networked object is that “it knows the right time” is very curious, in that it refers to what may be the primordially alienating regime to which human life is subjected. Is it the case, therefore, that exposure to such objects or abjects cannot help but reinforce an estrangement from the world and from being-in-the-world?

Talkin’ Everyware, tomorrow night in San Francisco

I’m pleased to pass along the news that tomorrow evening, Wednesday the 18th of August, 2010, the UX Book Club SF will be discussing my Everyware: The dawning age of ubiquitous computing. It’ll be interesting to hear what people think of the book after some four and a half years in the wild, at a time when it feels quaintly obsolescent to me, yet is objectively more relevant than it was on the day of its publication.

The meetup is at 1501 Mariposa Street, between 19.00 and 21.00. I’m going to be there, of course, but will do my damndest to maintain a dignified silence, somewhere in the back of the room. You can RSVP on the Facebook page or, I’m sure, just show up. Drinks afterward.

The overarching vision

One of the responses I often get after a talk is that, while I may have offered a critical take on existing and emergent trends in ubiquitous interactivity, I’ve failed to frame an affirmative vision of what I believe such interactivity ought to look like.

To be honest, I’m not sure this is entirely a fair cop. Sometimes the duty of an observer is simply to point out that a given situation is unfortunate, unaesthetic or undesirable, that one or another emperor is prancing around the block all buck-nekkid. This is an especially important thing to do when consensus might otherwise seal around the essential OKness of something that is really, truly Not OK.

But there’s another sense in which the complaint is not without a certain justice. There’s nothing less fair to the working designer than some dilettante, someone without any skin in the game, second-guessing their decisions — especially when the sideline sniper hasn’t had to squander their hope and energy along the nigh-endless gantlet of compromises, arguments, negotiations and endless meetings that constitutes the contemporary corporate development experience. Somewhere inside, I do feel that if I’m going to make a decent chunk of my living taking the efforts of others to pieces, it’s only right and proper that I throw something of my own on the table, to expose my own notions to the rigorous vetting I demand of any other. Some part of me feels like I should be sketching some kind of overarching, affirmative vision.

It was only liminally intentional, at best, but it’s now clear to me that over the last few months, I’ve been setting forth the building blocks of just such a thing here on Speedbird. For a variety of reasons, I’m not so hot on grand Statements of Intent at this point in my life, so it’s nowhere near as coherent as a purpose-built manifesto like this…but if you string the following (mostly rather wordy) posts together, the outlines of my stance ought to become pretty clear.

In 2010, anyway, this is my own personal vision of informatic technology at the service of the full range of human desire and complexity. Not a word of it is intended as a “solution” to what are inevitably and correctly local social or political challenges…but it is intended to give people everywhere better tools with which to join such struggles. I hope you find it useful, and invite you to subject its claims and assumptions to the same skepticism I’ve applied to other visions of ubiquitous technology.

Context

Understanding, first, the complexity of the environment in which any intervention will take place, and what kind of disciplinary tools might be useful in framing sensitive interventions.

- Toward urban system design

Frameworks for citizen responsiveness

Using ubiquitous informatics to reinforce a sense of public life and one’s own agency. Inviting new urban actors to the stage.

- Part I
- Part II: Toward a read/write urbanism

Transmobility

Using the envisioned frameworks instrumentally, to help people manage what is all-but-invariably among the most vexing challenges faced by citydwellers: getting around.

- Part I
- Part II
- Free mobility, social mobility…transmobility: Part III

Every user a developer

Arguing that the true gains will be made not by offering people powerful tools, but the ability to make their own tools of equal power.

- Part I: A brief history, with hopeful branches
- Part II: Momcomp

Respectful interfaces

Arguing that, while entirely new technical possibilities ultimately demand interface metaphors that convey the full measure of their power, they should also be designed in recognition of the environment in which they’ll be deployed.

- What Apple needs to do now
- jnd: An emergent vocabulary of form for urban screens

Every user a developer, part II, or: Momcomp

I wanted to take up a challenge Mike Migurski inadvertently laid down in comments the other day, in his response to my piece on the ongoing democratization of development for interactive systems.

I get where he’s coming from, especially his point about the moving-goalpost definition of “ease of use.” But I’m not convinced that there isn’t a whole lot further we could take tools like App Inventor toward making them painless for ordinary people to use, and I think — if you’ll forgive me, Mike — he’s mistaking my point about alternatives to the app paradigm. (It’s a little inside-baseball, but in brief: I acknowledge that the contemporary thrust of development is about things that happen in the browser, and that many “apps” are essentially specialized browsers. I just happen to believe that, despite the relative accessibility of tools like Apple’s iOS SDK, this whole model is still unnecessarily intimidating, and based on paradigms most users simply won’t get.)

So I thought I’d try a little thought experiment, to see if I couldn’t do a better job of getting my point across. I thought I’d start with a real, non-technically-inclined person, my mom, and a real challenge she used to confront on a several-times-a-month basis. And then I tried to imagine a toolkit which would allow her — as she is, and with little or no additional training — to build a custom module of functionality that would help her address this challenge, that she could use on an ad-hoc or near-realtime basis, and that would effectively lower the net frustration she experienced.

I have to say, right up front, that what I came up with is heavily, heavily dependent on circumstances which might never come to be. It posits a world in which there are widely-shared specifications for the description of networked objects we might encounter, whether those objects are people, places, things, or other kinds of system resources. And, of course, the open, shared, widely-adopted interoperability frameworks and standards that would allow us to bind these resources together and animate their interaction in useful ways. This, to put it mildly, is not the world we live in today. But it’s a world I’d like to see come to life, and if the best way to predict the future is to invent it, well: here’s my shot.

This is the use case. My mom lives in the Princeton, NJ area, a reasonably typical sweep of American suburbia that’s almost entirely predicated on automobility. Somewhere between two and five times a month, though, on a not-always-predictable basis, she has to drive to the nearest New Jersey Transit station, Princeton Junction, and there catch the train into New York City. Between the routine congestion of the area, the vagaries of the NJT timetable, and the hassle of finding parking at the station, she’s generally hugely stressed out by having to predict from among the available options the routing that will get her to the station in time, let her find a place to park, buy a ticket, and catch a given train. Our meetings in New York are generally subject to a back-and-forth flurry of last-minute phone calls: which train is she aiming for? What’s the traffic situation on Route 206? How about Route 1? Which train did she actually make? It’s not much fun, on either end, and yet something like this is how a great many people go about suturing their lives together, even in an age when information about most of the particulars here (time, location, traffic, timetable) exists, and is readily accessible from the device she has in her pocket.

Now my mom is not, in the slightest, a stupid woman. She just doesn’t like “technology.” And although she’s comfortable with (even delighted by) the iPhone UI, like a great many people she’s not the kind of person who’s going to switch back and forth between a Google Maps app and a New Jersey Transit app and whatever else she needs to come up with a relevant answer. So not only do I want to give her a single tool that offers her just the information she needs, and nothing else, but I want to give her the power to build that tool herself, so it speaks to her in something approaching her own voice.

What you see in this PDF, therefore, is a schematic representation of a constellation of plug-and-play objects she’d be choosing from and fusing together to make her ad-hoc service. Each of these objects is represented by a graphic icon and each is characterized under the surface by an arbitrary number of attributes and (inherent, dynamic and relational) attribute values. By selecting high-level, self-describing objects relevant to what she wants to do, and then using an enhanced text editor to compose what is effectively a rebus providing operators for these arguments, someone like my mom — with no technical background, or interest in or inclination toward acquiring one — can make herself a highly useful module of functionality, suited to her immediate and particular needs. She could even bundle it into a wrapper and upload it back to the network, either for someone else in nearly-identical circumstances to use as-is, or for others to deconstruct and rebuild according to their own requirements, given objects more relevant to their own local conditions.

Is it “an app”? No, not really. It’s something more, and less. It’s “just” a natural-language, textual interface layer to some reasonably complicated multivariate calculations running in the background. And in this telling of things, anyway, she built this layer herself, from available modular components fused together in an exceedingly lightweight, “intuitive” development environment. (You, Mike and the baby Jesus will have to forgive me: I’ve represented these components as something resembling Legos.)

Now I’m always a little concerned, when pushing something like this out, that I’m making myself look like that grizzled guy we’ve all encountered, wedged into a booth at All-Nite Donuts, guzzling serial cups of black coffee and scrawling his incoherent Grand Unified Theory of Everything across a stack of sweat-wrinkled legal pads. Nobody is more aware than me that there are holes in this schema you could drive a Northeast Corridor commuter train through. But I think it does a better job than I’ve yet been able to manage on two counts:

- It makes a better case than I was able to previously, regarding how easy the composition of complex functionality can and ought to be;

- And it lays out in black and white just what geomorphic feats of heavy lifting need to be taken care of in the background before any such vision could come to pass.

The things which I’ve painted as trivial here are admittedly anything but. But they are, I sincerely believe, how we’re going to handle — have to handle — the human interface to this so-called Internet of Things we keep talking about. Each of the networked resources in the world, whether location or service or object or human being, is going to have to be characterized in a consistent, natural, interoperable way, and we’re going to have to offer folks equally high-level environments for process composition using these resources. We’re going to have to devise architectures and frameworks that let ordinary people everywhere interact with all the networked power that is everywhere around them, and do so in a way that doesn’t add to their existing burden of hassle and care.

Momcomp, in other words. It’s an idea whose time I believe has come.

***
I hope you enjoy the PDF I ginned up to illustrate my above contentions. You’re free to take and use and rework it in any way you want and for what purpose you will, just so long as the use is noncommercial and you identify me as the source author. You can find the full terms of the Creative Commons license under which it’s provided to you here.

I’m shutting down threaded comments, by the way; regrettably, this otherwise-lovely theme doesn’t handle them particularly well. This has the particularly irritating consequence of rendering existing threaded discussions all but incoherent, for which I apologize. I’ve written to the theme author to see if there may be a solution. In the meantime, please try to make do. Thanks.

Every user a developer: A brief history, with hopeful branches

Google’s recent announcement of App Inventor is one of those back-to-the-future moments that simultaneously stirs up all kinds of furtive and long-suppressed hope in my heart…and makes me wonder just what the hell has taken so long, and why what we’re being offered is still so partial and wide of the mark.

I should explain. At its simplest, App Inventor does pretty much what it says on the tin. The reason it’s generating so much buzz is because it offers the non-technically inclined, non-coders among us an environment in which we can use simple visual tools to create reasonably robust mobile applications from scratch — in this case, applications for the Android operating system.

In this, it’s another step toward a demystification and user empowerment that had earlier been gestured at by scripting environments like Apple’s Automator and (to a significantly lesser degree) Yahoo! Pipes. But you used those things to perform relatively trivial manipulations on already-defined processes. I don’t want to overstate its power, especially without an Android device of my own to try the results on, but by contrast you use App Inventor to make real, usable, reusable applications, at a time when we understand our personal devices to be little more than a scrim on which such applications run, and there is a robust market for them.

This is radical thing to want to do, in both senses of that word. In its promise to democratize the creation of interactive functionality, App Inventor speaks to an ambition that has largely lain dormant beneath what are now three or four generations of interactive systems — one, I would argue, that is inscribed in the rhetoric of object-oriented programming itself. If functional units of executable code can be packaged in modular units, those units in turn represented by visual icons, and those icons presented in an environment equipped with drag-and-drop physics and all the other familiar and relatively easy-to-grasp interaction cues provided us by the graphical user interface…then pretty much anybody who can plug one Lego brick into another has what it takes to build a working application. And that application can both be used “at home,” by the developer him- or herself, and released into the wild for others to use, enjoy, deconstruct and learn from.

There’s more to it than that, of course, but that’s the crux of what’s at stake here in schematic. And this is important because, for a very long time now, the corpus of people able to develop functionality, to “program” for a given system, has been dwindling as a percentage of interactive technology’s total userbase. Each successive generation of hardware from the original PC onward has expanded the userbase — sometimes, as with the transition from laptops to network-enabled phones, by an order of magnitude or more.

The result, unseemly to me, is that some five billion people on Earth have by now embraced interactive networked devices as an intimate part of their everyday lives, while the tools and languages necessary to develop software for them have remained arcane, the province of a comparatively tiny community. And the culture that community has in time developed around these tools and languages? Highly arcane — as recondite and unwelcoming, to most of us, as a klatsch of Comp Lit majors mulling phallogocentrism in Derrida and the later works of M.I.A.

A further consequence of this — unlooked-for, perhaps, but no less significant for all of that — is that the community of developers winds up having undue influence over how users conceive of interactive devices, and the kinds of things they might be used for. Alan Kay’s definition of full technical literacy, remember, was the ability to both read and write in a given medium — to create, as well as consume. And by these lights, we’ve been moving further and further away from literacy and the empowerment it so reliably entrains for a very long time now.

So an authoring environment that made creation as easy as consumption — especially one that, like View Source in the first wave of Web browsers, exposed something of how the underlying logical system functioned — would be a tremendous thing. Perhaps naively, I thought we’d get something like this with the original iPhone: a latterday HyperCard, a tool lightweight and graphic and intuitive as the device itself, but sufficiently powerful that you could make real things with it.

Maybe that doesn’t mesh with Apple’s contemporary business model, though, or stance regarding user access to deeper layers of device functionality, or whatever shoddy, paternalistic rationale they’ve cooked up this week to justify their locking iOS against the people who bought and paid for it. And so it’s fallen to Google, of all institutions, to provide us with the radically democratizing thing; the predictable irony, of course, is that in look and feel, the App Inventor composition wizard is so design-hostile, so Google-grade that only the kind of engineer who’s already comfortable with more rigorous development alternatives is likely to find it appealing. The idea is, mostly, right…but the execution is so very wrong.

There’s a deeper issue still, though, which is why I say “mostly right.” Despite applauding any and every measure that democratizes access to development tools, in my heart of hearts I actually think “apps” are a moribund way of looking at things. That the “app economy” is a dead end, and that even offering ordinary people the power to develop real applications is something of a missed opportunity.

Maybe that’s my own wishful thinking: I was infected pretty early on with the late Jef Raskin’s way of thinking about interaction, as explored in his book The Humane Interface and partially instantiated in the Canon Cat. What I took from my reading of Raskin is the notion that chunking up the things we do into hard, modal “applications” — each with a discrete user interface, each (still!) requiring time to load, each presenting us with a new learning curve — is kind of foolish, especially when there are a core set of operations that will be common to virtually everything you want to do with a device. Some of this thinking survives in the form of cross-application commands like Cut, Copy and Paste, but still more of it has seemingly been left by the wayside.

There are ways in which Raskin’s ideas have dated poorly, but in others his principles are as relevant as ever. I personally believe that, if those of us who conceive of and deliver interactive experiences truly want to empower a userbase that is now on the order of billions of people, we need to take a still deeper cut at the problem. We need to climb out of the application paradigm entirely, and figure out a better and more accessible way of representing distributed computational processes and how to get information into and out of them. And we need to do this now, because we can clearly see that those interactive experiences are increasingly taking place across and between devices and platforms — at first for those of us in the developed world, and very soon now, for everyone.

In other words, I believe we need to articulate a way of thinking about interactive functionality and its development that is appropriate to an era in which virtually everyone on the planet spends some portion of their day using networked devices; to a context in which such devices and interfaces are utterly pervasive in the world, and the average person is confronted with a multiplicity of same in the course of a day; and to the cloud architecture that undergirds that context. Given these constraints, neither applications nor “apps” are quite going to cut it.

Accordingly, in my work at Nokia over the last two years, I’ve been arguing (admittedly to no discernible impact) that as a first step toward this we need to tear down the services we offer and recompose them from a kit of common parts, an ecology of free-floating, modular functional components, operators and lightweight user-interface frameworks to bind them together. The next step would then be to offer the entire world access to this kit of parts, so anyone at all might grab a component and reuse it in a context of their own choosing, to develop just the functionality they or their social universe require, recognize and relate to. If done right, then you don’t even need an App Inventor, because the interaction environment itself is the “inventor”: you grab the objects you need, and build what you want from them.

One, two, many Facebooks. Or Photoshops. Or Tripits or SketchUps or Spotifys. All interoperable, all built on a framework of common tools, all producing objects in turn that could be taken up and used by any other process in the weave.

This approach owes something to Ben Cerveny’s seminal talk at the first Design Engaged, though there he was primarily concerned with semantically-tagged data, and how an ecosystem of distributed systems might make use of it. There’s something in it that was first sparked by my appreciation of Jun Rekimoto’s Data Tiles, and it also has some underlying assumptions in common with the rhetoric around “activity streams.” What I ultimately derive from all of these efforts is the thought that we (yes: challenge that “we”) ought to be offering the power of ad-hoc process definition in a way that any one of us can wrap our heads around, which would in turn underwrite the most vibrant, fecund/ating planetary ecosystem of such processes.

In this light, Google’s App Inventor is both a wonderful thing, and a further propping-up of what I’m bound to regard as a stagnating and unhelpful paradigm. I’m both excited to see what people do with it, and more than a little saddened that this is still the conversation we’re having, here in 2010.

There is one further consideration for me here, though, that tends to soften the blow. Not that I’m at all comparing myself to them, in the slightest, but I’m acutely aware of what happens to the Ted Nelsons and Doug Engelbarts of the world. I’ve seen what comes of “visionaries” whose insight into how things ought to be done is just that little bit too far ahead of the curve, how they spend the rest of their careers (or lives) more or less bitterly complaining about how partial and unsatisfactory everything that actually does get built turned out to be. If all that happens is that App Inventor and its eventual, more aesthetically well-crafted progeny do help ordinary people build working tools, firmly within the application paradigm, I’ll be well pleased — well pleased, and no mistake. But in some deeper part of me, I’ll always know that we could have gone deeper still, taken on the greater challenge, and done better by the people who use the things we make.

We still can.

My back pages: Spimed

Originally published 17 October 2004 on my old v-2.org site. Very, very interesting for me to see how my feelings have evolved, and where they remain consistent; there are probably as many instances of the former as of the latter. Plus, all those “Sterlings” now feel so stilted and formal and unnatural. (Hi, Bruce!) At any rate: enjoy.

If spam simply isn’t annoying enough to suit your needs, or you’re the kind of person who’s disappointed by the disarming ease you encounter when upgrading your laptop’s operating system to a new version, then boy does Bruce Sterling ever have a vision of the future for you.

Refining the message of his much-linked speech from this year’s SIGGRAPH conference in a new piece for Wired, Sterling draws us a picture of a coming time when intelligent, deeply internetworked and self-authenticating objects dominate the physical world: an “expensive, fussy, fragile, hopelessly complex” world, where entirely new forms of “theft, fraud [and] vandalism” await us.

I preface my comments the way I do because Sterling isn’t warning us about this world. He’s enthusing about it.

To some degree, in the SIGGRAPH speech, Sterling’s thrown us a definitional curveball. Having previously defined a “blobject” as an artifact of digital creation “with a curvilinear, flowing design, such as the Apple iMac computer and the Volkswagen Beetle,” he now asks us to step back a level of abstraction, and understand the word instead to mean an object that contains its own history digitally. Possibly realizing that this bait-and-switch presents abundant opportunities for confusion, he rescues himself at the last moment by substituting for “blobject” a new coinage, “spime”: “an object tracked precisely in space and time.”

And then he proceeds to imagine a world in which this self-documenting, self-tracking, self-extending stuff he calls spime dominates utterly, or is allowed to become utterly dominant. (Whatever one thinks of this particular coinage and its descriptive utility, there clearly was the need for a word here. As Sterling quite correctly points out, this is a class of objects without precedent in human history.)

put this product into service

I have a lot to say about the notion of such chimeric object/product/service hybrids, both because I think Sterling’s onto something important and real, and because the direction he takes it in worries me.

He’s got unusually fine and sensitive antennae; as a novelist, fabulist, extrapolator, raconteur and ranter, he’s terrific. But as a designer and an organiser of design, oh…let’s just say Sterling’s taste leaves something to be desired. So when he starts talking about “an imperial paradigm…a weltanschauung and a grand schemata [sic],” for designed objects, my ears perk up.

And it’s when he suggests that we have little choice but to prepare ourselves for a world of

- spime spam (vacuum cleaners that bellow ads for dust bags);
- spime-owner identity theft, fraud, malware, vandalism, and pranks;
- organized spime crime;
- software faults that make even a mop unusable;
- spime hazards (kitchens that fry the unwary, cars that drive off bridges);
- unpredictable emergent forms of networked spime behavior;
- objects that once were inert and are now expensive, fussy, fragile, hopelessly complex, and subversive of established values…”

that I begin to get truly uncomfortable.

It’s not that Sterling’s identified the hazards improperly. Just the opposite: these are precisely (some of) the unpleasant eventualities we need to plan for in any setting of pervasive or ubiquitous “intelligence” (and which I discuss in an forthcoming article entitled “All watched over by machines of loving grace“). [Note: This article was essentially the genesis of Everyware.]

The problem is that he appears to be suggesting that “cop[ing] with” these headaches is about all that we can do, so obvious is the superiority of spime, and so inevitable its hegemony. Locked into technological determinism, he does little to challenge this here, beyond suggesting that, oh yeah, now that you mention it, this “imperial paradigm” might not necessarily be maximally convenient for its human subjects. This “ideal technology for concentration camps, authoritarian regimes, and prisons” is, yes, “a hassle. An enormous hassle.” But relax: “[I]t’s a fruitful hassle.”

With his unusually acute vision, Sterling can see something like this looming on the horizon and still be so cavalier as to suggest that, if we can only “cope with” these “hassles,” “spimes will be a massive improvement over the present closed, blind regime.” (Haven’t we heard all this better-living-through-chemistry noise before?) Such a stance strikes me as a not inconsiderable abdication of the role of anyone gifted with foresight. (It also strikes me as presuming a parallel abdication among designers, but we’ll get to that in a bit.)

It’s frustrating because I share, almost without exception, Sterling’s larger goals. He simply wants to save humanity from itself, from a situation in which we seem hellbent on drowning ourselves and whatever posterity we may achieve in tidal surges of our own noxious effluvia, and he’s looking for any help he can get from the technical side of the house. I get this, the essential good will undernetting the vision of spime.

But while I share a lot of Sterling’s faith in the ferment of human creativity, I’m not nearly as comfortable as he is with assuming that the results will always be “fruitful.”

the user and the used

I derive my suspicions not a little bit from what I know of the history of open-source software, in which applications that should by rights dominate their respective niches for their robustness or power or utility fail time and again to find the wider audience they deserve. I lay a lot of this to their user interfaces, which, designed by geeks for geeks as they are, almost invariably fail any other kind of user. The distributed nature of open-source creation seems to militate against the consistency required for a smooth, consumer-grade user experience.

Of course, one might point out that this inconsistency is inevitably implied in the core logic of open-source development, or anything like it: that notions of highly crafted user interfaces and content architectures are just so many farty, self-indulgent Rick Wakeman solos, bound to be cut down before the whirling DIY thresher of the new mutant thing.

Unless I’m badly mistaken, from what I’m able to gather from two decades of reading him this stance seems to capture something of Sterling’s position — that he doesn’t have much room for designers, mewling pitifully from the sidelines in all the impotence of their top-down, command-and-control obsolescence. Technology is destiny. The street will find its own uses; do what thou wilt shall be the whole of the law; great shall be the rejoicing.

It’s a weird thing to find myself on this side of history, given my other interests, and I’m not sure but that it may be a strategic mistake to even accept this framing of things, but here I go:

I do not believe that we want to live in a world where the best we can hope for is “wrangling” a surge of fast, cheap, out-of-control, autocatalytic blobjects. I simply do not believe that what we give up is worth less than what we are promised, even if what we are promised is delivered in anything close to full.

Control isn’t all DRM, you know. Control also means design with compassion, which is something whose complexities I believe we are just beginning to get a handle on. Control also means permitting (some) introduction of randomness in the service of a defined end. And for sure it means getting out ahead of foreseeable problems and taking measures to prevent their emergence.

To surrender this measure of control — to insist that all bottom-up, all the time is any kind of a path to a better world, and that all we can or should do is get out of the way — is fatuous, even negligent. (Indeed, “allowing otherwise avoidable dangers to manifest” defines negligence in the Anglo-American jurisprudential tradition.) Just in the last ten minutes, as I’m writing this, a correspondent tells me that an SMS-based survey inquiring as to who users believed the 100 Greatest South Africans to be had to be abandoned by its originators because the notorious fascist Eugene Terreblanche popped out at the top.

Importantly, I don’t believe that Bruce Sterling believes any such thing, either. I don’t think for a moment that he would propose that we accept, or accept himself, a situation in which people gave up all control over the things we build.

I just know, all too well, what happens to nuanced distinctions in the wild.

i contradict myself/i contain multitudes

Let me also take this opportunity to problematize even the notion that an object can usefully contain its own history. It’s a fetching, even an intoxicating idea, and you can easily see how all the ways in which such a thing might be desirable. But whose history are we talking about, exactly?

Nurri’s work on the New York Public Library’s African-American Migration Experience project provides us with a nice capsule illustration of some of the problems involved when an item is recursively accompanied by descriptive information as it travels down through time. One of her responsibilities at the Digital Library is verifying that archival images have accurate metadata, fields describing the contents of an image.

Imagine that she’s come across a picture from 1920′s Strivers’ Row, with a scrawled annotation on the back of it: “Some prominent local Negroes.” (This is not at all an atypical example.) An accurate provision of metadata, of course, requires transcribing the contemporaneous description word for word. But obviously, “prominent Negroes” is not going to fly as an object descriptor in 2004 — and nor should it, less from any feeling of political correctness (though there is that) than from the simple reason that few in 2004 are likely to search a database using the keyword “negroes,” unless it’s in a context like “Negro League baseball.”

And here the infinite regress beckons. Say you append both contemporary and historical tags to the image: “Images – Harlem – African-Americans” and “Caption – 1927 – ‘Some prominent local Negroes’”. You may have covered the obvious bases, but that’s nothing like a full history. To ensure the full understanding of someone arriving at the object from some context external in space, time, or both, you would also have to include information about the evolution of the English language and the society in which it’s used, just to explain why the 1927 label wasn’t considered appropriate a mere seventy-five years later. You see where this is going? (Sterling himself points out that “[o]nce we tag many things, we will find that there is no good place to stop tagging.”)

Sure, memory is cheap, and will be cheaper. It’s not storing such a bottomless effusion of autodescription that I’m concerned about. It’s how useful this metadata will be, any of it, when its reliability will be hard to gauge – when different parts of an object’s record, introduced at different junctures in “space and time,” may well have differing degrees of reliability, and little way to distinguish between them!

We know from the Web and from various p2p applications that, in the wild, metadata is close to useless because it can be gamed so easily; as a result, no credible search engine relies on it nor has done so for years. Is that really the new Metallica single, or is it five minutes of Lars Ulrich telling you to go fuck yourself? Is that really a captive about to be decapitated by Islamists, or is it a commercial for a crappy movie you never would have clicked on had it represented itself honestly? (Who has the authority to append metadata? Who has the responsibility, or even the technical wherewithal, to verify it?) I’m surprised that someone as savvy as Bruce doesn’t seem to grasp the implications of this for spime.

unspiming

I believe, with Bruce Sterling, that some watershed is fast approaching, past which ordinary objects will be endowed with such information-sensing ( -processing, -storage, -synthesis and -retransmission) power that both the way we understand them and the very language with which we refer to them will need to change.

Where I part ways with him, however, is in my belief that we don’t have to meekly bend over and try to “cope with” the negative consequences of any such development. As Lawrence Lessig rightly reminds us, in the destiny of any designed system, some possibilities are locked in, and others forestalled, at the level of architecture. And fortunately for all of us, when asked to submit to regimes of antihuman banality, some designers have historically had other ideas.

I can do little more than hope that this will always be the case: that those people endowed with the ability to see what’s coming over the horizon not merely describe what reaches their senses, but actively intervene to forestall the worst contingencies arising. Such an undertaking requires care and insight and discretion beyond that which we ordinarily display — myself as much as anyone else — but I firmly believe that we can choose our futures rather than have them imposed on us. In this season of decision, it is clear that in more ways than one, such a moment is now upon us.

If you want a closer look at the “spime metadata” I ginned up to serve as an illustration of this piece, it’s downloadable as a PDF here. It’s intended to represent the self-description (at time of first consumer purchase) of a notional Nike-brand t-shirt.

Join us in Helsinki on May 22nd for a Touchscapes workshop (updated)

Just in case folks here in town are feeling neglected, fear not: we’re doing events here as well.

As part of Helsinki’s World Design Capital 2010 Ideas Forum, and collaboration with our good friends at Nordkapp, I’m delighted to announce a workshop called “Touchscapes: Toward the next urban ecology.”

Touchscapes is inspired, in large part, by our frustration with the Symbicon/ClearChannel screens currently deployed around Helsinki, how little is being done with them, and how far short of their potential they’ve fallen. Our sense is that we are now surrounded by screens as we move through the city — personal devices, shared interactive surfaces, and now even building-sized displays — and if thinking about how to design for each of these things individually was hard enough, virtually nobody has given much thought to how they function together, as a coherent informational ecosystem.

Until now, that is, because that’s just what we aim to do in the workshop. Join us for a day of activity dedicated to understanding the challenges presented by this swarm of screens, the possibilities they offer for tangible, touch-based interaction, and their implications for the new urban information design. We’ll move back and forth between conceptual thinking and practical doing, developing solid ideas about making the most meaningful use of these emerging resources culturally, commercially, personally and socially.

Attendance is free, but spaces in the workshop are limited, so I recommend you sign up at Nordkapp on the Facebook event page as soon as you possibly can. See you on the 22nd!

How to bring a Systems/Layers walkshop to your town

Crossposted with Do projects.

The response to the Systems/Layers walkshop we held in Wellington a few months back was tremendously gratifying, and given how much people seem to have gotten out of it we’ve been determined to set up similar events, in cities around the planet, ever since. (Previously on Do, and see participant CJ Wells’s writeup here.)

We’re fairly far along with plans to bring Systems/Layers to Barcelona in June (thanks Chris and Enric!), have just started getting into how we might do it in Taipei (thanks Sophie and TH!), and understand from e-mail inquiries that there’s interest in walkshops in Vancouver and Toronto as well. This is, of course, wonderfully exciting to us, and we’re hoping to learn as much from each of these as we did from Wellington.

What we’ve discovered is that the initial planning stages are significantly smoother if potential sponsors and other partners understand a little bit more about what Systems/Layers is, what it’s for and what people get out of it. The following is a brief summary designed to answer just these questions, and you are more than welcome to use it to raise interest in your part of the world. We’d love to hold walkshops in as many cities as are interested in having them.

What.
Systems/Layers is a half-day “walkshop,” held in two parts. The first portion of the activity is dedicated to a slow and considered walk through a reasonably dense and built-up section of the city at hand. What we’re looking for are appearances of the networked digital in the physical, and vice versa: apertures through which the things that happen in the real world drive the “network weather,” and contexts in which that weather affects what people see, confront and are able to do.

Participants are asked to pay particular attention to:

- Places where information is being collected by the network.
- Places where networked information is being displayed.
- Places where networked information is being acted upon, either by people directly, or by physical systems that affect the choices people have available to them.

You’ll want to bring seasonally-appropriate clothing, good comfortable shoes, and a camera. We’ll provide maps of “the box,” the area through which we’ll be walking.

This portion of the day will take around 90 minutes, after which we gather in a convenient “command post” to map, review and discuss the things we’ve encountered. We allot an hour for this, but since we’re inclined to choose a command post offering reasonably-priced food and drink, discussion can go on as long as participants feel like hanging out.

Who.
Do projects’ Nurri Kim and Adam Greenfield plan and run the workshop, with the assistance of a qualified local expert/maven/mayor. (In Wellington, Tom Beard did a splendid job of this, for which we remain grateful.)

We feel the walkshop works best if it’s limited to roughly 30 participants in total, split into two teams for the walking segment and reunited for the discussion.

How.
In order for us to bring Systems/Layers to your town, we need the sponsorship of a local arts, architecture or urbanist organization — generally, but not necessarily, a non-profit. They’ll cover the cost of our travel and accommodation, and defray these expenses by charging for participation in the walkshop. In turn, we’ll ensure both that the registration fee remains reasonable, and that one or two scholarship places are available for those who absolutely cannot afford to participate otherwise.

If you’re a representative of such an organization, and you’re interested in us putting on a Systems/Layers walkshop in your area, please get in touch. If you’re not, but you still want us to come, you could try to put together enough participants who are willing to register and pay ahead of time, so we could book flights and hotels. But really, we’ve found that the best way to do things is to approach a local gallery, community group or NGO and ask them to sponsor the event.

At least as we have it set up now, you should know that we’re not financially compensated in any way for our organization of these walkshops, beyond having our travel, accommodation and transfer expenses covered.

When.
Our schedule tends to fill up 4-6 months ahead of time, so we’re already talking about events in the (Northern Hemisphere) spring of 2011. And of course, it’s generally cheapest to book flights and hotels well in advance. If you think Systems/Layers would be a good fit for your city, please do get in touch as soon as you possibly can. As we’ve mentioned, we’d be thrilled to work with you, and look forward to hearing from you with genuine anticipation and excitement. Wellington was amazing, Barcelona is shaping up to be pretty special, and Taipei, if we can pull it off, will be awesome. It’d mean a lot to us to add your city to this list. Thanks!

Transmobility, part II

Part II of our exploration of transmobility. I want to caution you, again, that this is very much a probe.

Perhaps it’s best to start by backing up a few steps and explaining a little better what I’m trying to do here. What I’m arguing is that the simple act of getting around the city is in the process of changing — as how could it not, when both paths themselves and the vehicles that travel them are becoming endowed with the power to sense and adapt?

Accordingly, I believe we need to conceive of a networked mobility, a transmobility: one that inherently encompasses different modes, that conceptualizes urban space as a field to be traversed and provides for the maximum number of pathways through that field, that gathers up and utilizes whatever resources are available, and that delivers this potential to people in terms they understand.

Yesterday, I posed the question as to how we might devise a transmobility that met all of these conditions, while at the same time acknowledging two additional, all-but-contradictory desiderata. These were the desire, on the one hand, to smoothen out our interactions with transit infrastructure until vehicular transportation becomes as natural as putting one foot in front of another, and on the other to fracture journeys along their length such that any arbitrary point can become a node of experience and appreciation in and of itself. Any system capable of meeting these objectives would clearly present us with a limit case…but then, I believe that limits are there to be approached.

Finally, I’m addressing all of these questions from a relatively unusual disciplinary perspective, which is that of the service, interaction or experience designer. The downside of this is that I’m all but certainly disinterring matters a professional transit planner or mobility designer would regard as settled questions, while missing the terms of art or clever hacks they would call upon as second nature. But there’s a significant upside, too, which is that I’m natively conversant with the interactive systems that will increasingly condition any discussion of mobility, both respectful of their power and professionally wary of the representations of reality that reach us through them.

So petrified, the landscape grows

In addressing the questions I posed yesterday, then, I’m inclined to start by holding up for examination some of the ways in which trips, routes and journeys are currently represented by networked artifacts. Maybe there’s something that can be gleaned from these practices, whether as useful insights or musts-to-avoid.

I would start by suggesting that the proper unit of analysis for any consideration of movement through urban space has to be the whole journey. This means grasping the seemingly obvious fact that from the user’s perspective, all movement from origin to destination comprises a single, coherent journey, no matter how many times a change from mode to mode is required.

I say “seemingly obvious,” because the interactive artifacts I’m familiar with generally haven’t represented circumstances this way.

Take a simple example: a trip that involves walking to the nearest bus stop, riding the bus downtown, and finally walking from the point you alight from the bus to your ultimate destination. Some of the more supple route-planning applications already capture this kind of utterly normal experience — HopStop, for example, is quite good, at least in New York City — but you’d be surprised how many still do not. To date, they’ve tended to treat journeys in terms solely of their discrete component segments: an in-car GPS system plots automotive routes, a transit route-planner provides for trips from station to station, and so on.

But people think about movements through the city in terms that are simultaneously more personal and more holistic. We think of getting to work, stopping off to pick up a few things for dinner on the way home, or heading crosstown to meet friends for drinks.

So contemporary representations already seem well-suited to one of our criteria, in that the seams between methods of getting around are stark and clear, and perhaps even stark and clear enough to imply the self-directed moments of experience that attend a journey on either side. As far as a GPS display is generally concerned, what happens in the car stays in the car, and what happens next is up to you.

Certainly as compared to some overweening, totalizing system that aimed at doing everything and wound up doing none of it well, there’s something refreshing about this humility of ambition. On the other hand, though, such systems manifestly do not lend themselves well to depicting an important variety of end-to-end trips through the city, which are those trips that involve one or more changes of conveyance.

Think back to our rudimentary example, above. It would be useful if, for the portion of the journey on which you take the bus, that bus “understood” that it was essentially functioning as a connector, a linkage between one segment traversed on foot and another.

And this is still truer of journeys involving intermodal junctures where both traffic and the systemic requirements of timetables and schedules permit you less freedom in planning than walking or cycling might. Such journey plans need to be adjusted on the fly, drawing in data from other sources to accurately account for unfolding events as they happen, with signaling carried through to the infrastructure itself so that some delay, misrouting or rupture in the original plan results in the traveler being offered a panoply of appropriate alternatives.

What if, instead of living with the vehicle, the representational system lived with the traveler, and could move with them across and between modes? On this count, we’re obviously most of the way there already: with turn-by-turn directions provided by Google Maps, the iPhone and its Android-equipped competitors spell howling doom for the single-purpose devices offered by Garmin and TomTom. The emergence of truly ambient approaches to informatic provisioning would guarantee that a traveler never lacked for situational awareness, whether or not they had access to personal devices at any given moment.

What if we could provide these systems with enough local intelligence to “know” that a specified endpoint offers n possibilities for onward travel? What if this intelligence was informed by a city’s mesh of active public objects, so that travel times and schedules and real-time conditions could all be taken into account? And finally, instead of presenting journey segments as self-contained, what if we treated them as if they enjoyed magnet physics?

Then, should you want (or be forced by exigencies beyond your control) to alter your travel plans, you could snap out the mode you’re currently using, and swap in another that met whatever bounding constraints you specified, whether those had to do with speed or accessibility or privacy or shelter from the weather. The RATP‘s head of Prospective and Innovative Design, Georges Amar, speaks of enabling transmodality, and this is just what we begin to approach here.

The distinction I’m trying to capture is essentially the same as that Lucy Suchman drew between global, a priori plans on the one hand and situated actions on the other. The result would be a more responsive journey-planning system that, given any set of coordinates in space and time, is capable of popping its head up, having a look around and helping you determine what your best options are.

Moments in modal culture

This isn’t to say that we don’t also conceive of mobility in terms of particular modes of travel, and all the allegiances and affinities they give rise to. As Ivan Illich put it, “Tell me how fast you go, and I’ll tell you who you are.”

It’s not simply the coarser distinctions that tell, either. These shades of meaning and interpretation are crucial even among and between people who share a mode of transport: a fixie rider self-evidently has a different conception of the human-bicycle mesh than a Brompton fan does, while New Yorkers will know perfectly well what I mean if I distinguish two friends by describing them respectively as a 6 train rider and a 7 type. (Though not directly analogous, you can summon up similar images by evoking the L Taraval versus the J Church, the Yamanote-sen against the Hibiya-sen, or the 73 bus against the 15.)

Those of us who ride public transit form personal connections with our stops, our stations and even with particular linkages between lines, and I can only imagine that both our cities and our lives would be impoverished if we gave that up. But there’s no particular reason we need to; all I’m suggesting here is that the total journey needs to be represented as such by all the networked systems traversed in the course of a given outing.

Neither, in devising our transmobility system, can we afford to neglect the specificities and particularities of the component systems that furnish us with its articulated linkages. If one train line isn’t interchangeable with another in the hearts and minds of their riders, the same is true of other kinds of frameworks.

For example, we can’t merely plug some abstract shared bicycle service into the mesh of modal enablers and call it a day. Consider the differing fates of two apparently similar bike-share networks, the Parisian Vélib and Barcelona’s Bicing. In their diverging histories, we can see how differences in business model wind up percolating upward to impact level of service. By limiting the right to use Bicing to residents, by requiring that users open accounts, and having those accounts tied in to the usual variety of identification data, the system provides would-be bad actors with a strong disincentive. You’re personally liable, accountable…responsible.

There are real and problematic downsides to this approach, but the difference this set of decisions makes on the street is immediate and obvious. A rank of Vélib bikes, even in a posh neighborhood, looks like a bicyclical charnelhouse, with maybe three or four out of every five saddles reversed, in what has become Parisians’ folk indicator to one another that a particular bike is damaged to the point that it’s unavailable for use. The Bicing installations that I saw, including ones seeing very heavy use in core commercial districts, aren’t nearly as degraded.

This goes to the point I was trying to make, earlier, by contrasting the older conception of a vehicle as an object to the emergent way of understanding it as a service. Even though they may be physically identical — may draw current from the same grid, may be housed in the same lot, may present the driver with the selfsame control interface — a ZipCar Prius doesn’t function in just exactly the same way as a City CarShare Prius does. You could design a transmobility system so it accounted for either or (preferably) both…but not interchangeably.

Smooth sailing

Again, though I want to enable smooth transitions, I’m not arguing for perfect seamlessness in transit, or anything like it. Kevin Lynch reminds us, in The Image of the City, that “[a]ny breaks in transportation — nodes, decision points — are places of intensified perception.” We ought to welcome some of this heightened awareness, as a counterpoint to the automaticity that can all too easily accompany the rhythms of transit ridership, especially when experienced on a daily or twice-daily basis. On the other hand, it’s true that some of this “intensified perception” is almost certainly down to the anxiety that attends any such decision under circumstances of time pressure, human density and the urgent necessity to perform modal transitions correctly — and this is the fraction I’d argue we’d be better off designing out of transmobility.

At most, I mean for transmobility systems to bolster, not replace, human intuition. Where alternative modes or routings exist, we’re already generally pretty good at using them tactically to optimize against one or another criterion. Sometimes you know the subway’s the only way you can possibly beat the gridlock and get to your appointment on time; other times you choose a taxi instead, because you need to arrive at a meeting looking fresh and composed. One day you have the time to take the bus and daydream your way downtown, and the next it doesn’t get you nearly close enough to where you need to be.

You know this, I know this. So if we’re going to propose any technical intervention at all, it had better be something that builds on our native nous for the city, not overwrites it with autistic AI.

And before we can even begin to speak credibly of integrated mobility services, we’d need to see existing systems display some awareness of the plenitude of alternatives travelers have available to them, some understanding of all the different real-time factors likely to influence journey planning.

To take the most basic example, journey planning for walkers requires a different kind of thinking about the city than, particularly, turn-by-turn directions for drivers. This isn’t simply for the obvious reasons, like car-centric routings that represent a neighborhood as a an impenetrable thicket, a maze of one-way streets all alike, that a walker would stroll on through placidly and unconcernedly.

It’s because, as thinkers from Reyner Banham to Jane Jacobs and Kevin Lynch to Ivan Illich have reminded us — and as anyone who’s ever ridden in a car already understands quite perfectly well — velocity is something like destiny. You simply attend to different cues as a walker than you do as a driver, you notice textures of a different gauge, different things matter. And of course the same thing is true for cyclists vis à vis both walkers and drivers.

Over the past eighteen months, I’ve finally seen some first sentinel signs of this recognition trickle into consumer-grade interactive systems, but we’ve still got a long, long way to go.

Musique concrète

A final step would be to design the built environment itself, the ground against which all journeys transpire, to accommodate transmobility. Why wouldn’t you, at least, plan and design buildings, street furniture and other hard infrastructure so they account for the fact of networked mobility services — both in terms of the hardware that underwrites their provision, and of the potential for variability, dynamism, and open-endedness they bring to the built landscape?

In other words: why shouldn’t a bus shelter be designed with a mobile application in mind, and vice versa? Why shouldn’t both be planned so as to take into account the vehicles and embedded sensors connected to the same network? When are we finally going to take this word “network” at face value?

Of course these technologies change — over time they get lighter, more powerful, cheaper. That’s why you design things to be easy-access, easily extensible, as modular as can be: so you can swap out the CAT5 cable and spool in CAT6 (or replace it with a WiMax transponder, or whatever). Nobody’s recommending that we ought to be hard-wiring the precise state of the art as it existed last Tuesday morning into our urban infrastructure. But anyone in a position of power who, going forward, greenlights the development of such infrastructures without ensuring their ability to accommodate networked digital interaction ought to be called to account by constituents at the very next opportunity.

You know I believe that we used to call “ubiquitous computing” is now, simply, real life. Anybody who cares about cities and the people who live in them can no longer afford to treat pervasively networked informatic systems as a novelty, or even a point of municipal distinction. It’s always hard to estimate and account for, let alone attach precise dollar figures to, missed opportunities, to count the spectral fruits of paths not taken. But given how intimate the relationship between an individual’s ability to get around and a region’s economic competitiveness is known to be, there is no excuse for not pursuing advantage through the adroit use of networked systems to enhance individual and collective mobility.

What we ought to be designing are systems that allow people to compose coherent journeys, working from whatever parameters make most sense to them. We need to be asking ourselves how movement through urban space will express itself (and be experienced as travelers as a cohesive experience) across the various modes, nodes and couplings that will necessarily be involved.

The challenge before us remains integrating this tangle of pressures, constraints, opportunities and affordances into coherent user-facing propositions, ones that would offer people smoother, more flexible, more graceful and more finely-grained control over their movements through urban space. Then we could, perhaps, begin to speak of a true transmobility.