And in Bild Medium Kunst  ed. Yvonne Spielmann & Gundolf Winter [Muenchen, Fink: 1999]

The theory of metaphor I studied years ago began with the observation that a metaphor involves an aberrant use of language—a use of language that breaks a rule of grammar, of semantics, of logic, or simply of common sense. Thus "the poet is a penguin, his wings are to swim with" (ee cummings) is aberrant because the reader knows that penguins do not write. The mind is jump-started by this knowledge into an interpretation that associates the two concepts poet and penguin . . . one imagines a penguin writing and wonders where he might hold his pen. What is that stub of a wing for, anyway, and how might a writer use it? And is a poet simply a writer? Isn’t a poet’s writing like the penguin underwater, gliding with those wing-stumps . . . a kind of flying? Confronted with poets equated to penguins, the mind struggles to find a way to relate some aspect of the poet’s persona to the penguin’s aquatic wings. In this way, we fly—or swim—around in language, applying a different kind of sense to the words than the ‘literal’ meaning.

Since Derrida, however, we accept the idea that all language is aberrant. There are no ‘normal’ or ‘literal’ usages in contrast to the metaphorical. Derrida undermines the rigidity of a rule of language; in this light, any use of language may depend on the kinds of devices we employ to understand metaphor.

Strange terms and acronyms emerge daily in discussions of computers. The development of digital systems for the storage and manipulation of all types of data has brought about a reconsideration of many concepts that previously seemed unproblematic. In order to make sense of the concept of a digital image, for example, the concept of an image has had to trade in elements of its accepted usage. In this essay I will try to understand how the metaphors we have become accustomed to in descriptions of images in the digital realm have affected our understanding of what an image is.

• Metaphors of Technology: Cyberspace and Virtual Reality

In Gibson’s early fiction, cyberspace is simultaneously concretized data, represented as multi-dimensional structures to navigate around and through, and the means of access to this data. It is both the material and the medium it inhabits, the files and what is now called "the interface." Cyberspace in Neuromancer requires several kinds of expertise, including visual acuity, right from the act of "jacking in." It is a region which makes specific demands on its visitors and offers a wealth of experiences in return. But Gibson’s 1984 description of cyberspace is not sharply focused. He pays more attention to what it feels like to operate on and in cyberspace than on the details of the imagined interface. Despite its vagueness, this picture of cyberspace captured the mind of a generation. Almost fifteen years later, actual technology is beginning to fulfill the expectations he raised with his fiction.

If Gibson’s cyberspace is a fictional territory, subsequent use of the term has transformed it into a metaphor. Cyberspace is now anything and everything stored in the international network of interlinked computers, from the words I am presently writing, to the data transmitted from the Hubble telescope to the laboratories on the ground, plus the images the data will be converted into. What was vague in Gibson’s novel has become gargantuan and ubiquitous, but hardly specific. In this context, however, I am more interested in how the term functions in language and how it works on us, than in the phenomenon it purports to describe.

A metaphor presents a picture, an implicit comparison, which we elaborate in order to comprehend it. One problem is pointed out by Wittgenstein: we can easily forget that the origin of an expression is poetic, and soon we find ourselves treating the expression as if it were not metaphorical. Wittgenstein finds the foundation of several classical philosophical problems—for example, the mind-body problem—in this misunderstanding of what "language game" we are currently playing.

"A picture held us captive," Wittgenstein writes. "And we could not get outside it, for it lay in our language and language seemed to repeat it to us inexorably." [Philosophical Investigations, ¶ 115]

The expressions "cyberspace," and "Virtual Reality" bring to mind the kinds of picture that hold us captive. VR is quite often spoken of as if it were a kind of reality, parallel to regular old reality, that we can "be in," and "explore." We can even meet friends and have sexual encounters in cyberspace. It is no longer a representation of space that we manipulate in very specific, somewhat limited, ways. No doubt very few people would admit to the belief that virtual reality is a kind of alternative reality. However, it is not the metaphor itself, but its extensions and implications that open the door to metaphysics. This is a metaphysics that can easily seize us as it becomes more and more fixed in the language—as the picture begins to hold us captive. Cyberspace becomes a variety of space, to be "colonized," "developed," "owned," and now "leased" and "subdivided"—all of which have severe social implications—rather than a shorthand term for data and the ways in which we represent, store and retrieve it. (But look how hard it is to maneuver in these swamps. "Storage," here, is another metaphor, as is "memory" when attributed to a machine—while "retrieval" hovers in the limbo between the metaphorical and non-metaphorical.)

The fundamental point is that metaphors do not live alone. Once we have embraced the concept of "cyberspace," and have implicitly begun to consider it a species of space—I always imagine something along the lines of the Biosphere project—a dolly of extra baggage is loaded. We become conquistadors and navigators as much as programmers and data manipulators. We begin to think about converting the natives to our language, our religion, and our culture, so that they will become consumers of our products, producers of raw materials for our industries, and carvers of primitive trinkets for our desktops. The extended family of the metaphor makes us forget that it is indeed a metaphor and not a literal description of a new world.

* * *

It is not difficult to elaborate reasons for the use of strongly metaphorical language in discussions about technological advances.The impact of computers on our everyday lives is so recent, so extreme, and so sudden, that it is a struggle to understand its implications. Normal uses of language are inadequate, and a bumper crop of metaphors has been harvested and applied, in an attempt to make sense of a range of phenomena.

Metaphor is useful as a marketing tool. Metaphors can easily appeal to morbid aspects of our personalities—for example an attraction to anything that can effortlessly provide us with instant power, instant creativity, or instant happiness. Here is a sampling from a magazine (MacWorld) that happens to be within arm’s reach.

Henry Jenkins and Mary Fuller, in their collaborative article "Nintendo and New World Travel Writing," [Cybersociety: Computer Mediated Communication and Community, ed. Steven G. Jones (Sage Press: 1995)] (http://www.rochester.edu/College/FS/Publications/FullerNintendo.html) , suggest that technology metaphors might be serving larger cultural goals. They advocate—with a gentle irony—using metaphors to discuss (and sell) technology as a means of redressing society’s historical lapses. Their example is the description of the conquest and parceling of cyberspace as colonization . . . without pain to the natives or ecological damage. We can also imagine slavery without suffering, in the descriptions of computers as willing and eager unpaid retainers; ethnic cleansing without genocide—both in Microsoft’s project for the elimination of competing operating systems and browsers, and in the ubiquitous anti-virus software. We can see computer software as granting licenses to commit serial murders without bodies to dispose of (Duke Nukem or the James Bond Nintendo 64 game GoldenEye). All of this leads up to war composed of surgical strikes without civilian casualties (Operation Desert Storm:: The Gulf War). [For a description of the "Highway of Death" in this ‘war without casualties,’ see: http://www.deoxy.org/wc/wc-death.htm]

Is it possible to describe computer technology in terms that carry less baggage? We can start by looking at four fundamental characteristics of the computer—digitization, interactivity, random access to data, and programmability.

• Digitization & Digital Phenomena

There are no images made of numbers. Numbers are abstractions. Numbers are non-visible, not even material, certainly not ingredients of a visual pie; while images have always been essentially visual. Their visual qualities are encoded into numerical form for electronic storage and manipulation. But this process is not a superimposition of a new structure or composition onto images. If mathematical form delineates the digital image, it is inseparably bound up with en- and de-coding. The ‘digital image,’ in other words, is not a particular kind of image, because it is not an image at all, if an image is understood to be a visual entity. The phrase "digital image" is a figure of speech, shorthand for the entire process of encoding, storage, mutatation, transmission, and decoding. Only at the beginnings and ends of the process are there visual entities—in the middle there are only numbers and electrical or magnetic charges, etc.

Just as the memory of a computer is not a silicon or magnetic cabinet of pictures in the form of numbers and electrical charges, so "digital" is not an attribute of the medium that carries the visual information. To describe an image as digital is not like describing it as painted or cinematic.

John F. Simon Jr’s Every Icon [http://www.numeral.com] is composed of digital images. Considered for visual qualities alone, however, this description—and indeed the piece itself—is difficult to understand. Only when the sense of "image" is augmented to incorporate some of its underlying process, is describing the images of the work as digital informative.

Every Icon starts from the fact that the Macintosh screen icon is based on a 32 by 32 grid. It is therefore feasible to generate every possible black and white icon by systematically counting through and displaying every combination of black and white elements of the grid. Every Icon attempts to do exactly that. The work is an icon generator. The problem is that if the process of icon generation begins with insertion of black squares in the upper left corner of the matrix, it will take an unthinkably long time—on the fastest super computer—to reach even one recognizable icon. Every Icon will not get beyond the first line for at least a year: 15/16ths of the tiny icon will remain white into the lifetimes of our great grandchildren. The length of time required to reach the lower rows of the grid is on the order of the age of the universe, Simon assures me.

Every Icon is an artwork distributed as a computer desktop ornament. As an artist, Simon fulfills the role of the native craftsman, providing trophies for the cyberspace explorer to display as evidence of travel in exotic virtual territories. Every Icon is a subtle, ironic work. Built on an apparently elementary concept, it opens a window into the cosmic and unthinkably vast. (I have discussed these implications on the web in "Elastic Language: How Not to Talk About Technology," Mediorama, June 1998.)

In this context, the question is whether a grid of black and white squares qualifies as a digital image. No image of Every Icon is comprised of numbers, either as material ‘numerals,’ or in the stricter sense of non-material number types. The argument in favor of classifying the images as digital is that they are isomorphic representations of a comprehensible algorithm. However, if this is what makes the image digital, it is precisely what interferes with its description as visual. Suppose the images shown on this page were not generated by an algorithm, but drawn by hand on paper. They would have the same visual qualities, though their mathematical basis, their ‘digitalness,’ would be lost. If the identity of an image is in its visual qualities, not the processes by which it is formed, the images of Every Icon are no more digital than the Sistine Chapel ceiling.

This can lead us in one of two directions. Either we reject, as a category error, the very idea of a digital image. In this case we would criticize the widespread use of the expression as slipshod and imprecise, if not nonsensical. But the notion of theorist as linguistic cop is outdated and irrelevant. It is more salient to describe and try to understand ‘the world as we find it’ than to make utopian demands. The algorithmic basis of Every Icon is integral to the images it generates. These images can only be understood if seen as the end result of a process. When we accept this, the notion of the image as fundamentally visual begins to erode. How an image is generated becomes an aspect of its identity.

What would be a feature common to all digital images? They can originate in almost any kind of visual object, from a photograph to a painting to an image editing application. And the methods computers use to store images are diverse, since any number of different algorithms for image encoding is in everyday use.

Perhaps the best candidate for a unifying feature is an underlying visual matrix. But this does not qualify. First, it is now common for the resolution of a bit-mapped image to be so fine that the constituent pixel grid is undetectable by the human eye. Secondly, not all encoding techniques rely on the analysis of an image into a lattice of elementary picture elements. Vector-based systems, by contrast, describe mathematically the curvature of the lines that compose an image, while 3D programs are based on descriptions of underlying polygons and relationships between them. Furthermore, not only is there wide variance in methods of encoding images into mathematical structures, but there is equal variation in decoding. The plotter translates the curvature of a line into the angular velocity of electric motors, which move paper in one direction and marking pens in another to produce genuinely curved lines. Even with devices that are based on a grid, like the monitor or the ink-jet printer, there is broad variation in the characteristics of the image. Different devices, for example, produce very different kinds of distortion (usually described metaphorically as "digital artifacts"), as a function of the absorption of the paper, or the way an output signal is configured for the video monitor or other display device. Thus, in line with the precepts of postmodernism, there are no visual features distinctive to the digital image—and we cannot undertake a modernist-type reduction of this supposed object to its essential visual qualities.

If there is something particular to the digital, it is related to the fact that it is necessarily countable. In other words, a digital image is at some point composed of discrete elements. A raster-based image, for example, is analyzed into picture elements, each of which might be described in terms of a numbered color or a brightness quotient. In this case color becomes a specific value of a defined, numerically limited, palette, rather than one of an infinity of shades of continuously variable light wavelengths. Vector-based images are described by their component curves, which must be mathematical functions. The common factor to both cases is that there are distinguishable, countable elements underlying the image, with implications—specific to each case—for the visual qualities of the picture output.

One way to express the main point I have made so far is to say that the atomic basis of the digital image carries no general visual implications. Though each method of encoding may have visual phenomena appended, there are no such phenomena common to all. The shared qualities are in storage, transportability, and such socio-economically significant factors as the disappearance of the existence of an ‘original.’

Timothy Binkley has been arguing along similar lines for ten years, in a notable series of papers that raise and resolve some philosophical dilemmas posed by computers.

"The Wizard of Virtual Pictures and Ethereal Places," Leonardo: Computer Art in Context supplemental issue, 1990

"Refiguring Culture," Future Visions: New Technologies of the Screen, eds. Philip Hayward and Tana Wollen (London: British Film Institute, 1993) [http://www.rochester.edu/College/FS/Publications/BinkleyCulture.html])

"The Vitality of Digital Creation," The Journal of Aesthetics and Art Criticism 55, Spring 1997

Binkley argues that the computer is neither a medium nor a tool, that interactivity is integral to it, and that its mathematical basis is conceptual rather than material. If there is a problem with Binkley’s position it is that he stops too early. The computer has no essential qualities—it is not necessarily a tool or a medium, though it can function as either one, depending on how it is used. The computer can become anything that can be described algorithmically. What it simulates is often changed in the process. If it is a tool, the PC is the universal tool, if a medium, the universal medium. The concepts of tool and medium are transformed in the transaction.

The PC is a penguin. Its wings are to swim with.

• New Images, New Effects

It is not my intention to undermine the enormously significant and unique changes brought about by the use of computers in image-production. Encoding facilitates the storage and transportation of complex data, and this is important enough. Even more important are the algorithmic transformations of encoded data enabled by the programmability of computers. Morphing and texture mapping are two of many possibilities. These visual effects can in principle be produced without the computer, but in practice they are beyond human limits—we are unable to undertake the repetitive tasks required at the degree of detail necessary. Even with the assistance of computers, achieving these results is labor intensive enough. It is not uncommon to see a room the size of an aircraft hangar full of workers equipped with Waco tablets, tracing images line by line for a big Hollywood film or TV commercial.

Though there is nothing to distinguish a digital image in general, there are indeed properties of images and types of images that can only be realized with computers. At this conference, we heard film theorist Vivian Sobchak deliver an elegant, moving analysis of morphing in film and video works, arguing that the familiar mechanisms of identification operate differently when the character on screen transforms seamlessly from one body shape to another. She is describing a remarkable change in the cinematic and our relationship to it, unthinkable before digitization.

Morphing, impossible before computers, writes the processes of production into the cinematic image itself. For the period of the morph’s transformation, the distinction between the photographic and the handmade is erased. In this chasm, the digital image emerges, quintessentially anchored to its algorithmic origins.

CAD videodisks store one video frame per track, a single circle of microscopic pits embedded in the surface of the plastic disc. There are 54,000 such tracks on a side, yielding 30 minutes of thirty-frame-per-second video. Video is played back by decoding the tracks one after another into the fluctuating waves that affect the cathode ray guns of the video monitor. I first saw a videodisk in 1981, so it is not a new technology by any measure, and indeed it is now widely thought of outdated. That a film could be stored on something that looked like an LP record was not very impressive, even eighteen years ago. That the frames could be randomly accessed, i.e. the frames of the film could be played back in any sequence, was—and still is—mind-boggling. It means that the filmmaker does not have to determine in advance the sequence that his images will get shown in—that this crucial transaction can be resolved at screening-time. This radically transforms the notion of a film-frame—instead of remaining fixed as an element of a particular sequence, any frame is now a potential element of any number of different sequences, depending on when it is decoded. Thus the videodisk is in principle a non-linear medium—one does not have to play through previous sections in order to reach a particular frame, and how frames form sequences can be left undecided. To go one step further: how frames form sequences may be determined by the viewer. In other words, random access to frames enables both non-linear cinematic structure and interactivity.

Manuel de Landa, "Real Time,’" Millennium Film Journal Nos. 20/21, Fall/Winter 1988-89, p. 74

Though it is now considered a fundamental, even defining, feature of computing and computer structures, interactivity is almost an accidental byproduct of technological development. As de Landa points out, it is something that hackers enlisted in their battle against the authority of the corporation. De Landa’s argues that interactivity is linked with computer processing speed. Like conversation, interactivity depends on the operator’s ability to react in an uninterrupted session to the machine’s response to his input. Early programmers relied on interactivity to test their software in "real time." Now it is something we expect in our daily dealings with, for example, a word processor or graphics program. I need to see the results of my changes to the text or drawing at the time I make them—without interactivity, the computer is no different from Tolstoy’s wife, who stayed up each night neatly copying his most recent manuscript, only to be confronted the next evening with her previous night’s text messily marked up for recopying.

Described in this way, interactivity and random access to data are interwoven. From the user’s point of view, the main advantage offered by random access is that an insignificant amount of time is required to retrieve a given piece of data. The limiting factror for interactivity is that the amount of time it takes the machine to respond must be unnoticeable, and this depends on the computer’s fetching a particular piece of data without needing to run through a mass of other data to get to it. Real Time is the determining factor. Without doubt, the practical and commercial implications of Real Time are profound. But are there any consequences for works of art? I have devoted a good part of my creative energies since 1981 to exploring the question.

For almost twenty years, the question I have been asking is: what happens to cinema when it is run through the wringer of random access and interactivity? Where does using interactivity as an additional expressive aspect of the cinematic take us? Other media artists look at the implications of digitization. Virtual reality attacks on both sides. And programmability constantly underlies the package. But for me, it is random access and interactivity that have opened up expressive possibilities that did not exist before computers. I have previously* advocated a response rather than a choice model for the aesthetics of interactivity—choice, as Ken Feingold points out, cannot take a viewer far beyond the concept of a candy machine. Allowing the system to incorporate a viewer’s response, on the other hand, opens the potential for a work that is like an "ocean of streams of story" (to quote Salman Rushdie’s quotation of the title of a 13th century Sanskrit/Hindi text.) In my conception of the interactive narrative, the viewer swims among narrative fragments in this ocean of story-streams, the turbulence created by his or her body mixing the fragments together, or separating them— a lava lamp effect—creating a narrative space that is always fluid and changing.

The computer acts according to instructions (it is almost impossible to avoid figurative language here— "act" ... "according to" ... "instructions"). Everything that I have discussed so far is dependent on a computer’s programmability—I bring it up only to make the point that a computer is a device that runs in time. It is not like architecture, painting, or literature, in that its operation essentially involves continuous change. We ought to keep this in mind when we are looking for images to describe computer operations. For example: the notion of the "page" or the "site" (as in web-) is a misnomer. It may well be these static descriptions of the internet that have produced so static a dataspace, driven, by and large, by conventional graphic design. The world wide web is still a great distance from Gibsonian cyberspace. (For a very useful detailed discussion of this issue, see Steven Johnson’s Interface Culture: How New Technology Transforms the Way We Create and Communicate [New York: Harper Collins Publishers, 1997], especially chapter 4, "Links.")

The computer is a dynamic entity. When it is not undergoing some kind of process, it has crashed. And it is partly with a quasi-Modernist motive—keeping close to the basic nature of the device in use—that I am committed to a dynamic aesthetic of interactivity.

• march

March is the most recent work in which I rehearse a dynamic aesthetic for interactivity. The piece was made in collaboration with architect James Cathcart, and exhibited in 1997 at the Anchorage in Brooklyn, New York (the cavernous vaulted chambers that support the Brooklyn Bridge.) The physical architecture of March is a steel and aluminum ramp embedded with sensors that detect the position of people on the ramp. The ramp can accommodate three or four people at once. Movement on the ramp affects two separate images. One is a large projection of the Rembrandt painting The Sacrifice of Isaac. A single detail of this painting fills the ten foot wide screen: Abraham’s large hand covering the face of his son, holding him in place; or the winged angel, his ethereal hand holding back the old man’s right hand, as he loses his grip on the sacrificial knife; the knife itself, caught in the middle of its drop to the ground; or the feet of the youthful Isaac, almost crossed in an prescient echo of the future crucified Christ. A ‘virtual camera’ pans and zooms across the surface of the image, picking up these details, its motions approximately echoing the viewer’s movements on the ramp.

Among other things, the Rembrandt painting is remarkable for its treatment of time. It depicts several moments simultaneously, moments that could not have occurred at the same instant—the ritual knife could not be suspended in the air at the precise instant the angel is grabbing Abraham’s right arm, while the old man’s left hand, carrying the weight of his whole body, is still leaning heavily on his son’s face, ready for the performance of the terrible sacrifice. The painting, in other words, is no frozen moment: it depicts what could never be photographed in a single frame (though of course with computer manipulation, it is suddenly in our time a possible photographic image.) The pans and zooms on this digitized painting bring it back from an impossible representation of a single moment to a representation of a series of moments in time. We can travel around the image, from the face of the child covered by his father’s hand, to the angel arriving in the nick of time, to the father’s surprised face, to the sacrificial knife in midair. Such a sequence of details is easily achieved by a viewer’s making the right series of moves around the ramp. On the other hand, a more fractured narrative is equally possible if viewers move more erratically. In this case other images might be accessed, including images of paintings of the Apocryphal Judith and Holofernes, or photographs of young Iraqi casualties of the shameful ‘Highway of Death’ episode of the Gulf War.

A large video monitor, mounted on a steel scaffold that towers over the ramp at an angle, displays cinematic images. The protagonist is an Angel on his way to prevent Abraham from performing the awful God-ordered sacrifice of his son. The angel speaks to himself as he rushes onward. He wants to complete his mission successfully but is repeatedly distracted by what he sees around him. In Heaven there is no desire, and nothing to feel desire for—this is the conceit of the piece—while landing on earth immediately releases an ocean of unfulfilled longing. The Angel-On-A-Mission is uncomfortably caught between the rock of duty and the hard place of desire. His monologue, written by Cathy Caplan and performed by Evert Eden as Angel, circles around this classical conflict. Movement on the ramp affects the image of the angel—changing his outfit or his mode of transportation, or replacing his image with pictures of Judith and Holofernes, the 1991 Gulf War Victory Parade, young Iraqi war victims, Manuel de Landa discussing drill and its history, or professional cantor John Trout singing Hebrew liturgical music. But even through the edits generated by viewer movement, the Angel tries to continue on his mission. His continuing monologue makes a kind of circular sense. We are caught in the middle of a story that has a direction and an ending. Though our sense of its direction keeps us in the story, we will never experience the ending except in the details of the Rembrandt painting on the large projection screen.

I have chosen to conclude this paper by describing my own work for two or three related reasons. The first is that I am primarily a practitioner, not a theorist, and my ideas about the aesthetics of media works grow out of my practice, not vice versa. In other words, I discover what I think about interactivity by making what I conceive as effective interactive works. Then, and only then, am I able to think more generally about what I did and why I made the creative choices I made. For this set of reasons it is only fitting that I should describe a recent work out of which the ideas stated earlier in this paper (more or less) developed. The second reason is that I think that the aesthetic position I am advocating can only make sense if it is realizable. This is not, of course, to say that I have perfectly (or even imperfectly) realized the position.. At least I have a concrete example (actually steel and aluminum) that I can refer to in my concluding discussion. It goes without saying (and, as John Barth always says, I shall therefore say it) that it would be better if the reader could see the piece itself—but, circumstances being what they are, text being what it is, a description and a couple of pictures will have to do.

Viewers of march do not make choices about what they will see next. They walk on the ramp, responding to the pull of gravity (the slope is quite steep) and to the pull of the various stories interwoven in the interactive images. The images on the two screens respond back to them, and, if everything goes well, sooner or later viewer and images are connected in a dance, or a conversation, or a call-and-response song. The logic of the piece is easy to discern. Moving closer to the screen gives the viewer more control over the Angel character, while moving from side to side at the lower end of the ramp produces parallel images. When this simple logic is internalized by the viewers on the ramp, they find themselves free both to move and to exert some control over the story. At the highest level, lateral movement puts the angel in different costumes—from the sublime winged Angel (riding a girl’s bicycle), to the pious country priest (walking), to the earthbound housepainter (on a child’s scooter), to the ridiculous derelict (on his slow, swollen feet). The angel’s anguished monologue continues throughout. The viewer’s feels always in the middle of the story, a story with a conclusion that we know but will never reach. Choice or selection is never required, for it would involve emotionally exiting the narrative. The viewer is constantly aware that his or her movement affects the texture of the story space. This is an interactivity where one remains within the story while acting on it. Thus it is dynamic, always poised between the predictable and the unpredictable. The Angel moves constantly forward, while paradoxically trapped in an endless present—until we find Manuel de Landa, whose discussion about military drill, its history and utility, grounds us in the dirty reality of the real world with its real conflicts—real and dirty when the bombs and artillery hit the ground, even if at home we experience these conflicts through the other end of a simulation.