The Semiotics of Database The dynamics which exist between database and narrative are not unique in new media. The relation between the structure of a digital image and the languages of contemporary visual culture is characterized by the same dynamics. As defined by all computer software, a digital image consists of a number of separate layers, each layer containing particular visual elements. Throughout the production process, artists and designers manipulate each layer separately; they also delete layers and add new ones. Keeping each element as a separate layer allows the content and the composition of an image to be changed at any point: deleting a background, substituting one person for another, moving two people closer together, blurring an object, and so on. What would a typical image look like if the layers were merged together? The elements contained on different layers will become juxtaposed resulting in a montage look. Montage is the default visual language of composite organization of an image. However, just as database supports both the database form and its opposite — narrative, a composite organization of an image on the material level supports two opposing visual languages. One is modernist-MTV montage — two-dimensional juxtaposition of visual elements designed to shock due to its impossibility in reality. The other is the representation of familiar reality as seen by a photo of film camera (or its computer simulation, in the case of 3-D graphics). During the 1980s and 1990s all image making technologies became computer-based thus turning all images into composites. In parallel, a Renaissance of montage took place in visual culture, in print, broadcast design and new media. This is not unexpected — after all, this is the visual language dictated by the composite organization. What needs to be explained is why photorealist images continue to occupy such a significant space in our computer-based visual culture.
It would be surprising, of course, if photorealist images suddenly disappeared completely. The history of culture does not contain such sudden breaks. Similarly, we should not expect that new media would completely substitute narrativeby database. New media does not radically break with the past; rather, it distributes weight differently between the categories which hold culture together, foregrounding what was in the background, and vice versa. As Frederick Jameson writes in his analysis of another shift, in this case from modernism to post-modernism: "Radical breaks between periods do not generally involve complete changes but rather the restructuration of a certain number of elements already given: features that in an earlier period of system were subordinate became dominant, and features that had been dominant again become secondary."
Database — narrative opposition is the case in point. To further understand how computer culture redistributesweight between the two terms of opposition in computer culture I will bring in a semiological theory of syntagm and paradigm. According to this model, originally formulated by Ferdinand de Saussure to describe natural languages such as English and later expanded by Roland Barthes and others to apply to other sign systems (narrative, fashion, food, etc.), the elements of a system can be related on two dimensions: syntagmatic and paradigmatic. As defined by Barthes, "the syntagm is a combination of signs, which has space as a support." To use the example of natural language, the speaker produces an utterance by stringing together the elements, one after another, in a linear sequence. This is the syntagmatic dimension. Now, lets look at the paradigm. To continue with an example of a langauge user, each new element is chosen from a set of other related elements. For instance, all nouns form a set; all synonyms of a particular word form another set. In the original formulation of Saussure, "the units which have something in common are associated in theory and thus form groups within which various relationships can be found." This is the paradigmatic dimension.
The elements on a syntagmatic dimension are relatedin praesentia, while the elements on a paradigmatic dimension are related in absentia. For instance, in the case of a written sentence, the words which comprise it materially exist on a piece of paper, while the paradigmatic sets to which these words belong only exist in writer's and reader's minds. Similarly, in the case of a fashion outfit, the elements which make it, such as a skirt, a blouse, and a jacket, are present in reality, while pieces of clothing which could have been present instead — different skirt, different blouse, different jacket — only exist in the viewer's imagination. Thus, syntagm is explicit and paradigm is implicit; one is real and the other is imagined.
Literary and cinematic narratives work in the same way. Particular words, sentences, shots, scenes which make up a narrative have a material existence; other elements which form an imaginary world of an author or a particular literary or cinematic style and which could have appeared instead exist only virtually. Put differently, the database of choices from which narrative is constructed (the paradigm) is implicit; while the actual narrative (the syntagm) is explicit.
New media reverses this relationship. Database (the paradigm) is given material existence, while narrative (the syntagm) is de-materialised. Paradigm is privileged, syntagm is downplayed. Paradigm is real, syntagm is virtual. To see this, consider the new media design process. The design of any new media object begins with assembling a database of possible elements to be used. (Macromedia Director calls this database "cast," Adobe Premiere calls it "project",ProTools calls it a “session,"but the principle is the same.) This database is the center of the design process. It typically consists from a combination of original and stock material distributed such as buttons, images, video and audio sequences; 3-D objects; behaviors and so on. Throughout the design process new elements are added to the database; existing elements are modified. The narrative is constructed by linking elements of this database in a particular order, i.e. designing a trajectory leading from one element to another. On the material level, a narrative is just a set of links; the elements themselves remain stored in the database. Thus the narrative is more virtual than the database itself. (Since all data is stored as electronic signals, the word "material" seem to be no longer appropriate. Instead we should talk about different degrees of virtuality.)
The paradigm is privileged over syntagm in yetanother way in interactive objects presenting the user with a number of choices at the same time — which is what typical interactive interfaces do. For instance, a screen may contain a few icons; clicking on each icon leads the user to a different screen. On the level of an individual screen, these choices form a paradigm of their own which is explicitly presented to the user. On the level of the whole object, the user is made aware that she is following one possible trajectory among many others. In other words, she is selecting one trajectory from the paradigm of all trajectories which are defined.
Other types of interactive interfaces make the paradigm even more explicit by presenting the user with an explicit menu of all available choices. In such interfaces, all of the categories are always available, just a mouse click away. The complete paradigm is present before the user, its elements neatly arranged in a menu. This is another example of how new media makes explicit the psychological processes involved in cultural communication. Other examples include the already discussed shift from creation to selection, which externalizes and codifies the database of cultural elements existing in the creator's mind; as well as the very phenomena of interactive links. New media takes "interaction" literally, equating it with astrictly physical interaction between a user and a screen (bypressing a button), at the sake of psychological interaction. The psychological processes of filling-in, hypothesis forming, recall and identification — which are required for us to comprehend any text or image at all — are erroneously equated with an objectively existing structure of interactive links.
Interactive interfaces foreground the paradigmatic dimension and often make explicit paradigmatic sets. Yet, they are still organized along the syntagmatic dimension. Although the user is making choices at each new screen, the end result is a linear sequence of screens which she follows. This is theclassical syntagmatic experience. In fact, it can be compared to constructing a sentence in a natural language. Just as alanguage user constructs a sentence by choosing each successive word from a paradigm of other possible words, a new media user creates a sequence of screens by clicking on this or that icon at each screen. Obviously, there are many important differences between these two situations. For instance, in the case of a typical interactive interface, there is no grammar and paradigms are much smaller. Yet, the similarity of basic experience in both cases is quite interesting; in both cases, it unfolds along a syntagmatic dimension.
Why doesnew mediainsist on this language-like sequencing? My hypothesis is that it follows the dominant semiological order of the twentieth century — that of cinema. Cinema replaced all other modes of narration with a sequential narrative, an assembly line of shots which appear on the screen one at a time. For centuries, a spatialized narrative where all images appear simultaneously dominated European visual culture; then it was delegated to "minor" cultural forms as comics or technical illustrations. "Real" culture of the twentieth century came to speak in linear chains, aligning itself with the assembly line of an industrial society and the Turing machine of a post-industrial era. New media continues this mode, giving the user information one screen at a time. At least, this is the case when it tries to become "real" culture (interactive narratives, games); when it simply functions as an interface to information, it is not ashamed to present much more information on the screen at once, be it in the form of tables, normal or pull-down menus, or lists. In particular, the experience of a user fillingin anon-line form can be compared to pre-cinematic spatialised narrative: in both cases, the user is following a sequence of elements which are presented simultaneously.
A Database Complex To what extent is thedatabase form intrinsic to modern storage media? For instance, a typical music CD isa collection of individual tracks grouped together. The database impulse also drives much of photography throughout its history, from William Henry Fox Talbot's "Pencil of Nature" to August Sander's monumental typology of modern German society "Face of Our Time," to the Bernd and Hilla Becher's equally obsessive cataloging of water towers. Yet, the connection between storage media and database forms is not universal. The prime exception is cinema. Here the storage media supports the narrative imagination. We may quote once again Christian Metz who wrote in the 1970s, "Most films shot today, good or bad, original or not, 'commercial' or not, have as a common characteristic that they tell a story; in this measure they all belong to one and the same genre, which is, rather, a sort of 'super-genre' ['sur-genre']." Why then, in the case of photography storage media, doestechnologysustain database, while in the case of cinema it gives rise to a modern narrative form par excellence? Does this have to do with the method of media access? Shall we conclude that random access media, such as computer storage formats (hard drives, removable disks, CD-ROMs), favors database, while sequential access media, such as film, favors narrative? This does not hold either. For instance, a book, this perfect random-access medium, supports databaseforms, such as photo-albums, and narrative forms, such as novels.
Rather than trying to correlate database and narrative forms with modern media and information technologies, or deduce them from thesetechnologies, I prefer to think of them as two competing imaginations, two basic creative impulses, two essential responses to the world. Both have existed long before modern media. The ancient Greeks produced long narratives, such as Homer's epic poems The Iliad and The Odyssey; they also produced encyclopedias. Thefirst fragments of a Greek encyclopedia to have survived were the work of Speusippus, a nephew of Plato. Diderot wrote novels — and also was in charge of monumental Encyclopédie, the largest publishing project of the 18th century. Competing to make meaning out of the world, database and narrative produce endless hybrids. It is hard to find a pure encyclopedia without any traces of a narrative in it and vice versa. For instance, until alphabetical organization became popular a few centuries ago, most encyclopedias were organized thematically, with topicscovered in a particular order (typically, corresponding to seven liberal arts.) At the same time, many narratives, such as the novels by Cervantes and Swift, and even Homer's epic poems — the founding narratives of the Western tradition — traverse an imaginary encyclopedia.
Modern media is the new battlefield for the competition between database and narrative. It is tempting to read the history of this competition in dramatic terms. First themedium of visual recording — photography — privileges catalogs, taxonomiesand lists. While the modern novel blossoms, and academicians continue to produce historical narrative paintings all through the nineteenth century, in the realm of the new techno-image of photography, database rules. The next visual recording medium — film — privileges narrative. Almost all fictional films are narratives, with few exceptions. Magnetic tape used in video does not bring any substantial changes. Next storage media -- computer controlled digital storage devices (hard drives, removable drives, CD-ROMs, DVD-ROMs) privilege database once again. Multimedia encyclopedias, virtual museums, pornography, artists' CD-ROMs, library databases, Web indexes, and, of course, the Web itself: database is more popular than ever before.
Digital computer turns out to be the perfect medium for the database form. Like a virus, databases infect CD-ROMs and hard drives, servers and Web sites. Can we say that database is thecultural form most characteristic of a computer? In her 1978 article "Video: The Aesthetics of Narcissism," probably the single most well-known article on video art, art historian Rosalind Krauss argued that video is not a physical medium but a psychological one. In her analysis, "video's real medium is a psychological situation, the very terms of which are to withdraw attention from an external object — an Other — and invest it in the Self." In short, video art is a support for the psychological condition of narcissism. Does new media similarly function to play out a particular psychological condition, something which can be called a database complex? In this respect, it is interesting that database imagination hasaccompanied computer art from its very beginning. In the 1960s, artists working with computers wrote programs to systematically explore the combinations of different visual elements. In part they were following art world trends such as minimalism. Minimalist artists executed works of art according to pre-existent plans; they also created series of images or objects by systematically varying a single parameter. So, when minimalist artist Sol LeWitt spoke of an artist's idea as "the machine which makes the work," it was only logical to substitute the human executing the idea by a computer.At the same time, since the only way to make pictures with a computer was by writing a computer program, the logic of computer programming itself pushed computer artists in the same directions. Thus, for artist Frieder Nake a computer was a "Universal Picture Generator," capable of producing every possible picture out of a combination of available picture elements and colors. In 1967 he published a portfolio of 12 drawings which were obtained by successfully multiplying a square matrix by itself. Another early computer artist Manfred Mohr produced numerous images which recorded various transformations of a basic cube.
Even more remarkable were films by John Witney, the pioneer of computer filmmaking. His films such as "Permutations" (1967), "Arabesque" (1975) and others systematically explored the transformations of geometric forms obtained by manipulating elementary mathematical functions. Thus they substituted successive accumulation of visual effects for narrative, figuration or even formal development. Instead they presented the viewer with databases of effects. This principle reaches its extreme in Witney's earlier film which was made using analog computer and was called "Catalog." In his Expanded Cinema (1970) critic Gene Youngblood writes about this remarkable film: "The elder Whitney actually never produced a complete, coherent movie on the analog computer because he was continually developing and refining the machine while using it for commercial work... However, Whitney did assemble a visual catalogue of the effects he had perfected over the years. This film, simply titled Catalog, was completed in 1961 and proved to be of such overwhelming beauty that many persons still prefer Whitney's analogue work over his digital computer films." One is tempted to read "Catalog" as one of the founding moments of new media. Today all software for media creation arrives with endless "plug-ins" — the banks of effects which with a press of a button generate interesting images from any input whatsoever. In parallel, much of the aesthetics of computerised visual culture is effects driven, especially when a new techno-genre (computer animation, multimedia, Web sites) is just getting established. For instance, countless music videos are variations of Witney's "Catalog" — the only difference is that the effects are applied to the images of human performers. This is yet another example of how the logic of a computer — in this case, the ability of a computer to produce endless variations of elements and to act as a filter, transforming its input to yield a new output — becomes the logic of culture at large.
Database Cinema: Greenaway and Vertov Although database form may be inherent to new media, countless attempts to create "interactive narratives" testify to our dissatisfaction with the computer in the sole role of an encyclopedia or a catalog of effects. We want new media narratives, and we want these narratives to be different from the narratives we saw or read before. In fact, regardless of how often we repeat in public that the modernist notion of medium specificity("every medium should develop its own unique langauge") is obsolete, we do expect computer narratives to showcase new aesthetic possibilities which did not exist before digital computers. In short, we want them to be new media specific. Given the dominance of database in computer software and the key role it plays in the computer-based design process, perhaps we can arrive at new kinds of narrative by focusing our attention on how narrative and database can work together. How can a narrative take into account the fact that its elements are organised in a database? How canour new abilities to store vast amounts of data, to automatically classify, index, link, search and instantly retrieve it lead to new kinds of narratives?
Peter Greenaway, one of the very few prominent film directors concerned with expanding cinema's language, complained that "the linear pursuit — one story at a time told chronologically — is the standard format of cinema." Pointing out that cinema lags behind modern literature in experimenting with narrative, he asked: "Could it not travel on the road where Joyce, Eliot, Borges and Perec have already arrived?" While Greenaway is right to direct filmmakers to more innovative literary narratives, new media artists working on the database — narrative problem can learn from cinema "as it is." For cinema already exists right in the intersection between database and narrative. We can think of all the material accumulated during shooting forming a database, especially since the shooting schedule usually does not follow the narrative of the film but is determined by production logistics. During editing the editor constructs a film narrative out of this database, creating a unique trajectory through the conceptual space of all possible films which could have been constructed. From this perspective, every filmmaker engages with the database-narrative problem in every film, although only a few have done this self-consciously.
One exception is Greenaway himself. Throughout his career, he has been working on a problem of how to reconcile database and narrative forms. Many of his films progress forward by recounting a list of items, a catalog which does not have any inherent order (for example, different books in Prospero's Books). Working to undermine a linear narrative, Greenaway uses different systems to order his films. He wrote about this approach: "If a numerical,alphabetic color-coding system is employed, it is done deliberately as a device, a construct, to counteract, dilute, augment or compliment the all-pervading obsessive cinema interest in plot, in narrative, in the 'I'am now going to tell you a story school of film-making." His favorite system is numbers. The sequence of numbers acts as a narrative shell which "convinces" the viewer that she is watching a narrative. In reality the scenes which follow one another are not connected in any logical way. By using numbers, Greenaway "wraps" a minimal narrative around a database. Although Greenaway's database logic was present already in his "avant-garde" films such as The Falls (1980), it has also structured his "commercial" films from the beginning. Draughtsman's Contract (1982) is centered around twelve drawings being made by the draftsman. They do not form any order; Greenaway emphasizes this by having draftsman to work on a few drawings at once. Eventually, Greenaway's desire to take "cinema out of cinema" led to his work onaseries of installations and museum exhibitions in the 1990s. No longer having to conform to the linear medium of film, the elements of a database are spatialized within a museum or even the whole city. This move can be read as the desire to create a database at its most pure form: the set of elements not ordered in any way. If the elements exist in one dimension (time of a film, list on a page), they will be inevitably ordered. So the only way to create a pure database is to spatialise it, distributing the elements in space. This is exactly the path which Greenaway took. Situated in three-dimensional space which does not have an inherent narrative logic, a 1992 installation "100 Objects to Represent the World" in its very title proposes that theworld should be understood through a catalog rather thananarrative. At the same time, Greenaway does not abandon narrative; he continues to investigate how database and narrative can work together. Having presented "100 Objects" as an installation, Greenaway next turned it into an opera set. In the opera, the narrator Thrope uses the objects to conduct Adam and Eve through the whole of human civilization, thus turning a 100 objects into a sequential narrative. In another installation "The Stairs-Munich-Projection" (1995) Greenaway put up a hundred screens — each for one year in the history of cinema — throughout Munich. Again, Greenaway presents us with a spatialised database — but also with a narrative. By walking from one screen to another, one follows cinema’s history. The project uses Greenaway's favorite principle of organization by numbers, pushing it to the extreme: the projections on the screens contain no figuration, just numbers. The screens are numbered from 1895 to 1995, one screen for each year of cinema's history. Along with numbers, Greenaway introduces another line of development. Each projection is slightly different in color.The hundred colored squares form an abstract narrative of their own which runs in parallel to the linear narrative of cinema’s history. Finally, Greenaway superimposes yet a third narrative by dividing thehistory of cinema into five sections, each section staged in a different part of the city. The apparent triviality of the basic narrative of the project — onehundred numbers, standing for onehundred years of cinema’s history — "neutralizes" the narrative, forcing the viewer to focus onthe phenomenon of the projected light itself, which is the actual subject of this project.
Along with Greenaway, Dziga Vertov can be thought of as a major "database filmmaker" of the twentieth century. His Man with a Movie Camera is perhaps the most important example of database imagination in modern media art. In one of the key shots repeated few times in the film we see an editing room with a number of shelves used to keep and organize the shot material. The shelves are marked "machines," "club," "the movement of a city," "physical exercise," "an illusionist," and so on. This is the database of the recorded material. The editor — Vertov's wife, Elizaveta Svilova — is shown working with this database: retrieving some reels, returning used reels, adding new ones.
Although I pointed out that film editing in general can be compared to creating a trajectory through a database, in the case of Man with a Movie Camera this comparison constitutes the very method of the film. Its subject is the filmmaker's struggle to reveal (social) structure among the multitude of observed phenomena. Its project is a brave attempt at an empirical epistemology which only has one tool — perception. The goal is to decode the world purely through the surfaces visible to the eye (of course, its natural sight enhanced by a movie camera). This is how thefilm's co-author Mikhail Kaufman describes it: An ordinary person finds himself in some sort of environment, gets lost amidst the zillions of phenomena, and observes these phenomena from a bad vantage point. He registers one phenomenon very well, registers a second and a third, but has no idea of where they may lead... But the man with a movie camera is infused with the particular thought that he is actually seeing the world for other people. Do you understand? He joins these phenomena with others, from elsewhere, which may not even have been filmed by him. Like a kind of scholar he is able to gather empirical observations in one place and then in another. And that is actually the way in which the world has come to be understood. Therefore, in contrast to standard film editing which consists in selection and ordering of previously shot material according to a pre-existent script, here the process of relating shots to each other, ordering and reordering them in order to discover the hidden order of the worldconstitutes the film's method. Man with a Movie Camera traverses its database in a particular order to construct an argument. Records drawn from a database and arranged in a particular order become a picture of modern life — but simultaneously an argument about this life, an interpretation of what these images, which we encounter every day, every second, actually mean.
Was this brave attempt successful? The overall structure of the film is quite complex, and on the first glance has little to do with a database. Just as new media objects contain a hierarchy of levels (interface — content; operating system — application; web page — HTML code; high-level programming language — assembly language — machine language), Vertov's film consists of at least three levels. One level is the story of a cameraman filming material for the film. The second level is the shots of an audience watching the finished film in a movie theater. The third level is this film, which consists from footage recorded in Moscow, Kiev and Riga and is arranged according to a progression of one day: waking up — work — leisure activities. If this third level is a text, the other two can be thought of as its meta-texts. Vertov goes back and forth between the three levels, shifting between the text and its meta-texts: between the production of the film, its reception, and the film itself. But if we focus on the film within the film (i.e., the level of the text) and disregard the special effects used to create many of the shots, we discover almost a linear printout, so to speak, of a database: a number of shots showing machines, followed by a number of shots showing work activities, followed by different shots of leisure, and so on. The paradigm is projected onto syntagm. The result is a banal, mechanical catalog of subjects which one can expect to find in the city of the 1920s: running trams, city beach, movie theaters, factories...
Of course watching Man with a Movie Camera is anything but a banal experience. Even after the 1990s during which computer-based image and video-makers systematically exploited every avant-garde device, the original still looks striking. What makes its striking is not its subjects and the associations Vertov tries to establish between them to impose "the communist decoding of the world" but the most amazing catalog of the film techniques contained within it. Fades and superimpositions, freeze-frames, acceleration, split screens, various types of rhythm and intercutting — what film scholar Annette Michelson called "a summation of the resources and techniques of the silent cinema" — and of course, a multitude of unusual, "constructivist" points of view are stringed together with such density that the film can't be simply labeledavant-garde. If a "normal" avant-garde film still proposes a coherent language different from the language of mainstream cinema, i.e. a small set of techniques which are repeated,Man with a Movie Camera never arrives at anything like a well-defined language. Rather, it proposes an untamed, and apparently endless unwinding of cinematic techniques, or, to use contemporary language, "effects," as cinema's new way of speaking.
Why in the case of Witney's computer films and music videos arethe effects just effects, while in the hands of Vertov they acquire meaning? Because in Vertov's film they are motivated by a particular argument, this being that the new techniques to obtain images and manipulate them, summed upby Vertov in his term "kino-eye," can be used to decode the world. As the film progresses, "straight" footage gives way to manipulated footage; newertechniques appear one after one, reaching a roller coaster intensity by the film's end, a true orgy of cinematography. It is as though Vertov re-stages his discovery of the kino-eye for us. Along with Vertov, we gradually realize the full range ofpossibilities offered by the camera. Vertov's goal is to seduce us into his way of seeing and thinking, to make us share his excitement, his gradual process of discovery of film's new language. This process of discovery is film's main narrative and it istold through a catalog of discoveries being made. Thus, in the hands of Vertov, a database, this normally static and "objective" form, becomes dynamic and subjective.More importantly, Vertov is able to achieve something which new media designers still have to learn — how to merge database and narrative merge into a new form.