Tools for an Interactive Virtual Cinema
David Blair and Tom Meyer
Abstract. This paper describes our experiments with narrative structure in nonlinear, interactive and networked forms of media. Through these explorations, we have begun to anticipate what types of tools will be useful to tell stories in multi-user 3D immersive environments incorporating synthetic actors.
Because of the complex nature of this work, which attempts to create a broad fusion of artistic, social, and technical ideas, this paper is als structured differently from a traditional research paper. Many parts ol tins paper use poetic and allusive techniques to describe the work; gene-rally, we have chosen the particular vocabulary which most succinctly describes the ideas at hand.
1 Traditional Nonlinear Narrative Methodologies
An associative narrative methodology takes advantage of the fact that rative impulse has already been jump-started within the majority of us, for example, as an ability or inability to tell lies. The narrative impulse fakes what is at hand and presents itself through a partial recombination of these immediate facts. Since anyone can tell a story, what distinguishes the expert or at least enthusiastic amateur from the uninterested is the use of techniques or tools, which range from the simple algorithm of focused interest, used to parse the world for certain types of "material", to any of the complicated and specific toolsets of media composition. The interested user makes use of memory or tools of memory to extend the range of what is compositionally at hand, past the simple moment or the single association of a joke, in order to compose the story and complicate the composition by putting it into extended time.
The aesthetic of associative narrative refers back to the techniques used to compose the story, and the possible fact that people are only watching your story in order to make their own. Its takes advantage of the autogenic nature of narrative in order to create a type of story "engine" that seems to lay its own track through an artificial landscape of potential narratives, and then puffs along that track of its own accord, releasing associations and continuities in the engine-steam, which disappear in a constant stream as they spread across the landscape. In essence, it creates both an artificial world and a way of narrativizing that world, making it easy for the "enduser" of the story to either dream through it or actively finish the meanings.
For centuries, the "grotesque" in fiction (e.g. Cervantes, Pynchon) has rec-onized both the mechanical nature of narrative, and the intrusion of strange reorganizing forces (often of ethical significance) through the very mechanisms of story. Within this thematically defined genre, story machines have often either been spoken about, or attempted. Tools like the "exquisite corpse" of the surrealists (something like the children's game "telephone"), Burrough's "cutup" (where scissored narratives, when pasted together, could reveal secrets and change the world), and the physical structure of Cortazar's novel early 60's novel -"Hopscotch" (numbered chapters read two extremely different ways according to a list at the front of the book) are specific examples.
In the 80's, American hypertext theorists Michael Joyce and Jay Bolter, both with strong interests in AI and associative meaning creation, helped created both a hypertext system (Storyspace [3]) and a new literature. This was based on their ideas of spatialized narrative, in which story was a landscape over which the reader wandered, with meaning created much in the same way landscape space is made coherent in the wanderer's mind, through the constant associative triangulation of "facts" in the artificial world. In both the "grotesque" and in hypertext, the attempted story machine was first a device for story experts. With the Storyspace software, the very machine used to create the narrative is also used to present the narrative, though often with the writing tools taken out.
2 The Narrative of Wax
Wax or the discovery of television among the bees, by David Blair, is a film told from the wandering point of view of an semi-intelligent agent, perhaps a literal artificial intelligence lost in a missile, before, during and after a somewhat Cain-like act of revenge. This narrator-agent is also one Jacob Maker, an apparently real person who has been forced to interact perhaps too long with an automated story generation system (represented by the bees of the film's title).
Fig. 1- Shots from one 20-second scene of Wax
Wax started as a open set of vaguely determined ideas which were concretized first as one or two simple jokes. Then, building on the autogenic narrative principle (which in turn builds upon the basic instability of the original composition substance, i.e. thought, and meaning), research material and video material were collected, organized, transformed, and put into temporary place as the composition began to take shape (Fig.l). The specific sort of work that resulted was linage-processed narrative, where both the images and the narrative were processed, and where the mechanisms of this processing to a great extent created the composition. Both images and story elements were "re-recognized" after this processing, both instantiating story possibilities, and leading the story into new directions.
3 Narrative Methodology of the Hypertext
Waxweb, developed by the authors as the World-Wide Web version of this film. contains a number of automated story generation (or story extension) systems, developed through a hybridization of some common aesthetic principles, and some off-the-shelf network-based writing tools.
The film Wax was created in the context of grotesque lief ion. and as it passed on, several years later, to 11 axweb, we fried to exploit the story-machine idea in order to find both the proper mode of hypermedia composition and presentation. First, a large descriptive hypertext was created and sent with the Storyspa.ee software to a. number of writers connected by the Internet, in the oxpocl.at.ion that they would add some additional, connected material to if.
Next, we placed the hypertext inside an object-oriented database [()]. As this project has developed, our database structure has allowed for several types of procedural recombination of story elements and story media. We index our media based on:
The linear path, which connects each node with the one immediately following it in the film.
- A tree structure, which establishes each node in its proper position in the
act/scone hierarchy.
- Shot overview maps. Clicking on an active still takes you to an index of similar pictures in the film, e.g., all the other pictures of hands.
- Textual thematic paths. The text of the film has embedded links between words corresponding to the overall themes of the film, e.g., the path through all the instances of the word "darkness".
- A random index, which consists of a page filled with X's, each connected to a random starting point in the film.
In its present incarnation, Waxweb resembles the process of its creation, starting at an unspeaking. dense, and empty monadical point (actually a 3-D hexagon), and descending through over 15 layers which offer increasing amounts of narrative detail, altogether acting rhetorically like a Powers of Ten zoom into the Waxworld. The obvious fact that the world has only a certain amount of detail is compensated for at about level 12 of the crosslinking, with the introduction of "image processing", which gets more narrative detail out. of the existing elements by performing simple recombinations, involving lookup tables and various sorts of manual parsing.
Of course, this recornbinatory use of an appropriately constructed database does not automatically lead to story generation; it just creates a great many more actual and potential moments of story. The intention of this, instead, is to use the ordinary accidents of recornbinatory prose-poetry to point towards true machine-made meaning.
4 MOO-based WWW Narrative Server
The system just described runs on a peculiar combination of technology, combining MUD-based interaction with WWW-based content presentation (through HTML, video, pictures, text, and VRML). Through this, we can explore the complex narrative possibilities afforded once many people begin to interact with a pre-existing story made of rich multimedia elements.
4.1 MUDs
MUDs, or Multi-User Domains, evolved out of multi-player Adventure-style games in the early 80s. These began as hack-and-slash style games, but some of the MUDs began to evolve into more social areas, somewhat reminiscent of chat lines. Although many of the earlier systems relied on hard-coded behaviors, MUD systems began to incorporate internal scripting languages. One particularly flexible server, MOO (MUD Object-Oriented), is now being widely used by the research community to support collaborative work, due to the ease of modifying the environment to support scholarship and sharing information.
The MOO server is distributed by Pavel Curtis as part of his research at Xerox PARC, studying collaborative computer systems [4]. Although there continue to be a large number of MOOs solely devoted to socializing, MOO systems have been established at the MIT Media Lab (collaborative environment for media researchers), the University of Virginia (postmodern theorists), CalTech (astronomers), and the Weizmann Institute of Science in Israel (biologists).
People at. several MOOs have developed software which allows a. MOO to serve as a WWW server, responding to http requests and dynamically formatting the requested information. This is made especially easy due to the interpreted nature of the MOO language1-. The first, MOO-based WWW server was developed at JaysHouseMOO, a. MOO devoted to exploring networking issues (they also developed a. MOO-based gopher interface). We have modified and extended their techniques to apply to large-scale multimedia servers, and have also added collaborative hypertext authoring tools to both the MOO- and WWW-based interfaces.
4.2 HTML Multimedia Representations
The model used in our system consists of a central database which contains the text of each node, plus additional information specifying the type, project info, and language of other multimedia information. This high-bandwidth multimedia information is typically stored at a number of distributed servers on a per-project basis. Therefore, the centralized database contains low-bandwidth information (text and content coding) which is likely to change, while the distributed multimedia servers contain high-bandwidth data (audio and video information) which will be modified much less frequently. At run-time, the central database server creates a specification of the text (based on current language and media preferences), with pointers to (he multimedia information on the closest distributed server for that project.
We use an SGML-based set of markup tags to represent the internal structure of (lie database (scenes, shots, etc). This representation allows us a great deal of flexibility in how we present the information to the user. Typically, these tags are dynamically parsed by the server into HTML [9], to be viewed in Netscape; we support rendering into pure text, and additional rendering formats are planned in the future (for example, to deal with the increasing number of dialects of HTML).
4.3 VRML
The recent development of VRML (Virtual Reality Modeling Language) [1] has allowed us to develop the structuring of flic database in entirely novel directions, through the use of 3D models with attached hyperlinks. These 3D models are taken from the original models used during the creation of the film Wax, and represent the narrator s journey through a virtual space, the "Land of the Dead". Many of these models represent, dynamic knowledge structures (televisions, telescopes, cities, etc), so we have1 adapted them as structuring metaphors for the hypertext.
Like the HTML, this VRML is also dynamically generated from the internal database [7], based on the same SGML-based representation of the content structure. The default representation is to arrange the shots associated with a. page, texture mapped onto an expanded icosahedron (in the film, this is the bee-television, which is used to direct, the narrator to where in the idea-space; he should travel next).
In order to take advantage of the individual strengths of the various media types (2D graphical design and layout, versus 3D information presentation), we encourage the participant to use both an HTML and a VRML browser, tiled side-by-side (Fig.2). Most, pages consist of text, pictures, and 3D images, so we use the VRML browser to display the 3D objects and associated pictures, and use the HTML browser to show the text and pictures, so that they remain relatively readable (3D is a notoriously bad environment for viewing text).
5 Proposed Narrative Experiments
Although several existing systems allow for the animation of simulated actors, and sometimes even incorporate higher-level, semi-autonomous goals for these actors [5, 2], these systems do not. yet fulfill the requirements which content, developers will place upon them. In this section, we describe two possible scenarios which could provide a litmus test, for artificial actors possessing some degree of narrative intelligence.
One of the most important reasons for choosing these scenarios as a test, base is that, they are extremely limited; there are many pertinent roles where the
Fig. 2. A typical configuration for using Waxweb, showing Netscape on the left and Webspace on the right
participants do not tend to be particularly chatty about subjects not pertaining to their immediate tasks.
5.1 Gatekeeper Scenario
One of the- simplest narrative scenarios we can imagine is that of a. participant approaching a gatekeeper, who is charged with preventing him or her from passing. This allows for a range of experiments, beginning with a. simple version where the gatekeeper may be a simple sphere which attempts to anticipate1 the user's motions and block him or tier from passing through. This simple scenario can be extended by providing for realistic representation and motion, more complex methods of bypassing the gatekeeper (bribery, deception, etc.), and the maintenance of internal state, so thai one's relationship with the gatekeeper may change in an interesting way over time.
5.2 Guide
Another narrative role suitable for a synthetic actor is that of the guide. Several research projects have begun to explore this connection [10]. Although this role is being explored in a variety of current products (e.g., Microsoft's Bob), these products are generally concerned with either conveying technical information or adding glitz to otherwise conventional interfaces.
A narrative guide is a useful way to direct the plot of interactive stories, and provides a motivational factor to encourage the user to proceed through chosen paths. As a simple test for the usefulness of a narrative guide, we proposea situation where a simple guide gives a brief overview of a large database such ' that used in Waxweb, and is able to respond to simple questions as it travels
6 Works in Progress
Here we describe two narrative projects currently under development by the authors, and what tools may be necessary in order to develop interesting synthetic narratives for them.
6.1 Jews In Space
One of the authors, David Blair, is working on a second feature-length project Jews m Space (.IIS), intended to be released both in theaters as a film and on networks as a large hyper-film. Automated story-generation is an important research topic for Jews in Space.
JIS, like WAX before it, is a heavily associative him, a type of hybrid construction very much in the tradition of the encyclopedic narrative, whose con,position requires the creation, collation, and preservation of huge numbers of historical and imaginary associations. Research material comes from onlme databases such as the hyperl.nked Encyclopedia Brifannica, as well as Iron, printed material integrated into the hypertext database through optical character recogu.f.on and keyboard input. This is then processed and extended through traditional and hypertextual writing techniques. However, composition from or within very large datasets can become unwieldy. This problem, however, is not specific to the system, but to current authoring paradigms. The construction of meaning from huge amounts of raw material continues as it has since even before the availability of cheap paper, as a form of intellectual handicraft, which m general resists mechanization.
"Association machines", software-based solutions to this problem, can serve as a partial solution to this problem, for this particular type of film. Such machines can act as amplifiers of the associative composition process, parsing large amounts of inputted raw material, to present the author with processed associative clusters. The author can then select a few meaningful proto-compositiona elements from the offered choices and use them as the beginning work of new plot sections or reuse them in the machine as the iterative seeds of new associative processing at least until the noise plays too loud. Very large scale hypernar-ratives may not be able to be practically constructed without the aid of such serm-semi-intelhgent meaning-engines.
In the current environment of onlme authoring systems and hopes for general integration of both tools and media types, there isn't much difference between production and distribution. Jews in Space will be authored, as much as possible, in an open-system "Virtual Studio". At the end of the production, the tools used by the movie-making professionals to intercommunicate, and perhaps some of the'production tools will be recycled with the entire data-set as an on-line virtual movie theater", an extremely large hypermedia site that will be. released in parallel with the small-cinema release of the celluloid film. Semi-intelligent authoring tools, if they can be. found or constructed, will be used by the author in the virtual studio. We hope that these composition tools could then be turned around, and serve as end-user tools, perhaps even engines for the creation of a somewhat intelligently recombining Story-Place.
G.2 Gr animation
Another project currently underway, which will incorporate synthetic actors in an multiple-participant 3D environment, is Grarnm.atron, being developed by 'Tom Meyer in association with the writer Mark Amerika. The setting of this project is the virtual city Prague-23, situated in a shared 3D environment, narratively situated ten to fifteen years in the future.
The story describes an artist, Abe Colam, who is asked to write an article about this virtual city, and his encounters in the "Interzone" situated there. His journey is directed by the other main character, Cynthia Kitchen, who is attempting to bring Abe Golarri into a psychogeographical state, where they may be able to be reunited.
Although we will allow for multiple participants to be present in the MUD-Iike social zones of the space1, as in [12], many of the narrative sections will involves the participant taking on the role of Abe Golam or Cynthia Kitchen as they interact with simulated actors throughout the city. In addition, parts of the story may require that, the participants add to the story, so we would like the authoring tools to be as simple and open as possible1.
Because this project is being designed as a general virtual environment system, it will require an extremely fast distributed database1, capable of tens of thousands of updates per second. The emerging union of VRML [8] and Java [II] may prove1 to be1 quite* useful for developing an architecture-independent client lor such a elat abase, and at the current rate of ha.rwa.re1, software, and network-bandwidth evolution, we1 expect that suitable1 clients and servers for this project may be1 realizable1 within the next two years.
7 Conclusion
Unfortunately, there has been a lot of research emphasis on prototyping run-time systems, and not a lot of emphasis on open tool-kits with functionalities that even hybridly imitate story creation, which then can be used by authors, and, in the best of all possible worlds, reused in distribution as a dumbed-down enduser toolkit. We believe that if you can get the machine to help compose the story out of 'omaterial", character comes easily next, as a further complexification of the autogenic story-making impulse.
8 Acknowledgements
Thanks to the Brown University Graphics Group, especially Andries van Dam, David Klaphaak, Suzanne Hader, Emma Johnson, Laura Mullen, and Brook Conner. John Unsworth of the Institute for Advanced Technology in the Humanities at the University of Virginia provided a machine and disk space for the project. Mark Pesce, Jan Hardenburgh, and the Open Inventor Group at SGI o-ave feedback and earlv bug-testing of the VRML version of Waxweb.
This work was supported in part by grants from NSF, ARPA, NASA, IBM, Taco, Sun, SGI, Ilk, Microsoft. Xing, Okidata, ONR grant N0O0M-91-.D4052, ARPA order 8225, and tin' New York State Council for the; Arts.
9 Bibliography
References
1. Bell G., Parisi A., Pesce M. (1995) "The Virtual Reality Modeling Language Version 1.0 Specification".
2. Blumberg B. and Galyean T.A. (1995) "Multi-Level Direction of Autonomous Creatures for Real-Time Virtual Environments". In Proceedings of SIGGRAPH 95, pp.47- 54
3. Bolter .1.13., Joyce M. (1987) "Hypertext and Creative Writing". Proceedings of Hypertext '87. November J 987. pp.41-50
4. Curtis P. (1992) "Mudding: Social Phenomena in Text-Based Virtual Realities". Proceedings of the 1992 Conference on Directions and Implications of Advanced Computing. Berkeley. May 1992
5. Goldberg A. (1990) "Behavioral Scripting: Method acting for synthetic characters".
< URLdittp://found.cs.nyu.edu/M RL/behaviour/behaviour.html>
6. Meyer T., Blair D., Hader S. (1994) "WAXweb: A MOO-based Hypermedia System for WWW". In Proceedings of the Second International WWW Conference.
7. Meyer T., Blair D., Conner D.B. (1995) "WAXweb: Toward Dynamic MOO-based VRML". In Proceedings of VRML 95.
8. Meyer T. and Conner D.B. (1995) "Adding Behavior to VRML". In Proceedings of VRML 95. < URL :http://www.cs.brown.edu/research/graphics/publications/ papers/ vr ml behaviors, lit ml >
9. Raggett D. (1995) "HyperText Markup Language, Version 3.0". Internet Draft.
< URL: http://www.w3.org/hypertext/WVVW/MarkUp/html3/CoverPage.litml>
10. Rich C. et al. (1994) "Demonstration of an Interactive Multimedia Environment". In IEEE Computer 27(11). pp.15-22.
11. Sun Microsystems. (1995) "The Java Language Specification". 1995.
12. Worlds Inc. (1995) "Worlds Chat".
Robert Trappl * Paolo Petta (Eds.)
Creating Personalities for Synthetic Actors
Towards Autonomous Personality Agents
Springer
