Dynamic Systems of Engagement

Scott Murray, MFA ’10

thesis abstract

We now have the tools and technology to create just about any interactive system imaginable. But how can we ensure that our designs are engaging? Dynamic Systems of Engagement illuminates how dynamic, interactive, computationally based systems offer new opportunities for engagement with participants and third-party observers. Through numerous case studies, I explore three core themes: data visualization, dynamic systems, and engagement.

I consider data visualization broadly as a process of interpreting and expressing data of all kinds, not just numbers and text. I explore principles of systems design to illustrate how dynamic systems differ from works of static, pre-composed media, like painting, film, and television. Finally, I connect these themes to methods of interaction and engagement.

My past projects illustrate a range of design possibilities grounded in these ideas. From Gesture Project, which responds to physical gestures with patterns of rotating, color-changing discs, to the ASCII Photo Booth, a high-tech, low-fi interpretation of a traditional photo booth, these interactive studies illuminate nontraditional uses of data visualization, systems design, and interface concepts.

Although the concepts are valuable, more important is how real people respond to the designs. That is, what is the experience like? I conduct extensive user research with each project, the findings of which are used to refine the designs and inform future projects.

I adopt a framework of challenge and reward for sustaining engagement, which I then employ for two primary thesis projects, Practice and Cheeky. Although each project has its own distinct content and approach, both elicit engagement by employing visual mirroring, establishing tension and ambiguity, and finally resolving that ambiguity, providing closure to the experience. Both projects address the questions: How can we challenge someone while keeping them engaged, and how can we incentivize participants to overcome the discomfort of the challenge?

Practice is a new interactive video piece that employs metaphors of stillness (physical and psychological) and reflection (visual and personal). While most interactive video installations reward motion, Practice rewards stillness, and in so doing tests participants’ tolerance for physical discomfort and emotional ambiguity.

Practice employs computer vision methods of face detection and face tracking to identify participants’ presence and level of engagement, so that mere visual stillness, without engaged users, elicits no reward. Visual and aural cues incentivize users to overcome the discomfort of the challenge, by establishing anticipation of the rewards to come. And through it all, the system collects data on participation, which is analyzed and visualized.

Cheeky, a second interactive video piece, is introduced and shown to apply the same principles of experience design to engaging and humorous ends.

Introduction to Thesis

Although the meaning of the term “design” shifts when applied to different disciplines, broadly speaking, it is a process of problem-solving within constraints. A successfully designed umbrella keeps its owner dry. A successfully designed bridge bears the weight of transport moving over it. A successfully designed pacemaker keeps ticking, year in and year out.

Those objects, when well-designed, are more or less invisible. Yet, if the designs fail, they suddenly become highly visible. Works of graphic design function in just the opposite way: visibility is key to their success. They must draw attention to themselves in order to communicate and serve their intended function.

Interactive design, a relatively young amalgamation of several fields — graphic design, film, video, sound design, computer science, and human-computer interaction — must also draw attention to itself in order to succeed. But visibility is not enough; participants must be engaged and motivated to interact, to communicate back to the system. Without engagement, there is no interaction between human and system, the communication flows in only one direction, and the design remains isolated, unexperienced. If the audience is not engaged, the design is a failure.

Engagement, therefore, is essential to an interactive project’s success. The degree, duration, and depth of engagement required depends upon the project’s goals. For an online advertisement, a fleeting moment of engagement may be all that’s needed in order to register one brand impression. In that case, the message is brief, task-oriented, and not meaningful (“Click me!”). I am interested in creating meaningful and memorable experiences, the digital equivalents of a coastal hike on a beautiful day, or a great party with your closest friends. These experiences stay with us as happy memories that are kept alive through periodic retellings: “Remember that time when…?” Designing and facilitating such significant experiences requires deep, sustained levels of engagement with participants.

Traditional media have many tools for engagement available to them, including motion, audio, and captivating narrative elements. But computationally based media offer opportunities for new levels of engagement through interaction. In this context, the “design” encompasses not just a system of communication, but also a system of interaction. Just as the structural engineer can design a bridge, the interaction designer can consciously and deliberately design systems that successfully engage participants.

Fortunately, unlike bridge-builders, we have the luxury of working mostly with software, not steel, which allows us a great deal of flexibility in our methodology. My approach is to first define the problem, then sketch out possible solutions, and test them with potential users as early and often as possible. Testing provides me with data — both raw usage data and anecdotal data about the user’s personal experience with the project — which then informs subsequent refinements to the design.

My thesis projects, Practice and Cheeky, are the results of one year of intensive design research — plus two additional years of graduate study, and a lifetime of fascination with dynamic media. To appreciate where I am now, I must first explain where I came from.

Personal History

At age four, I was given my first interactive, dynamic, digital object: a Speak ‘n’ Spell. The clunky orange box pronounced words that I then learned to spell on its keypad. This is my earliest memory of a device with true interactivity: it would prompt the user, accept input, and then respond with feedback.

At age five, I faced a monumental ethical dilemma. I wanted to attend a class on LOGO, the early computer graphics programming language, but it was for ages six and up only. At my parents’ suggestion, I lied about my age to the friendly staff at the Junior Museum in Palo Alto. This deceit caused me a great deal of distress, but perhaps the anxiety motivated me to learn as much as possible before being found out and removed from the classroom.
FORWARD 100?
RIGHT 90?
PENUP?
FORWARD 100?
PENDOWN

So began one of my first computer programs, although I didn’t realize it was programming at the time. All I knew was that I could tell the computer what to do, and it would do it. Also, it would do it correctly, every time. If a mistake was made, it was my own, and I could correct it. My simple circles, triangles, and squares weren’t beautiful, but they existed because I had learned to speak the computer’s own language. There was something attractive and intriguing about that. Maybe part of it was feeling like an insider, knowing a secret language that only I, the computer, and my classmates spoke. Part of it, too, was satisfaction from solving the puzzle — figuring out how to write a program that would execute my vision. But it was also about simply providing instructions and watching the machine carry them out. The more complex the instructions, the more complex the final image. Using recursion (a word I didn’t know at the time), I could even repeat parts of the instructions, building up patterns with spirograph-like complexity.

That same year, my family bought its first computer. Our Macintosh 512KE was very friendly to regular users, but not to aspiring programmers. My neighbor’s Commodore 64 was the opposite — just switch it on, and begin entering BASIC commands at the prompt. I remember returning home, excited to try BASIC on my own computer, only to be sorely disappointed by the Mac’s non-responsive and indifferent blinking question mark.

A few years later, I discovered HyperCard, an application that made it easy to create buttons, text fields, and pictures, and write code so that behaviors could be attached to those elements. I played with HyperCard for hours. I made Cipher, a program that could encrypt or decrypt text using a basic substitution cipher. (This was perhaps my first project to incorporate audio. When encryption was complete, which could take minutes, a prerecorded sample would play: “Sa-sa-sa-sa-cipheeeerrr…”) In sixth grade, I made an interactive, choose-your-own-adventure style adaptation of the book Killing Mr. Griffin, in lieu of a traditional, linear, written book report. Ms. Dolan didn’t totally “get it,” but she understood enough to know that this was a new form of creative expression. (Many creative liberties were taken, as the hidden cave and massively explosive finale don’t actually occur in the book.) Even though it didn’t resonate with her, she encouraged me to pursue similar projects, and she still gave me an A.

Following that, I created Adventure, a non-linear exploration through an underground cave, where clicking on doors and passageways led to other screens, all hand-drawn with the mouse, each one containing some hidden message or scary sound. HyperCard’s built-in transitions (basic dissolves, slides, and wipes) gave the piece a cinematic quality, and I could even animate transitions — making a bookcase swivel or flame flicker — albeit slowly. Some people loved it, while many couldn’t understand it — my first taste of interactive design’s power to engage others ?(or fail to do so).

Around that time, I acquired my first modem. I had found a magazine on Bulletin Board Systems at the local newsstand, loaded with pages of phone numbers that could be dialed by computer — what a novel idea! Within fifteen minutes of opening the box, I had connected the modem to the phone line, issued my first AT commands, and signed on to a BBS — my first experience with networked computing. In the following weeks, I tested every BBS in the 303 area code, and my frequent requests of “Don’t pick up the phone!” reminded my family that I was “online,” that word’s first entry into our daily lives. After the next phone bill arrived, I was told to scale back my time online, since I had racked up a hundred dollars in so-called local-long distance calls. I waited until late at night, when the calls were cheaper, and explored the worlds of FidoNet, Usenet, text filez, and many other things I can barely recall.

My middle school had a computer lab full of Apple IIEs, preloaded with BASIC, just like the old Commodore 64. For some reason, Ms. Danzer, the physical education instructor who made us run laps around the field, was also the computer and typing instructor. Her degree of disappointment with me on the track reversed as soon as we came inside to the lab, where she was impressed with my quick grasp of procedural instructions, logic, and variables. While my classmates poked around the Oregon Trail and searched for Carmen Sandiego, I made ASCII art and interactive knock-knock jokes.

Also that year, I completed and mailed a form to the Boulder Valley School District. They wrote back a couple of weeks later to say that my first email address was ready: scmurray@bvsd.k12.co.edu. I dialed into BVSD’s server with my modem and typed “mail” at the prompt — my first Unix command. This definitely was not a Macintosh, but a hardcore mainframe, just calling out to be explored. I experimented, though gently, as I didn’t want to break anything and have my account revoked. I wrote emails to my only other friend with email about how cool it was that we could write messages to each other without stamps or interference from nosy teachers.

In eighth grade, I somehow begged a hundred dollars to buy Morph, a software package that mimicked the visual effects made famous in Michael Jackson’s “Black or White” music video a year earlier. A friend and I set to work, using his new LC II’s video capture card and Adobe Premiere 1.0. I crafted a sequence of our friends smiling and laughing as they morphed into one another. We used the clip in our History Day submission, an 8-minute video on the history of communication technologies, rendered entirely on the computer. Our piece made it to the state-level competition, where it won only fourth place, so we didn’t get to take our groundbreaking video to nationals in Washington, DC. That was how I learned that non-geeks will never appreciate the technology alone — the whole experience has to win them over. Although our project was cutting-edge technologically, it occupied only 320 by 240 pixels in the center of the TV screen. The judges complained that it was too small, and thus hard to see, disregarding my explanation that making a full-screen video would have required twice as much expensive RAM and 48 hours to render, instead of only 24.

I made my first website in 1993 at the age of fifteen. My high school had received a grant of $100,000 from Pfizer, which covered the cost of about fifty Macintosh LC IIIs, an Apple QuickTake digital camera, and our own T-1 connection to the Internet, which was still unknown to most people. To thank Pfizer and demonstrate what we did with all that money, my teacher asked me to put together a small website. I taught myself HTML and filled a floppy disk with several inter-linked pages of student work and photographs of our fancy computers with color displays.

At college, I worked at Vassar’s computer center, recovering theses from near-death experiences on demagnetized floppy disks. I continued making websites, both for myself, as a creative outlet, and for student groups. Upon my graduation, the alumni association hired me to revamp and expand their tiny site. This was my first professional, paid experience in design, and I did a decent job, considering my inexperience. I remember acknowledging that the redesign had to welcome and accommodate a specific audience, including older alumni with limited web experience. This was in sharp contrast to the work I had done until then, which was purely for my own satisfaction.

On September 11 of that year I built an online check-in system, so New York-based alumni could reassure others that they were safe. Since phone lines were down, people could post notes like “I spoke to so-and-so this afternoon, and she made it out of midtown safely…” We emailed the link to about 20,000 alumni. The server nearly ground to a halt, but that project probably had a greater emotional impact on its audience than anything else I’ve worked on. Alumni wrote and called us in tears, overflowing with gratitude.

Six years and three jobs later, I had learned a great deal about web standards and usability research, but I left work every day with a headache, bored in a position that was about ten percent creativity and ninety percent bureaucratic struggle. So I applied to graduate programs in design. My focus was on graphic design programs (I had a fantasy of being a type designer, which has since passed), but I knew that technological expertise was my primary asset. I couldn’t speak the language of letterpress, but hoped that Illustrator and InDesign would get me in the door.

Thankfully, I was encouraged to look into MassArt by a friend, and the language on the DMI website made perfect sense to me, in contrast with other art schools’ self-important copy. So I applied, and at the interview I could see that we were speaking the same language. But I still thought that DMI was a graphic design program. I didn’t understand the distinction between that and dynamic media design until halfway through my first semester. Time-based, interactive, non-static, databased, generative, systems… These terms floated through my brain until I suddenly got it: This isn’t about posters, books, business cards, or even websites! This is about designing the point of interface between people and their machines. The machine needs a system of rules to follow, and the people need a pleasant, satisfying psychological experience. I began to see how all my past experiences had prepared me for this new direction.

During that first semester, though, I grew restless creating only conceptual designs — I had come to school to get creative and make new things. I wanted them to actually work! Keynote was not bad for faking it, but then I heard about Processing, an open-source, free application then emerging from MIT. I downloaded it, bought a book (the best forty dollars I’ve ever spent), and got cracking.

What a surprise: This was the modern-day LOGO! The triangular turtle had been superseded by far more advanced visual capabilities, but it was essentially the same idea: a programming language for creating visual art, digitally. Processing, unlike LOGO, could also be programmed to accept user input and react in kind. So the process of using LOGO may have been dynamic, but its output was not. Processing, however, could “output” both static images and dynamic experiences. The system designer interacts with Processing by writing code, while the participant (e.g., end-user, client, visitor) interacts with the finished, compiled work. As the designer, I get to not only engage in the creative puzzle of translating a concept into computer-speak, but I also get to define the terms of the puzzle itself. I decide which elements and influences will be included in the system, and I then enjoy watching others interact with that system and figure it out.

Methods of Engagement

We now have the tools and technology to create just about any interactive system we can dream of. So how can we make them engaging? Dynamic systems offer three new avenues for engagement: motion, interaction, and content.

Visual motion is a powerful tool for attracting and maintaining attention. Motion is processed pre-attentively, meaning that we perceive motion before we are consciously aware of it (Ware 2004). That’s why it can be so hard to tune out the visually busy animations of online advertisements; your eye will jump straight to them, even though, consciously, you are already aware of the source of motion and have decided it’s no longer relevant to you. Motion can be used to capture (and re-capture) attention, even against the participant’s conscious will.

Motion can also be used to communicate information, such as through data visualizations or interface feedback, within certain limits. Yet our perceptual abilities are limited in how much motion they can comprehend at once (Ware 2004). While large numbers of independently moving visual objects may be beautiful, they cannot be interpreted meaningfully. (Later, I will review my Gesture Project, which captivated participants on a visceral level through hundreds of moving shapes.)

Once users are paying attention, they can be engaged by interactivity. I define interactivity as the two-way feedback loop between system and user. The faster that feedback is provided, the better. A more responsive system is always more engaging.

Dynamic systems can employ any number of interaction methods, but I find one to be most effective: mirroring. Visual mirroring is a medium-specific method as it requires live video input. By capturing an image of the participant, flipping it horizontally, and projecting it back to them, the system acts as a mirror enhanced with computational power. The “mirrored” image, then, can be augmented with additional visuals or replaced entirely with some algorithmic interpretation, as done by the ASCII Photo Booth.

According to artist Camille Utterback, a pioneer of interactive video installations and 2009 MacArthur Fellowship awardee, people intuitively understand how to move and interact with mirrored video by virtue of their experience in the physical world (i.e., using real mirrors) (2009). Psychological research supports and expands on that view. English psychoanalyst Donald Winnicott was one of the first in the field to study the powerful role of mirroring in child development. Winnicott summarized his findings in the statement, “When I look I am seen, so I exist” (1967). Mirroring is such a powerful technique for engagement in part because it shows us that we are being seen. Our actions and very existence are validated and proven to be real. I explore the psychology behind mirroring in more depth later in this document.

After motion and interaction, the third new opportunity for engagement is through data as content. Backed by computational power and network connectivity, dynamic systems can draw data from practically anywhere, either the user’s mouse movements or a database on the other side of the planet. The potential for engaging users through unexpected, current, time-sensitive content is enormous. And this, more than anything else, is why we must think of these projects in terms of systems: Unlike highly composed paintings, sculptures, photographs, and even films, the content of a dynamic system is not fixed and may not be defined in advance. Rather, the designer defines the scope and structure of the system’s contents. Then, only once the system is executed does it collect, transform, and express that data.

The use of quasi-random content from networked sources is an exciting opportunity, for it practically guarantees that each experience with the system will be unique. But designing a system to handle a limitless variety of content is not possible; some restrictions must be put in place. And creating such a system requires a new design approach.

Projects Overview

Although my thesis process revolved primarily around one major project (Practice) and a minor one (Cheeky), these projects cannot be understood in a vacuum, as they were preceded by so many other projects. All of my prior projects at DMI have influenced my thesis projects, so I have elected to include descriptions of the most relevant ones here.

Data visualization has been an important theme in my work, and I discuss several projects, ranging from Search Explorer, an interactive tool for voyeuristic exploration of web search queries, to Relationship Visualizer, a tool for visualizing connections in network data. My iTunes Library Visualization was also a valuable learning experience, as was the BART Trains Visualization, which taught me that innovative designs are not necessarily successful designs. I briefly explore how data might sound — call it data auralization — with two projects, Audible Particles and Aural Data Plot.

With Dictionary Words and Questions & Answers, I explore elements of interactive, nonlinear narrative structures, to varying degrees of success. The Gesture Project and ASCII Photo Booth introduced me to the engagement potential of visual mirroring. Anticipation Study, my very first project created with Processing, continues to inform my thinking about how to communicate without words, and how to imply what is not yet seen.

Goals of This Document

In the pages that follow, I document and reflect on my personal process of engagement with dynamic media, concluding with in-depth analyses of my thesis projects. By the end, I hope to have shared the most pertinent and valuable insights and discoveries made during this journey at DMI.

My hope is that this document will be accessible, enjoyable, and valuable for anyone interested in design and interactivity.