Expressive Code

Kate Nazemi, MFA ’06

project

Who are the makers in the digital world, and what is their form? Since physical objects are more likely reveal the process of making done by the hand, I wanted to investigate the process of making in a digital environment to find visual clues of making. The question–where does this take place–led me to investigate the programmer’s role in making. What happens in the process of programming and its visual result, code, is quite interesting. Take for example the programmer’s process of making a change. When the programmer changes something, it is usually an act of replacement–substituting one word for another that works, and commenting–a process of inserting notes only to be interpreted by another programmer. These changes are a part of the creative process, yet have no visual residue in the final outcome. Thus users are confronted with the results, such as an interface to a software program, that bear little human evidence of the process of making that went into it. It is this kind of thinking that led me to consider ways of revealing the humanity behind the code.

I am not a programmer, so for me the programming language is esoteric, it feels cold, and it is a silent language. But the language is composed of words, and in the world of poetry, words have a voice. I looked at a variety of programming languages and began reading (sometimes performing) lines of code: int SUBU arg! (treated typographically in book). Line after line I read these strange words out loud: loud, low, excited, slow, high, quiet. I was enamored with how expressive (and funny) the words were. It was this expressive potential found right in the language of code that began my journey of transforming code from a silent fixed language into one that is tactile, sculptural, phonetic, dynamic, and performative.

OVERVIEW
Expressive Code is a gesture driven object-based interface and printed book. It is my largest case study contributing to the development of my final thesis project: Code Performs (not sure of title). With Expressive Code, both print and interactive domains are examined: the interface allows participants to explore the visual and aural elements of code though a touch screen, while the book provides a historical foundation for learning various interpretations of the programming language C. The book is an analytical examination of the transformation of the scientific language of code to the poetic interpretation of code. The interface was developed in Flash and runs on a touch screen monitor. Participants, presented with a continuous left to right flow of code, are encouraged to explore the poetry through touch. Interactive specifics are discussed later under the heading “Relationships.” Together, these methods blend the nonlinear, dynamic and unique explorative attributes of interactive media with the linear, static, and analytic properties of the printed book for a complete multi-sensory experience.

Each method of representation rigorously investigates various levels in, and representations of, a programming language in order to relate it to the broader context of human experience and learning. This is observed in the following ways: 1. Translation — code to computer, fixed language of science to the fluid language of poetry, visual to sound; 2. Relationships — size, weight, value, contrast, rhythm, space, and 3. User Experience.

CONCEPT: TRANSLATION
Digital code is the requisite material used by scientists to process information and produce results, and now by artists who use it as a creative tool. Here, we find a link between two seemingly disparate disciplines. But the connection goes deeper than shared material and the challenge is how to communicate it across the linguistic divide. Where artists and scientists differ in their unique forms of language, they are alike in their approaches to creativity. For example, the artist and the scientist are curious investigators of the unknown. As Leonard Shlain states “while the scientist demonstrates that A equals B or that X is the same as Y, artists often choose signs, symbols and allegories to equate a painterly image with a feature of experience. Both of these techniques reveal previously hidden relationships.” (20, Shlain, Art and Physics).

To translate is to transform. This process is one of conversion for the purposes of learning and understanding, and takes on many forms. In one sense, to translate is to express in another language precisely the original sense. In another, to translate is to “convey from one form or style to another” (American Heritage Dictionary). In accordance with this definition, Expressive Code uses the processes of translation to identify and communicate unique connections between the diverse languages of science and art.

CODE TO COMPUTER
What do the many expressions of code look like (or even sound like)? What connects the process of translation from human thought down to the interpretation of binary code by the microprocessor? Where does one expression begins and another end?

Given the relative secrecy that code operates in (its activity is invisible to most computer users), the search for answers suggests both an analytical and creative approach.

Beginning analytically, code is expressed in several distinct forms before being interpreted by the microprocessor. Take for example the following line from the highest-level language of code:

sum.c #include int main (int argc, char *argv [])

These characters represent the first step in transforming the programmer’s instructions into specific tasks the computer can perform–in this case, to count from 0 to 100. Curiously, high-level language is considered closest in relationship to human linguistic communication.

Before high-level language can be interpreted by the microprocessor though, it is reduced to assembly language through the substitution of mnemonics and operational commands:

sum.s .text .align 2 .globl main .ent main 2 main: subu $sp, 32 sw $31, 20($sp)

This translation is silent, private and exact. Finally, assembly language is reduced to binary code, called machine language.

This lowest level language has the smallest vocabulary, yet is largest in volume:

00100111101111011111111111100000101011111011111100000
00000101001010111110100100000000000010000010101111101
001010000000000100100101011111010

And to think that this three step transformation takes place in the blink of an eye and in complete silence! (code courtesy of Professor James Larus, The University of Wisconsin)

Perhaps well understood by the computer programmer, the degrees of abstraction apparent in the translation of code make this language difficult to understand. Therefore, code remains elusive to the many who depend on it for creative work. It is with this assumption that the search to convey code in more approachable forms begins.

LANGUAGE OF SCIENCE TO LANGUAGE OF POETRY
One of the best ways to understand something better is by breaking it down into smaller, more manageable parts. A visual language offers countless creative ways to express complex information, and is where the transformation of code from a language of science to a language of poetry starts.

The expressive and communicative potential of this traditionally silent and purely computational material begins with the isolation of unique letter combinations from each line of code:

sum.c #include int main (int argc, char *argv [])
sum.c #include int main (int argc, char *argv [])

This process of isolation, when applied to the entire programming language, detaches code from its original sense and creates a reduced vocabulary subject to a new set of parameters and a different kind of analysis. Code, in its new context, is freed from a silent and fixed form, and transformed into a rhythmic array of visual language:

sum.c #include int main (int argc, char *argv [])

The parameters that govern this visual language originate with the Avant-garde artists of the 1920s who pioneered the reduction and restructuring of language to form new relationships to space and time, and thus new ways to perceive and create meaning. Of particular influence are the concrete poets “who reduced words to their elements of letters (to see) syllables (to hear)” (Concrete Poetry: A World View). This approach to language came from a belief that accepted grammatical-syntactical standards were inadequate to express certain ideas of the time. It is in this sense that the transformation of code from a scientific to a poetic language makes a historical connection to the Avant-garde artists.

VISUAL TO SOUND
When we read we give voice to the world of words. Tapping into multiple layers of perception, phonetic poetry (a subcategory of concrete poetry) explores this combination of sound and visuals through the juxtaposition of sound and typography. Here, it is the expressive aural and visual communication channels that transform the scientific language of code into its new poetic form: int SUBU arg! (treated typographically in book). This new poetic form not only takes on the properties of phonetic poetry (letters are seen and syllables are heard), it also transforms code from 2d to 3d by considering each word as an immediate and tactile object that responds to human interaction. It is this union between sight, sound, object and touch that creates an explorative environment to experience the emotive qualities found in the poetics of code.

CONCEPT: RELATIONSHIPS
To relate is to connect. Expressive Code fuses traditional formal elements in design: scale, contrast, rhythm and space, with properties of dynamic media: modularity, variability, and time, to make clear visual and aural connections between content and form. In unison with the poetic language of code, this creates an environment to explore and experience these unique relationships through physical manipulation. In keeping with the properties of an explorative environment, Expressive Code has no explicit rules, allows users multiple paths for interaction and is generally ambiguous.

Integral to Expressive Code is the typographic scale of code. Here, distinctions in typographic size distinguish inactive typography–small type flowing from left to right–from active typography–that, when touched, scales in relation to the volume of the word heard. Here, the following relationship between scale and sound are formed: the larger the word, the louder the pronunciation. Another relationship between scale and sound is found in the rate of word flow: the larger the word, the slower it moves from left to right. These variations in typographic scale and sound create contrast and rhythm in the interactive environment. It is within this environment that poetry is created and new expressive relationships are realized.

Adding to this sense of contrast and rhythm are the vocal expressions of the two typefaces chosen: Mrs. Eaves and Franklin Gothic. The voice of words set in Mrs. Eaves (a feminine typeface) is female, while Franklin Gothic (a masculine typeface) is male. Within each voice, alterations in volume range from high to low depending on speed. Here, relationships between dynamism and sound are how the poetry is created. Assigning voice to code is paramount to what brought me to this piece in the first place: my need to reveal the humanity behind code. While typography is a great visual expression of voice, it is only in combination with human voice that gives rise to a phonetic performance.

Space and time are also explored through the developing user relationship within the interactive environment. Through the touch of a hand, changes in size, content organization, and time are investigated. The fluid gestures of a hand allow participant the freedom of isolating certain words and re-organizing them within the given space. Although typography streams across the composition continuously, the system can also be paused to allow a more thorough investigation of the poetry.

THE USER EXPERIENCE
Allowing participants to intuitively examine code as poetry through interactions leading not to specific results, but rather to individual explorations is the interactive objective of Expressive Code. Interacting with programming code through touch directly relates to expressing the humanity of code. Here, guiding principals of intuition and nuance lead the way to investigating the poetic representation of code, where natural physiologic considerations like touch, pressure and object manipulation work in creating poetry.

REFLECTION
Expressive Code was a featured exhibition in the Boston 2005 CyberArts Festival and the 2005 Language of Dynamic Media Show. These two shows provided me with great feedback which influenced my final project, a revised version of Expressive Code, called Code Performs.

At first, I was ill prepared for the feelings I had while observing others interact with my work. It was incredibly intimidating. All I could think about was how terrible the project must be. But, I the warm reception I received helped mitigate my unease.

Overall, people seemed to understand the conceptual thinking behind the project, which was very rewarding. I think people found the transformation of a programming language into sound poetry to be intriguing. Many people found humor in the project (the way the strange words were pronounced), which made me feel good.

Right away though, I recognized people struggling to interact with the streaming words. It was clear I really confused people by not revealing any interactive tips. Only through individual demonstration did interaction make sense. Therefore, one of my goals in developing Code Performs was to create a transparent interface — one that encourages self-directed exploration — one that could be easily understood and explored through independent interaction.

Although I kept the interface physical by using a touch screen, I still felt that the project did not fully reflect the high degree of humanity I sought in my previous work. I think this had mostly to do with the screen-based environment the project was in. Thus, another goal of Code Performs was to better integrate materiality into the interface.

The overall exhibit lacked cohesiveness. The environment was not designed to evoke the kind of poetic response I was hoping for. The code words needed to sing, yet, they were drowned out by competing noise and such a small visual field to interact with them in. By placing the book next to the screen, rather than integrating it interestingly it into the space, I did little to entice people to read it. Another goal of Code Performs then is to make better use of the book form by transforming it into a sculptural object that communicates content more openly and clearly while for fully occupying the exhibit space.

The essence of sound poetry is found through live performance. I was left wanting to further explore this through the expressive qualities of speech and more sophisticated typography and interaction. These are the primary investigations in Code Performs.

Expressive Code was included in the American Institute of Graphic Arts, Boston 2005 Best of New England Juried Show, the American Institute of Graphic Arts Best of New England On the Road Show, and exhibited at the American Institute of Graphic Arts 2005 National Design Conference.

Keywords: code, interaction, print