Synaesthesia as a Model for Dynamic Media

Colin Owens, MFA ’09

thesis abstract

Thesis Abstract
Throughout human evolution our senses have evolved, and the tools we have invented have augmented our sensory exchanges with our surroundings. The extent to which we engage with the tools we invent, and how we use our senses to engage with them, has had a tremendous impact on our understanding of how these tools function.

This research examines the interrelationship between our senses as a means of more intuitive control of the computer-based tools we create and use. It challenges historical assumptions about audiovisual synaesthetic relationships, and proposes the adoption of perceptual relationships based on natural metaphor for building more useful experiences with these tools.

I have focused on the computer audio mixer interface in my research as an example of the type of human-computer interface that can be created by using natural motion and space cues, as well as other cues found in both the auditory and visual realm.

Thesis Introduction
What if you woke up one morning and you discovered you could hear colors? What if instead you thought certain numbers had personalities? If you could taste shapes? For a small population this is a permanent condition. It’s called synaesthesia. Its Latin root words are “together” and “with the senses,” meaning that in the person’s mind, aspects of one sense are clearly associated with aspects of another sense.

My friend, Violet, has a form of synaesthesia that allows her to see whole numbers as colors. She said that learning multiplication tables was easier for some numbers than it was for others. The number 9 was a gorgeous color purple and that made all equations using the number 9 elegant. However, she didn’t like the combination of the numbers 4 and 6, a leaf green and an indigo, because the resultant color was ugly to her.

We all have some form of synaesthesia or at least we did when we were very young and, if we held onto it, were able to develop strategies for memorizing and deciphering vast quantities of information. Cross-modal transference, a developmental form of synaesthesia, is linked to intelligence in early infancy. Intelligence is associated with the ability to identify a stimulus that had only been experienced through one sense by using another sense. “A baby who is able to recognize by sight a screwdriver that she has previously only touched, but not seen, is displaying cross-modal transference.” 1 The level of this cross-modal transference by the time the infant reaches its first year correlates with intelligence scores later in life (Spelke, 1987; Rose, Feldman & Jankowski, 1999, 2003).

Our minds are capable of putting together powerful associations by the way we sense and how we perceive those senses. If infants who hold onto aspects of their synaesthesia are considered more intelligent later on, then synesthesia should be considered a useful model for how we work, the tools we use and the work we produce with those tools.

This thesis aims to use cross-modal sensory cues as a basis for creating those tools, specifically for computer software interface.

Click here to learn more about Colin’s major thesis project, “Shape Mix”.

Project: Dynamic Charted Music

dynamic charted music from Colin Owens on Vimeo.

This is a visualization of music I composed called “Number Station.” The visualizations are generated by the volume of each channel from the original multitrack file.

Generated using Processing and Minim libraries with 14 channels of streaming audio on an unbelievably fast machine.

Download “Synaesthesia as a Model for Dynamic Media” (PDF, 9 MB).