Sonic Shapes

Visualizing Vocal Expression

Contact: Mary Pietrowicz

Sound has been an overlooked modality in visualization. Why? Because it is ephemeral. We experience sound as it happens, in the context of time, and often in community with others. Then, the sound is gone. Furthermore, sound in human communication is multidimensional. It includes the semantic meaning of words, the meaning of expressive verbal gestures (paralingual and prosodic components), the nonvocal gestures, and relational gestures. Even though we are able to record sound and play it back, researchers have typically focused more on the semantic meaning of the words than on vocal expression.

Img This work targets vocal expression, specifically, the development of a voice analytics toolkit and related visualizations which highlight the relational and expressive verbal channels in speech. By making the overlooked channels of human communication visible and persistent, we make it possible to see beneath the surface of our words. This insight will potentially enable the develoment of new applications for speech therapy, the quantization and visualization of vocal trends common to speakers with medical conditions such as autism spectrum disorder (ASD), the characterization and visualization of communication patterns common in different relationships and cultures and the development of new kinds of creative, multimodal works.