PDCON:Conference/A Pd framework for the Xth Sense: enabling computers to sense human kinetic behaviour

Biosensing musical technologies use biological signals of a human subject to control music. One of the earliest applications can be identified in Alvin Lucier's Music for Solo Performer (1965). Alpha waves generated when the performer enters a peculiar mind state are transduced into electrical signals used to vibrate percussion instruments. Over the past thirty years biosensing technologies have been comprehensively studied and presently notable biophysical-only music performances1 are being implemented at SARC by a research group lead by the main contributor to the Bio Muse project Ben Knapp.

Whereas biological motion and movement and music are arising topics of interest in neuroscience research, the biologic body is being studied by music researchers as a mean to control virtual instruments. Although such approach has informed gestural control of music, I argue that it overlooks the expressive capabilities of biological sounds produced by the body. They are inaudible but may retain a meaningful vocabulary of intimate interactions with the musicians' actions.

To what extent could biologic sounds be employed musically? In which ways could the performer's perceptual experience be affected? How could such experimental paradigm motivate an original perspective on musical performance?

This paper seeks to outline methods underlying the development of the Xth Sense project, an ongoing research which investigates exploratory applications of biophysical sound design for musical performance and responsive milieux. Presently the Xth Sense technology consists of a low-cost, biosensing wearable hardware device and a Pure Data based application for capture, analysis and real time processing of human muscle sounds.

Firstly, an exploratory study of the acoustics of body sounds, namely muscle sounds is illustrated. I describe the development of an audio synthesis model for muscle sounds which offered a deeper understanding of the body sound matter and provided the ground for further experimentations in signal processing and composition in Pd. Then follows a description of the development and design of the Xth Sense - a wearable hardware sensor device for capturing biological body sounds. Eventually I describe the development of the DSP modular application, focusing on its main features such as: a MMG features extraction methods, a GOP routing system for dynamic mapping of gesture to sound, real time audio processing models and GUI utilities, which were developed to provide a fast prototyping environment.

In conclusion, I elaborate on the implementation of the Xth Sense framework in the composition of Music for Flesh II, a solo musical performance; which realization process prompted a better understanding of muscle sounds capabilities for biophysical generation and control of music, thus providing further prospects for the inquiry.