GMU:VR Experience Designers/Sandra Anhalt & Erkan Rojahn/'''einhändig, zweistimmig'''

einhändig, zweistimmig

a gesturally controlled music synthesis program

The project einhändig, zweistimmig is a gesturally controlled music synthesis program which enables the user to play two voices simultaneously with one hand. But not only the pitch of the voices is changing with the position of the hand in the air, it is also the rythm and diverse effects which are influenced by the hand and finger movement which gets tracked by the leap motion controller. The project is about creating a tool which makes the immersion of your body into the instrument possible.

attempts of visualisation

As we wished to provide not only an auditiv, immersive experience of the synthesiser to the user we had the goal to create a visualisation in a three dimensional space to suggest to the user an experience of fusion with the musical instrument at hand. We wished to create a playful, new experience for the user with the new instrument. First attempt was simply to represent the moving hand in the 3d space.

As we got confronted with osc connectivity issues, we decided after a consultation with Jörg to go along with his suggestion to simplify the visualisation. Therefore we took six spheres in the Unity scene: one sphere represents the palm (purple); for the five fingertips we placed five coloured spheres into the scene as such: thumb (red), pointing finger (silver), middle finger (green), ring finger (yellow), little finger (white).

scene in Unity

scene in Unity game view

Here, we had the plan to implement particle systems, mesh divider and fire trails to the spheres / fingertips. Certain auditive inputs were meant to trigger special visual effects in the background of the scene in the 3d space.

effect 1

effect 2

As we did´t succeed to connect the fingertips of the hand (represented by six coloured spheres) as well as the two effects via osc protocol with the Leap Motion Controller we decided to change our path and try a graphical visual representation in Processing. Unfortunately, we also had osc connectivity issues here. Finally we dropped the vision of graphical, visual representation for the project and concentrated our energy on the design of the sound of the instrument.