EKK:LoFi Sounds in HiFi Spaces/Immersive Collage

Idea

some first thoughts on the environment

How do we think of places that we‘ve never been to before? How do we imagine a place in San Diego, if we only hear its soundscape? How do image movements affect the perception of unknown soundscapes?

First of all, if we hear things of an unknown place, we are projecting images of our own experiences onto these “unknown” places, which therefore become “mappings” that are provided with our own mind.

But – due to internet services like google maps or google streetview, nowadays it‘s not a big deal anymore to demistify and to discover unknown places via internet. A massive amount of available image-data tells us anything about the far places. Still the place we expire via the internet is only a fragmental space.

edit: the following thoughts are formulated for one direction (San Diego to Weimar), but of course it should work vice versa, too.

So, what happens if people in Weimar are confronted with a new random environment of San Diego?

The idea is to play with these thoughts and build an immersive space that can be discovered by visitors in weimar. The visitor should be confronted with some new, made-up space, an audiovisual environment that is fed by two different sorts of data:

• 4 channel (live-)audio-stream from San Diego
• googleMaps/panoramio images from San Diego which have been shot on a location that is close to the current position of the recording device

edit:if its any trouble with copyrights we have to think about using other images or maybe attaching webcams to the adc~ unit...

The aim is to let a computer system discover new relations between San Diego soundscapes, google/panoramio-image data. The audiovisual result will then impact the associations each visitor would have on his own, since a generated movement of pixels may be obviously related to the sound, yet the concrete content of the images is not always clear due to the fragmental/blurred/distorted way of displaying.

Technical approach

some first thoughts on the processing chain

The current plan is to analyze the incoming live audio data via some semantical analysis (timbreID for Pd) and use the results to trigger a generative collage and of the picture-footage that is found on the internet or shot by the webcams evey now and then(?). A goal is to design some nice algorithms that put together different parts of images in relation to the sonic events. As an example: If you hear some waves of San Diego beach or some car traffic, the computer decodes the sound as a wave, which then causes the pixels from the images to appear in a wavelike movement that is related to the sound etc...(potential latencies are not a problem) The different parts of the images would then fade-out gently after some moments to be displaced by new pixel data.

Current experiments

• Setting up some timbreID- Pd patch. If we can train the system to recognize typical weimar sounds, then what will happen if it is employed on soundscapes from San Diego later on? Which sounds work fine, which don't? Are there interesting misinterpretations?
• Setting up an audiovisual data stream
• Collecting ideas and visualizations for image transformations. What looks nice? What are subjective associations one could try to code? e.g: noise: bird sounds → visualization: flocking; noise: surf → visualization: undulation …
• Writing an application (openFrameworks) that can implement the ideas. Experiments on generative (stereoscopic?) pixel transformations.

some first pixel fun

some first pixel fun

Video

<videoflash type=vimeo>67899634|450|250</videoflash> here are some first renderings. i'm sorry for the terrible quality, but my macbook had some hard times analyzing the audio, generating the animation and video-capturing the screen at the same time... Special credits go to Jonas, who was a great help explaining openGL&Pointclouds to me :D

Participants

• Alex
• Kevin

Links, Literature

• Corinne Vionnet: Photo Opportunities Crowdsourced photography.
• Ryoichi Kurokawa The impressive audiovisual installation "rheo" shows some interesting correspondences in sound and pixel processing.
• University of Dayton Interactive Wall another example, this time interesting in terms of two visual levels: the partially displayed image content and the movement of the enclosed animation.
• Jörn Loviscach A series of video tutorials that give a great overview and introduction to semantical audio analysis. Unfortunately in german only...

more to come