At first we had to set up the Kinect with Ubuntu. The tutorial from www.kdab.com came in very handy for a first try. Later we decided to use the openframeworks toolkits. It is an awesome collection of libraries and it contains the “ofxKinect” addon, which comes with some basic examples fitting perfectly for our purposes. Furthermore it’s written in C++ and it is open source.
We used the basic example for ofxKinect and adapted it.
For our design we need to track the actors on the stage in a defined area (in front of the projection). Therefore we decided to mount the Kinect to the ceiling, to leave the stage open to every possible design and keep the view from the audience to the stage clear.
To track just the actors we implemented a possibility to define the minimum rectangle one can draw around a blob the actor builds. This way we can set the height and width of this rectangle so it won’t track incorrectly the screen or a prop on the stage. It’s not working perfectly save now, because if a person touches an element of the stage it’s blob connects with the contour of the element. Then the new blob is mistakenly tracked as an actor and the tracked values are wrong.
The example came with the possibility to define the nearest and farthest threshold the Kinect is detecting. Of course we kept this feature and made it, with the others too, easy accessible through a graphical user interface, build with the help of the addon “ofxGUI”.
For the feature to draw with hands into the projection we used the possibility of Kinect to get the depth of every pixel the picture holds. We can set a range in which the height (we’re still beneath the ceiling) of the closest point to the projection plane (the upper range of the screen) of the blob is measured. Because humans tend to stretch their arms for drawing gestures, it was the easiest way to assume that the nearest point is the crucial and important one.
For the task of visualisation and projection it seemed to be good to use processing. It is easy to handle, has a lot of examples to mess around with and most designers like it. To use the data Kinect has tracked in processing , we decided to send messages containing our data via OSC. OSC is also easy to set up and you can design your messages at your wish. Right now we just have to send some float values, like the centre values of the actor and the x-/y-value of the closest point the actor gives, but of course we can send a lot more data if we want to. Because we use OSC for sending messages we are also able to set up the tracking on one laptop and give the projection task to another, faster one.
The processing sketches are basically drawing rectangles or ellipses on a beautifully designed background-image.