GMU:Tutorials/Networking/Controlling Processing with The Captury

!! See functioning Processing sketch provided by Felix

Introduction

Processing is usually used for the electronic arts, new media art and visual design.
If you want to have a creative work in processing with Captury, you need to receive the OSC messages from the Captury and then use the data as the variable.
This includes OSC with Processing and an example of controlling the video in Processing with Captury.
Follow this Step by Step tutorial for using PROCESSING to play the VIDEO with CAPTURY!

Tips Before Beginning

In this tutorial, you will see how to use a single value getting from the Captury to control processing. For further processed data, referencing the tutorial by Qianqian Li: Measuring Motion with Processing

Before the coding in Processing, make sure that the following libraries have been installed: Sketch > Import Library > Add Library
- oscP5: An Open Sound Control (OSC) implementation
- Video: GStreamer-based video library for Processing

Prepare the video with a high resolution that can be showed on the video wall well. Because the change of the play speed, the video need a high frame rate to play fluently even with low speed. Here is the information about the video used here as the example:
- Format: .mov (H.264 Codec)
- Resolution: 1920*1280 px
- Length: 10s
- Frame Rate: 60 FPS
- Frame: 0~599

Use The Data From The Captury In Processing

Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.

PART 01 – Connect the Captury and Processing

To build a connection between the Captury and Processing, you need to define port number. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.

int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages

Port for the Captury to Send Messages

Start oscP5, listening for incoming messages at local port.

osc = new OscP5(this, localport);

Remote is a NetAddress, which includes an IP address and a port number. Here it used as parameter to sending OSC messages back to the Capury.

remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);

PART 02 – Receive data from the Captury

Inform Processing the name of the skeleton and the bone in use. Remember to rename your own skeleton to tell the difference from the other.

String skeleton1 = "LS";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use

Name of Skeleton and Bone

Incoming OSC messages are forwarded to the oscEvent method and this function runs every time you receive data. Print the Address Pattern and the Typetag of the received OSC messages in the console.
- Address Pattern is used to differentiate between different messages you are sending on one port. Here to differentiate between different bones and skeletons.
- Typetag is used to define what the data type is. For the data of position x, y, z receiving from the Captury, they are all float 32 type and will be showed in console as 'f' ('i' for int 32, 's' for OSC-string, 'b' for OSC-blob, etc.)

void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}

Data Received in Processing

To make sure that all the bones in use are available and refresh them every second or so. And send the message back to the Captury.

void refreshSubscriptions() {
  subscribeBone(skeleton1, bone1);
}

if (frameCount % 10 == 0) {
refreshSubscriptions();
}

void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");  //Create OSC message
  msg.add(50.0);  //Add a float to the OSC message
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);  //Send the message
}

Data Received in the Captury

PART 03 – Pass data to the object

Define a function to plug a specific bone to a object.

void plugBone(Object target, String skeleton, String bone) {
  String path =  "/" + skeleton + "/blender/" + bone + "/vector";
  osc.plug(target, "update" + bone, path);
}

Declare and construct a object (sphere) from the class Sphere, which is the receiver of the data.

Sphere sphere1;

sphere1 = new Sphere(1);

sphere1.draw();

Use plugBone function to pass position data of LS's Root bone to the sphere object

plugBone(sphere1, skeleton1, bone1);

PART 04 – Use the data as a variable

In the class Sphere, define the float variable for the position of the sphere.

float x, y, z;

Update the position of the root node. This function will be plugged to OSC.

public void updateRoot(float x, float y, float z) {
  this.x = map(x, 0, 100, -width/2, width/2);
  this.y = map(y, -100, 100, -height/2, height/2);
  this.z = map(z, 0, 100, 0, depth);
}

Then you can have a sphere moving follow the data from the Captury! For further application, you can use x, y, z to control whatever you want in Processing!

Control The Video By Processing

Since we can successfully use the data from the capture, it can be creatively to use the position of the object to control whatever you want in Processing!
The following is an example of using position values to play or reverse the video and change the play speed of the video.
This video is about a 3D Chinese Knot tied from loose to tight. The 3D animation is made by 3DS MAX.

PART 01 – Play a video in Processing

Basic code to play a video in Processing.

import processing.video.*;  //Import libraries

Movie myMovie;  //Declare a object as Movie

void setup() {
  size(1920, 1080);  //or use 'fullScreen(P3D);'
  frameRate(60);
  background(0);
  myMovie = new Movie(this, "Chinese Knot_Final.mov");  //Load a movie, normally followed with 'play()' or 'loop()'
}

Draw the movie by the class object (Sphere)

void draw() {
    image(myMovie, 0, 0, width, height);  //Position to put the movie
}

Called every time a new frame is available to read.

void movieEvent(Movie m) {
  m.read();
}

PART 02 – Play or reverse the video and change the play speed

First thing to know is about the relationship between coordinate axis in the Captury and the Performance Platform. Here is the diagram:

Coordinate Axis of the Performance Platform

Diagram

Using position x as the condition. If user stand in the center of the platform, the speed will be slowest. The farther away from the center user stand, the faster the movie play.
Using position y as the condition. If user stand at the side close to the Video Wall, the movie will play as usual, if else the movie will reverse.

int play;  //Declare a value to judge play or reverse the video

Get the total number of the movie frames.

  int getLength() {
      return int(myMovie.duration() * myMovie.frameRate);
  }

Get current frame.

  int getFrame() {    
      return ceil(myMovie.time() * 60) - 1;
  }

Pay attention that if the movie play to the end, it must be reversed. And if the movie reverse to the beginning, it must play as usual.

    if(x < -50 && getFrame() < (getLength() - 1)){
      play = 1;
      myMovie.play();
    }
    else {
      play = -1;
      myMovie.play();
    }

    float newSpeed = map(abs(y)/10, 0, 150, 0*play, 2*play);  //abs() to have absolute position value
    myMovie.speed(newSpeed);

Let's Enjoy The Example File

Thanks for your attention to this tutorial, and feel free to debug and improve it.

Here is the complete code for this tutorial uploading on GitHub: Complete Code of Wool's World
Example File including the movie is shared on Dropbox: Example File of Wool's World

Demonstration

Demonstration Video of Controlling the Video in Processing with Captury