GMU:Tutorials/Networking/Controlling Processing with The Captury

Introduction

Processing is usually used for the electronic arts, new media art and visual design.
If you want to have a creative work in processing with Captury, you need to receive the OSC messages from the Captury and then use the data as the variable.
This includes OSC with Processing and an example of controlling the video in Processing with Captury.
Follow this Step by Step tutorial for using PROCESSING to play the VIDEO with CAPTURY!

Tips Before Beginning

• In this tutorial, you will see how to use a single value getting from the Captury to control processing. For further processed data, referencing the tutorial by Qianqian Li: Measuring Motion with Processing

• Before the coding in Processing, make sure that the following libraries have been installed: Sketch > Import Library > Add Library
  • oscP5: An Open Sound Control (OSC) implementation
  • Video: GStreamer-based video library for Processing

• Prepare the video with a high resolution that can be showed on the video wall well. Because the change of the play speed, the video need a high frame rate to play fluently even with low speed. Here is the information about the video used here as the example:
  • Format: .mov (H.264 Codec)
  • Resolution: 1920*1280 px
  • Length: 10s
  • Frame Rate: 60 FPS
  • Frame: 0~599

Use The Data From The Captury In Processing

Data sending and receiving between the Captury and Processing over a network connection require OSC (Open Sound Control) protocol. The following is the anatomy of OSC in Processing.

• To build a connection between the Captury and Processing, the listening port and the port of the remote location address need to be same. This could basically any number, and using a 4- or 5-digit number to stay out of the range of most common ports.

int localport = 12000;
int remoteport = 1065;  //Port number the Captury used to send messages

• Start oscP5, listening for incoming messages at local port.

osc = new OscP5(this, localport);

• Remote is a NetAddress, which includes an IP address and a port number. Here it used as parameter to sending OSC messages back to the Capury.

remote = new NetAddress("kosmos.medien.uni-weimar.de", remoteport);

• Inform Processing the name of the skeleton and the bone in use.

String skeleton1 = "LiQianqian";  //Skeleton name same with the Captury
String bone1 = "Root";  //The bone in use

• To make sure that all the bone in use can be refreshed every second or so.

refreshSubscriptions();

void refreshSubscriptions() {
  subscribeBone(skeleton1, bone1);
}

if (frameCount % 10 == 0) {
refreshSubscriptions();
}

void subscribeBone(String skeletonId, String bone) {
  OscMessage msg = new OscMessage("/subscribe/" + skeletonId + "/blender/" + bone + "/vector");
  msg.add(50.0);
  msg.add(0.0);
  msg.add(100.0);
  osc.send(msg, remote);
}

• capture all OSC events

void oscEvent(OscMessage msg) {
  if(debug) {
    print("### received an osc message.");
    print(" addrpattern: "+msg.addrPattern());
    println(" typetag: "+msg.typetag());
  }
}

• plug a specific bone

void plugBone(Object target, String skeleton, String bone) {
  String path =  "/" + skeleton + "/blender/" + bone + "/vector";
  osc.plug(target, "update" + bone, path);
}

• objects in space

Sphere sphere1;

sphere1 = new Sphere(1);

• pass position of Asha's root bone to the first sphere object

plugBone(sphere1, skeleton1, bone1);

class Sphere {
  // position of the sphere
  float x, y, z;
  Sphere( color c) {
    this.c = c;
  }
  // draw the sphere
  void draw() {
  
  }
  // update the position of the root node 
  // (this function will be plugged to OSC)   
  public void updateRoot(float x, float y, float z) {
    this.x = map(x, 0, 100, -width/2, width/2);
    this.y = map(y, -100, 100, -height/2, height/2);
    this.z = map(z, 0, 100, 0, depth);}}

Play The Video By Processing

STEP 01 –