Kinect

Introduction

A pixel is the smallest unit of an image.

In a black and white image, the pixel will be some shade of gray between 100% black and 100% white. In a color image, pixels are described in terms of RGB values. In all images, a pixel is described by a number that represents its intensity. The larger the number the more memory the pixel needs. A colored image needs more memory than a grayscale image.

The image captured by the Kinect is 640 pixels by 480 pixels (or 307,200 pixels). Each pixel's number represents both its grayscale intensity and its distance in space. You can use these numbers to either interpret the data as an image or to calculate the distance of the object that the pixel is a part of.

In the captured frame, each pixel can describe the distance between the object and the Kinect. The brightest part of the image (closest to 255) is closest to the Kinect, while the darkest part of the image (closer to 0),is the farther away from the camera.

Lesson

Read Chapter 2 of Greg Borenstein's Making Things See
Setting up SimpleOpenNI
Simple SimpleOpenNI
Images and Arrays
Getting Interactive

Setting up SimpleOpenNI

SimpleOpenNI is a library that provides access to all of the data from the Kinect, as well as a series of tools and helpers.

Install PrimeSense's OpenNI in your applications folder. This application communicates with the Kinect in order to access and process the data

MAC
Download OpenNI_NITE_Installer-OSX-0.xx.zip and Max Rheiner's SimpleOpenNI
1. On the MAC you should see the following:
  - install.sh
  - LGPL.txt
  - Readme.txt
  - data
2. Open Terminal (Command_CTRL+U)
3. Change directory to the downloaded folder (type cd and a space, then drag the folder into the terminal window, then press RETURN)
4. Run the installer with the command:
```
sudo ./install.sh
```
  Type your password when prompted.
Windows
Installing OpenNI requires a number of pieces of supporting software that all have to be installed for things to work. A company called ZigFu provides a oneclick installer that puts all the pieces in place for you.
1. Download the ZigFu DevBundle (or download all the files from here.
  
  If you’ve installed the Microsoft Kinect SDK, you need to uninstall it. The OpenNI software does not work with the Microsoft Kinect drivers.
Install the SimpleOpenNI library in your Processing>Libraries folder
Quit and restart Processing if it was open
Plug in the Kinect and make sure it is facing you.
Test out the install by running one of the built-in examples that ships with SimpleOpenNI. Inside of OpenNI, find DepthImage and double-click it. This is a basic sketch. Hit the play button in the top left corner of the Processing window to run the sketch.(Slow start up is normal when using SimpleOpenNI.)
If the sketch fails, you might see this
```
Invalid memory access of location 0x8 eip=0xb48d8fda
```
If you see that your Kinect is not plugged in or not fully powered. Make sure your Kinect is connected to a power outlet and plugged into your computer’s USB port. If you’re working on a laptop also make sure that your laptop is plugged into power. Some laptops will provide inadequate power to their USB ports when running off a battery.

Simple Kinect Sketch

The color of each pixel in the depth image represents the distance of that pixel in the scene.

Objective:When you print on the image, Processing will print out information on individual pixels

Connect your Kinect
Open Processing and create a new sketch

Paste in the following:

import SimpleOpenNI.*;
SimpleOpenNI kinect;

void setup(){
// double the width to display two images side by side
    size(640*2, 480);
    kinect = new SimpleOpenNI(this);
    kinect.enableDepth();
    kinect.enableRGB();
}
void draw(){
    kinect.update();
    
    PImage depthImage = kinect.depthImage();
    PImage rgbImage = kinect.rgbImage();

    image(depthImage, 0, 0);
    image(rgbImage, 640, 0);
}

void mousePressed(){
    color c=get(mouseX, mouseY);
    println("r: "+red(c)+"g: "+green(c)+"b: "+blue(c));
}

Run the sketch. You might find that some of the values are surprising. This is because when you look at the world you see colors as relative to the environment. But this is not how a digital camera sees them.To test this out, look for the whitest part of a color image. It if were 100% white, you should get 255 or each R, G, B values
If you click on te depth image, you will get brightness data (shades of gray). But how do you translate brightness to distance? The kinect can detect distances between 20 inches and 25 feet with millimeter precision. You can figure out that a brightness value of 96 is 37% between 0 and 255, or about 14.5 feet away. Chances are this distance is not correct. The issue is that an 8bit pixel is manageable to work with, but is not as accurate as it can be. The kinect can be more accurate, but the processing time would also get extended. But if you use the more precise piel sparingly, you should be able to get accurate distances without slowing your application down too much.

Images and Arrays

While we think of images as columns and rows, an image is actually described with a 1-dimensional array.

Accessing the array of pixels directly allows you to run your program faster, and allows you to work with higher resolution data, as well as enabling you to work with more than one pixel at a time.

Open Photoshop by clicking on the icon on the dock
Create a new 8-bit image by pressing Command+N
Create a new layer (Command+Shift+N)
Select Edit>Fill and fill image with 50% gray
Save the file as Photoshop Raw
This file is stored as a 1 dimensional array, which means every pixel is assigned a number in sequential order.
1 2 3 4

5 6 7 8
Open Audacity.
Click on Project>Import Raw Data
You should see a straight line because the image is uniform (there is no change).
Back in Photoshop, add some noise
Resave and re-open in Audacity.
Now back in Photoshop add splotches of black and white.
Resave and reopen in Audacity.
Audacity plays the image file from top left to bottom right, reading each pixel in sequential order. The frequency is the width of the image.

The first pixel is 0, 0.
Even though graphics programs stores image data in a one-dimensional array, you want to be able the think about the data in terms of two-dimensions.

Each pixel can be described with this formula:
```
int pixel=x+(y*width_of_image)
```

Getting Interactive

Tracking the nearest object

If you create a sketch that tracks an object closest to the kinect (let's say your hand), you could have your hand control shapes or pixels.

Finding the closest pixel is like finding a minimum or maximum value. You store the first value of the array as your target, you iterate through the array and compare each subsequent value to the target and update the target as necessary.

import SimpleOpenNI.*;
SimpleOpenNI kinect;

int closestValue;
int closestX;
int closestY;

void setup(){
// double the width to display two images side by side
    size(640, 480);
    kinect = new SimpleOpenNI(this);
    kinect.enableDepth();
  
}
void draw(){
    //set closestValue to the farthest possible value (8000)
    ____________________________

    kinect.update();
    
    //create an array of ints to hold the depth values
    ____________________________
    
    //create a for loop with y as the variable.
    //start at 0 and continue as long as y is less than the //height of the image, increment by 1
    ____________________________
         
          //create a for loop with x as the variable.
          //start at 0 and continue as long as x is less than //the width of the image, increment by 1
         for (int x=0;x<640;x++){
             
             //create a variable to hold the pixel
             //remember that the formula is
             //int pixel=x+(y*the_width_of_the_image)
             ____________________________
             
             //set the currentDepth to the current iteration in the depth value array 
             ____________________________
             
             //compare if the current depth is greater than 0 and less than 
             //the closestValue
             ____________________________
                 //update the closest value
                 ____________________________
                 //update the closestX
                 ____________________________
                 //update the closestY
                 ____________________________
             }
        }
    }
    
    PImage depthImage = kinect.depthImage();

    image(depthImage, 0, 0);
    //pick a fill color
    ____________________________
    
    //draw a circle at the closestX and closestY
   ____________________________
    
}

Run the sketch and move your hand in front of you and in front of the kinect.

Modify the previous sketch so that instead of an ellipse, you draw a line.

You'll need:
Two new variables of type float:
1. lastX
2. lastY
Set the background to a color (Do this in the setup())
Don't display the depth image (comment the appropriate line out)
Set the variable that was holding the pixel , to hold the mirrored pixel, or translated a distance from the right edge:
```
	int reversedX = 640-x-1;
	//int pixel=x+(y*width_of_image)
	int pixel=_____________;
	
```
Only set the current X and Ys if the the current depth is less than the closest depth and the current depth is between 2 and 5 feet (610 and 1525)
Perform a linear interpolation (lerp). lerp() takes three arguments, the first two are the two numbers to interpolate between and the last is a float that specifies how much to move between the two points (with 1.0 being all the way to the second point and 0.0 being none). Here's how lerp() works:
```
float x = lerp(-20.0, 60.0, 0.0); // Assign -20.0 to x
float y = lerp(-20.0, 60.0, 0.5); // Assign 20.0 to y
float z = lerp(-20.0, 60.0, 1.0); // Assign 60.0 to z
```
In your sketch set interpolatedX and interpolatedY:
```
float interpolatedX = lerp(your_last_x, your_closest_x, 0.3f); 
```
and do the same for the y
Instead of fill(), set the stroke() to a color.
Set the width of the line (strokeWeight())
Use the line() function. This should take the last point and the interpolated point.
update the last variables (lastX and lastY) to hold the interpolated X and Y values
Add a way to clear the screen.

Assignment 1:Thresholding, or grabbing a slice

Create a Processing Sketch where you analyze the depth image and if the pixels are within a range, color the pixels blue, otherwise don't display the pixels.

You'll need an instance variable of type PImage. Name it myImage
Create instance variables of type int that will hold your min and max values. Set them to 0 and 750 respectively.
Set your PImage to a new PImage with the appropriate dimensions.
Inside the draw() method loop through all the depth pixels, setting the pixels within the threshold range to the color blue and everything else to black.
```
 yourPImage.pixels[i]=0x0000FFFF;
 yourPImage.pixels[i]=0;
```
You'll need to update the pixels of your PImage:
```
yourPImage.updatePixels();
```
Draw the image to the screen.

Assignment 2

Do the minority report examples from chapter 2 of Making Things See by Greg Borenstein.

/ / navigation