Leap Motion C API  4.1.0
The API to the LeapC library.
Images

The Leap Motion controller uses infrared stereo cameras as tracking sensors. The LeapC API provides an image buffer containing the sensor brightness values and a distortion buffer containing the camera calibration map. The distortion map can be used to correct lens distortion in the image data. The function LeapRectilinearToPixel() can be used to undistort specific pixels in an image and the function LeapPixelToRectilinear() can be used to find the 3D location corresponding to a matched pair of stereo pixels.

Using Images

Images are sent automatically if the policy for images is set.

When a stereo image pair is available, a LEAP_IMAGE_EVENT is added to the event queue and can be accessed via LeapPollConnection(). The image event contains image properties, distortion correction data, and a pointer to the buffer.

Both images from the stereo cameras are written to the same buffer; left image first and then the right image.

To get an image:

  1. Open the connection.
  2. Set image policy.
  3. Poll for images.
  4. Continue polling for image frames (optionally also poll for tracking frames and status messages). When an image is available, a LEAP_IMAGE_EVENT message is provided by LeapPollConnection(). The LEAP_IMAGE_EVENT struct contains the images, a description of the images, and the distortion map.

Image Buffer Size

The image buffer size depends on the device type as well as the image type and format. The buffer size depends on the width, height, and pixel format (bytes per pixel) of the image. A single buffer must hold stereo images. For example, if the current images produced by a Leap Motion device is 640x240, 1-byte pixels, then the buffer size must be: 640 x 240 x 1 x 2 = 307,200 bytes.

Image Distortion

When a ray of light enters one of the Leap Motion cameras, the lens bends the ray so that it hits the sensor, which records it as a greyscale brightness value at a specific pixel location. Of course, no lens is perfect, so a ray of light does not land on the sensor in the optically perfect spot. The distortion map provides data to correct this imperfection, allowing you to calculate the true angle of the original ray of light. You can use the corrected angle to generate a rectified image, and, using the angles from both images in the stereo pair, you can triangulate the 3D location of a feature identified in both images.

For image rectification, the distortion map can be fed to a shader program that can efficiently interpolate the correction applied to rays of light to produce a texture containing the rectified image. For getting the true angle for a small set of points, you can use the LeapPixelToRectilinear() function (but this is not typically efficient enough to transform a full bitmap at a high frame rate).

Rectifying Image Points

To rectify individual points in an image, you can call the LeapRectilinearToPixel() function. Given a ray from the camera, this function returns the pixel coordinates in the image buffer that contain the light recorded from that direction, corrected for lens distortion.

The following code takes a target texture size and determines the correct pixel brightness value. For each pixel, the code computes the ray direction to the point in the scene that would illuminate the target pixel given an ideal optical system. The code then corrects for optical distortion by calling LeapRectilinearToPixel(), which provides the pixel coordinates in the image buffer that actually contains the brightness value for the target pixel. The code copies the brightness value to the target texture buffer. The result is a rectified image.

//The pixel size of the textures we write the undistorted images to
#define TEX_WIDTH 400
#define TEX_HEIGHT 400
//Max field of view varies by device, use 8 for the peripheral, 22 for Rigel
#define MAX_FOV 22
//image_buffer contains the image data
for( float row = 0; row < TEX_HEIGHT; row++ ) {
for( float col = 0; col < TEX_WIDTH; col++ ) {
//Normalize from pixel xy to range [0..1]
LEAP_VECTOR input;
input.x = col/TEX_WIDTH;
input.y = row/TEX_HEIGHT;
//Convert from normalized [0..1] to ray slopes
input.x = (input.x - .5) * MAX_FOV;
input.y = (input.y - .5) * MAX_FOV;
int dindex = (int)floor(row * TEX_WIDTH + col);
int pindex = (int)roundf(pixel.y) * image_width + (int)roundf(pixel.x);
if(pixel.x >= 0 && pixel.x < image_width && pixel.y >=0 && pixel.y < image_height){
undistorted_image_left[dindex] = ((char*)image_buffer)[pindex];
} else {
undistorted_image_left[dindex] = 128;
}
}
}

For the right image, remember to offset the index into the image_buffer array since the right image follows the left.

You can get the actual device field of view from the LEAP_DEVICE_INFO struct. Use tan(fov/2) in the calculation above instead of MAX_FOV.

Note that LeapRectilinearToPixel() may not be fast enough to rectify the image to a high resolution texture in real-time. Generally you should limit its use to individual points, image patches, or low-resolution textures. For better performance for full images, you can use a shader to rectify the image.

See Rectifying Points Example

Rectifying with a Shader

To rectify an entire image with a shader, create a texture containing the data in the distortion buffer. The distortion buffer is contained in the distortion_matrix field of the LEAP_IMAGE_EVENT struct. This data is set up to use a shader's normal uv interpolation mechanism. You can use the interpolated uv coordinates to look up the corrected brightness in the image texture, as in the following fragment shader program:

uniform sampler2D rawTexture;
uniform sampler2D distortionTexture;
void main()
{
vec2 distortionIndex = texture2D(distortionTexture, gl_TexCoord[0].st).xy;
float hIndex = distortionIndex.r;
float vIndex = distortionIndex.g;
if(vIndex > 0.0 && vIndex < 1.0 && hIndex > 0.0 && hIndex < 1.0)
{
gl_FragColor = vec4(texture2D(rawTexture, distortionIndex).rrr, 1.0);
}
else
{
gl_FragColor = vec4(0.2, 0.0, 0.0, 1.0);
}
}

To interpolate correctly, the distortion texture must use the following filter and wrap parameters:

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

The distortion texture itself must use two floating point values per texel and contain 64x64 texels:

glTexImage2D(GL_TEXTURE_2D, 0, GL_RG, 64, 64, 0, GL_RG, GL_FLOAT, distortion_buffer_left);

The distortion map itself rarely changes – the only time it changes is if the Leap Motion hardware is swapped with a different device or if the device is rotated so that the user's hands enter from the opposite side of the vertical field of view. In this case the Leap Motion software inverts the image, and the distortion map, to automatically maintain the proper orientation. (Auto-orientation can be turned off in the Leap Motion service configuration.)

See Rectifying with a Shader Example

Detecting when the Distortion Map Changes

The distortion map contains a grid of values that can be used to rectify the images. Rectification should be performed if you are aligning 3D objects with pixels in the images or if you are performing stereo triangulation on the image pair.

Each image complete event contains the distortion map for that image, however, since translating the map to an interpolation array or shader texture can itself be an expensive task, you typically only want to do this when the distortion map actually changes – not once for every image request. The distortion map can only change when the images are flipped due to automatic or manual re-orientation or if the device itself is changed. (For internal reasons) there is no way to tell that a reorientation has occurred. However, you can use the LEAP_IMAGE_EVENT::matrix_version field to detect when the distortion map changes. When the matrix_version changes between images, you know that the distortion map has changed. The matrix version number ALWAYS increases when the distortion map changes and never returns to a prior value.

See Rectifying with a Shader Example