Hello World

This article demonstrates how to connect to the Leap Motion controller and access basic tracking data. After reading this article and following along with your own basic program, you should have a solid foundation for beginning your own application development.

First, a little background...

How the Leap Motion Controller Works

The Leap Motion controller encompasses both hardware and software components.

The Leap Motion hardware consists primarily of a pair of stereo infrared cameras and illumination LEDs. The camera sensors look upward (when the device is in its standard orientation). The following illustration shows how a user’s hands look from the perspective of the Leap Motion sensor:

https://di4564baj7skl.cloudfront.net/documentation/images/Leap_View.jpg

The Leap Motion software receives the sensor data and analyzes this data specifically for hands, fingers, arms, and tools. (Tracking for other types of objects could be added in the future, but this is the current set of tracked entities). The software maintains an internal model of the human hand and compares that model to the sensor data to determine the best fit. Sensor data is analyzed frame-by-frame and the service sends each frame of data to Leap Motion-enabled applications. The Frame object received by your application contains all the known positions, velocities and identities of tracked entities, such as hands, fingers, and tools. Constructs such as gestures and motions that span multiple frames are also updated each frame. For an overview for the tracking data provided by the controller, read API Overview.

The Leap Motion software runs as a service (Windows) or daemon (Mac and Linux) on the client computer. Native Leap Motion-enabled applications can connect to this service using the API provide by the Leap Motion dynamic libraries (provided as part of the Leap Motion SDK). Web applications can connect to a WebSocket server hosted by the service. The WebSocket provides tracking data as a JSON-formatted message – one message per frame of data. A JavaScript library, LeapJS, provides an API wrapping this data. For more information read System Architecture.

Set Up the Files

This tutorial also uses command line compilers and linkers (where needed) in order to focus on the code rather than the environment. For details on how to set up projects using the Leap Motion SDK in popular IDEs, see Setting Up a Project.

  1. If you haven’t already, download and unzip the latest Leap Motion SDK from the developer site and install the latest Leap Motion service.

  2. Open a terminal or console window and navigate to the SDK samples folder.

  3. Sample.cs contains the finished code for this tutorial, but to get the most out of this lesson, you can rename the existing file, and create a new, blank Sample.cs file in this folder. Keep the existing file for reference.

  4. In your new Sample.cs file, add code to reference the Leap Motion libraries:

    using Leap;
    
  5. Add the “structural” code to define a Java command-line program:

    class Sample
    {
        public static void Main ()
        {
            // Keep this process running until Enter is pressed
            Console.WriteLine ("Press Enter to quit...");
            Console.ReadLine ();
        }
    }
    

This code simply prints a message and then waits for keyboard input before exiting. See Running the Sample for instructions on running the program.

Get Connected

The next step is to add a Controller object to the program – which serves as our connection to the Leap Motion service/daemon.

class Sample
{
    public static void Main ()
    {
        Controller controller = new Controller ();

        // Keep this process running until Enter is pressed
        Console.WriteLine ("Press Enter to quit...");
        Console.ReadLine ();
    }
}

When you create a Controller object, it automatically connects to the Leap Motion service and, once the connection has been established, you can get tracking data from it using the Controller.Frame() method.

The connection process is asynchronous, so you can’t create the Controller in one line and expect to get data in the next line. You have to wait for the connection to complete. But for how long?

To Listen or not to Listen?

You can add a Listener object to the Controller, which provides an event-based mechanism for responding to important Controller state changes. This is the approach used in this tutorial – but it is not always the best approach.

The Problem with Listeners: Listener objects use independent threads to invoke the code that you implement for each event. Thus, using the listener mechanism can introduce the complexities of threading into your program. It becomes your responsibility to make sure that the code you implement in your Listener subclass accesses other parts of your program in a thread-safe manner. For example, you might not be able to access variables related to GUI controls from anything except the main thread. There can also be additional overhead associated with the creation and cleanup of threads.

Avoiding Listeners: You can avoid using Listener objects by simply polling the Controller object for frames (or other state) when convenient for your program. Many programs already have a event- or animation-loop to drive user input and animation. If so, you can get the tracking data once per loop – which is often as fast as you can use the data anyway.

The Listener class in the API defines the signatures for a function that will be called when a Controller event occurs. You create a listener by creating a subclass of Listener and implementing the callback functions for the events you are interested in.

To continue this tutorial, add the SampleListener class to your program:

class SampleListener : Listener
{
    private Object thisLock = new Object ();

    private void SafeWriteLine (String line)
    {
        lock (thisLock) {
                Console.WriteLine (line);
        }
    }

    public override void OnConnect (Controller controller)
    {
        SafeWriteLine ("Connected");
    }


    public override void OnFrame (Controller controller)
    {
        SafeWriteLine ("Frame available");
    }
}

Note that the SafeWriteLine function exists because the Listener is multi-threaded. Calling Console.WriteLine() directly occasionally causes a threading conflict.

If you have already taken a look at the finished file, you may have noticed that several more callback functions are present. You can add those too, if you wish, but to keep things simple, we will concentrate on OnConnect() and OnFrame().

Now create a SampleListener object using your new class and add it to your controller:

class Sample
{
    public static void Main ()
    {
        SampleListener listener = new SampleListener ();
        Controller controller = new Controller ();
        controller.AddListener (listener);

        // Keep this process running until Enter is pressed
        Console.WriteLine ("Press Enter to quit...");
        Console.ReadLine ();

        controller.RemoveListener (listener);
        controller.Dispose ();
    }
}

Now is a good time to test your sample program. Follow the directions in: Running the Sample.

If everything is correct (and your Leap Motion hardware is plugged in), then you should see the string “Connected” printed to the terminal window followed by an rapid series of “Frame available”. If things go wrong and you cannot figure out why, you can get help on our developer forum at developer.leapmotion.com.

Whenever you run into trouble developing a Leap Motion application, try opening the diagnostic visualizer. This program displays a visualization of the Leap Motion tracking data. You can compare what you see in your program to what you see in the visualizer (which uses the same API) to isolate and identify many problems.

On Connect

When your Controller object successfully connects to the Leap Motion service/daemon AND the Leap Motion hardware is plugged in, The Controller object changes its IsConnected property to true and invokes your Listener.OnConnect() callback (if there is one).

When the controller connects, you can set controller properties using the Controller.EnableGesture() and Controller.SetPolicy() methods. For example, you could enable the swipe gesture with the following onConnect() function:

On Frame

All the tracking data in the Leap Motion system arrives through the Frame object. You can get Frame objects from your controller (after it has connected) by calling the Controller.Frame() method. The OnFrame() callback of your Listener subclass is called when a new frame of data becomes available. When you aren’t using a listener, you can compare the Id value to that of the last frame you processed to see if there is a new frame. Note that by setting the history parameter of the Frame() function, you can get earlier frames than the current one (up to 60 frames are stored). Thus, even when polling at a slower rate than the Leap Motion frame rate, you can process every frame, if desired.

To get the frame, add this call to Frame() to your OnFrame() callback:

Then, print out some properties of the Frame object:

public override void OnFrame (Controller controller)
{
    Frame frame = controller.Frame ();

    SafeWriteLine ("Frame id: " + frame.Id
             + ", timestamp: " + frame.Timestamp
             + ", hands: " + frame.Hands.Count
             + ", fingers: " + frame.Fingers.Count
             + ", tools: " + frame.Tools.Count
             + ", gestures: " + frame.Gestures ().Count);
}

Run your sample again, put a hand or two over the Leap Motion device and you should now see the basic statistics of each frame printed to the console window.

I’ll end this tutorial here, but you can look at the rest of the code in the original sample program for examples on how to get all the Hand, Finger, Arm, Bone and Gesture objects in a frame.

Running the Sample

To run the sample application:

  1. Compile the sample application:

    • On Windows, make sure that Sample.cs and either LeapCSharp.NET3.5.dll or LeapCSharp.NET4.0.dll are in the current directory. Run the following command in a command-line prompt (using the proper library reference for the .NET framework you are using):

      csc /reference:LeapCSharp.NET4.0.dll /platform:x86 /target:exe Sample.cs
      

      Note: use the ``csc`` compiler from the appropriate version of the .NET framework.

    • On Mac, you can use the Mono project to compile C# programs. Make sure that Sample.cs and either LeapCSharp.NET3.5.dll or LeapCSharp.NET4.0.dll are in the current directory:

    • For the .NET 3.5 framework, run:

      gmcs /reference:LeapCSharp.NET3.5.dll /platform:x86 /target:exe Sample.cs
      
    • For the .NET 4.0 framework, run:

      dmcs /reference:LeapCSharp.NET4.0.dll /platform:x86 /target:exe Sample.cs
      
  2. Run the sample application:

    • On Windows, make sure that Sample.exe, LeapCSharp.dll, Leap.dll and either LeapCSharp.NET3.5.dll, or LeapCSharp.NET4.0.dll are in the current directory. Use the libraries in the lib\x86 directory for 32-bit projects. Use the libraries in the lib\x64 directory for 64-bit projects. Run the following command in a command-line prompt:

      Sample.exe
      
    • On Mac, make sure that Sample.exe, libLeapCSharp.dylib, libLeap.dylib and either LeapCSharp.NET3.5.dll, or LeapCSharp.NET4.0.dll are in the current directory and run the following command in a terminal window:

      mono Sample.exe