Hello World

This article discusses the C# sample application included with the Leap SDK. After reading this article, you should be ready to access Leap hand tracking data from your own C# applications.

In the Leap SDK folder, you can find the following files used for this article:

  • LeapSDK/samples/Sample.cs — C# sample application
  • Windows:
    • LeapSDK/lib/LeapCSharp.NET3.5.dll — Leap C# library for .NET 3.5
    • LeapSDK/lib/LeapCSharp.NET4.0.dll — Leap C# library for .NET 4.0
    • LeapSDK/lib/x86/LeapCSharp.dll — 32-bit Leap C# library for Windows
    • LeapSDK/lib/x64/LeapCSharp.dll — 64-bit Leap C# library for Windows
    • LeapSDK/lib/x86/Leap.dll — 32-bit Leap library for Windows
    • LeapSDK/lib/x64/Leap.dll — 64-bit Leap library for Windows
  • Mac (Mono):
    • LeapSDK/lib/LeapCSharp.NET3.5.dll — Leap C# library for .NET 3.5
    • LeapSDK/lib/LeapCSharp.NET4.0.dll — Leap C# library for .NET 4.0
    • LeapSDK/lib/libLeapCSharp.dylib — Leap C# library for Mac
    • LeapSDK/lib/libLeap.dylib — Leap library for Mac

Overview

In a nutshell, the Leap Motion tracking device detects and tracks hands and fingers placed within its field of view. The Leap captures this data one frame at a time. Your applications can use the Leap API to access this data.

The sample application demonstrates how to use the Leap API to listen for frame events dispatched by the Leap and how to access the hand and finger data in each frame. The application is a small command-line program that prints information about detected hands and fingers to standard output. The application is contained in a single file, Sample.cs.

The sample application uses most of the key classes in the Leap API, including:

  • Controller — the interface between the Leap and your application
  • Listener — used to handle events dispatched by the Leap
  • Frame — contains a set of hand and finger tracking data
  • Hand — contains tracking data for a detected hand
  • Finger — contains tracking data for a detected finger
  • Vector — represents a 3D position or directional vector
  • Gesture — represents a recognized gesture.

Creating a Controller object

The Controller class provides the main interface between the Leap and your application. When you create a Controller object, it connects to the Leap Motion software running on the computer and makes hand tracking data available through Frame objects. You can access these Frame objects by instantiating a Controller object and calling the Controller.Frame method.

If your application has a natural update loop or frame rate, then you can call Controller.Frame as part of this update. Otherwise, you can add a listener to the controller object. The controller object invokes the callback methods defined in your Listener subclass whenever a new frame of tracking data is available (and also for a few other Leap events).

The sample application creates a Controller object in the Sample class’s Main method and adds an instance of a Listener subclass to the Controller with the Controller.AddListener method:

class Sample
{
    public static void Main()
    {
        // Create a sample listener and controller
        SampleListener listener = new SampleListener();
        Controller controller = new Controller();

        // Have the sample listener receive events from the controller
        controller.AddListener(listener);

        // Keep this process running until Enter is pressed
        Console.WriteLine("Press Enter to quit...");
        Console.ReadLine();

        // Remove the sample listener when done
        controller.RemoveListener(listener);
        controller.Dispose();
    }
}

The application then runs until you press the Enter key. In the meantime, the controller calls the appropriate listener callback methods when events occur. Before running the program, though, you must create your own subclass of the Listener class.

Subclassing the Listener class

The sample application defines a Listener subclass, SampleListener, which implements callback methods to handle events dispatched by the Leap Motion controller. The events include:

  • OnInit — dispatched once, when the controller to which the listener is registered is initialized.
  • OnConnect — dispatched when the controller connects to the Leap Motion controller and is ready to begin sending frames of motion tracking data.
  • OnDisconnect — dispatched if the controller disconnects from the Leap Motion controller (for example, if you unplug the device or shut down the Leap Motion software).
  • OnExit — dispatched to a listener when it is removed from a controller.
  • OnFrame — dispatched when a new frame of motion tracking data is available.

For three lifecycle event callbacks, OnInit, OnDisconnect, and OnExit the sample application simply prints a message to standard output. For the OnConnect and OnFrame event, the listener callback does a bit more work. When the controller calls the OnConnect callback, the function enables recognition for all the gesture types. When the controller calls OnFrame, the function gets the latest frame of motion tracking data and prints information about the detected objects to standard output.

Getting a Frame of data

The Controller calls the OnFrame callback method when the Leap generates a new frame of motion tracking data. You can access the new data by calling the Controller.Frame() method, which returns the newest Frame object. (A reference to the Controller object is passed to the callback as a parameter.) A Frame object contains an ID, a timestamp, and a list containing a Hand object for each physical hand in view.

The following code from the sample application’s OnFrame implementation gets the most recent Frame object from the controller, retrieves the list of hands from the frame and then prints out the frame ID, timestamp, and the number of hands in the list:

// Get the most recent frame and report some basic information
Frame frame = controller.Frame();
SafeWriteLine("Frame id: " + frame.Id
            + ", timestamp: " + frame.Timestamp
            + ", hands: " + frame.Hands.Count
            + ", fingers: " + frame.Fingers.Count
            + ", tools: " + frame.Tools.Count);

The method goes on to examine the first hand in the list:

if (!frame.Hands.IsEmpty)
{
    // Get the first hand
    Hand hand = frame.Hands[0];

A Hand object contains an ID, properties representing the hand’s physical characteristics, and a list of Finger objects. Each Finger object contains an ID and properties representing the characteristic of the finger.

Once it has retrieved a hand, the method checks it for fingers and then averages the finger tip positions, printing the result and the number of fingers:

// Check if the hand has any fingers
FingerList fingers = hand.Fingers;
if (!fingers.IsEmpty)
{
    // Calculate the hand's average finger tip position
    Vector avgPos = Vector.Zero;
    foreach (Finger finger in fingers)
    {
        avgPos += finger.TipPosition;
    }
    avgPos /= fingers.Count;
    SafeWriteLine("Hand has " + fingers.Count
                + " fingers, average finger tip position: " + avgPos);
}

Next, the method prints the radius of a sphere fit to the hand’s curvature, along with the hand’s palm position:

// Get the hand's sphere radius and palm position
SafeWriteLine("Hand sphere radius: " + hand.SphereRadius.ToString("n2")
            + " mm, palm position: " + hand.PalmPosition);

Finally, the OnFrame method uses Vector methods to calculate the hand’s pitch, roll, and yaw angles from the hand’s normal vector and the direction vector. The angles are converted from radians to degrees:

// Get the hand's normal vector and direction
Vector normal = hand.PalmNormal;
Vector direction = hand.Direction;

// Calculate the hand's pitch, roll, and yaw angles
SafeWriteLine("Hand pitch: " + direction.Pitch * 180.0f / (float)Math.PI  + " degrees, "
            + "roll: " + normal.Roll * 180.0f / (float)Math.PI + " degrees, "
            + "yaw: " + direction.Yaw * 180.0f / (float)Math.PI + " degrees\n");

Getting Gestures

To receive gestures from the Leap, you first have to enable recognition for each type of gesture you are interested in. You can enable gesture recognition any time after the controller connects to the Leap (IsConnected is true). In the sample program, all gestures are enabled in the OnConnect() callback function using the EnableGesture() methods defined in the Controller class:

public override void OnConnect (Controller controller)
{
    SafeWriteLine ("Connected");
    controller.EnableGesture (Gesture.GestureType.TYPECIRCLE);
    controller.EnableGesture (Gesture.GestureType.TYPEKEYTAP);
    controller.EnableGesture (Gesture.GestureType.TYPESCREENTAP);
    controller.EnableGesture (Gesture.GestureType.TYPESWIPE);
}

The Leap adds Gesture objects representing each recognized movement pattern to the gestures list in the Frame object. In the OnFrame() callback, the sample application loops through the gesture list and prints information about each one to the standard output. This operation is performed with a standard for-loop and switch statement.

The gesture API uses a base Gesture class which is extended by classes representing the individual gestures. The objects in the gesture list are Gesture instances, so you must convert the Gesture instance to an instance of the correct subclass. Casting is not supported, instead each subclass provides a constructor that performs the conversion. For example, a Gesture instance representing a circle gesture can by converted into a CircleGesture instance with the following code:

CircleGesture circle = new CircleGesture (gesture)

If you try to convert a Gesture instance into the wrong subclass, the constructor functions return invalid Gesture objects.

It is sometimes useful to compare the properties of a gesture in the current frame to those from an earlier frame. For example, the circle gesture has a progress attribute that describes how many times the finger has traversed the circle. This is the total progress, however; if you want the progress between frames, you must subtract the progress value of the gesture in the previous frame. You can do this by looking up the gesture in the previous frame using the gesture ID. The following code calculates the progress since the previous frame to derive the angle in degrees:

// Calculate angle swept since last frame
float sweptAngle = 0;
if (circle.State != Gesture.GestureState.STATESTART) {
    CircleGesture previousUpdate = new CircleGesture (controller.Frame (1).Gesture (circle.Id));
    sweptAngle = (circle.Progress - previousUpdate.Progress) * 360;
}

The full code for the gesture loop is:

// Get gestures
GestureList gestures = frame.Gestures ();
for (int i = 0; i < gestures.Count; i++) {
    Gesture gesture = gestures [i];

    switch (gesture.Type) {
    case Gesture.GestureType.TYPECIRCLE:
        CircleGesture circle = new CircleGesture (gesture);

            // Calculate clock direction using the angle between circle normal and pointable
        String clockwiseness;
        if (circle.Pointable.Direction.AngleTo (circle.Normal) <= Math.PI / 4) {
            //Clockwise if angle is less than 90 degrees
            clockwiseness = "clockwise";
        } else {
            clockwiseness = "counterclockwise";
        }

        float sweptAngle = 0;

            // Calculate angle swept since last frame
        if (circle.State != Gesture.GestureState.STATESTART) {
            CircleGesture previousUpdate = new CircleGesture (controller.Frame (1).Gesture (circle.Id));
            sweptAngle = (circle.Progress - previousUpdate.Progress) * 360;
        }

        SafeWriteLine ("Circle id: " + circle.Id
                       + ", " + circle.State
                       + ", progress: " + circle.Progress
                       + ", radius: " + circle.Radius
                       + ", angle: " + sweptAngle
                       + ", " + clockwiseness);
        break;
    case Gesture.GestureType.TYPESWIPE:
        SwipeGesture swipe = new SwipeGesture (gesture);
        SafeWriteLine ("Swipe id: " + swipe.Id
                       + ", " + swipe.State
                       + ", position: " + swipe.Position
                       + ", direction: " + swipe.Direction
                       + ", speed: " + swipe.Speed);
        break;
    case Gesture.GestureType.TYPEKEYTAP:
        KeyTapGesture keytap = new KeyTapGesture (gesture);
        SafeWriteLine ("Tap id: " + keytap.Id
                       + ", " + keytap.State
                       + ", position: " + keytap.Position
                       + ", direction: " + keytap.Direction);
        break;
    case Gesture.GestureType.TYPESCREENTAP:
        ScreenTapGesture screentap = new ScreenTapGesture (gesture);
        SafeWriteLine ("Tap id: " + screentap.Id
                       + ", " + screentap.State
                       + ", position: " + screentap.Position
                       + ", direction: " + screentap.Direction);
        break;
    default:
        SafeWriteLine ("Unknown gesture type.");
        break;
    }
}

Running the sample

To run the sample application:

  1. Compile the sample application:
  • On Windows, make sure that Sample.cs and either LeapCSharp.NET3.5.dll or LeapCSharp.NET4.0.dll are in the current directory. Run the following command in a command-line prompt (using the proper library reference for the .NET framework you are using):

    csc /reference:LeapCSharp.NET4.0.dll /platform:x86 /target:exe Sample.cs

    Note: use the ``csc`` compiler from the appropriate version of the .NET framework.

  • On Mac, you can use the Mono project to compile C# programs. Make sure that Sample.cs and either LeapCSharp.NET3.5.dll or LeapCSharp.NET4.0.dll are in the current directory:

  • For the .NET 3.5 framework, run:

    gmcs /reference:LeapCSharp.NET3.5.dll /platform:x86 /target:exe Sample.cs

  • For the .NET 4.0 framework, run:

    dmcs /reference:LeapCSharp.NET4.0.dll /platform:x86 /target:exe Sample.cs

  1. Plug the Leap device into a USB port and place it in front of you.

  2. If you haven’t already, install the Leap software.

  3. The Leap Motion software should start automatically.

    The Leap Motion control panel icon appears in the notification area of the task bar (on Windows) or finder bar (on Mac) and turns green when ready. A service or daemon runs in the background and provides data to client applications through the Leap Motion API. You can use the diagnostic visualizer to check whether the software is set up and running correctly.

  4. Run the sample application:

  • On Windows, make sure that Sample.exe, LeapCSharp.dll, Leap.dll and either LeapCSharp.NET3.5.dll, or LeapCSharp.NET4.0.dll are in the current directory. Use the libraries in the lib\x86 directory for 32-bit projects. Use the libraries in the lib\x64 directory for 64-bit projects. Run the following command in a command-line prompt:

    Sample.exe

  • On Mac, make sure that Sample.exe, libLeapCSharp.dylib, libLeap.dylib and either LeapCSharp.NET3.5.dll, or LeapCSharp.NET4.0.dll are in the current directory and run the following command in a terminal window:

    mono Sample.exe

You should see the messages “Initialized” and “Connected” printed to standard output when the application initializes and connects to the Leap. You should then see frame information printed each time the Leap dispatches the onFrame event. When you place a hand above the Leap, you should also see finger and palm position information printed.

Now that you have seen how to access motion tracking data with the Leap Motion API, you can begin developing your own C# applications that integrate the Leap Motion controller.