Tracking Hands, Fingers, and Tools

Hands, fingers, and tools are the basic entities tracked by the Leap Motion system. This article discusses the details of gettting and using the objects representing these entities.

Overview

The Leap API defines a class representing each of the basic tracked objects.

The Frame object provides access to lists tracked entities:

  • Hands — All hands.
  • Pointables — All fingers and tools as Pointable objects.
  • Fingers — All the fingers.
  • Tools — All the tools.
  • Gestures — All the gestures that started, ended, or which had an update.

Finger and Tool objects are examples of pointable objects and can be treated together using the PointableList or separately using the FingerList and ToolList classes. The three pointable lists contain every pointable object detected in a frame. You can access those pointables associated with a specific hand through Hand objects in the list of hands. Note that a finger or tool may not be associated with a hand if the user’s hand is only partially within the Leap Motion field of view.

The physical characteristics of hands, fingers, and tools are reported in reference to the Leap coordinate system (measured in units of millimeters). The Leap SDK provides a Vector class to describe points and directions. The Vector class provides several useful math functions for working with vectors.

If you are tracking an individual object, such as a finger, from frame to frame, you can use the ID associated with that object to look it up in each new frame. You can look up Hand, Finger, Tool, Pointable, and Gesture objects by ID. The lookup functions return a reference to the corresponding object if it exists in the current frame. If the object no longer exists, then a special invalid object is returned. Invalid objects are well-defined, but do not contain valid tracking data. This technique helps reduce the amount of null checking you have to do to when accessing Leap Motion tracking data.

Hand, and Pointable Lists

The list classes all have a similar structure. They are designed to act like vector-style arrays and support iterators. You cannot remove or alter the member objects of lists received from the Leap API, but you can combine lists of the same object type.

To use an iterator with one of these lists:

    foreach (Hand hand in handList) {
        Console.WriteLine (hand.ToString ());
    }

The HandList, PointableList, FingerList, and ToolList classes define additional functions for getting members of the list based on their relative position within the Leap coordinate system. These functions include leftmost(), rightmost(), and frontmost(). The following snippet illustrates a few ways to use these functions:

    Finger farLeft = frame.Fingers.Leftmost;
    Finger mostForwardOnHand = frame.Hands [0].Fingers.Frontmost;
    Tool rightTool = frame.Tools.Rightmost;

A more complex example calculates the walls of a bounding box containing all detected Pointable objects. This example defines its own functions for getting the top, bottom, and rear pointables since these are not included in the API:

    float left = frame.Pointables.Leftmost.TipPosition.x;
    float right = frame.Pointables.Rightmost.TipPosition.x;
    float front = frame.Pointables.Frontmost.TipPosition.z;

    float back = backmost(frame.Pointables).TipPosition.z;
    float top = topmost(frame.Pointables).TipPosition.y;
    float bottom = bottommost(frame.Pointables).TipPosition.y;

    Pointable backmost(PointableList pointables)
    {
        if(pointables.Count == 0) return Pointable.Invalid;
        Pointable backmost = pointables[0];
        for( int p = 1; p < pointables.Count; p++ )
        {
            if( pointables[p].TipPosition.z > backmost.TipPosition.z)
                backmost = pointables[p];
        }
        return backmost;
    }

    Pointable topmost(PointableList pointables)
    {
        if(pointables.Count == 0) return Pointable.Invalid;
        Pointable topmost = pointables[0];
        for( int p = 1; p < pointables.Count; p++ )
        {
            if( pointables[p].TipPosition.y > topmost.TipPosition.y)
                topmost = pointables[p];
        }
        return topmost;
    }

    Pointable bottommost(PointableList pointables)
    {
        if(pointables.Count == 0) return Pointable.Invalid;
        Pointable bottommost = pointables[0];
        for( int p = 1; p < pointables.Count; p++ )
        {
            if( pointables[p].TipPosition.y < bottommost.TipPosition.y )
                bottommost = pointables[p];
        }
        return bottommost;
    }

Hands

The Hand class represents a physical hand detected by the Leap. A Hand object provides access to lists of its pointables as well as attributes describing the hand position, orientation, and movement. Get Hand objects from a Frame:

    Frame frame = controller.Frame(); // controller is a Controller object
    HandList hands = frame.Hands;
    Hand firstHand = hands[0];

Or, if you know the ID from a previous frame:

   Hand knownHand = frame.Hand(handID);

You can also get hands by their relative positions in the frame:

    Frame frame = controller.Frame(); // controller is a Controller object
    HandList hands = frame.Hands;

    Hand leftmost = hands.Leftmost;
    Hand rightmost = hands.Rightmost;
    Hand frontmost = hands.Frontmost;

Note that the the leftmost() and rightmost() functions only identify which hand is farthest to the left or right. The functions do not identify which hand is the right or left hand.

Getting the Hand Characteristics

A hand is described by its position, orientation, and motion:

  • Palm Position — The center of the palm measured in millimeters from the Leap Motion origin.
  • Palm Velocity — The speed of the palm in millimeters per second.
  • Palm Normal — A vector perpendicular to the plane formed by the palm of the hand. The vector points downward out of the palm.
  • Direction — A vector pointing from the center of the palm toward the fingers.

The hand’s position is given by its palm position attribute, which provides a vector containing the 3-dimensional coordinates of the palm center point in millimeters from the Leap Motion origin. The hand’s orientation is given by two vectors: the direction, which points from the palm center towards the fingers, and the palm normal, which points out of the palm, perpendicular to the plane of the hand.

The movement of the hand is given by the velocity attribute, which is a vector providing the instantaneous motion of the hand in mm/s. You can also get motion factors that translate how a hand has moved between two given frames into translation, rotation, and scaling values.

The following code snippet illustrates how to get a Hand object from a frame and access its basic attributes:

    Hand hand = frame.Hands.Rightmost;
    Vector position = hand.PalmPosition;
    Vector velocity = hand.PalmVelocity;
    Vector direction = hand.Direction;

Getting the Fingers and Tools

You can get the fingers and tools associated with a hand as a list or individually using an ID obtained in a previous frame.

By list:

    // hand is a Hand object
    PointableList pointables = hand.Pointables; // Both fingers and tools
    FingerList fingers = hand.Fingers;
    ToolList tools = hand.Tools;

By ID from a previous frame:

    Pointable knownPointable = hand.Pointable(pointableID);

To get a finger or tool by relative position within the Leap field of view, use the right-, left- and frontmost functions of the matching list class:

    // hand is a Hand object
    Pointable leftPointable = hand.Pointables.Leftmost;
    Finger rightFinger = hand.Fingers.Rightmost;
    Tool frontTool = hand.Tools.Frontmost;

Note that these functions are relative to the Leap Motion origin, not to the hand itself. To get the fingers relative to the hand, you can use the Leap Matrix class to transform the finger positions into the hands frame of reference.

Computing the Hand Orientation

You can compute the hand orientation angles using the Hand direction and normal vectors.

devguide/../../../images/Leap_Palm_Vectors.png

The normal vector points perpendicularly out of the hand; the direction vector points forward.

The Vector class defines functions for getting the pitch (angle around the x-axis), yaw (angle around the y-axis), and roll (angle around the z-axis):

    float pitch = hand.Direction.Pitch;
    float yaw = hand.Direction.Yaw;
    float roll = hand.PalmNormal.Roll;

Note that the roll function only provides the expected angle when used with a normal vector.

Transforming Finger Coordinates into the Hand’s Frame of Reference

Sometimes it is useful to obtain the coordinates of the fingers of a hand with respect to the hand’s frame of reference. This lets you sort the fingers spatially and can simplify analysis of finger positions. You can create a transform matrix using the Leap Matrix class to transform the finger position and direction coordinates. The hand frame of reference can be usefully defined with the hand’s direction and palm normal vectors, with the third axis defined by the cross product between the two. This puts the x-axis sideways across the hand, the z-axis pointing forward, and the y-axis parallel with the palm normal.

    Frame frame = leap.Frame();
    for( int h = 0; h < frame.Hands.Count; h++ )
    {
        Hand leapHand = frame.Hands[h];

        Vector handXBasis =  leapHand.PalmNormal.Cross(leapHand.Direction).Normalized;
        Vector handYBasis = -leapHand.PalmNormal;
        Vector handZBasis = -leapHand.Direction;
        Vector handOrigin =  leapHand.PalmPosition;
        Matrix handTransform = new Matrix(handXBasis, handYBasis, handZBasis, handOrigin);
        handTransform = handTransform.RigidInverse();

        for( int f = 0; f < leapHand.Fingers.Count; f++ )
        {
            Finger leapFinger = leapHand.Fingers[f];
            Vector transformedPosition = handTransform.TransformPoint(leapFinger.TipPosition);
            Vector transformedDirection = handTransform.TransformDirection(leapFinger.Direction);
            // Do something with the transformed fingers
        }
    }

Pointables

Pointable objects represent fingers and tools — i.e. things that can be pointed. You can get the fingers and tools associated with a particular hand from the Hand object. You can also get all detected pointables from a Frame object. Pointables are not necessarily associated with a Hand object — the physical hand itself may be out of the field of view or blocked by another hand. Thus the list of pointables from the frame can include fingers and tools not found in any list associated with a hand.

devguide/../../../images/Leap_Finger_Model.png

Finger tipPosition and direction vectors provide the positions of the finger tips and the directions in which the fingers are pointing.

The Leap Motion software classifies a detected pointable object as either a finger or a tool. Use the Pointable::isTool() function to determine which one a Pointable object represents.

devguide/../../../images/Leap_Tool.png

A Tool is longer, thinner, and straighter than a finger.

Pointable objects have many attributes describing the characteristics of the represented finger or tool:

  • Tip position — the instantaneous position in mm from the Leap Motion origin.
  • Tip velocity — the instantaneous velocity in mm/s.
  • Stabilized tip position — the position filtered and stabilized using velocity and past positions.
  • Direction — the current pointing direction vector.
  • Length — the apparent length of the finger or tool.
  • Width — the average width.
  • Touch distance — the normalized distance from the virtual touch plane.
  • Touch zone — the pointable’s current relation to the virtual touch plane.

The following example illustrates how to get a pointable object from a frame and access its basic characteristics:

    Pointable pointable = frame.Pointables.Frontmost;
    Vector direction = pointable.Direction;
    float length = pointable.Length;
    float width = pointable.Width;
    Vector stabilizedPosition = pointable.StabilizedTipPosition;
    Vector position = pointable.TipPosition;
    Vector speed = pointable.TipVelocity;
    float touchDistance = pointable.TouchDistance;
    Pointable.Zone zone = pointable.TouchZone;

Converting a Pointable Object to a Finger or Tool

To convert a Pointable object to its proper Finger or Tool subclass, use the appropriate Finger or Tool constructor (one of the few times that you should use the constructor for a Leap class).

    if (pointable.IsTool) {
        Tool tool = new Tool(pointable);
    } else {
        Finger finger = new Finger(pointable);
    }

Calculating the Position of the Base of a Finger

If you need to calculate the position of the base of the finger, you can use the finger tip position and direction as follows:

    Vector basePosition = -pointable.Direction * pointable.Length;
    basePosition += pointable.TipPosition;