Introducing the Skeletal Tracking Model

Version 2.0 of the Leap Motion API introduces a new skeletal tracking model that provides additional information about hands and fingers and also improves overall tracking data.

By modeling a human hand, the Leap Motion software can better predict the positions of fingers and hands that are not clearly in view. Five fingers are always present for a hand and hands can often cross over each other and still be tracked. Of course, the controller still needs to be able to see a finger or hand in order to accurately report its position. Keep this in mind when designing the interactions used by your application. Avoid requiring complex hand “poses” or subtle motions, especially those involving non-extended fingers.

https://di4564baj7skl.cloudfront.net/documentation/images/Visualizer_Hand_Model.png

Explore the behavior of the new hand model in the Diagnostic Visualizer

The API changes are not as dramatic as you might expect and existing API classes still work mostly the same whether or not you use the new features.

The additions include:

  • Reporting of a confidence rating based on the correlation between the internal hand model and the observed data
  • Identification of right or left handedness
  • Identification of digits
  • Reporting of the position and orientation of each finger bone
  • Reporting of grip factors indicating whether a user is pinching or grasping
  • Reporting of five fingers for each hand
  • Reporting whether a finger is extended or not

Perhaps the most significant change for existing applications is the improved persistence of hands and fingers. This should improve the usability of most Leap Motion-enabled apps. However, if you relied on fingers disappearing when curled into the hand or when they touched, then you may have to make some changes. For example, if you count visible fingers or otherwise distinguish between fingers extended outward and those retracted toward the hand, you will have to use the new isExtended() finger properties to know which fingers are extended or not. If, on the other hand, you relied on fingers disappearing when they touched each other, then you will have to establish a minimum distance threshold between the finger tips instead, since the fingers will no longer disappear.

Note: Finger.jointPosition() will be removed (in favor of the Bone class) in an upcoming SDK release.

The new tracking API in Java

The API changes in the new tracking model include:

Hand class:

Hand.confidence(), a value between 0 and 1, rates how well the internal hand model fits the observed data. A low value indicates that there are significant discrepancies; finger positions, even hand identification could be incorrect. The significance of the confidence value to your application can vary with context. Gestures and motions can be valid even without a high confidence in the hand data. On the other hand, you should make sure the confidence value is high before triggering significant application events based on postures or hand properties.

Hand.isLeft() and isRight() are self explanatory. One of these will always be true. However, the Leap Motion controller needs to see enough of the hand to make an accurate classification. If, for example, you put your closed fist into the controller field of view, the software has very little to go on and may come to the wrong conclusion. If the software realizes it has misclassified a hand, the current hand is replaced with a Hand object having a new ID value (and corrected finger identification).

The Hand.grabStrength() and pinchStrength() properties are values between 0 and 1, indicating whether your hand is in a pinching or grabbing posture. A strength value is close to zero for a hand with flat, extended fingers. For a grab, the strength increases toward 1.0 as you curl your fingers into a fist. For a pinch, the strength increases as you curl the thumb and any finger toward each other. You can use these strength properties to trigger grabs or selections in your application when the value exceeds a certain threshold.

Hand.palmWidth() provides an estimation of the width of the palm when the hand is in a flat position. Hand.basis() is a convenience method for getting the orthonormal basis vectors defining the orientation of the hand. The three vectors in the basis matrix are the palm normal, the hand direction, and the cross product between them.

Hand.arm() provides the information about the forearm to which a hand is attached. The properties of the forearm are estimated based on typical human anatomy when the entire forearm is not in view.

PointableList and FingerList classes

The extended functions return the members of the current list that are considered extended. This includes fingers that are more or less pointing straight out from the hand and, in the case of a PointableList object, any tools in the list. The fingerType() function returns all fingers in a list of the specified finger type (i.e all pinkies, all index fingers, etc).

Pointable and Finger classes:

The Finger.type() function identifies the finger name. The values returned are an enumeration, [0..4], representing: thumb, index, middle, ring, and pinky. The Finger.bone() function returns the Bone object representing the specified bone of a finger. Each digit, including the thumb, has four defined bones – including the thumb (even though it only has three anatomical bones).

Intermediate phalanges Proximal phalanges Distal phalanges Metacarpals 0-length thumb metacarpal

Note that Pointable.direction() is parallel to the line running between the base and the tip of the finger. It is not necessarily parallel to the last bone of the finger (the distal phalanx). The more a finger is curved, the greater the difference between the two directions will be. This method of calculating Pointable.direction is consistent with earlier versions of the API, but may look odd if you project a ray from a visual representation of the hand along the Pointable.direction().

New Bone class

The Bone class describes be position and orientation of each of the bones of a finger as well as the metacarpal bones, which connect the fingers to the base of the hand.

New Arm class

The Arm class, similar in structure to the Bone class, provides the physical positions of the end points of the forearm as well as basis vectors specifying how the arm is oriented in space.

JSON data changes

The new tracking model is also reflected in the tracking data supplied via the Leap Motion WebSocket interface. As with the native API, the biggest difference you are likely to notice in existing applications is that hands will always have 5 fingers now.

To access the new API features, use the v6.json endpoint (ws://127.0.0.1:6437/v6.json).

Each hand in the frame hands array has the following new properties:

  • armWidth – the average width of the forearm.
  • confidence – indicates how well the skeleton model and the observed data fit.
  • elbow – the position of the elbow of the arm to which the hand is attached
  • grabStrength – a value between 0 and 1.
  • pinchStrength – a value between 0 and 1.
  • type – a string indicating whether the hand is a left or a right hand.
  • wrist – the position of the wrist

Each finger in the frame pointables array has the following new properties:

  • type – an integer code for the finger name.
  • bases – the basis vectors for each bone, in index order (wrist to tip).
  • btipPosition – position of the extreme end of the distal phalanx as an array of 3 floating point numbers.
  • carpPosition – position of the base of metacarpal bone as an array of 3 floating point numbers.
  • dipPosition – position of the base of the distal phalanx as an array of 3 floating point numbers.
  • pipPosition – a position vector as an array of 3 floating point numbers
  • mcpPosition – a position vector as an array of 3 floating point numbers
  • extended – a boolean indicating whether the finger is pointing or not.