Touch Emulation

The Leap Motion API provides information that you can use to implement touch emulation in your application. Touch information is provided by the Pointable class.

Overview

The Leap defines an adaptive touch surface that you can use to orchestrate interaction with 2D elements of your application. This surface is oriented roughly parallel to the x-y plane, but adapts to the user’s finger and hand position. As the user reaches forward with a finger or tool, the Leap reports whether that pointable object is close-to or touching this imaginary surface. The API reports touches with respect to the surface with two values: the zone and the distance to the touch plane.

devguide/../../../images/Leap_Touch_Plane.png

The virtual touch surface

The touch zone identifies whether the Leap Motion software considers a Pointable as hovering near the touch surface, as penetrating the touch surface, or as too far from the surface (or pointing in the wrong direction). The zones are “hovering,” “touching,” and “none.” The transition between zones tends to lag behind the touch distance. This lag is used to prevent abrupt and repeated transitions. If you are implementing touch interaction within an application, you may not need to consider the touch zone very often.

The touch distance is valid only when a Pointable is within the hovering or touching zones. The distance is a normalized value in the range [+1..-1]. When a Pointable first enters the hovering zone, the touch distance is +1.0 and the distance decreases toward 0 as the Pointable nears the touch surface. When the Pointable penetrates the surface, the distance is 0. As the Pointable pushes deeper into the touch zone, the distance approaches, but never exceeds, -1.

You can use the zone value to decide when to update UI elements based on hover or touch. You can use the distance to further modify UI elements based on proximity to the touch plane. For example, you can show the highlight state of a control when a finger is over the control and in the hovering zone and change the cursor based on distance to provide feedback about how close the user is to touching the control.

As part of the touch emulation API the Leap Motion provides a stabilized position for Pointable objects in addition to the standard position. The Leap Motion software stabilizes the position using an adaptive filter that smooths and slows the motion to make it easier for the user to interact with small regions on the screen (like buttons and links). The smoothing is greater when the movement is slow so that the user can zero in and touch a particular point more easily.

Getting the Touch Zone

The touch zone is reported by the touchZone attribute of the Pointable class. The zones are identified using the Zone enumeration, which defines the following states:

  • NONE — the pointable is either too far from the touch surface to be considered touching, or it is pointing back toward the user.
  • HOVERING — the pointable tip has crossed into the hovering zone, but isn’t considered touching.
  • TOUCHING — the pointable has crossed the virtual touch surface.

The following code snippet illustrates how to retrieve the zone of the forward-most finger:

	Frame frame = leap.Frame ();
	Pointable pointable = frame.Pointables.Frontmost;
	Pointable.Zone zone = pointable.TouchZone;

Getting the Touch Distance

The touch distance is reported by the touchDistance attribute of the Pointable class. The distance ranges from +1 to -1 as the finger moves to and through the virtual touch surface. The distance does not represent a physical quantity, but rather how close to touching the Leap Software considers the pointable.

The following code snippet illustrates how to retrieve the touch distance of the forward-most finger:

    Frame frame = leap.Frame();
    Pointable pointable = frame.Pointables.Frontmost;
    float distance = pointable.TouchDistance;

Getting the Stabilized Position of a Pointable

The stabilized position is reported by the stabilizedTipPosition attribute of the Pointable class. This position is reported in reference to the standard Leap Motion coordinate system, but has a context-sensitive amount of filtering and stabilization.

The following code snippet illustrates how to retrieve the stabilized position of the forward-most finger:

    Frame frame = leap.Frame();
    Pointable pointable = frame.Pointables.Frontmost;
    Vector stabilizedPosition = pointable.StabilizedTipPosition;

Converting from Leap Motion Coordinates to Application Coordinates

When implementing touch emulation, you must map the Leap Motion coordinate space to the screen space of your application. To make this mapping easier, the Leap Motion API provides the InteractionBox class. The InteractionBox represents a rectilinear volume within the Leap Motion field of view. The class provides a function that normalizes positions within this volume to coordinates in the range [0..1]. You can normalize a position and then scale the resulting coordinate by the application dimensions to get a point in application coordinates.

For example, if you have a window with a client-area size represented by the variables windowWidth and windowHeight, you can get the 2D pixel coordinates of a touch point within this window using the following code:

    Frame frame = leap.Frame();
    Finger finger = frame.Fingers.Frontmost;
    Vector stabilizedPosition = finger.StabilizedTipPosition;

    InteractionBox iBox = leap.Frame().InteractionBox;
    Vector normalizedPosition = iBox.NormalizePoint(stabilizedPosition);
    float x = normalizedPosition.x * windowWidth;
    float y = windowHeight - normalizedPosition.y * windowHeight;

TouchPoints Example

The following example uses the touch emulation APIs to display the positions of all detected Pointable objects in an application window. The example uses the touch zone to set the color of the points and uses the touch distance to set the alpha value. The stabilized tip positions are mapped to the application window using the InteractionBox class.

devguide/../../../images/Leap_Touch_Point_Example.png

The TouchPoints example

using System;
using System.Collections.Generic;
using System.Windows;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Shapes;
using System.Windows.Ink;

using Leap;

namespace TouchPoints
{
    public partial class MainWindow : Window
    {
        Controller leap = new Controller();
        float windowWidth = 1400;
        float windowHeight = 800;
        DrawingAttributes touchIndicator = new DrawingAttributes();
        public MainWindow()
        {
            InitializeComponent();
            CompositionTarget.Rendering += Update;
            touchIndicator.Width = 20;
            touchIndicator.Height = 20;
            touchIndicator.StylusTip = StylusTip.Ellipse;
        }

        protected void Update(object sender, EventArgs e)
        {
            paintCanvas.Strokes.Clear();
            windowWidth = (float)this.Width;
            windowHeight = (float)this.Height;

            Leap.Frame frame = leap.Frame(); 
            InteractionBox interactionBox = leap.Frame().InteractionBox;

            foreach(Pointable pointable in leap.Frame().Pointables)
            {
                Leap.Vector normalizedPosition = interactionBox.NormalizePoint(pointable.StabilizedTipPosition);
                float tx = normalizedPosition.x * windowWidth;
                float ty = windowHeight - normalizedPosition.y * windowHeight;

                int alpha = 255;
                if(pointable.TouchDistance > 0 && pointable.TouchZone != Pointable.Zone.ZONENONE)
                {
                    alpha = 255 - (int)(255 * pointable.TouchDistance);
                    touchIndicator.Color = Color.FromArgb((byte)alpha, 0x0, 0xff, 0x0);
                }
                else if(pointable.TouchDistance <= 0)
                {
                    alpha = -(int)(255 * pointable.TouchDistance);
                    touchIndicator.Color = Color.FromArgb((byte)alpha, 0xff, 0x0, 0x0);
                }
                else
                {
                    alpha = 50;
                    touchIndicator.Color = Color.FromArgb((byte)alpha, 0x0, 0x0, 0xff);
                }
                StylusPoint touchPoint = new StylusPoint(tx, ty);
                StylusPointCollection tips = new StylusPointCollection(new StylusPoint[] { touchPoint });
                Stroke touchStroke = new Stroke(tips, touchIndicator);
                paintCanvas.Strokes.Add(touchStroke);
            }
        }
    }
}

<Window x:Class="TouchPoints.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" Height="350" Width="525">
    <Grid>
        <InkPresenter Name="paintCanvas"></InkPresenter>
    </Grid>
</Window>

The example uses Microsoft WPF, but the Leap Motion-related code is applicable to all C# projects.