ALLPCB
Overview
Positioning and input are key to VR technology. Leap Motion uses a recognition method based on computer vision principles. This article summarizes the main concepts from an API perspective.
Sensor structure
Broadly, the Leap sensor reconstructs the 3D motion of the hand in real-world space from images captured by its two onboard cameras from different angles. The detection range is approximately 25 mm to 600 mm above the sensor. The detection volume is roughly an inverted frustum.
Coordinate system
The Leap Motion sensor establishes a right-handed Cartesian coordinate system. The origin is at the center of the sensor. The X axis runs parallel to the sensor and points to the right of the screen. The Y axis points upward. The Z axis points away from the screen. Units are millimeters in real-world measurements.
Frames and tracked objects
During operation, the Leap Motion sensor periodically sends data about hand motion; each such data packet is called a frame. Each frame contains the detected:
- List of all hands and their data
- List of all fingers and their data
- List of tools (thin, straight objects longer than fingers, such as a pen) and their data
- List of all pointable objects, i.e., all fingers and tools and their data
The sensor assigns a unique identifier (ID) to each tracked object. These IDs remain constant while the hand, finger, or tool stays within the sensor's field of view. Using these IDs, an application can query each tracked object via Frame::hand(), Frame::finger(), and similar API calls.
Motion detection between frames
Leap can compute motion information by comparing the current frame with the previous one. For example, if both hands move in the same direction, the system interprets that as a translation; if hands rotate like turning a ball, it is interpreted as rotation; if the hands move closer or farther apart, it is interpreted as scaling. The computed motion data includes:
- Rotation axis vector
- Rotation angle (clockwise positive)
- Rotation matrix
- Scale factor
- Translation vector
Hand-level data
For each detected hand, the following information is available:
- Palm position (3D vector relative to the sensor origin, in millimeters)
- Palm velocity (mm/s)
- Palm normal vector (perpendicular to the palm plane, pointing outward from the palm)
- Palm direction
- Virtual sphere center determined by palm curvature
- Virtual sphere radius determined by palm curvature
In addition, per-hand transforms such as translation, rotation (for example, wrist-induced palm rotation), and scaling (for example, fingers moving apart or together) can be detected. These transforms include the same parameters as the global transforms: rotation axis vector, rotation angle (clockwise positive), rotation matrix, scale factor, and translation vector.
Tools and pointable objects
In addition to fingers, Leap can detect tools: thin, straight objects that are longer than a finger, such as a pen. Fingers and tools are collectively referred to as pointable objects. Each pointable object provides:
- Length
- Width
- Direction
- Tip position
- Tip velocity
Application
Using global frame information, motion transforms, and the detailed data for hands, fingers, and tools, developers can implement interactive input for games and applications.
