The Hand Tracking and Face Tracking extensions use your computer's camera to follow the player's hand or face. Your game can then react to where the hand or face is, and to what it does, like pinching, closing a fist or opening the mouth. These extensions are not available by default. To learn how to add them to your game, see Game Builder - Extensions.
The tracking runs in the browser. The camera images are not recorded, collected or stored.
Hand Tracking
Hand tracking finds 21 points on the player's hand, including the fingertips, the knuckles and the wrist. The outline of the hand is drawn over the game, and it changes color when the player pinches or closes their hand into a fist. Only one hand is tracked at a time.
The Hand Tracking tab
Camera Opacity: how much of the camera image shows behind the game, from 0 (hidden) to 100.
Hand Color: the color of the hand outline.
Pinch Color: the color of the outline while the player is pinching.
Fist Color: the color of the outline while the player's hand is closed into a fist.
Hand Tracking functions
Function | Description |
fingerX('index') | Returns the x position of a point on the player's hand, in game coordinates. The point can be thumb, index, middle, ring, pinky or palm. |
fingerY('index') | Returns the y position of a point on the player's hand, in game coordinates. The point can be thumb, index, middle, ring, pinky or palm. |
isPinching() | Returns true while the player is pinching (the thumb and the index finger are touching), and false otherwise. |
onPinch (position) => | Runs the code inside it once, each time the player pinches (brings the thumb and the index finger together). |
onPinchRelease (position) => | Runs the code inside it once, each time the player opens a pinch. |
isFistClosed() | Returns true while the player's hand is closed into a fist, and false otherwise. |
onFist (position) => | Runs the code inside it once, each time the player closes their hand into a fist. |
onFistRelease (position) => | Runs the code inside it once, each time the player opens their fist. |
@enablePinchToDrag() | Lets the player pick up the sprite by pinching over it. The sprite follows the hand until the player opens the pinch. |
@disablePinchToDrag() | Stops the player from picking up the sprite with a pinch. If the sprite is being held, it is dropped. |
@enableFistToDrag() | Lets the player pick up the sprite by closing their hand into a fist over it. The sprite follows the hand until the player opens the fist. |
@disableFistToDrag() | Stops the player from picking up the sprite with a fist. If the sprite is being held, it is dropped. |
The position that onPinch, onPinchRelease, onFist and onFistRelease get is where the pinch or the fist happened. It has an x and a y, and you can pass it to getDistanceFrom to check how close it is to a sprite.
Example
To let the player grab a hammer with their fist, write this in the hammer's code. When the player closes their hand over the hammer, they can move it around the game, and when they open their hand, the hammer is dropped.
@enableFistToDrag()
In this example, the sprite hides when the player pinches close to it.
onPinch (position) =>
if @getDistanceFrom(position) < 50
@hide()
Face Tracking
Face tracking finds the player's face and the position of its parts, like the nose, the eyes, the mouth and the top of the head. Dots are drawn on the face, and the mouth dots change color when the player opens their mouth. The dots are useful for checking that your code works. You can hide them when your game is ready.
The Face Tracking tab
Camera Opacity: how much of the camera image shows behind the game, from 0 (hidden) to 100.
Dots Opacity: how visible the face dots and outline are.
Face Color: the color of the face dots and outline.
Mouth Open Color: the color of the mouth dots while the mouth is open.
Show Face Dots: shows or hides the dots.
Show Face Outline: shows or hides the outline of the face and eyes.
Face Tracking functions
Function | Description |
faceX('nose') | Returns the x position of a point on the player's face, in game coordinates. The point can be nose, leftEye, rightEye, mouth, leftEar, rightEar, forehead, chin, headTop, leftHeadTop or rightHeadTop. |
faceY('nose') | Returns the y position of a point on the player's face, in game coordinates. The point can be nose, leftEye, rightEye, mouth, leftEar, rightEar, forehead, chin, headTop, leftHeadTop or rightHeadTop. |
headWidth() | Returns the width of the player's head, from ear to ear, in game units. Use it to make a sprite grow and shrink as the player moves closer to or farther from the camera. |
headHeight() | Returns the height of the player's head, from the chin to the top of the head, in game units. |
headTilt() | Returns how much the player's head is tilted, in degrees. It is 0 when the eyes are level and grows clockwise, like a sprite's rotation. |
isMouthOpen() | Returns true while the player's mouth is open, and false otherwise. |
onMouthOpen (position) => | Runs the code inside it once, each time the player opens their mouth. The position is where the mouth is. |
onMouthClose (position) => | Runs the code inside it once, each time the player closes their mouth. The position is where the mouth is. |
Example
To put a cat nose on the player's nose, write this in the nose sprite's code. The nose follows the player's nose and tilts with the head.
@onUpdate = () =>
@setX faceX('nose')
@setY faceY('nose')
@setRotation headTilt()
In this example, the sprite shows while the player's mouth is open, and hides when it closes.
onMouthOpen (position) =>
@show()
onMouthClose (position) =>
@hide()
Tips for Good Tracking
Use good light. The camera finds the hand and face more easily in a well lit room.
Keep the hand or face in view. Tracking stops when the hand or face is partly outside the camera, or too close to it.
Don't move too fast. Fast movements are harder to follow.
Keep game objects away from the edges. The camera has trouble seeing a hand near the bottom of the screen.
Relevant Articles: