The Hand Tracking and Face Tracking extensions use your computer's camera to follow the player's hand or face. Your game can then react to where the hand or face is, and to what it does, like pinching, closing a fist or opening the mouth. These extensions are not available by default. To learn how to add them to your game, see Game Builder Blocks - Extensions.
The tracking runs in the browser. The camera images are not recorded, collected or stored.
Hand Tracking
Hand tracking finds 21 points on the player's hand, including the fingertips, the knuckles and the wrist. The outline of the hand is drawn over the game, and it changes color when the player pinches or closes their hand into a fist. Only one hand is tracked at a time.
The Hand Tracking tab
Camera Opacity: how much of the camera image shows behind the game, from 0 (hidden) to 100.
Hand Color: the color of the hand outline.
Pinch Color: the color of the outline while the player is pinching.
Fist Color: the color of the outline while the player's hand is closed into a fist.
Hand Tracking blocks
Block | Description |
 x of (hand point) | Returns the x position of a point on the player's hand, in game coordinates. Choose the point from the dropdown: thumb, index finger, middle finger, ring finger, pinky or palm. |
 y of (hand point) | Returns the y position of a point on the player's hand, in game coordinates. Choose the point from the dropdown: thumb, index finger, middle finger, ring finger, pinky or palm. |
 Is pinching | Returns true while the player is pinching (the thumb and the index finger are touching), and false otherwise. |
 On Pinch | Runs the blocks inside it once, each time the player pinches (brings the thumb and the index finger together). |
 On Pinch Release | Runs the blocks inside it once, each time the player opens a pinch. |
 Is fist closed | Returns true while the player's hand is closed into a fist, and false otherwise. |
 On Fist | Runs the blocks inside it once, each time the player closes their hand into a fist. |
 On Fist Release | Runs the blocks inside it once, each time the player opens their fist. |
 Enable pinch to drag | Lets the player pick up the sprite by pinching over it. The sprite follows the hand until the player opens the pinch. |
 Disable pinch to drag | Stops the player from picking up the sprite with a pinch. If the sprite is being held, it is dropped. |
 Enable fist to drag | Lets the player pick up the sprite by closing their hand into a fist over it. The sprite follows the hand until the player opens the fist. |
 Disable fist to drag | Stops the player from picking up the sprite with a fist. If the sprite is being held, it is dropped. |
Example
To let the player grab a hammer with their fist, add Enable fist to drag to the hammer's On Run block. When the player closes their hand over the hammer, they can move it around the game, and when they open their hand, the hammer is dropped.
Face Tracking
Face tracking finds the player's face and the position of its parts, like the nose, the eyes, the mouth and the top of the head. Dots are drawn on the face, and the mouth dots change color when the player opens their mouth. The dots are useful for checking that your code works. You can hide them when your game is ready.
The Face Tracking tab
Camera Opacity: how much of the camera image shows behind the game, from 0 (hidden) to 100.
Dots Opacity: how visible the face dots and outline are.
Face Color: the color of the face dots and outline.
Mouth Open Color: the color of the mouth dots while the mouth is open.
Show Face Dots: shows or hides the dots.
Show Face Outline: shows or hides the outline of the face and eyes.
Face Tracking blocks
Block | Description |
 x of (face point) | Returns the x position of a point on the player's face, in game coordinates. Choose the point from the dropdown: nose, left eye, right eye, mouth, left ear, right ear, forehead, chin, top of head, top left of head or top right of head. |
 y of (face point) | Returns the y position of a point on the player's face, in game coordinates. Choose the point from the dropdown: nose, left eye, right eye, mouth, left ear, right ear, forehead, chin, top of head, top left of head or top right of head. |
 Head width | Returns the width of the player's head, from ear to ear, in game units. Use it to make a sprite grow and shrink as the player moves closer to or farther from the camera. |
 Head height | Returns the height of the player's head, from the chin to the top of the head, in game units. |
 Head tilt | Returns how much the player's head is tilted, in degrees. It is 0 when the eyes are level and grows clockwise, like a sprite's rotation. |
 Is mouth open | Returns true while the player's mouth is open, and false otherwise. |
 On Mouth Open | Runs the blocks inside it once, each time the player opens their mouth. |
 On Mouth Close | Runs the blocks inside it once, each time the player closes their mouth. |
Example
To put a cat nose on the player's nose, add an On Update block to the nose sprite. Inside it, add Set X with x of nose and Set Y with y of nose. To make the nose tilt with the head, add Set Rotation with Head tilt.
Tips for Good Tracking
Use good light. The camera finds the hand and face more easily in a well lit room.
Keep the hand or face in view. Tracking stops when the hand or face is partly outside the camera, or too close to it.
Don't move too fast. Fast movements are harder to follow.
Keep game objects away from the edges. The camera has trouble seeing a hand near the bottom of the screen.
Relevant Articles: