The AI extensions are designed to create predictive AI models that use your computer's webcam to make a prediction based on the captured image. This prediction can then be used to execute different code in your game.
Adding the Extensions to Game Builder
Both of these options are not available by default, but can be added by clicking the + sign on the game builders tabs.
When you click the + button, you will have the option to add either AI - Image or AI - Pose. Only one can be added to your game and if you have one enabled and go to add the other one, it will replace the existing extension.
AI - Image versus AI - Pose
AI Image and AI pose work in a similar manner. Both use the webcam to create image sets and make predictions, however the difference is that the AI - Image analyzes a general image while the AI - Pose looks specifically for a person and analyzes their body posture.
If you are trying to create an option that will look for a person in a particular pose, such as having their hands up versus down, then the AI - Pose would be the better option, otherwise you should choose the AI - Image. Either way, the process below will be the same.
Creating a Model
Before using the AI predictions, the first step is to create a model. This is done by clicking the Add New Model button in the AI - Image/AI - Pose tab.
Creating a model involves several steps.
Name Your Model
Before getting started, the first thing you should do is name your model. The name should be descriptive and unique, not just for this game, but all of your games as the same model can be used in any of your games.
Creating Classes
By default, the model starts with two classes. For each possible prediction that you want to make, you need a separate class. For example, if you wanted a model to predict hands up versus hands down, you would rename the two existing classes to represent those two options. If you wanted a model to predict a cat, dog, or duck, you would use the two existing classes and then click the button to add another class.
Adding Image Samples
After you have created the classes for you model, you will add sample images to each class. The sample images will be used to train the model. You must add sample images to each class before you will be able to train the model.
To add images, click the record button. You may need to grant permission to your computer's webcam. Once enabled, you will see a preview window from the webcam.
You can either click and hold the yellow button or enable the timer to right. Without the timer, the model will capture sample images while the button is being held down and start as soon as the button is pushed.
If the timer is enabled, clicking the button will start a 5 second count down. After that, the images will automatically be captured for approximately 5 seconds. While recording, the timer button will appear red, and then it will turn back to green once completed.
Training the Model
Once all classes have sample images, you can train the model. Click the Train Model button and you can watch the progress. In general, larger your sample set is, the longer the model will take to train.
Previewing and Saving
After the model is trained, you will have an opportunity to preview the model. By either holding up an object, or striking a pose, you should see the prediction in the window.
If you are not satisfied with the accuracy of the prediction, you can either add additional images or remove all samples and start over for either a particular class or the entire model.
Once you are satisfied that the model is predicting to a high level, click the save button.
Using the Model
In order to use the model in your game, you will need to add a webcam widget to the game. Find this under the Widget tab. The webcam can be placed anywhere in the game and does not have to be visible in order to work. If you do not want the webcam on the screen, you can hide it by clicking off the show button.
Once the model is created, you can use it by selecting the model in the AI - Image/ AI - Pose tab. When selected, you will see a green check mark next to the model.
There are two AI commands available.
Command | Description |
Event statement used to execute code based on the class name being predicted. | |
Used to return the value of the prediction as a decimal between 0 and 1. |
onPrediction className
The onPrediction function (event handler) is called each time the specified class is predicted from the model. The event triggers once the probability of the class is greater than 95%.
This function is triggered by an event and does not need to be called explicitly.
Arguments
Name | Type | Description |
className | string | The name of the model class that you are creating an event trigger for. |
Syntax
onPrediction 'className', () =>
Example
In the example below, if this code was placed in a monkey sprite's code, when the camera detects a banana, the monkey would step 50.
onPrediction 'banana', () =>
@step 50
predict className
Returns the model prediction score as a number between 0 and 1. A 1 would represent that the model is 100% certain that it is predicting that object or pose.
Returns
Number - the model's predicted score.
Arguments
Name | Type | Description |
className | string | The name of the model class that you are creating an event trigger for. |
Syntax
predict('className')
Example
This example will continuously display the model prediction score for the Standing class.
@onUpdate = () =>
text.text = predict('Standing')
Tips To Improve Models
Your model will only be as good as the images that you use to train your model on. Getting the right image set is key. Here are a few tips to help get useful images.
Move the object/person around in the camera. You want to have some images with the object centered, but also some with the object to one side and the other. You should also vary the size of the object in the screen by moving it forward and backward relative to the camera.
Provide some variations. If you have an object that you are using, make sure you rotate the object. If you are doing a pose, make sure you have some slight variations. For example, if you are holding your arms up, have a few variations where your elbows might be slightly bent, or your arms are tilted to one side.
Find a neutral background. If possible, use a background that provides contrast to the object or person and doesn't have much else in the image other than what you are trying to capture.
Use the timer, especially for poses. It is important that your images be only the poses or objects that you want. By using the timer, it allows you time to get into the correct pose before the images start to record. Make sure you also hold the pose until the timer button turns back to green.
Make sure you have enough samples. You should cover all the variations and options discussed above and do so with a decent number of samples. When using the timer, it is good to use at least 2 timer cycles. When holding to record, you should aim for at least 200 sample images.
Relevant Articles: