No Controller Needed Ski With Your Body Using AI Pose Recognition Sensor

Last time I built an AI penalty shootout goalkeeper game that used gesture recognition, and I learned that the AI Vision Posture and Gesture Sensor can also learn custom poses. That made me wonder: if hand gestures can control a game, why not use the whole body? So I set out to build a motion-sensing skiing mini-game where leaning left, leaning right, flipping, and accelerating are all done with real body movements. No keyboard, mouse, or game controller is involved. In this project I will show you how I taught the Sensor six distinct poses, mapped each pose ID to a game command, and connected everything to an HTML skiing game. By the end, you will be able to stand in front of the Sensor and control a virtual skier with your own body.

1 x SEN0670 AI Vision Posture and Gesture Sensor

1 x USB cable

1 x Computer with the official learning tool installed

1 x Pose skiing game HTML file

Before you start, learn which body pose triggers which game action. This game uses six human poses: Lean Left makes the character glide left, Lean Right makes it glide right, Front Flip triggers a 360° front flip, Left-Side Front Flip triggers a 360° left-side front flip, Right-Side Front Flip triggers a 360° right-side front flip, and Accelerate enters the acceleration state. Once you can hold each pose clearly, you are ready to teach them to the Sensor.

Download the official software that comes with the SEN0670 AI Vision Posture and Gesture Sensor. It is Attachment 1 at the end of this article. After extracting the archive, double-click the .exe file to launch the software. Connect the computer to the SEN0670 with a USB cable, and the software will automatically detect the serial port for the Sensor. Select the correct serial port and open it.

Select pose estimation mode in the software. Stand about 2 meters away from the Sensor and strike a custom pose. I will use the lean left pose as an example. After you strike the pose, you can see the human key points and the human detection box in the left-side view. Before learning is complete, the human key points are shown in green.

After striking the lean left pose, click the learning button and hold the pose still for about 3 seconds. When the color of the body joint points changes from green to another color, learning has succeeded. You can then customize the gesture name and call it "left". The real-time data display below shows the coordinates of each joint point.

Using the same method, continue learning right, power, left360, right360, and fast, until all six game control poses are learned.

Below is the specific pose learning process. You can also refer to the official pose learning tutorial: https://wiki.dfrobot.com/sen0670/docs/23979

After pose learning, the next step is to let the web game receive the pose estimation results returned by the SEN0670. The complete pose skiing game program is Attachment 2 at the end of this article. Open the HTML file to view the full source code.

The program is divided into four parts: Access Layer, Perception Layer, Decision Layer, and Presentation Layer. In the Access Layer, the program establishes communication with the SEN0670 and receives the data returned by the Sensor. After the Sensor recognizes a human pose, it returns the corresponding pose ID. Once the program obtains this ID, it can move on to decision-making.

In the Perception Layer, the program needs to turn the raw pose ID into something the game can use. For example, assume the pose ID currently recognized by the Sensor is:

Here, detectedPoseId represents the human pose currently recognized by the Sensor. Directly using numeric IDs is not intuitive, so I created a pose ID to game action name mapping table.

Through this mapping table, the numeric IDs returned by the Sensor become intuitive game action names. For example: ID 1 → left → glide left; ID 2 → right → glide right; ID 6 → fast → accelerate. This step is the bridge between AI perception results and game control logic.

In the Decision Layer, the program decides what operation the player wants to perform. For example, left and right gliding can be implemented with the following code:

Here, the program first checks whether a valid steering pose has been detected, and it also requires the recognition confidence conf to reach a certain threshold.

When the recognition result is left, the character moves left.

When the recognition result is right, the character moves right. In this way, the player's left and right body movements become left and right movements of the game character.

Different human poses correspond to different game behaviors. Ultimately, the entire game forms a complete data chain, which is the core implementation concept of the whole motion-sensing skiing game.

The core program first reads the pose ID returned by the SEN0670, then uses the pre-established pose mapping table to convert different poses into game control commands. IDs 1 and 2 control the character gliding left and right, IDs 3–5 correspond to different flip actions, and ID 6 triggers acceleration. This achieves a direct mapping from self-learned human poses to virtual character actions.

When the player strikes the corresponding pose and the Sensor reads it, the program enters the matching game control logic. Here, left and right change the skiing character's movement direction, fast controls the acceleration state, and power, left360, and right360 enter the trick action decision logic. This completes the conversion from pose estimation results to actual character actions.

After selecting the game mode you want to challenge, stand within the recognition range of the Sensor. In actual use, a distance of about 2 meters from the Sensor is appropriate. Once the game starts, strike the corresponding pose according to the prompts on the screen. The Sensor detects your body movements in real time and returns the currently recognized pose ID. The program converts different poses into game control commands.

For example: body leaning left is recognized as left, so the game character glides left; body leaning right is recognized as right, so the character glides right; striking a front flip pose is recognized as power, which triggers a front flip. This creates the complete interaction chain from human body movement to AI pose estimation to game command to virtual character action.

Below is the entire gameplay experience. Download the attachment at the end of the article and follow the steps in the video to quickly try the skiing level challenge based on pose estimation.

Full Project Code

The complete pose skiing game program is provided as Attachment 2 at the end of the article. Open the HTML file to view the full source code. The key code sections are shown below.

Final

The game is ready to play once the Sensor is connected and the HTML page is open. Stand about 2 meters in front of the Sensor, choose a mode, and follow the on-screen pose prompts. The Sensor detects your body movements in real time and returns the recognized pose ID, which the program converts into game commands. You can watch the full gameplay experience in the video below.

This project taught me how naturally pose estimation fits motion-sensing interaction. The SEN0670 AI Vision Posture and Gesture Sensor can recognize different human poses and build its own action categories through custom learning. I designed six poses based on the actual gameplay and mapped the returned pose IDs to game control commands, adding stability checks, debouncing, and action trigger logic for real-time response. Next, I want to combine more pose data and application scenarios, such as smart device control and robot interaction. One pitfall to watch: if you stand too close or too far from the Sensor, recognition becomes unreliable, so keep roughly 2 meters of distance and hold each pose steady during learning.

Attachments

Attachment 1: SEN0670 AI Vision Posture and Gesture Sensor companion learning tool:

SEN0670AI Visual Gesture and Posture Sensor PC Software.zip

Attachment 2: Pose skiing game HTML file:

Trick Skiing Game.zip

Appendix

GitHub Link: https://github.com/mengyali878-jpg/trick_skiing_game