No Controller Needed: a Hand Gesture-Based Penalty Shootout Game

Have you ever thought about how the most thrilling part of a football game — the penalty shootout — can only be played by pressing arrow keys or pushing a joystick to make a save? In real life, goalkeepers rely on physical reactions and anticipation to stop the ball, so why do games force us to memorize button combinations?

Once, while playing a football game with a friend, he was taking a penalty and I was in goal. I pressed the wrong direction key, and the ball flew into the net on the opposite side. At that moment, a thought struck me: What if the goalkeeper's movements weren't triggered by buttons, but directly controlled by my own body movements — wouldn't that be far more immersive?

Coincidentally, I had an AI vision Sensor on hand that supports gesture recognition and hand keypoint detection, so I started experimenting with turning "human actions" directly into "game commands." Players don't need to memorize any button combinations or hold a game controller — just open your palm and tilt it in the direction the ball is coming, and the goalkeeper on screen will make the corresponding save. This way, the penalty shootout is no longer about "watching a character move" but about "I see the ball coming, and my body moves to save it."

1 x Gravity: AI Vision Posture and Gesture Sensor (SEN0670)：Core gesture recognition Sensor, supports gesture recognition, pose estimation, and hand keypoint output

1 x USB Cable (Type-C): Connects the Sensor to the computer for power and data transmission

1 x Computer (Windows/macOS): Runs the Sensor companion software and HTML game page

No additional development board needed! The SEN0670 has built-in AI processing capability and can be used directly by connecting to a computer via USB.

The gameplay is very simple — players don't need a keyboard, mouse, or game controller, let alone complex programs or high-computing-power development boards. First, open the Sensor companion software (Attachment 1) on your computer to train the save gesture, then open my HTML webpage (Attachment 2), connect the gesture recognition Sensor, and select the game mode you want to play to experience an immersive penalty-saving session.

Understanding the Mapping Between Save Direction and Gesture

Before entering the game, you can learn the correspondence between palm direction and the goalkeeper's save position: When the palm tilts left, the goalkeeper dives left; when the palm stays upright, the goalkeeper defends the center; when the palm tilts right, the goalkeeper dives right.

After selecting the mode you want to play, simply place your hand in front of the gesture Sensor, open one palm, and you're ready for the penalty-saving challenge. The left side shows the Sensor's real-time feed and hand keypoints, so players can see whether their palm is within the recognition range and what direction their palm is currently judged to be pointing. The bottom-left corner also provides real-time feedback on the corresponding action based on palm direction.

Game Vedio

Below is the complete gameplay experience. You can download the attachments at the end of this article and follow the steps in the video to quickly experience the gesture-based penalty goalkeeper game.

The core of this Project is the Gravity: AI Vision Posture and Gesture Sensor (SKU: SEN0670). The image above shows the key specifications of the sensor.

Official Wiki: https://wiki.dfrobot.com/sen0670/

The SEN0670 connects directly to your computer via a USB cable — no additional wiring needed:

1. Connect the SEN0670 Sensor using a Type-C cable.
2. Plug the other end of the cable into a USB port on your computer.
3. Once the Sensor is powered on, the indicator light will turn on. Wait for the computer to recognize the device.

Gesture recognition is a technology that allows machines to determine what gesture a person is currently making through a camera or Sensor. The system first captures hand images or keypoints, then determines which gesture it belongs to based on motion characteristics. The image above compares common gesture recognition approaches.

From the comparison above, both Sensor-based and computer vision-based gesture recognition approaches have their limitations. I was wondering if there was a Sensor that was relatively affordable and didn't require setting up a vision environment and computing power on the computer — that's when I noticed the SEN0670 AI Vision Gesture and Posture Sensor.

It belongs to the computer vision-based gesture recognition approach: it captures images through a camera and then uses AI vision algorithms to analyze human hand movements. The difference is that it integrates gesture recognition, hand keypoint detection, and other functions directly into the Sensor itself, combining the advantages of both approaches. This means the computer doesn't need to build a vision recognition environment from scratch — on-device AI processing makes it much more convenient to use. Additionally, this Sensor comes with three recognition modes: custom gesture recognition (supports up to 8 types), custom pose estimation (supports up to 8 types), and fixed gesture recognition (13 types). Learners can get started easily, and the gesture recognition process outputs coordinate information for hand joint keypoints.

When designing the game control scheme, my first consideration was: how can I establish a natural mapping between the player's hand movements and the goalkeeper's save direction?

Initially, I considered training multiple independent gestures — for example, using different gestures to correspond to the three save directions: left, center, and right. This approach was relatively straightforward to implement, but after actual testing, I found that players had to deliberately switch between different gestures while tracking the ball's flight direction — the correspondence between actions and actual saves didn't feel natural enough. Real save movements typically involve an open palm, with the main difference being the palm's orientation and direction of movement.

So, I shifted my thinking from "recognizing multiple different gestures" to "recognizing one base gesture, then analyzing the directional changes of that gesture."

After selecting "open palm" as the base gesture, I discovered that even when the palm tilts or swings in different directions, the model still recognizes it as the same "open palm" gesture. This way, I could keep the gesture category unchanged and further analyze the palm orientation changes, mapping them to the three different save directions: left, center, and right.

After diving deeper into the DFRobot SEN0670 AI Vision Gesture and Posture Sensor, I found that it can not only recognize trained gestures but also obtain coordinate information for each hand keypoint during recognition. This gave me an idea: instead of training multiple different gestures, I could use the coordinate relationships between these hand keypoints to calculate the palm's tilt direction, thereby establishing a "palm posture — save direction" mapping.

So in the end, I chose "open palm" as the base control gesture. When the Sensor recognizes an open palm, the program first confirms that the player has entered a valid control state; then it reads the palm keypoint coordinates, determines the palm's tilt direction by analyzing the positional relationships between keypoints, and maps it to one of the three save directions: left, center, or right. With this design, players don't need to learn or switch between different control gestures — they just open their palm like a real goalkeeper and naturally move it in the desired direction to control the game.

1. Download the official companion software for the SEN0670 AI Vision Gesture and Posture Sensor (Attachment 1). Unzip it and double-click the .exe file. Once opened, connect the computer to the Sensor — it will automatically detect the device port. Then open the serial port to see the camera feed.
2. Select gesture recognition mode from the modes. Make the "open palm" gesture at a distance of about 50–60 cm from the Sensor. The left screen will display the current gesture's joint keypoints and bounding box — they appear green before training.
3. After making the "open palm" gesture, click the training button and keep your palm still for about 3 seconds. When the finger joint keypoints change from green to another color, training has succeeded. You can now customize the gesture name — I named it "Plam" as the base gesture for subsequent save recognition. The real-time data display area below will show the coordinates of each joint keypoint.

Here is the specific gesture training process. You can also refer to the official gesture training tutorial: https://wiki.dfrobot.com/sen0670/docs/23979

The complete gesture save game is available in Attachment 2 at the end of this article. You can view the HTML file's source code as shown below.

The entire AI gesture goalkeeper program code can be divided into four parts: Access Layer, Perception Layer, Decision Layer, and Presentation Layer.

In the game, after recognizing the "open palm" gesture, the program further reads the 21 hand keypoints output by the SEN0670 AI Vision Gesture and Posture Sensor. Keypoint 0 is selected as the wrist position, and keypoint 12 is selected as the middle fingertip position.

The core program establishes a 2D direction vector from "wrist → middle fingertip" and calculates the angle of this vector. Based on the angle range, the palm direction is mapped to three save commands: left, center, and right.

The program divides the angle into three ranges: 22°–70° is determined as right, 70°–110° as center, and 110°–158° as left. For example, when the palm is basically upright, the direction vector is close to 90°, and the program outputs "center"; when the palm tilts right, the angle approaches 45°, determined as "right"; when tilting left, the angle approaches 135°, determined as "left."

The complete Project code includes two attachments:

Attachment 1: SEN0670 AI Vision Gesture and Posture Sensor Companion Learning Tool

SEN0670AI Visual Gesture and Posture Sensor PC Software.zip

Attachment 2: Save Game HTML File

AI Gesture Goalkeeper Game.zip

Download Attachment 2, unzip it, and open the HTML file in a browser to run the game. The core logic of the game code is shown in Section 6.5. Refer to the attachment for the complete source code.

Through this project, I gained a more intuitive understanding of gesture recognition. The SEN0670 AI Vision Gesture and Posture Sensor can not only recognize different gestures but also output hand keypoint and posture data. By further analyzing this data, we can convert information like palm direction and movement trajectory into control commands for games.

In this penalty goalkeeper game, I used "open palm" as the base gesture and calculated the palm direction through 21 hand keypoints, achieving control over three save directions: left, center, and right.

This also made me realize that gesture recognition isn't just about "recognizing gestures" itself — it can serve as an interaction method that connects real body movements with the virtual world.

This penalty goalkeeper game is just the beginning. If you want to keep exploring, you could try adding a "reaction speed score" to the saves — calculating points based on the time difference between the ball being shot and the save movement, making the game more challenging. You could also expand the save directions from 3 Extensionss to 5 (adding upper-left and upper-right) for finer control. You could even send the Sensor data to a phone or smart home platform via serial communication to create a home application like "gesture-controlled TV channel switching."

Pitfall Warning: Make sure the lighting is sufficient when training gestures — backlit or backlit environments can cause unstable keypoint recognition. Additionally, the recommended recognition distance for the Sensor is around 0.5m; Being too close or too far will affect the accuracy of angle calculation.

Appendix

GitHub link: https://github.com/mengyali878-jpg/ai-gesture-goalkeeper-game