Sign Language Translator

Communication through sign language is very important for people with hearing and speech impairments. However, many people do not understand sign language, making communication difficult. In this project, we build a Sign Language Translator using Quarky and PictoBlox with Artificial Intelligence (AI).

The AI model is trained to recognize hand gestures such as Like, Dislike, Point Finger (Ask Me a Question), Call Me, and Victory/Peace. When a user shows one of these gestures to the webcam, the system detects it in real time and displays the corresponding message on the screen.

This project is simple, interactive, and a great way to learn about AI, Machine Learning, and Computer Vision while creating a useful accessibility tool.

1. Pictoblox Software: Download PictoBlox | Windows, MacOS, Linux, Chromebook, Android & iOS
2. Laptop/PC
3. Inbuilt or External Camera Setup
4. Speaker or Headphones
5. Quarky

Open PictoBlox and connect Quarky to your computer. Then, add the Hand Pose Classifier extension and turn on the webcam.

The Hand Pose Classifier detects your hand and allows you to create custom gesture classes. Create classes for the gestures you want to recognize, such as:

1. 👍 Like
2. 👎 Dislike
3. 👉 Point Finger (Ask Me a Question)
4. 🤙 Call Me
5. ✌️ Victory/Peace

After setting up the Hand Pose Classifier, you are ready to collect training images for each gesture.

Now it's time to train the AI model to recognize your hand gestures.

For each gesture class (Like, Dislike, Point Finger, Call Me, and Victory/Peace), capture multiple images from different angles and hand positions. Make sure the lighting is good and your hand is clearly visible in the webcam.

After collecting enough samples for all the classes, click Train to create the AI model. Wait for the training process to finish, then test the model to check whether it correctly identifies each gesture.

Now, connect Quarky to your computer using a USB cable and make sure it is detected in PictoBlox.

1. Connect Quarky to your computer.
2. Open PictoBlox and select the Quarky board.
3. Click Connect and choose the correct COM port.
4. Wait until the connection is successful.
5. Run the program to allow Quarky to work with the Hand Pose Classifier.

Once Quarky is connected, it can respond to the detected hand gestures and execute the programmed actions in real time.

After training the Hand Pose Classifier, create the following block program in PictoBlox.

1. Start the Program

1. Use the When Green Flag Clicked block to start the project.
2. Turn on the webcam and set the video transparency to 0% so the camera feed is clearly visible.
3. Display a welcome message such as "Hi! Welcome Lakshith" for 2 seconds.
4. Make Quarky speak "Hi! Welcome Lakshmi".
5. Show the neutral face emotion on the LED matrix.

2. Continuously Detect Hand Gestures

1. Add a Forever loop.
2. Inside the loop, use the Analyze Image from Webcam block so the AI continuously checks the hand gesture.

3. Detect the "Good Job 👍" Gesture

1. Check if the identified class is Good Job.
2. Display "Yeah! Good Job!" for 2 seconds.
3. Make Quarky speak "Yeah! That's great! Good Job."
4. Play the Good Detected sound.
5. Show the Party Colors LED pattern.

4. Detect the "Dislike 👎" Gesture

1. Check if the identified class is Dislike.
2. Display "Oh! Sorry" for 2 seconds.
3. Make Quarky speak "Oh! Sorry."
4. Play the Woo Detected sound.
5. Show the Happy LED emotion.

5. Detect the "Question 👉" Gesture

1. Check if the identified class is Question.
2. Display "Please ask my any question!" for 2 seconds.
3. Make Quarky speak "Please ask my any question."
4. Play the Bravo Detected sound.
5. Show the Basic LED emotion.

6. Detect the "Peace ✌️" Gesture

1. Check if the identified class is Peace.
2. Display "Cool! Peace" for 2 seconds.
3. Make Quarky speak "Cool! Peace."
4. Play the Found sound.
5. Show the Celebrations LED pattern.

7. Detect the "Call Me 🤙" Gesture

1. Check if the identified class is Call Me.
2. Display "Please call me" for 2 seconds.
3. Make Quarky speak "Please call me."
4. Play the Alive Detected sound.
5. Show the Fear LED animation.

8. Repeat the Process

1. Add a 1-second wait block.
2. The Forever loop repeats continuously, allowing the AI to recognize hand gestures in real time and respond with text, speech, sounds, and Quarky's LED expressions.

Now it's time to test your project.

1. Click the Green Flag to start the program.
2. Stand in front of the webcam and make sure your hand is clearly visible.
3. Show each trained gesture one by one:
4. 👍 Good Job
5. 👎 Dislike
6. 👉 Question
7. ✌️ Peace
8. 🤙 Call Me
9. Observe the output on the screen.
10. Verify that Quarky:
11. Detects the correct hand gesture.
12. Displays the correct text message.
13. Speaks the corresponding message.
14. Plays the appropriate sound.
15. Shows the correct LED animation or emotion.

If any gesture is not recognized correctly, collect more training images and retrain the Hand Pose Classifier to improve its accuracy.

Communication assistance

Smart classrooms

AI learning projects

Accessibility tools

Robotics demonstrations

STEM education

Congratulations! You have successfully built a Sign Language Translator using Quarky and PictoBlox. The system uses the Hand Pose Classifier to recognize hand gestures in real time and translates them into meaningful text. Quarky also responds with speech, sounds, and LED animations, making the project interactive and engaging.

This project demonstrates how Artificial Intelligence (AI) and Computer Vision can be used to improve communication and create accessible technology for everyone. It is an excellent beginner-friendly project for learning AI, Machine Learning, and robotics while solving a real-world problem.