Control a Robotic Hand With Your Bare Hand — No Glove, Just a Webcam!

Most DIY gesture-controlled hands rely on a glove stuffed with flex sensors. They work, but you have to wear the thing, calibrate every sensor, and deal with wiring across your knuckles.

GRIP (Gesture Responsive Intelligent Prosthetic)-5 does it with nothing but a webcam. You hold your hand up in front of the camera and a robotic hand copies it — not in a clunky "finger is either open or closed" way, but smoothly and proportionally. Curl a finger halfway and the robot finger goes halfway. Open your hand slowly and the robot opens slowly; snap it open and the robot snaps open too.

The trick is entirely in software: Google's MediaPipe finds 21 points on your hand, a bit of geometry turns each finger's bend into a number, and two layers of smoothing make the motion fluid instead of jerky. The hardware is just five servos and an Arduino.

This article focuses on the electronics and the code — the part that makes any robotic hand gesture-responsive. If you already have a printed or purchased servo hand, this is the brain you bolt onto it.

Electronics

1. Arduino Uno R4 WiFi ×1 (the WiFi is what enables the wireless mode later — a plain Uno works for USB-only)
2. Servo motors ×5 (one per finger — SG90 / MG90S class is fine for a light hand)
3. A robotic/prosthetic hand with tendon-driven fingers (3D-printed or purchased)
4. 7.4 V (2S) rechargeable Li-ion/LiPo battery — powers the whole rig untethered
5. Buck (step-down) converter — drops 7.4 V to a regulated 5 V for the servos and board (LM2596-class, sized for your servos' stall current)
6. USB-C charging breakout board — for recharging the pack (must support 2S charging — see the note in Step 2)
7. Jumper wires
8. Breadboard or a small protoboard (power rails carry the regulated 5 V)
9. Multimeter — to set the buck output before connecting anything
10. USB-C cable (for flashing the R4)

Computer side

1. A PC/laptop with a webcam
2. Python 3.9+

Software / libraries

1. MediaPipe
2. OpenCV (opencv-python)
3. NumPy
4. PySerial
5. MediaPipe's hand_landmarker.task model file (free download)

Here's the whole pipeline, from your hand to the robot's:

1. The webcam captures a frame.
2. MediaPipe finds 21 landmarks on your hand in 3D.
3. For each finger, we measure the angle at the middle knuckle — straight ≈ 180°, fully curled is much smaller.
4. That angle becomes a number from 0 (closed) to 9 (open).
5. The five numbers form a string like "90743" (index → thumb).
6. The string is sent to the Arduino over USB or WiFi (through UDP packets).
7. The Arduino eases each servo toward its target, so motion is smooth.

The two ideas that make it feel natural — instead of like a robot snapping between poses — are explained in Steps 6 and 7. If you just want it working, keep going; if you want to understand why it's smooth, those are the good bits.

This build runs entirely off a 7.4 V battery (the diagram's 9V battery is just for representative purposes), so it's portable. The battery is stepped down to a clean 5 V by a buck converter, that 5 V feeds a pair of breadboard power rails, and everything — the five servos and the Arduino — draws from those rails. One regulated rail, one shared ground.

⚠️ Do this FIRST: Set the Buck Voltage

Before connecting the buck to anything downstream, power it from the battery and turn its trim-pot until the output reads 5.0–6.0 V on a multimeter. The Arduino's 5V pin is being back-fed directly here — it bypasses the board's onboard regulator and its protection — so a buck left at 9 V or 12 V (where many ship from the factory) will instantly kill the board. Measure, confirm, then wire it in.

Wiring, step by step

1. Battery → buck input. Battery + to the buck IN+, battery − to IN−. (Don't connect the output yet.)
2. Set the output to 6 V with a multimeter, as above. Disconnect the battery once set.
3. Buck output → breadboard rails. Buck OUT+ to the red (+) rail, OUT− to the blue (−/GND) rail.
4. Servos → rails. Every servo's red wire to the + rail, every brown/black wire to the GND rail.
5. Arduino power. Arduino 5V pin to the + rail, an Arduino GND pin to the GND rail. This shares the regulated 5 V and — critically — ties the Arduino's ground to everything else.
6. Servo signal wires → Arduino pins per the information below. Signal is the third servo wire (usually orange/yellow).

Why the common ground matters

The servo signal from the Arduino is measured relative to ground. If the Arduino and the servos don't share the same ground, that signal has no stable reference — the servos jitter, drift, or don't respond at all. Steps 4 and 5 both land on the GND rail for exactly this reason. It's the single most common wiring mistake.

A few things that'll save you grief

1. Add a big capacitor across the 5 V rail. Five servos moving at once cause current spikes that make the rail dip; the Arduino shares that dip and can brown out into a reset (the same random-reset symptom you'd otherwise blame on code). A 470–1000 µF electrolytic across +/GND near the servos smooths it out.
2. Size the buck for stall current. Servos pull far more when stalled than when free. An LM2596-class buck handles ~2 A continuous before it gets hot, which is marginal for five servos under load — check your buck's rating against your servos.
3. Charging needs 2S support. A 7.4 V (2S) pack cannot be charged from a plain 5 V USB-C breakout — that only passes 5 V. You need a proper 2S charging board (with balance/BMS) in the charge path. Confirm your charging breakout is rated for 2S, or the pack won't charge and can be damaged.
4. One 5 V source while flashing. Plugging USB-C into the R4 to upload adds a second 5 V source next to the buck. Cleanest is to unplug the battery while flashing, then run on battery afterward.

💡 Why not just power the servos from the Arduino's 5V pin? Because then all that servo current flows through the board's traces and regulator, browning it out. Here the servos draw straight from the buck rail and the Arduino merely shares the regulated rail — which is exactly the right way to do it.

The Arduino's job is simple: read a 5-digit string, and move each servo smoothly toward the position that string asks for.

The heart of it is the easing loop — this is what turns discrete commands into smooth motion. Instead of jumping a servo straight to its target, we move it a fraction of the remaining distance every cycle:

The full firmware reads commands, maps each digit 0–9 to a servo angle, and runs this easing loop continuously. Grab mcu.ino from the repo, open it in the Arduino IDE, select your Uno R4 WiFi as the board, and upload.

Once flashed, the hand should boot to a closed fist (all servos at the closed angle), ready for commands.

On your computer, create a project folder, drop in the hand_landmarker.task model, and install the libraries:

The Python code is split into small modules:

1. config.py — all the tunable numbers (thresholds, smoothing, pins-adjacent settings)
2. gesture_logic.py — the geometry (turning landmarks into finger values)
3. transport.py — sending data over USB or WiFi
4. drawing.py — drawing the landmarks on the preview window
5. main.py — ties it all together

Run it wired:

A preview window opens with your hand skeleton drawn on it. Make a fist, open your hand — the robot should follow.

This is the part I'm proudest of, because the obvious approach fails and the fix is elegant.

The naive way to detect a bent finger is to compare the fingertip's height to a knuckle's height in the image. It works — until you point your palm straight at the camera. Then a curling finger moves toward the lens, its 2D position barely changes, and the code thinks the finger is still straight.

The fix: MediaPipe gives you 3D world landmarks — real metric coordinates that don't depend on the camera angle. We compute the actual joint angle at each finger's middle knuckle using three points:

Because it's a true 3D angle, it reads correctly whether your palm faces the camera or is tilted. We then map that angle (e.g. 45°→150°) to a 0.0–1.0 openness value for each finger.

The thumb is special. It doesn't curl like the other fingers — it folds across the palm. After trying a couple of distance-based measures that failed (the thumb read "open" whenever the other fingers were extended), the reliable answer was to use the thumb's own joint angle, just like the fingers.

A robot hand that teleports between poses looks robotic. Two cheap tricks make GRIP-5 look alive:

1. Smooth the camera signal (on the PC). MediaPipe's landmarks jitter slightly frame to frame. An exponential moving average blends each new reading with the previous one, killing the jitter:

2. Ease the servos (on the Arduino). This is the loop from Step 3. Because each step moves a fraction of the remaining distance, the speed automatically scales with how far the target jumped — so if you open your hand fast, the gap is big and the servo rushes; open slowly and it drifts. The robot's speed ends up tracking your hand's speed for free.

Together these give you fluid, rate-responsive motion from what is really just a stream of discrete 0–9 numbers.

Everyone's hand is different, so a quick calibration makes tracking crisp. In config.py, set DEBUG = True to print live finger values, then:

1. Fingers: fully extend, note the angle; make a fist, note the angle. Set FINGER_OPEN_ANGLE / FINGER_CLOSED_ANGLE to bracket your range.
2. Thumb: tuck it, note the value; splay it, note the value. Set THUMB_CLOSED / THUMB_OPEN just inside those two readings.

Do this with your palm facing the camera — the hardest case. Once each finger sweeps the full 0–9 range as you open and close, set DEBUG = False and you're dialed in.

This is where the Uno R4 WiFi earns its place. Instead of USB, the PC sends commands over WiFi as UDP packets:

You set the Arduino's IP address in config.py and it just works — if your network cooperates. Many do not (more on that in the next step). The robust trick: have the Arduino host its own WiFi network. Your laptop connects directly to the hand, no router involved. It works anywhere, which is perfect for a portable demo.

Honestly, this section is half the value of the project. Every one of these cost me time:

1. Fingers snapped instead of moving smoothly. I was sending only 0 or 1 per finger. Switching to 0–9 levels plus easing fixed it. (Steps 5–6)
2. Tracking only worked when I tilted my hand. Classic 2D-projection trap — fixed by using MediaPipe's 3D world landmarks. (Step 5)
3. The thumb was a nightmare. It read "open" during a four-fingers-up gesture. Distance metrics failed; its own joint angle worked. (Step 5)
4. Servos twitched / the Arduino reset randomly. Powering servos from the board. Separate 5V supply + common ground fixed it. (Step 2)
5. No serial output after re-flashing the R4. The R4's native USB can wedge. Double-tap the reset button to enter bootloader mode, then re-upload. Lifesaver.
6. WiFi connected but showed IP 0.0.0.0. The board associated but never got an address — a guest network blocking it. Solution: Arduino-hosted access point. (Step 8)
7. The R4 is 2.4 GHz only — it can't join a 5 GHz network at all.

The finished hand mirrors your gestures in real time, smoothly, at any orientation, wired or wireless. Total electronics cost is modest, and the only "smart" part is software you can read end to end.

Ideas to extend it:

1. Add wrist rotation with a sixth servo
2. Recognize specific gestures (peace sign, thumbs-up) to trigger actions
3. Run the vision on a Raspberry Pi for a fully standalone unit
4. Add force feedback so the hand grips gently

If you build one, I'd love to see it — drop a photo in the comments.

And last but not the least, I would like to extend a special thanks to Prashant Basnet and Maftunabonu Mukumjonova from Sejong University, alongside the MINES Lab. This project simply would not have been possible without their incredible support and contributions. Explore the GitHub repository here!