A few months ago I started playing with MediaPipe's hand tracking just to see what it could do. I drew a dot on my index finger, then a line between my thumb and finger, then sparks when they touched. One thing led to another, and now there are over 100 effects in a site called VisionPipe3D.
You open the page, allow the camera, pick an effect, and move your hands. That's the whole thing.
Some of the effects
Most of them started as "wouldn't it be funny if..." ideas. A few I keep coming back to:
- Finger Gun: point at the targets and flick your thumb to shoot. It keeps score, which turned out to be weirdly addictive.
- Air Piano: tap the air and play notes.
- Clap Shockwave: clap and the screen ripples out from your hands.
- Energy Orb: hold your hands apart and a ball of light grows between them.
- Laser Eyes and Googly Eyes: these use face tracking instead of hands.
- Replace Background: put a beach or a nice office behind you, no green screen needed.
There are also a few genuinely useful ones. Presenter Mode is handy for recording tutorials, Countdown Photo takes a picture without touching the keyboard, and Nod / Shake Poll counts yes or no answers from head movement.
You can also record a clip in the browser and download it, which makes it easy to post a short video.
Try it here: visionpipe3d.cappuai.com/playground
How it works
Everything runs in your browser. The video never leaves your computer, and I didn't want to deal with storing people's faces anyway.
The tracking comes from @mediapipe/tasks-vision, which runs as WebAssembly. I use three of its models:
- the gesture recognizer, which returns 21 points per hand plus a guess at the gesture (open palm, thumbs up, and so on)
- the face landmarker, for the face mesh and expression scores like smiling or blinking
- the image segmenter, which gives a mask of where the person is, for background effects
The hand model loads right away. The face and background models only load when you pick an effect that needs them, so the first load stays reasonably quick.
After that, almost every effect follows the same steps. Read the points, turn them into a number, draw something on a canvas over the video. A pinch, for example, is just the distance between two fingertips:
// point 4 is the thumb tip, point 8 is the index fingertip
const thumb = hand[4];
const index = hand[8];
const d = Math.hypot(thumb.x - index.x, thumb.y - index.y);
if (d < 0.05) {
spawnSparks(index.x, index.y);
}
Once you know which point is which, a new effect is usually a short evening of work.
The site itself is Next.js with TypeScript and Tailwind. Clerk handles login, and the docs use Fumadocs.
A few things I'd tell myself at the start
The mirror will confuse you. Webcam previews are flipped, so your right hand shows up on the left. MediaPipe's left/right labels follow the flipped image too. Pick one coordinate system early and convert everything into it once, in one place.
Smooth the points. Raw tracking jitters a little every frame. Moving each point part of the way toward its new position makes everything feel calmer.
Be generous with gestures. My first Finger Gun needed almost perfect aim, and nobody enjoyed it. When I loosened the thresholds and made the targets bigger, people actually started playing it.
Don't load what you don't need. The face and background models are big. Loading them only on demand made a noticeable difference.
Pricing
The playground runs on credits. New accounts get some free ones, and after that you can top up:
- Starter: $1 for 1 credit
- Growth: $5 for 6 credits
- Scale: $20 for 30 credits
There's also an API if you want to build this into your own app. Details are on the pricing page and in the docs.
That's it
If you try it, I'd love to hear which effect you liked, or which one felt broken. And if you have an idea for a new effect, leave it in the comments. A few of the current ones came from friends saying "can it do this?"
Top comments (0)