Throw Hands: a webcam fighting game where your body is the controller

Andrew Yin — Mon, 26 Oct 2026 05:36:44 +0000

This is a submission for the MLH x DEV Writing Challenge

What I Built

Throw Hands is a 3D fighting game that runs in a browser tab. Your webcam tracks your body, and a fighter in a UFC-style octagon copies your arms, torso and head. Throw a real jab and it jabs; raise your forearms and it blocks; step toward the camera and it walks forward. You can fight a CPU, a friend on the same camera, or friends on other laptops and phones by room code. It's live at https://throwhands.tech, with no download, account or controller.

AI shows up in three places. A pose model in the browser (MediaPipe Pose Landmarker) is the controller. Fighter Forge turns a sentence like "lava golem with a crown" into a fighter with a name, backstory and card art, using Meta's Muse Spark and Muse Image. Gemini commentates the fight (details below).

I built it solo at HackTX 2026 in about 24 hours, working with two Claude Code agents.

A punch at the camera is almost invisible

My first jab detector measured arm extension, the distance from shoulder to wrist. It missed punches thrown straight at the lens. One webcam measures depth poorly: on a labeled clip, a full jab only changed the measured arm length from 0.24 m to 0.34 m.

The camera measures angles well. The elbow snaps open from about 75° to 120° in a tenth of a second, whatever the depth. Detecting peaks in elbow angle took recall on the labeled jab clip from 4 of 5 to 6 of 6.

Test at 15 fps, not just 30

The second detector used velocity over a 100 ms window. At 30 fps it worked. At 15 fps, which a phone or a busy laptop delivers, its recall fell to 20%, because a window holding one frame can't give a velocity.

Strikes are now peaks: a local maximum of elbow angle (jab), wrist height (uppercut) or sideways sweep (hook) that rose far enough within 350 ms. That costs one frame of latency and behaves the same at 10, 15 and 30 fps. An evaluation tool replays recorded body points at any frame rate, so every change is scored at both 30 and 15 fps.

Real people as the webcam, in CI

I couldn't stand in front of a camera all night, so the end-to-end tests use downloaded video of real people as the webcam. Chromium's --use-file-for-fake-video-capture plays an MJPEG clip as a camera. For WebKit, the test injects a getUserMedia that paints a clip into a canvas stream.

The best test opens two browsers driven by two pro fighters' shadowboxing footage. They create and join a room through the real UI and play a full best-of-3 while each client logs every state snapshot it receives. Comparing the two logs: 2,160 shared ticks, 0 mismatches in health, round or timer.

The tests found real bugs. Safari never started pose tracking, for three separate reasons:

The camera <video> sat inside a display:none element, and WebKit won't load a stream into it.
MediaPipe needs WebGL inside a Web Worker, which WebKit lacks, so pose detection falls back to the main thread there.
The main thread needs MediaPipe's classic wasm loader, not the ES-module one.

A pipeline for labeled movement data

Shadowboxing clips got the gestures working, but my first session at a laptop broke movement: leaning back walked my fighter forward. A real player at a laptop showed problems no downloaded clip had, so I built a pipeline for collecting labeled movement data: a research mode with guided drills. You can try it at https://throwhands.tech/?research=1&drills=1.

Each drill shows one prompt at a time ("stand still", "big hook", "step back") while I try to copy the move. The game records the 33 body points in every frame and tags them with the prompt and the rep number, so the labels come from the drill script instead of hand annotation. After a consent screen, the body points go to the server under a random session ID; the camera video is saved only on the player's own device. That gave me basically correctly labeled data of me trying to copy each move, which let me iterate fast: record a session, replay it through the same recognizer, fix the rule, and replay again until the reps registered.

Laptops hide the hips. Forward and back used to compare shoulder depth with hip depth, but at a laptop the hips are out of frame. Movement now compares your apparent body size with "your spot", the median size over the last 10 seconds you stood still. Stepping back went from never registering to 3 of 3.
Big punches aren't small punches. Real big hooks open the elbow to 160° and sweep the wrist about 1 m, so a 150° elbow limit called them jabs. Classifying big swings by the whole motion (sweep, rise and torso turn) took hooks from 0 of 4 to 3 of 4 and uppercuts from 0 of 3 to 3 of 3.

One simulation for every mode

One deterministic TypeScript simulation handles health, guard, supers, grappling, cage collisions and KOs. It runs in the browser for local play and on the server for online rooms. Every message carries a fighter slot ID, and bots send the same inputs humans do, so a 1v1 duel, an 8-player Brawl and winner-stays-on are all configurations of one room.

Demo

Claude Code made this video too. The implementer agent recorded the gameplay in a headless browser, rendered the motion graphics, generated the voiceover with ElevenLabs and cut it together with ffmpeg. I reviewed every cut and sent back fixes with timestamps.

Play: https://throwhands.tech. Allow the camera, stand back until your upper body is in frame, and hold your guard up for 3 seconds to calibrate.
Code: https://github.com/AndrewYin04/throwhands
Devpost: https://devpost.com/software/throw-hands

Partner Technologies

Gemini API. Every 3 to 4 seconds, the server sends a batch of match events (hits, guard breaks, supers, cage slams, KOs, health) to a Gemini Flash-Lite model, which returns one roast or hype line about the fight. In online rooms the server writes one line per room, so every player and spectator sees the same call and the free tier lasts longer. Gemini TTS was supposed to voice the lines, but its free quota ran out during the hackathon. Live lines now appear as captions while a matching pre-recorded announcer roast plays. Next time I'd build the caption fallback first and add the voice on top.

Meta Muse API. Fighter Forge sends the player's one-line prompt to Muse Spark (muse-spark-1.3), which returns JSON with a name, backstory, three move names and two colors in about 2 seconds. Muse Image (muse-image-1.0) then paints the fighter card; setting reasoning_strength: "low" cut that from about 17 seconds to about 10. The fighter's suit takes the card's dominant color. After a two-player online match, the server makes one rivalry card per match (a trash-talk line from Muse Spark and a fight poster from Muse Image) and sends it to both players. Three things I learned:

Muse Spark 1.3 sometimes answered model_not_found while 1.2 kept working, so text calls fall back to 1.2.
Meta's API returned an empty model list to the server's IPv6 addresses, so outbound calls are pinned to IPv4.
The server caches every forged fighter and stops calling Meta at an estimated $20 of spend, after which players get premade house fighters. A busy day can't run up the bill.

Tiger Data. Hits, blocks and supers go into a match_events hypertable, and continuous aggregates compute the Elo leaderboard. Players can opt in to share their body-point coordinates (never video) for a motion-data program, and those streams go into a pose_streams hypertable. My first schema stored each frame as an array, and compression only reached 1.7×. Storing one smallint column per coordinate let TimescaleDB's columnar compression work: 19.7 MB became 5.9 MB per player-hour, 3.4×. Drill sessions from research mode go into a third hypertable, research_frames, which is what the replays read from.

Vultr. One Vultr VM in Dallas, the smallest plan (1 vCPU, 1 GB RAM), serves the game over HTTPS through Caddy. It also runs the WebSocket server and the simulation for online rooms, and it makes the Gemini, Meta and Tiger Data calls so no API key reaches the browser. An 8-browser Brawl test ran against it with 0 mismatches between clients.

.Tech domain. "Throw hands" is slang for fighting, so the game lives at throwhands.tech.

Hackathon Experience

HackTX 2026 ran October 24 to 25 at the University of Texas at Austin, with about 695 hackers. I entered solo and split the work between two Claude Code agents:

A planner on my laptop ran ideation, wrote the spec, and handled everything behind a login: Devpost, the domain and YouTube.
An implementer on my home server built, deployed and tested the game, and rendered the demo video.

They talked through files in the public repo: SPEC.md for what to build, REVIEW.md for my feedback, STATUS.md for progress. My job was the part neither agent could do: standing in front of the webcam at 4 AM, throwing hooks that didn't register, and recording drills until they did.

What I'll remember most is how bad AI can be at perceiving videos and animations until you review them yourself. In one cut of the demo video, two voiceovers played over each other at 50 seconds; in a later cut, the Shoulder Throw was still animated wrong at 56 to 58 seconds. Both agents got better once I gave them specific pointers, which I passed through the planner agent so it could build a better, more specific prompt, and once the agent reviewing the video looked at it at a finer grain, frame by frame. The game's own visual tests work the same way: scenes are rendered frame by frame and measured (foot slide, fist-to-target distance) before a fresh AI reviewer scores them. Frame-by-frame review is really slow across an entire video, though, so a human still has to step in periodically to review UI, animation and video changes.

DEV Community: Andrew Yin