AirDeck: I Turned My Webcam Into a DJ Controller and a Game Console

This is a submission for the MLH x DEV Writing Challenge

What I Built

AirDeck turns your webcam into a DJ controller and a game console, right in the browser. You don't hold anything; your hands are the controller.

The idea started with Just Dance: the fun of moving to the beat, but with your hands in front of a webcam instead of a controller. For the look, we went back to the Windows XP "Bliss" wallpaper and the Frutiger Aero era: blue skies, soft clouds and glossy glass buttons.

AirDeck has four modes:

• DJ Deck: pinch the air to change speed and filter, and the distance between your hands sets the volume. Fist gestures switch effects (reverb, echo, bitcrush, flanger) and live visuals.
• AI DJ: tell it a vibe, like "chill into hype", and Gemini picks the next songs while ElevenLabs gives the DJ a voice.
• Arcade · 2P: two players, one camera, five hand-tracked games: Snake Battle, Bubble Trails, Air Pong, Bubble Pop and Hand Duel (rock-paper-scissors on the beat).
• AirMix: turns any playlist into an osu!-style rhythm game. You hit targets with your fingertips, and every hit mixes the tracks live, controls filters and follows the beat.

What I learned:

• You can do a lot in the browser alone: real-time ML hand tracking, audio processing and 3D, with nothing to install.
• Designing for the body is very different from designing for a mouse. The gesture has to feel natural first, and the precision comes after. Our first Snake moved on a grid and was too hard to steer by hand, so we rebuilt it to follow your fingertip.
• Always build a fallback around an AI API.
• A strong visual identity helps a team project feel like one product.

Demo

Partner Technologies

Google Gemini powers the AI DJ. A small Python server sends Gemini the vibe the user asks for and gets back song picks as structured JSON, so the app can queue the tracks directly without parsing free text. At a live event, an API can hit rate limits or lose connection, so if Gemini fails, the DJ falls back to picking songs by genre and tempo.

ElevenLabs gives the AI DJ its voice. The same server turns the DJ's intros into speech before each song. If ElevenLabs isn't available, the browser's built-in voice takes over, so the show never stops.

The API keys stay on the server, and the server only answers requests from localhost, so no keys are exposed in the browser.

The rest of the stack: MediaPipe Hand Landmarker tracks up to two hands in real time, and a One Euro filter smooths the jittery landmarks. All the audio runs through the Web Audio API and an AudioWorklet, including the effects, tempo and downbeat detection, and the beat-matched transitions between songs. Three.js and WebGL shaders handle the 3D club and visuals, and Canvas 2D draws the arcade.

Hackathon Experience

I attended CodeRED Orion, organized by the University of Houston. I traveled from Monterrey, Mexico, to Houston, and it was my first hackathon abroad: taking the plane, getting to know a new city and meeting new people. I didn't want to wait for the perfect moment to take on a challenge like this.

Building in parallel was a challenge of its own. My teammates, Chris Dong, Alexander Betancourt and Pete Sankar, and I each worked on different pages in separate branches, so keeping links, file names and one visual identity in sync took constant coordination.

What I'll remember most is a piece of advice from Zaquariah Holland, a mentor and tech industry expert I met at the event: "If you do things differently, you'll be different from the rest and you'll have more opportunities."