Have you ever had a brilliant idea for a gesture-controlled game… like playing flappy bird by pinching your fingers...and then immediately realized how painful real-time computer vision can be to implement? We’ve been there — over and over again.
People often have amazing ideas: hand-controlled games, facial-gesture communication tools, accessibility interfaces…
But execution? Hours of debugging due to coding agent hallucinations or computer-vision spaghetti.
Real-time motion tracking sounds fun… until you try to vibe-code hand, face, and head landmark detection into a fully functional app. Coding agents often hallucinate MediaPipe functions that don't exist, CV logic breaks, and suddenly a simple idea becomes hours of debugging and prompting. As we started exploring Kiro’s features for Kiroween 2025, we realized Kiro was the perfect tool to help bridge this gap. We get how frustrating it is when you can’t quickly bring a creative idea to life, and we want to fix that. That’s why we created the InteractionKit.
A Code Template that Saves the Headache
InteractionKit is a clean, starter-friendly template for building gesture-controlled games, apps, and interactive experiences using MediaPipe (Google’s computer-vision framework) and Python. It gives you a flexible foundation for turning real-time hand, face, and head tracking into intuitive controls that you can plug directly into any project.
- Want to build a Mario-style platformer where jumping = pinch your fingers?
- Make music just by moving your head?
- Draw in mid-air with your fingertip for a fresh twist on Pictionary?
These are just a few ideas.
But the real superpower isn’t just our template alone — it’s what happens when you use it inside the Kiro IDE. We configured hooks, steering docs, and MCP so Kiro handles your app architecture, wires up gesture logic, and generates reliable CV code from your spec without hallucinations. Together, InteractionKit + Kiro let you vibe-code robust motion-tracking apps in minutes, not hours!
Featured Applications Built With InteractionKit + Kiro
We built these two, distinct apps entirely inside the Kiro IDE using vibe-coding + spec-driven development on top of our InteractionKit template:
| 🎃 Math-O-Lantern | ✏️ Holo-Board |
|---|---|
 |
 |
| Slice pumpkins & solve math problems by pinching with your hand in this fast, playful Pygame experience. | A lightboard-style recording tool that lets you draw in mid-air using hand gestures — perfect for creative content. |
What You Get in the Template
- Pre-configured environment for real-time computer vision MediaPipe, OpenCV, PyGame, PyAutoGUI, and camera setup are handled automatically. Clone → run setup → start building.
- Scaffolded Python interaction architecture A clean structure that separates tracking, gesture state, and app behavior so your project stays maintainable as it grows.
- Robust, real-time hand, head, and face tracking out of the box Fast detection running inside a high-performance app loop.
- 17 ready-to-use gestures ✋🤘👍 🙂↔️ Pinch, cursor control, thumbs up/down, nods, blinks and more — instantly usable in any interaction. (You can also ask Kiro to include custom gestures that integrate automatically.)
- Built-in automation for your coding workflow Hooks, steering docs, and MCP setup for Kiro to keep your architecture clean, wire gestures automatically, and fetch accurate MediaPipe + PyGame references.
Instant Setup for Developers
Getting started is easy — just clone the repo, open it in the Kiro IDE and run the setup script:
git clone https://github.com/Harpita-P/Kiro-InteractionKit.git
cd Kiro-InteractionKit
./setup.sh # macOS / Linux
.\setup.bat # Windows
# Test Math-O-Lantern Application 1
python my_apps/Math-O-Lantern/main.py
# Test Holo-Board Application 2
python my_apps/Holo-Board/main.py
#To customize the images in your video, please add your images to the Annotate folder.
This will set up a Python virtual environment, install MediaPipe, OpenCV, PyGame, and other required libraries, and verify that your camera is working. Once setup completes, you can instantly try the pre-defined gestures in our template or test the two demo apps we created. Then, create a Spec with Kiro to start building your app idea. For full details, see the Github Repo.
How We Used Kiro
Here’s how each of Kiro’s core capabilities supercharged our project:
| Kiro Feature | How We Used It | Why It Mattered |
|---|---|---|
| Vibe-Coding | We kicked things off by chatting with Kiro through Vibe Sessions to brainstorm and design the core InteractionKit skeleton. We discussed how the MediaPipe tracking loop should run, how gesture states should transition, and where controllers should sit in the architecture. Kiro wrote full modules based on simple explanations. | It let us build fast without getting buried in boilerplate code. From natural prompts, Kiro produced a real gesture to state to action pipeline. We could iterate instantly and stay in a creative mindset. |
| Agent Hooks | We built three custom hooks that automate everything needed when a new gesture is added. Kiro detects a new gesture file, updates imports, attaches state fields, wires logic into the live tracking loop, updates test scripts and documents the gesture automatically. | It saved us from editing multiple files over and over again. Seventeen gestures were integrated in seconds and developers using InteractionKit can create custom gestures without touching the underlying architecture. |
| Spec-Driven Development | Once the template was solid, we wrote two simple specs and Kiro built complete apps on top of InteractionKit. These specs described gameplay, gesture inputs and UI structure for Math-O-Lantern and Holo-Board. Kiro handled the rest. | Specs improved code structure and accuracy while still letting us build fast. We could preview full design before generation, avoid mistakes and end up with polished apps that feel production ready. |
| Steering Docs | We created two extra steering docs inside the project. One documented clean game architecture in Python and the other explained how to test CV modules. Kiro referenced these documents during generation. | Kiro consistently followed our architectural rules. CV logic stayed separate from game logic and files remained organized across the entire project. Clean project structure was maintained automatically. |
| MCP (Context7) | We linked Kiro to Context7 MCP for real MediaPipe and PyGame documentation. Kiro used it to confirm correct landmark indices for facial gestures (like EAR, MAR) and proper PyGame design patterns during code generation. | This removed hallucinations, giving us confidence that every gesture implemented by Kiro behaves exactly as expected. |
When developers don’t have to worry about the complicated setup behind computer vision, they can focus on building innovative features and getting their ideas into people’s hands faster. And that’s the gist of why we built this template – whether you’re creating accessibility tools, gesture-based games, AR-inspired experiences, or something entirely new, InteractionKit + Kiro remove the technical friction and let you jump straight into prototyping what matters. We are proud to have built a project that supports a real need in the community, helping both beginners getting started and experienced developers who want to move from computer-vision idea to prototype more quickly. We're super excited to share the interaction kit template with the community and can't wait to see the creative ideas you build.
Visit the Github Repo
This project was built for Kiroween 2025.
Check out our Project on Devpost
Top comments (0)