This is a submission for Weekend Challenge: Passion Edition
What I Built
Game Whisperer
AI-Powered Sound Design for Games
Traditional game development forces indie developers to spend days hunting for the right footstep, the right coin jingle, the right sword swing — browsing stock libraries, negotiating licenses, hiring voice actors.
Game Whisperer makes that entire process disappear.
Game Whisperer is an AI-native game audio pipeline that transforms sound design from a bottleneck into a one-click operation. The platform combines AI vision, ElevenLabs audio generation, and a custom Godot 4 plugin to give any developer — regardless of audio experience — a complete, production-ready sound design system.
Key Features
AI Sound Design Brief
Drop any game screenshot into the web app. The AI vision model analyzes every visual element — terrain type, characters, objects, environment — and returns a structured sound design brief in under ten seconds. Every sound comes with a name, a description of exactly how it should sound, and a reason why it belongs in your game.
ElevenSFX — Custom Godot Node
A custom @tool node that runs inside the Godot editor. Give it a name, write a plain English description, hit Generate. The ElevenLabs API generates the MP3 and saves it directly into your project. Any developer can have professional audio in thirty seconds. Zero audio experience required.
ElevenDialog — AI Voice Acting
NPC dialog that actually speaks. Type the lines, pick a voice, hit Generate. The audio plays back with the text typewriting itself out in perfect sync with the voice duration — no manual timing needed.
AiBgm — AI Background Music
Describe a mood, get a full background music track. AI-generated ambient audio that fits your game's atmosphere.
AiAnalyze — One-Click Full Pipeline
The flagship feature. Click Analyze — the AI scans your entire scene, identifies every sound it needs, and auto-imports a complete scene of pre-configured ElevenSFX nodes ready to generate. One click. Complete audio pipeline.
Demo
🎬 Video Demo
https://www.loom.com/share/afb07def125e4edabed4370a491630ba
Code
https://github.com/atuljha-tech/gamewhisperer
How I Built It
Game Whisperer is built as two tightly integrated systems — a Next.js web app and a Godot 4 editor plugin — that share the same AI backbone.
Architecture
Screenshot → Falcon AI Vision → JSON SFX Brief → ElevenLabs API → MP3 in Godot
The Web App (Next.js 15 + TypeScript)
The web app is a single-page tool built with Next.js 15, Tailwind CSS, and react-dropzone. The API route converts the uploaded image to base64 inline — no external storage needed — and sends it directly to the Falcon AI vision model. The model returns structured JSON with the full SFX brief which renders as interactive cards.
The Godot Plugin (GDScript @tool nodes)
Godot's @tool annotation lets scripts run inside the editor itself. Each node makes an HTTP request to the ElevenLabs API from inside the editor, streams the response binary, and writes the MP3 directly to the project filesystem.
The AiAnalyze node calls Falcon AI vision with a screenshot, parses the JSON response, dynamically creates ElevenSFX and ElevenDialog nodes, and saves them as a packed scene. The entire SFX library for a scene can be scaffolded in one click.
The Demo Game
A fully playable Godot 4 top-down RPG where every audio file was generated by AI. Not recorded. Not purchased. Generated by typing a sentence.
- Footsteps with random pitch variation so they never sound robotic
- Axe impact that fires exactly on animation frame 6
- Coins that float up with an AI jingle on every collect
- NPC dialog with typewriter text synced to audio duration
- AI-generated background music
Prize Categories
🏆 Best Use of ElevenLabs
ElevenLabs is the core audio engine of Game Whisperer and powers two distinct capabilities:
Sound Effects Generation — The ElevenSFX node calls ElevenLabs /v1/sound-generation with a plain English description and receives a game-ready MP3 in under a second.
Text-to-Speech Dialog — The ElevenDialog node calls /v1/text-to-speech to voice-act NPC lines. The response audio length is used to calculate the typewriter animation duration — so the text always finishes typing at exactly the same moment the voice finishes speaking.
ElevenLabs transforms what was previously a multi-day audio production task into a thirty-second workflow inside the game engine.
Team
Atul Jha
Project Lead, AI Engineering, Godot Plugin Development, Web App
DEV Profile: https://dev.to/atul_jha_27
Ayush Kumar
Co-Developer, Product Architecture
DEV Profile: https://dev.to/ayushkumar2601
Top comments (0)