The latest articles on DEV Community by Santhosh (@santhosh_120). https://dev.to/santhosh_120 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3874725%2F6d07f402-959b-4538-bb76-0c6225eff9c2.png https://dev.to/santhosh_120 en Santhosh Sun, 12 Apr 2026 10:29:01 +0000 https://dev.to/santhosh_120/building-a-voice-controlled-ai-agent-with-groq-gradio-15pb https://dev.to/santhosh_120/building-a-voice-controlled-ai-agent-with-groq-gradio-15pb <h2> What I Built </h2> <p>A voice-controlled AI agent that: records your voice → transcribes it → <br> understands your intent → executes the right action → shows everything in a UI.</p> <p>Built for the Mem0 AI Internship assignment.</p> <h2> Architecture </h2> <p>Voice Input → Groq Whisper (STT) → LLaMA 3.3 70B (Intent) → Tool Executor → Gradio UI</p> <h2> Model Choices </h2> <p><strong>Speech-to-Text: Whisper-large-v3 via Groq</strong><br> I initially planned to run Whisper locally using HuggingFace, but on a CPU-only <br> Windows machine this was extremely slow (30+ seconds per clip). Groq's API gives <br> sub-second transcription for free, making the user experience much better.</p> <p><strong>LLM: LLaMA 3.3 70B via Groq</strong><br><br> Running a 70B model locally requires 40GB+ VRAM. Groq's free tier handles this <br> instantly. The model is prompted to return structured JSON for reliable intent parsing.</p> <h2> The Intent Classification Trick </h2> <p>The key insight was prompting the LLM to return ONLY JSON — no explanation, <br> no markdown. Combined with low temperature (0.1), this gives very consistent results.</p> <p>It also supports compound commands: "Write a retry function AND save it" <br> correctly returns intents: ["write_code", "create_file"].</p> <h2> Challenges </h2> <ol> <li><p><strong>JSON parsing</strong> — LLMs sometimes wrap JSON in markdown fences. <br> Fixed with regex: <code>re.sub(r"</code><code>json\s*|\s*</code><code>", "", raw)</code></p></li> <li><p><strong>Audio formats</strong> — Gradio passes temp file paths. <br> Opening with "rb" and letting Groq handle format detection solved this.</p></li> <li><p><strong>Safety</strong> — Used os.path.basename() to prevent path traversal attacks <br> when creating files.</p></li> </ol> <h2> What I'd Add With More Time </h2> <ul> <li>Local model support via Ollama (offline mode)</li> <li>Wake word detection</li> <li>Web search tool integration</li> </ul> <h2> GitHub &amp; Demo </h2> <ul> <li>Code: <a href="https://github.com/SANTHOSH-120/voice-agent.git" rel="noopener noreferrer">https://github.com/SANTHOSH-120/voice-agent.git</a> </li> <li>Demo: <a href="https://youtu.be/Q2vEAmrYtYg" rel="noopener noreferrer">https://youtu.be/Q2vEAmrYtYg</a> </li> </ul> agents ai llm showdev