🎙️ VoiceAgent AI — Local AI Agent with Voice Control
Fully-functioning voice-controlled local AI Agent for Mem0 AI/ML & Generative AI Developer Intern Assignment. The system accepts audio input, transcribes it, classifies intent with an LLM, and runs local tools, all presented in a sleek, dark-themed Streamlit UI.
🏗️ Architecture
The architecture includes a few components such as audio input, speech-to-text, intent classification, and a tool dispatcher.
┌─────────────────────────────────────────────────────────┐
│ VoiceAgent AI │
│ │
│ ┌──────────┐ ┌──────────┐ ┌───────────┐ │
│ │ Audio │──▶│ STT │──▶│ Intent │ │
│ │ Input │ │ (Whisper │ │ Classify │ │
│ │ .wav │ │ via │ │ (LLaMA │ │
│ │ .mp3 │ │ Groq) │ │ 3.3 70B │ │
│ └──────────┘ └──────────┘ │ via Groq)│ │
│ └─────┬─────┘ │
│ │ │
│ ┌───────────────────────┼────────────────┐ │
│ │ Tool Dispatcher │ │ │
│ │ ▼ │ │
│ │ create_file │ write_code │ summarize │ │
│ │ │ general_chat │ │
│ └─────────────────────────────────────── ┘ │
│ │ │
│ ▼ │
│ output/ folder │
│ (sandboxed, safe) │
└─────────────────────────────────────────────────────────┘
The architecture diagram above shows this.
Module Breakdown
app.py | Streamlit UI — Pipeline Display, Session History, Human-in-the-loop. intent_classifier.py | LLaMA 3.3 70B prompt + JSON parsing + graceful fallback. tools.py | Tool handlers: create_file, write_code, summarize, general_chat. requirements.txt | Minimal dependencies (Streamlit + Groq SDK).
🔑 Hardware Note & Workaround
Why Groq API instead of local models?
My local machine does not have any dedicated GPU. Running Whisper Large v3 or LLaMA 3.3 70B locally would require at minimum 8GB VRAM (Whisper) and 40GB+ RAM/VRAM (LLaMA 70B quantized). Inference using the models locally will result in accuracy degradation that’s prohibitively high.
Solution: Groq’s free API tier offers:
Whisper Large v3 for STT — state-of-the-art accuracy, ~2-3 seconds per audio file LLaMA 3.3 70B Versatile for intent + code generation — extremely fast (~200 tokens/sec on Groq hardware)
This is fully in compliance with the assignment’s hardware workaround policy. The whole pipeline is running at API speed (3 – 6 seconds, end-to-end).
✨ Bonus Features Implemented
Compound Commands — “Summarize and save to file” deals with more than one action Human-in-the-Loop — Checkbox confirmations for any writing-to-file commands Graceful Degradation – when JSON parsing fails, no intent corresponds to the request, or the audio message makes no sense, general_chat with a useful message is triggered. Session Memory: The entire history of the actions is displayed in the UI for the session. Safe Sandbox — All file operation limited to output/ folder with path traversal safeguard
📁 Project Structure
voice-agent-ai/ ├── app.py # The main Streamlit UI ├── intent_classifier.py # LLM-based intent classification ├── tools.py # Tool execution handlers; ├── requirements.txt # Dependencies ├── output/ # All generated files (gitignored) README.md
🎬 Demo Video
YouTube Unlisted Link — Demonstrates:
- Voice input → “Create a python file with a retry decorator” →
write_codeintent → file saved 4. Provide a voice input: “What is the difference between RAM and ROM?" → intent of general_chat → get a response;
📝 Technical Article
Medium / Dev.to Link — Architecture, model selection, Groq’s speed advantage, challenges.
🛡️ Safety
All file writes are limited to output/ directory, using an os.path.basename() stripping Stripping Path traversal (“../”) Human-in-the-loop confirmation before any destructive file operation
📦 Dependencies
streamlit>=1.35.0 # UI framework groq>=0.9.0 # Groq SDK (STT + LLM)
No Bulky ML Libraries Needed. No heavy ML libraries required; can run on any machine with Python 3.9+.
YOU CAN CHECK MY WORK
Top comments (0)