Over 10 days of the VoiceForBharat challenge, I built Saksham, a conversational voice agent that helps Indian learners practice reading, speaking, and asking questions naturally.
Saksham runs on Murf Falcon (fast, Indian-accented TTS) and LiveKit Agents on the backend to speak and listen in real time. Its key features: memory (it remembers returning users and context), built-in tools (fetching quiz questions), outbound calling (it can call a user proactively), human escalation (hands off to a real tutor when needed), and specialist handoffs (math questions go to a dedicated Math bot).
This post walks through what Saksham does, its architecture, the hardest problems I hit (and how I solved them), and the setup steps so you can build your own voice agent, plus code snippets, diagrams, and a suggested LinkedIn post at the end.
The Problem and the Users
My focus for the challenge was Learning & Literacy in India. Students, adults learning a new language, or people with limited literacy all benefit from conversational practice, and typing is often a barrier. It's easier to speak a question or read aloud than type it out. Saksham fills that gap: it acts like a tutor or friend who listens and talks back.
For example, a user might say, "Mujhe English grammar mein help chahiye" (I need help with English grammar), and Saksham responds in kind, guiding them or asking quiz questions.
Voice lowers the entry barrier: anyone can talk instead of type. And in a multilingual country like India, code-mixed speech (Hindi + English) is the norm, not the exception. Saksham uses Murf Falcon's Indian voice to sound natural, and I tailored the prompts specifically to understand Hinglish. As I put it in the Day 2 demo: "Saksham is designed to support learning and literacy by helping users improve their reading, speaking, and communication skills through natural conversations." The goal, simply: learning should be accessible to everyone.
What Saksham Does
Saksham grew from a simple voice prototype into a full learning companion over the 10 days. The standout features:
Indian-accented TTS (Murf Falcon): Saksham speaks in an Indian-English voice with minimal latency (under 130ms to first audio), so the conversation feels native. Users can speak Hindi or code-switch mid-sentence, and Saksham handles it gracefully.
A clear persona and guardrails: A short system prompt at startup defines Saksham's goal: help the learner practice and stay on topic. Safety guardrails keep it from answering anything inappropriate or out of scope; it refuses off-limits requests the way a human tutor would.
A stateful web UI: The Next.js frontend shows Saksham's current state (Ready, Connecting, Listening…, Speaking…, Call Ended), driven by LiveKit's agent framework, so users always know what's happening.
Memory of returning users:
With permission, Saksham stores recent conversation snippets per user and prepends them to the LLM prompt. So on a later call it might say, "Namaste Anusha! Last time we talked about English Grammar." The tricky part was keeping this bounded so the prompt didn't blow up, more on that below.
Built-in tools (quiz generator): When a user asks for practice questions, Saksham calls a helper function to generate them. Ask "Give me 3 multiplication questions" and it responds with fresh problems, spoken aloud. Tools stay inside the LLM flow, so Saksham remains the interface while specialized functions do the heavy lifting.
Outbound calling: Using LiveKit's SIP telephony support, Saksham can dial a user proactively, for example, for a scheduled English-practice call. This was my first real use of LiveKit for outbound (not just inbound) voice.
Human escalation: When Saksham detects it's out of its depth (confusing follow-ups, or something clearly beyond a voice tutor's scope), it says, "Let me get a human tutor to help you better," and hands off to a person on the call.
Call analytics dashboard:
A simple Chart.js dashboard logs each session's outcome: success, escalation, satisfaction rating, duration. Over time this surfaces patterns, e.g. if 80% of math questions escalate, the math tool needs work.
Specialist handoff (multi-agent): On Day 9 I added a second "Math Practice Specialist" agent. When Saksham detects math intent, it transfers the LiveKit call to the math agent, carrying over conversation history: "Let me transfer you to our Math tutor." This uses LiveKit's multi-agent handoff pattern, so Saksham doesn't need to be an expert in everything; it can call in specialists.
Architecture
User audio flows through Deepgram STT to Gemini to Murf TTS, with LiveKit handling the real-time stream. The agent logic layer manages memory, tools, escalation, and specialist handoffs.
LiveKit sits in the middle as the real-time transport, Deepgram handles speech recognition, Gemini powers the conversation, and Murf Falcon TTS speaks the response. LiveKit Agents also handles telephony; that's how Saksham makes outbound phone calls, letting a user join from a regular phone instead of the web app.
Tech Stack:
| Component | Saksham (this project) | Alternatives |
|---|---|---|
| Speech-to-Text | Deepgram (streaming) | Google STT, Azure Speech, Whisper |
| LLM | Google Gemini | OpenAI GPT-4, Claude, LLaMA |
| Text-to-Speech | Murf Falcon (Indian voice) | ElevenLabs, Amazon Polly, Google Wavenet |
| Transport | LiveKit Agents (WebRTC) | Twilio, Daily, Agora |
| Memory / DB | In-memory / simple store | SQLite, Pinecone, Redis, Weaviate |
| Frontend | React / Next.js | Vanilla WebRTC UI, Flutter, native app |
| Telephony | LiveKit SIP Gateway | Twilio SIP, Zoom SDK |
This is the same stack Murf's own starter kit recommends, which gave me a fully asynchronous voice pipeline with built-in turn detection and tool support out of the box — so I could focus on Saksham's personality and skills instead of low-level streaming bugs.
Timeline
Getting Started
Full code is in the public GitHub repo.
Prerequisites: Python 3.10+, Node.js 18+, uv (Astral's package manager), and a LiveKit project (Cloud or self-hosted). The LiveKit CLI (lk) is handy but optional.
git clone https://github.com/ianusha08/murf-livekit-starter.git
cd murf-livekit-starter
cd backend && uv sync
cd ../frontend && pnpm install
The repo is based on Murf's official starter kit, so the WebRTC boilerplate and audio pipeline are already there — the real work is customizing it for your use case.
Environment variables — copy .env.example to .env.local in both backend/ and frontend/, then fill in:
LIVEKIT_URL=<your LiveKit URL>
LIVEKIT_API_KEY=<your LiveKit API key>
LIVEKIT_API_SECRET=<your LiveKit secret>
MURF_API_KEY=<your Murf API key>
DEEPGRAM_API_KEY=<your Deepgram key>
GOOGLE_API_KEY=<your Gemini API key>
These are mandatory — without them Saksham can't connect to STT, LLM, or TTS. Never commit them; keep .env.local in .gitignore.
Download models (if needed):
uv run python src/agent.py download-files
Run it:
start LiveKit locally, or point LIVEKIT_URL at LiveKit Cloud
brew install livekit && livekit-server --devbackend
uv run python src/agent.py devfrontend
pnpm dev
Open the frontend, hit "Call Agent," and talk. Watch backend logs for STT transcripts and LLM responses to debug.
To extend it, edit backend/src/agent.py, for example, calling tools.get_math_questions(topic, count) on certain intents, or setting an escalation flag. Because LiveKit Agents is built to be extensible, adding tools and handoffs is mostly a matter of plugging into session callbacks.
Code Snippets
Memory, recent turns per user, trimmed to keep the prompt bounded:
`user_id = session.user_id
if user_id not in memory_store:
memory_store[user_id] = []
memory_store[user_id].append(user_input)
history = " ".join(memory_store[user_id][-10:])
prompt = f"Previous: {history}\nUser: {user_input}\nSaksham:"
response = llm.generate(prompt)
Tool call — triggered when the LLM detects a practice-question intent:
python
if user_intent == 'ask_practice':
topic = extract_topic(user_input) # e.g. "algebra"
questions = math_tool.generate_questions(topic=topic, count=3)
agent_response = "Here are some practice questions on " + topic + ":\n" + "\n".join(questions)
Human escalation — triggered on low-confidence answers or explicit confusion:
python
session.logger.info("Escalating to human agent due to confusion")
session.end("HANDOFF", final_context="The user needs a human tutor.")
session.dispatch_event("escalate")
Full implementations live under backend/src/agent.py and backend/src/actions in the repo.
Challenges and lessons :
Memory overload. My first pass just appended every past turn to the prompt. It blew past the context window fast, cost went up, and the model started hallucinating from stale context. Fix: a FIFO queue capped at the last few turns, with occasional summarization of older content. Coherence improved immediately. Lesson: keep memory short and relevant, not exhaustive.
Escalation logic. Getting Saksham to know when not to answer was harder than expected. I started with hard-coded catch-phrases (e.g., flag responses containing "ask a tutor"), which was too blunt; it escalated on trivial queries. I moved to a two-step check: after each response, Saksham asks itself "Should I escalate?" and combines that self-classification with a small keyword list. Much more reliable after testing against a batch of sample dialogues.
Multilingual input. This was the toughest one. Deepgram's language detection on Hinglish (code-mixed Hindi-English, often mid-sentence) was inconsistent; it would lock onto one language and mangle the other. I worked around it by tuning the STT language settings and adding code-mixed examples directly into the system prompt, so the LLM side could compensate even when transcription wasn't perfect. It's noticeably better now, but still not solved: full code-mixed STT is a real, still-open problem worth digging into further.
Real-time performance. Murf Falcon's TTS latency and LiveKit's streaming kept things fast overall, but occasional network hiccups caused stutter. Enabling LiveKit's default jitter buffering and only showing "Speaking…" once audio had actually started fixed the perceived lag. Testing on mobile and weaker networks mattered more than I expected.
UI/backend state sync. Keeping the frontend in sync with the agent's internal state (e.g., showing "Transferring…" during a handoff) needed explicit events. I emit LiveKit metadata (session.publish_state) whenever internal state changes, and the UI listens for it. Small thing, but it made the whole experience feel much more real-time..
How to Build Your Own Voice Agent
- Know your components. Any voice agent needs STT, an LLM, TTS, and a real-time transport. Swap in whatever fits: Azure Speech, OpenAI, AWS Polly. The pattern stays the same.
- Start from a template. Murf's murf-livekit-starter (Python backend + React frontend) already has WebRTC, TURN setup, and the basic audio pipeline wired up.
- Get your API keys. Murf (TTS), Deepgram or similar (STT), and an LLM provider (Gemini, OpenAI, etc.), all via environment variables, never hardcoded.
- Write the agent logic. In agent.py, define the persona, memory handling, tool/handoff triggers, and guardrails.
- Wire up the frontend. Point the "Call Agent" button at your LiveKit URL and keys, or build a fully custom UI as long as it joins the LiveKit room.
- Test iteratively. Start with "Hello, Saksham," check STT/LLM logs, then test escalation and handoff flows explicitly with real sample dialogues.
- Write it up and share. Link your repo (minus your keys!) and post a demo clip if you can.
What's Next for Saksham
- Retrieval-augmented answers: let Saksham fetch real content (news, reference material) instead of relying purely on the LLM's own knowledge.
- More Indian languages: Tamil, Marathi, and better native Hindi TTS to widen the audience.
- Personalized learning plans: track progress and recommend lessons based on level.
- Real analytics backend: a proper database + BI layer instead of the current lightweight dashboard.
- Production hardening: auth, error monitoring, and clustering LiveKit servers for concurrent learners.
Links
Top comments (0)