Building Saira: How I Built a Real-Time Voice AI Learning Assistant for Bharat in 10 Days

Md Ibrahim — Sat, 15 Aug 2026 16:35:09 +0000

"If you talk to a man in a language he understands, that goes to his head. If you talk to him in his language, that goes to his heart." — Nelson Mandela

Over 250 million students in India step into classrooms every day. Many possess boundless curiosity, but hesitate to raise their hand.

Traditional educational software still treats learning like a silent exam: cold text boxes, rigid grammar checks, and intimidating interfaces.

When Murf AI launched the 10 Days of Voice Agents — #VoiceForBharat Edition, I set out to build an alternative: Saira, an empathetic, bilingual AI Learning and Literacy Voice Assistant that converses naturally with Indian learners, remembers their journey, and guides them patiently through spoken dialogue.

Here is the story of how Saira came to life, the architectural decisions behind sub-second voice latency, the engineering hurdles conquered, and how you can run this voice agent yourself.

🌟 The Vision: Why Voice is the True Equalizer for Bharat

In Bharat, learning does not happen in isolated, monolingual silos. It happens in vibrant code-mixed sentences — a seamless blend of English, Hindi, and regional dialects, seasoned with unique local cadences.

Text chatbots fail in this demographic because typing creates friction. Voice eliminates that friction entirely.

With Saira, a 10-year-old student doesn't need to struggle with a keyboard to ask what "photosynthesis" means, or feel embarrassed about mispronouncing a word. Saira listens, encourages, hints, and teaches — just like a dedicated personal mentor sitting across the table.

🏛️ The Architecture: Orchestrating Ultra-Low Latency

Conversational voice AI lives or dies by one metric: Latency.

In natural human speech, conversational turns take between 200ms and 500ms. If an AI pauses for two or three seconds to "think," the illusion of presence shatters instantly.

To achieve fluid, lifelike conversations, Saira runs on a finely orchestrated pipeline:

┌─────────────────┐       Audio Stream       ┌──────────────────────┐
│ 🎙️ Learner Speaks│ ───────────────────────> │ 👂 Deepgram Nova-2   │
└─────────────────┘                          │    Speech-to-Text    │
                                             └──────────┬───────────┘
                                                        │ Transcribed Text
                                                        ▼
┌─────────────────┐      Sub-60ms Audio      ┌──────────────────────┐
│ 🔊 Learner Hears │ <─────────────────────── │ ⚡ Murf Falcon TTS   │
│    Saira Voice  │                          │    (Pooja Voice)     │
└────────┬────────┘                          └──────────▲───────────┘
         │                                              │ Streamed Tokens
         │ LiveKit WebRTC Transport                     │
         └──────────────────────────────────────────────┴───────────┐
                                                                    │
                                            ┌───────────────────────┴──────┐
                                            │ 🧠 Google Gemini 2.5 LLM     │
                                            │    Educational Guardrails    │
                                            └───────────────┬──────────────┘
                                                            │
                     ┌──────────────────────────────────────┼──────────────────────────────────────┐
                     ▼                                      ▼                                      ▼
             ┌──────────────┐                       ┌──────────────┐                       ┌──────────────┐
             │ 💾 Persistent│                       │ 📚 Live Quiz │                       │ 🔄 ARIA Math │
             │   Memory DB  │                       │   & Tools    │                       │   Handoff    │
             └──────────────┘                       └──────────────┘                       └──────────────┘

The Core Stack:

LiveKit WebRTC: Resilient, bidirectional audio channels with native browser compatibility and server-side room management.
Deepgram Nova-2: High-accuracy speech recognition optimized for Indian accents and acoustic environments.
Murf Falcon: The crown jewel of our voice pipeline — delivering sub-60ms model latency and 130ms time-to-first-audio with warm, natural Indian intonation (Pooja voice).
Google Gemini & Custom Guardrails: Structured reasoning tailored to prioritize pedagogical understanding over simple answer-giving.

⚡ What Makes Saira Special: 5 Architectural Highlights

1. Socratic Pedagogy Over Direct Answers
Most chatbots rush to give the answer away. Saira is explicitly instructed never to solve a problem for the student immediately. Instead, she breaks down concepts into intuitive bites, offers gentle hints, and celebrates when the student discovers the answer themselves.

2. Multi-Agent Specialist Handoffs (SAIRA ➔ ARIA)
A single agent shouldn't pretend to be an expert in everything. SAIRA specializes in literacy, language, and concept revision.

When a student asks to practice mental math or algebra, Saira seamlessly triggers a handoff tool that passes conversational control to ARIA, our dedicated Mathematics Practice specialist, preserving the learner's session history without missing a beat:

@function_tool()
async def transfer_to_aria(self, context: RunContext) -> tuple[Agent, str]:
    """Transfer the learner seamlessly to ARIA for mathematics instruction."""
    aria_agent = AriaAgent(
        chat_ctx=self.chat_ctx.copy(exclude_instructions=True)
    )
    return aria_agent, "Transferring you to our math specialist, ARIA."

3. Persistent Learner Memory

Returning students shouldn't have to introduce themselves every time. Backed by SQLite, Saira recalls:

Past topics covered
Recurring conceptual hurdles
Preferred difficulty level and language comfort

4. Live Domain Tools with Fail-Safe Fallbacks

Saira connects to a live dictionary API for phonetic spellings and word origins, as well as a structured exercise engine that scores spoken answers out loud. If a live API ever times out, Saira catches the error gracefully and provides verbal guidance rather than failing silently.

5. Empathetic Escalation & Safety Guardrails

If a learner expresses deep distress, frustration, or asks sensitive questions beyond an AI's remit, Saira gently suggests involving a human mentor and logs a structured escalation ticket with the session summary for human educators.

🛠️ The Hard Battles: Engineering Lessons from the Trenches

Building a real-time voice agent is vastly different from building a text bot. Here are the three hardest hurdles I faced and how I solved them:

1. The "Awkward Silence" vs. "Rude Interruption" Dilemma
Early prototypes either cut off the student mid-thought or hesitated too long after they stopped speaking.

The Fix: We combined Silero Voice Activity Detection (VAD) with LiveKit’s multilingual turn detector, configuring SentenceTokenizer(min_sentence_len=2) on Murf Falcon. This ensures natural breathing room without sluggishness.

2. The Context Amnesia Problem During Handoffs
When transferring from Saira to ARIA, the new agent initially had no recollection of what was just spoken.

The Fix: We created an active context cloning pattern that preserves conversation history while swapping the system instructions and voice profiles in real time.

3. Token-Stream Audio Jitter
Streaming TTS token-by-token can cause audio stuttering if network jitter occurs.

The Fix: Enabling Murf Falcon's text_pacing=True smoothed out the audio buffering, delivering silky-smooth voice cadence.

🚀 How to Build and Run Saira in 10 Minutes

Want to experience Saira or build your own voice agent on this foundation? Follow these steps:

Prerequisites

Python 3.10+ & uv
Node.js 18+ & pnpm
Free API keys from LiveKit, Murf AI, Deepgram, and Google AI Studio.

1. Clone & Set Up Environment

git clone https://github.com/<your-username>/murf-livekit-starter.git cd murf-livekit-starter
2. Create .env.local inside backend/ and frontend/:

LIVEKIT_URL=wss://your-project.livekit.cloud LIVEKIT_API_KEY=your_livekit_key LIVEKIT_API_SECRET=your_livekit_secret MURF_API_KEY=your_murf_falcon_key DEEPGRAM_API_KEY=your_deepgram_key GOOGLE_API_KEY=your_gemini_key

3. Launch the Application

Terminal 1: Backend Agent Engine

cd backend
uv sync
uv run python src/agent.py dev

Terminal 2: Interactive Frontend

cd frontend
pnpm install
pnpm dev`

Visit http://localhost:3000, grant microphone permissions, and say hello to Saira!

🔮 What Lies Ahead

Saira is only the beginning. The roadmap for Voice AI in education includes:

Regional Dialect Expansion: Native fluency in Tamil, Telugu, Marathi, and Bengali.
Dial-in Telephony (SIP): Enabling learners in rural communities to call Saira from basic 2G feature phones without requiring internet access.
Pronunciation Coaching: Real-time acoustic phoneme scoring to help students gain spoken English confidence.

💬 Final Thoughts

Building Saira across these 10 days proved that when you combine ultra-low-latency voice synthesis with thoughtful pedagogy, technology ceases to feel like a machine — it becomes a companion in learning.

A heartfelt thank you to Murf AI for organizing the #VoiceForBharat challenge and pushing the boundaries of real-time voice intelligence.

If you found this helpful, drop a reaction, star the repository, and let me know your thoughts in the comments below!

🔗 Project Links & Connect

💻 GitHub Repository: Check out the Saira Codebase on GitHub
💼 Connect on LinkedIn: Let's connect on LinkedIn
🎙️ Powered by: Murf AI Falcon TTS & LiveKit WebRTC

Built with ❤️ for the **10 Days of Voice Agents — #VoiceForBharat Edition.

DEV Community: Md Ibrahim