DEV Community

Cover image for Saathi (साथी) A Bilingual Multi-Agent Voice AI Tutor for Indian Students
sharvin shetty
sharvin shetty

Posted on

Saathi (साथी) A Bilingual Multi-Agent Voice AI Tutor for Indian Students

#ai #webdev #programming #voiceai

Over the last 10 days, I participated in the 10 Days of AI Voice Agents (Voice for Bharat Edition) challenge by Murf AI. My goal was to build a real-world voice application that solves an actual accessibility problem in India.

I chose the Learning & Literacy Track and built Saathi (साथी) — an AI-powered voice companion designed to help school students and first-generation learners practice concepts, play quizzes, and learn math in Hinglish.

Here is a breakdown of what I built, the architecture behind it, the hardest bugs I ran into, and how you can run the project yourself.

1. The Real-World Problem: The Indian Spoken English Paradox

In rural or under-resourced Indian households, many children are first-generation schoolgoers. While they study passive school material, they lack supportive, bilingual study partners at home to help clarify doubts, explain math equations, or review vocabulary.
Voice calls are intuitive for young students who find reading long textbook texts or typing queries on tiny screens difficult. A 30-second conversational call in Hinglish is much more effective than navigating websites.
I designed Saathi to act like a personal phone call tutor:

1. It greets students in natural bilingual Hinglish, recognizing returning profiles.

2. It offers interactive general knowledge trivia quizzes and English word definitions.

3. It protects student privacy by requiring verbal consent before storing names and study progress in SQLite.

4. If the student struggles or asks for a teacher, it creates an escalation ticket and sends alerts to supervisors.

5. If the student wants to practice math, it transfers them mid-session to a Specialist Math Tutor (Samar), who speaks in a dedicated male voice.

2. System Architecture

Here is how real-time audio and data flow through the application:

Now the end-to-end pipeline view, showing the same system as a sequence of turn-by-turn stages:

3. The Tech Stack

• Voice Synthesis (TTS): Murf Falcon Streaming TTS (latency <100ms)
• Real-Time Transport: LiveKit Agents SDK & WebRTC Rooms
• Speech Recognition (STT): Deepgram Nova-3 Multilingual
• Language Model (LLM): Google Gemini 3.5 Flash-lite
• Database: SQLite (saathi_memory.db)
• Frontend: Next.js 15, React, Tailwind CSS

Real-Time Transport: LiveKit Cloud manages bidirectional WebRTC audio tracks with adaptive jitter buffers and low-latency packet recovery.
Speech Recognition (STT): Deepgram Nova-3 (multilingual) transcribes audio, handling code-mixed Hinglish phrases.

Agent Core & LLM Orchestration: Configured with Google Gemini 3.5 Flash-lite to provide fast bilingual token generation with sub-400ms time-to-first-token latency.

Speech Synthesis (TTS): Murf Falcon streams synthesized speech frames over WebSockets.

Saathi (Main Agent): Configured with Murf Falcon voice "Anisha" (conversational Hinglish female voice).

Math Specialist (Specialist Agent): Configured with Murf Falcon voice "Samar" (patient male Indian math tutor voice).
Data Layer & Observability: Local SQLite database (memory.db) tracks 3 tables: users, escalations, and calls.

4. Visual Evidence from the Build

Here is how the system looks and behaves across core interaction touchpoints.

A. Landing & Practice Goals
The main interface allows learners to choose practice focus areas and configure daily practice schedules.

B. Active Voice Conversation & State Feedback
During an active session, the animated Voice Orb visualizer renders real-time state feedback (listening, thinking, speaking) alongside an active agent and voice indicator badge.

C. Human Support Escalation Drawer & Discord Alerts
When a learner requests human assistance or expresses distress, Saathi opens a support ticket (ESC-XXXX), displays it in the Human Help drawer, and dispatches a sanitized alert to academic counselors via Discord Webhooks.

D. Production Call Analytics Dashboard
The built-in analytics dashboard reads directly from the SQLite call_analytics table, presenting real session counts, success rates, failure breakdowns, and duration logs.

5. Key Features Built Over the 10 Days

A. Natural Hinglish Voice with Accent Scripting
Using Murf Falcon (Anisha voice), the coordinator responds almost instantly. To prevent a robotic English-accented voice, Saathi's system prompt strictly requires Devanagari script output (नमस्ते) for all Hindi phrases, prompting native cadence and natural rhythm.

B. Caller Profile Memory & Consent Guardrails
Instead of asking for information repeatedly, Saathi remembers students across calls. When a student calls back, the agent loads their saved profile from SQLite. To protect user privacy, Saathi is programmatically blocked from saving name or topic progress unless the student gives explicit verbal consent.

C. Structured Study Tools
I integrated tools that retrieve live trivia quiz questions from the Open Trivia Database, and define English words in Hinglish using dictionary APIs. Students can study vocabulary and test general knowledge mid-session.

D. Teacher Escalation & Analytics Dashboard
If a student repeatedly struggles, Saathi creates an escalation ticket, displays it on a Next.js /escalations page, and dispatches a sanitized warning embed notification directly to a Discord Webhook. Metrics including total calls, success rates, and duration logs are shown on a glassmorphism /dashboard page.

E. Multi-Agent Specialist Handoff
When a student asks to learn math, the main agent (Anisha) announces a handoff. The active chat context is copied and sent to a specialized MathSpecialist agent (Samar), who takes over the session in a distinct male voice (Samar) to solve equations step-by-step.

6. Verified Code Highlights (Evidence from our codebase)

Example 1: Context-Preserving Multi-Agent Handoff Tool
_Location: backend/src/agent.py_

@function_tool
async def transfer_to_maths(self, context: RunContext) -> tuple[Agent, str]:
    """Transfer the student to the Maths Specialist when they want to study math."""
    logger.info(f"Handoff triggered: transfer_to_maths for user_id: {self.user_id}")
    math_agent = MathSpecialist(
        user_id=self.user_id,
        chat_ctx=self.chat_ctx.copy(exclude_instructions=True)
    )
    return math_agent, "Main aapko hamare Maths specialist se connect karti hoon. Ek second rukiye."
Enter fullscreen mode Exit fullscreen mode

Example 2: Decoupled Multi-Threaded Python REST Server
_Location: backend/src/agent.py_

class EscalationsHandler(BaseHTTPRequestHandler):
    def do_OPTIONS(self):
        self.send_response(200)
        self.send_header("Access-Control-Allow-Origin", "*")
        self.send_header("Access-Control-Allow-Methods", "GET, OPTIONS")
        self.send_header("Access-Control-Allow-Headers", "Content-Type")
        self.end_headers()

    def do_GET(self):
        if self.path == "/stats":
            from database import get_call_stats
            try:
                stats = get_call_stats()
                self.send_response(200)
                self.send_header("Content-Type", "application/json")
                self.send_header("Access-Control-Allow-Origin", "*")
                self.end_headers()
                self.wfile.write(json.dumps(stats).encode("utf-8"))
            except Exception as e:
                self.send_response(500)
                self.wfile.write(str(e).encode("utf-8"))
Enter fullscreen mode Exit fullscreen mode

Example 3: SQLite Call Metrics Outcome Updates
_Location: backend/src/database.py_

def update_call_outcome(call_id: str, duration_seconds: int, name: str, outcome: str, failure_reason: str = None):
    conn = get_db_connection()
    cursor = conn.cursor()
    cursor.execute(
        """
        UPDATE calls
        SET duration_seconds = ?, name = ?, outcome = ?, failure_reason = ?
        WHERE call_id = ?
        """,
        (duration_seconds, name, outcome, failure_reason, call_id)
    )
    conn.commit()
    conn.close()
Enter fullscreen mode Exit fullscreen mode

Example 4: Windows Safe Unicode Terminal Logging Filter
_Location: backend/src/agent.py_

class SafeLoggingFilter(logging.Filter):
    def filter(self, record):
        try:
            if isinstance(record.msg, str):
                record.msg = record.msg.encode("ascii", errors="replace").decode("ascii")
            if record.args:
                new_args = [
                    arg.encode("ascii", errors="replace").decode("ascii")
                    if isinstance(arg, str) else arg for arg in record.args
                ]
                record.args = tuple(new_args)
        except Exception:
            pass
        return True
Enter fullscreen mode Exit fullscreen mode

7. Real Engineering Challenges & How I Solved Them

Here are the primary technical issues encountered during development and how they were resolved.

1. The LiveKit NOT_GIVEN Sentinel Issue
Problem: Subclassing Agent with tts=None in Assistant caused LiveKit to raise RuntimeError: 'tts_node' called but no TTS node is available during speech output.
Cause: In LiveKit Agents SDK, Agent uses NOT_GIVEN as the default sentinel to inherit the session-level TTS. Passing None explicitly disabled the TTS node.
Fix: Updated constructors to pass tts if tts is not None else NOT_GIVEN.
Lesson: Inspect framework-level sentinel objects (NOT_GIVEN) when subclassing agent classes.

2. Next.js 15 React Server Component (RSC) Boundary Crash
Problem: Adding a custom useActiveAgent hook resulted in Error: Could not find the module ... in the React Client Manifest and TypeError: __webpack_modules__[moduleId] is not a function.
Cause: The hook file omitted the 'use client'; directive. In Next.js 15 App Router, hooks utilizing useState or useEffect must explicitly declare client boundaries.
Fix: Added 'use client'; at the top of frontend/hooks/useActiveAgent.ts and rebuilt with pnpm build.
Lesson: Every custom hook that uses React lifecycle state in Next.js App Router must include the 'use client'; directive.

3. Latency in Multi-Turn Specialist Handoffs
Problem: Transferring between agents previously took multiple conversational turns (Ask permission → User confirmation → Transfer), making the transition feel delayed.
Cause: Prompt instructions required confirmation for all mentions of the specialist topic, even when the user explicitly asked for immediate practice.
Fix: Updated prompt rules so that when a user explicitly requests specialist practice, the agent triggers the handoff tool in the same turn.
Lesson: Reserve multi-step verbal confirmation for destructive operations (e.g. data deletion); routing to specialized skills should happen promptly upon explicit intent.

8. Practical Step-by-Step Build Guide

To set up and run the Saathi repository locally:

Prerequisites
Python 3.10+ with uv package manager
Node.js 18+ with pnpm
LiveKit Cloud Project (URL, API Key, API Secret from cloud.livekit.io)
Murf AI API Key (murf.ai/api)
Deepgram API Key (console.deepgram.com)
Google Gemini API Key or Groq API Key

Step 1: Clone and Configure Environment

git clone https://github.com/sharvinshetty10-hub/murf-livekit-starter.git
cd murf-livekit-starter
Enter fullscreen mode Exit fullscreen mode

Backend Configuration (backend/.env.local):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_gemini_api_key
GROQ_API_KEY=your_groq_api_key

DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/your_webhook_url
Enter fullscreen mode Exit fullscreen mode

Frontend Configuration (frontend/.env.local):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
Enter fullscreen mode Exit fullscreen mode

Security Note: Never commit .env.local files, webhook URLs, or private database files to version control.

Step 2: Start the System
Using the Startup Scripts:

Windows:

.\start_app.ps1
Enter fullscreen mode Exit fullscreen mode

macOS / Linux:

./start_app.sh
Enter fullscreen mode Exit fullscreen mode

Manual Startup:

# Terminal 1: Backend
cd backend
uv sync
uv run python src/agent.py dev

# Terminal 2: Frontend
cd frontend
pnpm install
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000 to call Saathi, and visit http://localhost:3000/dashboard to view the metrics panel.

Step 3: Run the Test Suite

# Backend unit & integration tests
cd backend
uv run pytest tests/test_analytics.py tests/test_escalation.py tests/test_memory_db.py tests/test_handoff.py -k test_specialist_murf_voice_configuration -v

# Frontend lint & build check
cd ../frontend
pnpm lint
pnpm build
Enter fullscreen mode Exit fullscreen mode

9. Practical Troubleshooting

Issue 1: Audio Playback Fails on Initial Page Load
Cause: Modern browsers block autoplay audio until a user interaction event occurs.
Fix: The frontend uses an explicit Start Audio button (StartAudioButton)to ensure an audio context is initialized before streaming starts.

Issue 2: Discord Webhook Times Out
Cause: Network egress latency or invalid webhook URL.
Fix: escalationtools.py wraps webhook requests in a non-blocking background task with a 5-second timeout, ensuring the voice dialogue continues even if the webhook call fails.

10. Conclusion & Links

Building Saathi demonstrates how voice, fast streaming TTS, and multi-agent setups can build accessible educational tools in India. Thanks to Murf AI for hosting the #VoiceForBharat challenge!

GitHub Repository: https://github.com/sharvinshetty10-hub/murf-livekit-starter
LiveKit Voice AI Docs: https://docs.livekit.io/agents
Murf Falcon TTS Docs: https://murf.ai/api/docs

11. References

1.Pratham Education Foundation (2023). Annual Status of Education Report (Rural) 2023: Beyond Basics. New Delhi: ASER Centre. Available at: https://www.asercentre.org
2.Horwitz, E. K., Horwitz, M. B., & Cope, J. (1986). Foreign language classroom anxiety. The Modern Language Journal, 70(2), 125-132.
3.National Council of Educational Research and Training (NCERT) (2020). Position Paper on the Teaching of English. National Curriculum Framework, Ministry of Education, Government of India.
4.LiveKit Agents Framework Documentation (2026). Real-Time Multimodal Voice Agent Framework. https://docs.livekit.io/agents
5.Murf AI Falcon Streaming API (2026). Ultra Low-Latency Text-to-Speech Engine for Conversational AI. https://murf.ai/api/docs

Top comments (0)