DEV Community

Cover image for Building Mo Saathi: An Odia-First AI Voice Learning Companion
Swayam Jethi
Swayam Jethi

Posted on

Building Mo Saathi: An Odia-First AI Voice Learning Companion

#ai #opensource #webdev #voiceai

TL;DR: What is Mo Saathi?

Feature Description
The Problem Students in Odisha lack digital learning tools in their native language (Odia).
The Solution Mo Saathi (ମୋ ସାଥୀ), an Odia-first AI voice learning companion.
Core Stack LiveKit (WebRTC/SIP), Sarvam AI (Odia STT), Google Gemini (LLM), Murf Falcon (TTS).
Key Features Persistent Memory, RAG (Syllabus), Practice Tools, Outbound Phone Calls, Teacher Escalation, Specialist Handoff.
Target Audience Odia-medium school students needing accessible, conversational tutoring.

Table of Contents

  1. The Problem and the Users
  2. How the System Works
  3. The 9-Day Engineering Journey
  4. Challenges and How I Overcame Them
  5. How You Can Build and Run It
  6. What I Would Improve Next
  7. Links and Resources

The Problem and the Users

A student should not have to translate their question into English before they can learn from an AI tutor. In real conversations, language is rarely perfectly formal. Students naturally mix Odia with English terms like Physics, Carbon, or Photosynthesis.

Most educational AI tools treat regional Indian languages as an afterthought, relying on slow, generic translation APIs. Mo Saathi was designed around an Odia-first interaction model for students in Odisha.

The goal was simple: A student says, "ଆଜି ଆମେ Physics ବିଷୟରେ ପଢ଼ିବା?", and the system understands it natively, responds conversationally, and teaches the concept without forcing the student to change how they speak.

How the System Works

Mo Saathi is split into two layers to separate real-time transport from application logic.

  1. The Client Layer (Next.js): A distraction-free, hand-drawn "pencil box" aesthetic interface.
  2. The Agent Layer (Python): A LiveKit Agents backend coordinating the STT, LLM, TTS, tools, and local SQLite memory.

Mo Saathi System Architecture

The final Mo Saathi architecture, showing the real-time voice layer, AI services, memory, tools and human support. View the architecture in HD
Stage Technology Role
Input LiveKit WebRTC Captures real-time student audio.
Transcription Sarvam AI Natively transcribes code-mixed Odia/English instantly.
Reasoning Gemini 3.5 Flash Lite Processes context, decides to explain, use a tool, or hand off.
Synthesis Murf Falcon (Anisha) Streams ultra-low latency conversational Odia speech back to the student.

The 9-Day Engineering Journey

I built Mo Saathi as part of the 10 Days of Voice Agents — VoiceForBharat Edition. Here is how it evolved from a simple script into a stateful educational platform.

Day 1–3: The Voice Pipeline and Persona

The first hurdle was simply making the agent hear and speak Odia naturally. I integrated Sarvam AI for native Indic speech-to-text, and Murf Falcon for blazing-fast speech generation. I crafted a strict system prompt to ensure the agent acted as a tutor—explaining concepts step-by-step rather than just behaving like a search engine. I also built the Next.js frontend with a hand-drawn, paper-like UI.

Mo Saathi Interface

Day 4: Memory and RAG (Grounding the AI)

A tutor is useless if it forgets you or hallucinates facts.
I implemented a local SQLite database to persist student profiles across sessions. When a student joins, the agent recalls their name and past struggles. To prevent hallucinations, I built a local RAG pipeline (sentence-transformers) that injects actual Class 9 and 10 Science and Maths syllabus content into the LLM context before it answers.

Day 5: Interactive Tools (Practice Exercises)

I didn't want the LLM hallucinating math problems. Instead, I gave the agent a get_next_exercise function tool. Now, when a student asks for practice, the LLM triggers the tool, fetches a structured question from a local question bank, and evaluates the student's spoken answer dynamically.

Day 6: Outbound SIP Phone Calls (Study Reminders)

Learning shouldn't depend on opening a web browser. I integrated LiveKit's SIP outbound trunking. A student can ask, "Remind me to study at 5 PM." A background Python scheduler polls the SQLite database, dials the student's actual phone number using Linphone, and greets them by name.
(See the live phone call demonstration on my LinkedIn post).

Day 7: Human Escalation (Knowing when to stop)

AI isn't perfect. If a student expresses emotional distress or repeatedly fails an exercise, the agent halts the lesson. It asks for consent and triggers a create_escalation tool. This fires a colored-priority email to a human teacher via the Resend API, keeping human oversight in the loop.

Day 8: Analytics and Call Outcomes

It’s easy to track if an AI talked, but did it teach? I built a real-time Next.js /analytics dashboard. A call is only marked "Successful" in the database if the student actively attempted a practice exercise via the function tool.

Analytics Dashboard

Day 9: Specialist Handoff (Vigyan Saathi)

A general-purpose prompt struggles with deep Physics or Biology nuances. So, I introduced a multi-agent handoff. If a student asks a complex science question, Mo Saathi (Anisha) announces a transfer: "I am connecting you to our science expert, Vigyan Saathi."

The system hot-swaps the active agent, passing the exact conversation context so the student doesn't have to repeat themselves. The new agent switches to a different Murf Falcon voice (Samar) and operates under a strict, science-focused prompt.

@function_tool
async def transfer_to_science_specialist(self, context: RunContext) -> tuple[Agent, str]:
    # Pass a copy of the active conversation matrix to the new agent
    science_agent = ScienceSpecialist(
        chat_ctx=self.chat_ctx.copy(exclude_instructions=True)
    )
    return science_agent, "ବିଜ୍ଞାନ ସାଥୀ ସହ ଯୋଡ଼ୁଛି..."
Enter fullscreen mode Exit fullscreen mode

Challenges and How I Overcame Them

1. Context and Turn Ordering Rules
Gemini is incredibly strict about turn order. When injecting RAG context, if the LLM then attempted to call a tool (like the handoff or memory tool), Gemini would reject it with a 400 Bad Request if the message history didn't strictly alternate between User and Model.
Solution: I had to refactor the RAG pipeline to dynamically append the retrieved textbook context directly into the current User message payload, rather than appending hidden system messages to the history.

2. Speech Output vs. Code-Mixed Text
The LLM would sometimes generate text with English translations inside brackets (e.g., "ବାଷ୍ପୀଭବନ (Evaporation)"). While correct as text, Murf TTS would read both languages awkwardly, breaking the conversational flow.
Solution: I strictly prompt-engineered the model to forbid bracketed translations, forcing it to pick one script and stick to it so the audio sounded like a natural human speaking.

How You Can Build and Run It

You can run this entire multi-agent system locally. You will need API keys for LiveKit, Google Gemini, Murf AI, Sarvam AI, and Resend.

1. Start the Backend

The backend runs on Python. Use uv for lightning-fast dependency resolution.

git clone https://github.com/Swayam42/voice-agent-for-bharat-2026.git
cd voice-agent-for-bharat-2026/backend

cp .env.example .env.local
# Add your API keys to .env.local (Never commit this file!)

uv sync
uv run python src/agent.py dev
Enter fullscreen mode Exit fullscreen mode

2. Start the Frontend

The client is a Next.js app.

cd ../frontend

cp .env.example .env.local
# Add your LIVEKIT_URL and LIVEKIT_API_KEY

pnpm install
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Visit http://localhost:3000 to talk to Mo Saathi, and http://localhost:3000/analytics to view real-time call outcomes.

What I Would Improve Next

  • Offline / Low-Bandwidth Mode: Students in rural Odisha shouldn't be blocked by poor internet. I'd like to implement local caching for practice exercises so the learning loop can survive network drops.
  • More Specialists: Expand the multi-agent system to include a dedicated Maths specialist and an English grammar specialist.
  • Teacher Insights: Enhance the escalation dashboard to highlight why the student is struggling, using the RAG history to pinpoint knowledge gaps.

Links and Resources

Building a voice agent is easy. Building a useful learning companion takes memory, safety boundaries, and an understanding of how people actually speak. That is what I want to keep building.

#VoiceForBharat #MurfAI #10DaysofAIVoiceAgents #LiveKit #AI #EdTech #BuildInPublic

Top comments (0)