DEV Community

Ishan Singh
Ishan Singh

Posted on

Building ShikshaBharat: How I Built an AI Voice Companion for Rural Primary Education in 10 Days

#ai #python #programming #webdev

Track: Learning & Literacy

Tech Stack: Python, LiveKit Agents SDK, Murf Falcon TTS, Google Gemini, Deepgram Nova-3, Next.js, SQLite, LiveKit SIP / Twilio

Public Code Repository: github.com/murf-ai/murf-livekit-starter

1. The Problem and The Users

In many rural primary schools across India, student-to-teacher ratios frequently exceed 40:1. Young children (ages 5–12) rarely receive personalized attention when practicing reading, pronunciation, basic vocabulary, or foundational science concepts.

Key Pain Points:

  • UI Friction for Early Learners: Text-heavy graphical user interfaces fail because early learners cannot read complex UI menus, buttons, or navigation instructions.
  • Typing Barrier: Traditional text chat interfaces create immense friction for young kids who speak and converse far better than they write.
  • Linguistic & Cultural Mismatch: Existing voice assistants often use Western accents that fail to pronounce Indian names, phonics, or Hinglish phrases naturally.

2. What The Voice Agent Does

ShikshaBharat is a voice-first educational platform designed to remove all UI friction. Children simply talk naturally in English, Hindi, or Hinglish, receive instant spoken feedback in a warm Indian accent, and view real-time visual learning cards on an interactive digital classroom chalkboard.

Two Dedicated AI Personas:

  1. Aarvi (Literacy Buddy): A warm, encouraging female AI companion (voiced by Murf Falcon Anisha) who guides students through reading passages, dictionary definitions, phonics, and guided spelling.
  2. Vigyan Buddy (Science Specialist): An enthusiastic male AI agent (voiced by Murf Falcon Karan) who handles science experiments, space, nature, and interactive Q&A.

3. How The System Works

ShikshaBharat uses an end-to-end real-time audio pipeline built with LiveKit Agents SDK, Murf Falcon TTS, Google Gemini LLM, and Deepgram Nova-3 STT.

Architecture Overview 

flowchart LR
    A[🎙️ User Speaks] -->|Audio Stream| B[Deepgram STT Nova-3]
    B -->|Transcribed Text| C[Google Gemini LLM]
    C -->|Tool Selection / Prompt| D[Murf Falcon TTS Streaming]
    D -->|Audio Stream| E[LiveKit WebRTC Room]
    E -->|Audio Output| F[🔊 User Hears Agent]
    C -.->|Data Packets| G[📺 Next.js UI Chalkboard]
Enter fullscreen mode Exit fullscreen mode

Core Pipeline Code (backend/src/agent.py) 

session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=google.LLM(model="gemini-3.5-flash-lite"),
    tts=murf.TTS(
        voice="Anisha",
        locale="en-IN",
        style="Conversation",
        tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=2),
        text_pacing=True,
    ),
    turn_detection=MultilingualModel(),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=True,
)
Enter fullscreen mode Exit fullscreen mode

4. The Most Important Features

Here is a breakdown of the core features built across the 10-day sprint:

🎙️ 1. Ultra-Fast Indian Voice Powered by Murf Falcon

  • Voices: Anisha (Warm Female) and Karan (Enthusiastic Male).
  • Performance: Murf Falcon streaming TTS delivers 55ms model latency and ~130ms time-to-first-audio, enabling natural, conversational turn-taking without awkward delays.

🗣️ 2. Indian Language & Hinglish Code-Mixing

  • Naturally handles code-mixed conversations (English + Hindi). Hindi words are rendered in Devanagari script (e.g., "नमस्ते! आज हम new words practice करेंगे") so Murf Falcon synthesizes native phonetic accents.

📺 3. Interactive Frontend Chalkboard & State Visualizer

  • Custom LiveKit data packets (RoomEvent.DataReceived) broadcast state updates to the Next.js frontend (ExerciseCard.tsx), rendering real-time chalkboard cards for reading passages, dictionary definitions, and teacher handoffs.

🧠 4. Returning User Memory & Privacy Consent Guardrails

  • SQLite Persistence (db.py): Remembers student name, reading level (Beginner, Intermediate, Advanced), and previous practice topics.
  • Consent Guardrail: Aarvi asks for explicit verbal permission before saving profile data or executing a "Forget Me" deletion request (forget_caller_info).

🛠️ 5. Real-Time Domain Tools & Tool Chaining

  • fetch_next_exercise: Retrieves graded reading passages. If level is omitted, automatically queries the student's stored SQLite profile (Tool Chaining).
  • lookup_word_definition: Fetches live dictionary definitions, IPA phonetics, and sample sentences.

📞 6. Outbound Practice Telephony Calls (SIP)

  • Uses LiveKit SIP & Twilio to initiate daily outbound practice nudge calls (outbound.py). Aarvi speaks first upon connection: "नमस्ते Rahul! This is Aarvi calling from ShikshaBharat..."

🚨 7. Consent-Guarded Human Teacher Escalation

  • Triggers when a student is upset or asks for a real teacher. Requests explicit consent before creating a reference ID (e.g., ESC-849201) and dispatching an automated email notification (email_service.py).

📊 8. Real-Time Call Analytics Dashboard

  • Accessible via /dashboard route in Next.js. Tracks Total Calls, Success Rate %, and call history logs with automatic privacy masking.

🔀 9. Multi-Agent Specialist Handoffs

  • Bi-directional handoffs between Aarvi and Vigyan Buddy (session.update_agent() and session._tts = murf.TTS(voice="Karan")) while updating the UI chalkboard in real time.

5. Challenges and How I Overcame Them

Building a real-time voice agent presents unique engineering challenges. Here are three major hurdles faced and solved:

Challenge 1: TTS Spoken Output Mismatching UI Chalkboard Text

  • The Cause: When fetch_next_exercise fetched a reading passage string for the LLM, the LLM paraphrased the passage when speaking it—causing the student to hear text different from what was displayed on screen.
  • The Solution: Implemented direct audio rendering inside the tool function using await session.say(passage, allow_interruptions=False) to speak the exact passage text, bypassing LLM paraphrasing.

Challenge 2: Audio Handshake Delays on Outbound Telephony (SIP)

  • The Cause: On outbound phone calls over Twilio/SIP, the media channel takes ~1–2 seconds to establish the RTP audio stream after answering, chopping off the first 2 seconds of the agent's greeting.
  • The Solution: Added a 2.0-second media settling delay (await asyncio.sleep(2.0)) specifically for outbound SIP sessions before triggering the greeting.

Challenge 3: Dynamic Voice & Agent Handoff Without Disconnection

  • The Cause: Swapping personas (Aarvi ↔ Vigyan Buddy) required changing both system prompts and TTS voices (Anisha ↔ Karan) inside an active LiveKit WebRTC room without disconnecting the caller.
  • The Solution: Called session.update_agent(ScienceAgent()) and re-instantiated session._tts = murf.TTS(voice="Karan") while publishing a LiveKit data packet (topic="agent_handoff") to update the frontend UI simultaneously.

6. How Readers Can Build and Run It

Step 1: Prerequisites

Step 2: Environment Variables

Create .env.local in backend/ and frontend/:

backend/.env.local:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key
Enter fullscreen mode Exit fullscreen mode

frontend/.env.local:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
AGENT_NAME=my-agent
Enter fullscreen mode Exit fullscreen mode

Step 3: Launch Backend & Frontend 

# Terminal 1: Backend Agent
cd backend
uv sync
uv run python src/agent.py dev

# Terminal 2: Frontend UI
cd frontend
pnpm install
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000 in your browser and click Start talking!

7. What I Would Improve Next

Looking ahead, here are three key enhancements planned for ShikshaBharat:

  1. Edge Offline Mode: Integrate lightweight local STT and TTS models for deployment on low-cost hardware in classrooms with zero internet connectivity.
  2. Expanded Regional Languages: Add support for regional Indian languages including Tamil, Telugu, Bengali, Kannada, and Marathi using localized Murf voices.
  3. Gamified Progress Badges: Introduce star badges, daily streaks, and printable achievement cards directly on the classroom chalkboard.

Check out the full source code and star the repo on GitHub:

👉 github.com/ishan200716/murf-livekit-starter

Top comments (0)