DEV Community

Ilhan Mansij Jidda
Ilhan Mansij Jidda

Posted on

Shiksha AI & Mitra AI: Real-Time Spoken English & Roleplay Voice Agent for Bharat

#agents #ai #education #showdev

Created for the Voice for Bharat Challenge 2026 — 10 Days of Voice Agents Challenge

Powered by **Murf Falcon* (the fastest TTS API), LiveKit Agents, Deepgram Nova-3, and Google Gemini.

1. The Problem and the Users

For millions of learners across India—students, job seekers, non-native English speakers, and young professionals—learning to speak English fluently is one of the most impactful skills for career and educational growth.

However, traditional language learning platforms focus primarily on reading comprehension, flashcards, and written grammar quizzes. When learners actually try to speak, they face two major barriers:

  1. Fear of Judgment & Social Anxiety: Learners feel self-conscious making grammar or pronunciation mistakes in front of peers or native speakers.
  2. Lack of Conversational Practice Partners: Most learners lack daily access to an English-speaking partner for real-life conversations like ordering food, asking for street directions, or attending job interviews.

Under the Learning & Literacy track, I built Shiksha AI (a patient, encouraging main tutor) and Mitra AI (a real-life scenario roleplay specialist) to provide a 24/7, judgment-free, voice-first English practice companion.

2. What the Voice Agent Does

Shiksha AI & Mitra AI provide an interactive, end-to-end voice experience over both Web Browsers and Telephone/SIP calls:

  • Empathetic Main Tutor (Shiksha AI): Greets the learner, conducts turn-by-turn spoken practice, models correct grammar gently, and never shames or criticizes mistakes.
  • Dynamic Specialist Handoff (Mitra AI): When the learner asks to practice a real-life situation (ordering at a fast-food restaurant, asking for metro directions, buying groceries, doctor visits), Shiksha AI instantly transfers the call live to Mitra AI, who steps into character for interactive roleplay.
  • Hinglish & Native Script Support: Seamlessly understands code-mixed Indian English (Hinglish) and writes non-English words in native Devanagari script (e.g., नमस्ते).
  • Live Knowledge Tools: Fetches real-time word definitions (Free Dictionary API) and analyzes spoken grammar (LanguageTool Engine).
  • Outbound Telephony (SIP Calls): Schedules and places automated daily practice calls directly to a user's SIP phone address (e.g. Linphone) using a background scheduler.
  • Human Escalation & Persistent Memory: Remembers returning learners across calls and logs human teacher support tickets whenever a learner feels overwhelmed.

3. How the System Works

The system connects real-time WebRTC audio streaming to a Python AI pipeline powered by LiveKit Agents and Murf Falcon TTS:

graph TD
    User([Learner - Web Browser / Linphone SIP]) <--> NextJS[Next.js 15 Frontend & LiveKit UI]
    NextJS <--> LiveKit[LiveKit Cloud WebRTC & SIP Gateway]
    LiveKit <--> Agent[Python LiveKit Agent Worker Process]

    subgraph Core Voice Pipeline
        Agent <--> STT[Deepgram Nova-3 Multi-Language STT]
        Agent <--> LLM[Google Gemini 2.5 Flash / Groq Llama 3.3 70B]
        Agent <--> TTS[Murf Falcon TTS - Anisha & Samar Voices]
    end

    subgraph Tools & Data Storage
        Agent <--> DictAPI[Live Free Dictionary API]
        Agent <--> GrammarAPI[LanguageTool Grammar Engine]
        Agent <--> DB[(SQLite Memory & Call Analytics DB)]
    end
Enter fullscreen mode Exit fullscreen mode

Core Architecture Components:

  • Speech-to-Text (STT): Deepgram Nova-3 (language="multi") for ultra-low latency Indian English & multi-lingual speech transcription.
  • Brain (LLM): Google Gemini 2.5 Flash / Groq Llama 3.3 70B for fast conversational reasoning and tool calling.
  • Text-to-Speech (TTS): Murf Falcon TTS—the fastest text-to-speech API—using Anisha (Conversational Indian English) for Shiksha AI and Samar (Energetic Male Voice) for Mitra AI.
  • Real-Time Transport: LiveKit Agents SDK 1.4 for sub-second WebRTC audio streaming, VAD turn detection, and LiveKit SIP Trunking for telephony calls.

4. The Most Important Features Built

Here are the key highlights built across my 10-day sprint:

Authentic Indian Voices with Murf Falcon

Using Murf Falcon's streaming API, my agent speaks with natural Indian accent rhythm, natural pauses (text_pacing=True), and sentence-level tokenization for zero perceptible lag.

Safety Guardrails & Cultural Rules

  • Never Shame: Always praise effort enthusiastically before modeling corrections.
  • Never Diagnose: Strictly forbidden from diagnosing learning or cognitive deficits.
  • Hard Refusals: Polite escalation refusal scripts for medical, legal, or financial queries.
  • Native Script Mandate: Non-English words are written in their native script (Hindi → Devanagari नमस्ते), avoiding romanized transliterations.

Live Specialist Agent Handoff (Day 9)

When a user says "Let's practice ordering food at a fast-food restaurant", Shiksha AI triggers @function_tool async def transfer_to_scenario_specialist(...). LiveKit session switches live to Mitra AI (Samar voice), who opens in character. Once done, return_to_main_tutor() returns the call to Shiksha AI cleanly!

@function_tool
async def transfer_to_scenario_specialist(self, context: RunContext, scenario_type: str) -> Agent:
    """Seamlessly transfer the active call to Mitra AI for scenario roleplay."""
    await context.session.say("Sounds fun! I will connect you to Mitra AI, our real-life scenario specialist.")
    return ScenarioSpecialist(
        room=self.room,
        participant_name=self.participant_name,
        scenario_type=scenario_type,
        parent_assistant=self,
        http_session=self.http_session,
    )
Enter fullscreen mode Exit fullscreen mode

Scheduled Outbound Telephony & Background Scheduler (Day 6)

Learners can schedule automated daily practice calls via an interactive Next.js modal. A Python background daemon polls SQLite and dispatches LiveKit SIP calls directly to the learner's Linphone address (sip:username@sip.linphone.org).

5. Challenges and How I Overcame Them

Building a real-time voice system comes with real production hurdles. Here are three major issues I faced and solved:

Challenge 1: SIP 486 Busy Here Caller ID Loopback Rejection

  • The Problem: When dialing a Linphone SIP address, Linphone rejected incoming calls with INVITE failed: sip status 486: Busy here.
  • Root Cause: Setting sip_number (Caller ID) to match the target username (user1) caused Linphone to treat the call as a self-loopback call.
  • Solution: Set sip_call_to to user1 while setting sip_number to a distinct caller ID (+18885550199 / Shiksha AI). Linphone now rings cleanly every time!

Challenge 2: HTTP Session Leaks Across Agent Handoffs

  • The Problem: Handoffs to Mitra AI hung because Murf Falcon TTS in the new specialist agent lacked the active aiohttp.ClientSession.
  • Solution: Explicitly passed http_session across OutboundAssistant and ScenarioSpecialist constructors, allowing Murf Falcon TTS to reuse existing network connections effortlessly.

6. How Readers Can Build and Run It

Want to build your own voice agent? Follow these steps to get running in minutes:

Step 1: Clone the Repository 

git clone https://github.com/Cheese-Cheese/voice-for-bharat-challenge-2026.git
cd voice-for-bharat-challenge-2026
Enter fullscreen mode Exit fullscreen mode

Step 2: Configure Environment Variables

Copy .env.example to backend/.env.local:

LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_gemini_api_key

SIP_TRUNK_ID=your_livekit_sip_trunk_id
MY_SIP_URI=sip:your_username@sip.linphone.org
Enter fullscreen mode Exit fullscreen mode

Step 3: Run the Backend Worker & Scheduler 

cd backend
uv sync
uv run python src/agent.py dev        # Main WebRTC Voice Worker
uv run python src/scheduler.py        # Outbound Call Scheduler
Enter fullscreen mode Exit fullscreen mode

Step 4: Run the Next.js Frontend 

cd frontend
pnpm install
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000 to start talking to your voice agent!

7. What I Would Improve Next

  1. Multilingual Speech Output: Extend Murf Falcon TTS voices across regional Indian languages (Hindi, Tamil, Telugu, Marathi).
  2. Pronunciation Pitch & Phoneme Scoring: Real-time visual feedback showing exact syllable stress and pitch contours during practice.
  3. WhatsApp / SMS Practice Digest: Automatically sending a post-call summary of new vocabulary & grammar rules practiced directly to the learner's phone.

8. Links to Code & Resources

Top comments (0)