Building Shiksha: What I Learned Creating a Real-Time AI English Coach in 10 Days

#ai #webrtc #voiceagents #buildinpublic

Building Shiksha: An AI English Coach for Indian Learners

For many Indian learners, the biggest barrier to speaking English fluently isn't a lack of vocabulary or grammar rules learned in school—it's speaking anxiety and the fear of making mistakes in front of peers or teachers.

Over the past 10 days, as part of the 10 Days of Voice Agents — Voice for Bharat Edition under the Learning & Literacy track, I built Shiksha: an interactive, real-time AI English Communication Coach designed to provide friendly, judgment-free spoken practice.

🌟 Why Voice?

Text chatbots don't build spoken confidence. Reading and typing are passive activities, whereas real-world conversations require instant auditory processing, cognitive framing, and spoken articulation.

Shiksha gives learners a low-latency, empathetic voice partner that understands Hinglish (code-mixed Hindi and English), allowing them to practice daily presentations, grammar rules, and workplace conversations without embarrassment.

🏗️ High-Level Architecture

User Speech (WebRTC / SIP) ──► LiveKit Audio Ingest
│
▼
Speech-to-Text (STT)
│
▼
LLM + Tools (agent.py + db.py)
│
▼
Murf Falcon (Ultra-Low Latency TTS)
│
▼
Audio Output ◄────────────── WebRTC Audio Sink

🚀 Key Features Built Over the 10 Days

Ultra-Low Latency Indian Voice: Powered by Murf Falcon TTS, Shiksha delivers natural, culturally resonant Indian English voice output with near-instant response times.
Persistent Conversational Memory (SQLite): Retains learner names, historical presentation goals, and specific practice needs across calls (agent_memory.db).
Curriculum-Driven Vocabulary Tools: Dynamically fetches context-specific vocabulary drills from exercises.json and evaluates sentences live.
Outbound Daily Practice Telephony (LiveKit SIP): Initiates automated daily check-in calls straight to a learner's phone.
Human-in-the-Loop Escalation & Privacy Guardrails: Detects severe learner frustration or explicit requests for human mentors, requests explicit permission, and logs sanitized support tickets with clear reference IDs.
Call Analytics Dashboard: A real-time Next.js dashboard displaying aggregated metrics (Total Calls, Successful Drills, Incomplete Calls) with zero personal transcripts exposed.
Multi-Agent Specialist Handoff: Dynamically transitions the call from Shiksha (general coach) to Arjun (Grammar Specialist with a distinct male voice persona) for complex syntactic queries without dropping the WebRTC session.

🛠️ Hardest Technical Challenges & Fixes

1. Hindi/Devanagari Pronunciation Glitches in TTS

Issue: Romanized Hindi text caused phonetic glitches in English voice models.
Fix: Structured the system prompt to output pure Hindi terms in native Devanagari script (नमस्ते!), allowing Murf Falcon to pronounce localized nuances cleanly.

2. Next.js Dashboard Real-Time Cache vs. SQLite

Issue: Call logs updated in SQLite, but the Next.js /dashboard served cached numbers.
Fix: Enforced dynamic rendering with export const dynamic = "force-dynamic" and export const revalidate = 0 at the top of the dashboard page.

3. Context Preservation During Specialist Handoff

Issue: Switching agents risked losing conversational context, requiring the user to repeat themselves.
Fix: Implemented dynamic prompt-state switching in the same LiveKit session loop, passing the handoff_reason and recent turns directly into Arjun's context.

💻 How to Run the Project Locally

1. Clone Repository & Setup Backend


bash
git clone [https://github.com/](https://github.com/)[YOUR_USERNAME]/[YOUR_REPO].git
cd shiksha-voice-agent/backend
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt


Configure Environment Variables
Create a .env.local file in both backend/ and frontend/:
LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_api_key
LIVEKIT_API_SECRET=your_api_secret
MURF_API_KEY=your_murf_falcon_api_key
OPENAI_API_KEY=your_llm_api_key

Start Backend Worker & Frontend UI

# Terminal 1 (Backend)
python agent.py dev

# Terminal 2 (Frontend)
cd ../frontend
npm install
npm run dev
Open http://localhost:3000, click Start Conversation, and begin speaking!

🔗 Links & Resources
📂 GitHub Repository: https://github.com/Spgamer0407/murf-livekit-starter_voice_agent/tree/day-10

💼 LinkedIn Profile: https://www.linkedin.com/in/srinivasa-puranik-911609369/

Built as part of the #10DaysOfVoiceAgents — Voice for Bharat Edition powered by @Murf.ai.

DEV Community