DEV Community

Cover image for Building BharatGuru AI: My Journey Creating an Intelligent Multilingual Voice Tutor with Murf Falcon and LiveKit
Md Fahad
Md Fahad

Posted on

Building BharatGuru AI: My Journey Creating an Intelligent Multilingual Voice Tutor with Murf Falcon and LiveKit

#ai #livekit #murfai #voiceai

Building BharatGuru AI: My Journey Creating an Intelligent Multilingual Voice Tutor with Murf Falcon and LiveKit

For the past ten days, I participated in the 10 Days of Voice Agents – VoiceForBharat Edition challenge. Instead of building just another chatbot, I wanted to create something that could genuinely help learners through natural conversations.

The result was BharatGuru AI—a multilingual voice tutor that remembers learners, adapts to conversations, provides practice questions, escalates difficult situations to humans, tracks analytics, and even switches into a Literature Specialist mode when needed.

This article shares what I built, how the system works, the challenges I faced, and how anyone can build a similar voice agent.

Why BharatGuru AI?

Millions of students in India are more comfortable speaking than typing.

Many learners hesitate to ask questions in English or struggle with complicated educational websites.

I wanted to build an AI tutor that feels like talking to a friendly mentor instead of using another search engine.

The goals were simple:

  • Help students learn through conversation.
  • Support English, Hindi, and code-mixed speech.
  • Personalize future conversations with memory.
  • Make learning accessible through voice.

Voice makes learning much more natural because users don't need to type long questions or navigate complex interfaces.

Tech Stack

My project combines several AI services into one real-time voice pipeline.

Component Technology
Real-time Transport LiveKit
Speech-to-Text Deepgram Nova-3
Language Model Google Gemini
Text-to-Speech Murf Falcon
Voice Activity Detection Silero
Turn Detection LiveKit Multilingual Model
Memory SQLite
Backend Python

System Architecture 

User
   │
   ▼
Microphone
   │
   ▼
Deepgram STT
   │
   ▼
Gemini LLM
   │
   ├──────────────► Memory
   │
   ├──────────────► Learning Tools
   │
   ├──────────────► Analytics
   │
   ├──────────────► Human Escalation
   │
   └──────────────► Specialist Mode
   │
   ▼
Murf Falcon TTS
   │
   ▼
User
Enter fullscreen mode Exit fullscreen mode

Feature 1 — Natural Indian Voice using Murf Falcon

One of my favorite parts of the project was integrating Murf Falcon.

Instead of robotic responses, BharatGuru AI speaks naturally and conversationally.

The response latency is low enough that conversations feel smooth instead of delayed.

Using Murf Falcon was surprisingly simple inside LiveKit.

tts = murf.TTS(
    voice="Samar",
    style="Conversation",
    text_pacing=True,
)
Enter fullscreen mode Exit fullscreen mode

The conversational style makes a huge difference in user experience.

Feature 2 — Multilingual Conversations

India is multilingual.

Instead of forcing users to speak one language, BharatGuru AI automatically mirrors the user's language.

Examples include:

  • English
  • Hindi
  • Hindi-English mixed conversations

Users can naturally switch between languages without changing settings.

Feature 3 — Persistent Memory

Returning users shouldn't have to introduce themselves every time.

I built a memory system that stores information only after explicit user permission.

It remembers things like:

  • learner's name
  • preferred language
  • learning level
  • completed topics
  • common mistakes

The assistant can later personalize learning automatically.

Example:

User:
Give me another programming question.

Assistant:
Welcome back! Here's another beginner programming question.
Enter fullscreen mode Exit fullscreen mode

Privacy was important, so memory is never stored without consent.

Feature 4 — Real Learning Questions

Instead of generating random quizzes, BharatGuru AI fetches real questions from the Open Trivia Database.

The tool selects questions based on:

  • topic
  • difficulty level

Example tool call:

result = get_learning_question(
    topic="computers",
    level="beginner"
)
Enter fullscreen mode Exit fullscreen mode

If the API is unavailable, the assistant responds naturally instead of exposing technical errors.

Feature 5 — Human Escalation

Sometimes AI isn't enough.

If a learner becomes frustrated or explicitly asks for a human teacher, BharatGuru AI creates an escalation request.

Before sharing any information, the assistant asks for permission.

Only after receiving consent does it create an escalation record.

This was one of the most useful safety features I implemented.

Feature 6 — Call Analytics

I wanted to know whether conversations were actually useful.

So every conversation records:

  • call duration
  • success
  • failure reason
  • tool failures

This allows future improvements based on real usage instead of assumptions.

Feature 7 — Literature Specialist Mode

One feature I was especially excited about was adding a Literature Specialist.

Originally, I planned to switch between two different LiveKit agents.

Later I discovered that LiveKit 1.4.5 doesn't support true runtime agent handoffs.

Instead of giving up, I redesigned the architecture.

Rather than switching agents, BharatGuru AI changes its internal mode.

current_specialist = "main"
Enter fullscreen mode Exit fullscreen mode

When a literature-related question appears:

current_specialist = "literature"
Enter fullscreen mode Exit fullscreen mode

The assistant announces:

"I'm transferring you to our Literature Specialist."

Internally, it's still the same agent—but now it answers only literature questions.

If the learner later asks about coding or careers:

"I'm returning you to BharatGuru AI."

From the learner's perspective, it feels like talking to two different experts even though only one agent is running.

Sometimes good engineering means adapting to platform limitations instead of fighting them.

Biggest Challenges

Runtime Agent Switching

This was easily the hardest problem.

I initially tried using:

await session.set_agent(...)
Enter fullscreen mode Exit fullscreen mode

Unfortunately, LiveKit 1.4.5 doesn't support runtime handoffs the way I expected.

After several debugging sessions, I redesigned the feature into a specialist mode using internal state.

The final solution was actually simpler and more reliable.

Managing Long Prompts

As the project grew, the system prompt became very large.

It included:

  • guardrails
  • memory rules
  • language behavior
  • escalation
  • specialist instructions

Keeping everything organized required separating responsibilities into dedicated modules.

Voice Latency

Real-time conversations need quick responses.

Balancing response quality with low latency required adjusting:

  • Gemini temperature
  • token limits
  • Murf pacing
  • Deepgram settings

Project Structure 

project/
│
├── agent.py
├── memory.py
├── analytics.py
├── escalation.py
├── learning_tools.py
├── handoff.py
├── literature_agent.py
└── requirements.txt
Enter fullscreen mode Exit fullscreen mode

Splitting the project into modules made debugging much easier.

How to Run the Project

Clone the repository.

git clone https://github.com/Fahad035/murf-livekit-starter
Enter fullscreen mode Exit fullscreen mode

Install dependencies.

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Create a .env.local file.

LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=

GOOGLE_API_KEY=

DEEPGRAM_API_KEY=

MURF_API_KEY=
Enter fullscreen mode Exit fullscreen mode

Never commit API keys to GitHub.

Start the agent.

python agent.py dev
Enter fullscreen mode Exit fullscreen mode

Open the LiveKit frontend and begin talking to BharatGuru AI.

What I Learned

Before this challenge, I thought voice AI was mostly about speech recognition.

Now I understand that a great voice assistant is really a combination of many systems working together:

  • speech recognition
  • language understanding
  • memory
  • tools
  • safety guardrails
  • analytics
  • user experience

Each component matters.

Future Improvements

There are many ideas I'd like to explore next.

  • Better emotional understanding
  • More specialist domains
  • Richer dashboards
  • Retrieval-Augmented Generation (RAG)
  • Regional language expansion
  • Parent and teacher portals

Repository

GitHub Repository

👉 https://github.com/Fahad035/murf-livekit-starter

Final Thoughts

Building BharatGuru AI over these ten days taught me much more than how to connect APIs.

It taught me how to design conversations, think about user trust, handle failures gracefully, and build systems that feel genuinely helpful.

Voice AI is becoming more accessible every day, and tools like LiveKit, Deepgram, Google Gemini, and especially Murf Falcon make it possible to create production-quality conversational experiences much faster than I imagined.

A huge thank you to Murf AI for organizing the 10 Days of Voice Agents – VoiceForBharat Edition challenge. It was an enjoyable and practical way to explore what modern voice AI can do.

If you're considering building a voice agent, my advice is simple:

Start small, keep conversations natural, and iterate based on real user interactions.

Thank you for reading!

Happy building! 🚀

Useful Links

Top comments (0)