Building Bhasha Academy: A Multi-Agent Hinglish Voice Tutor with Murf Falcon & LiveKit

Bharat Lashotra — Sat, 15 Aug 2026 08:51:09 +0000

Building Bhasha Academy: An AI Voice Tutor for English and Math

Language is meant to be spoken, not just read.

For millions of students and job seekers in India, being confident in English and basic math can open the door to better career opportunities. But many learners don't have access to a personal tutor. Even when they do, the fear of making mistakes in front of someone can make practicing uncomfortable.

So I built Bhasha Academy, an AI-powered voice tutor designed to give learners a private, judgment-free space to practice.

It can have conversations in Hinglish, help users practice English, solve basic math problems, remember learning progress, and even connect the learner with a human teacher when needed.

The interesting part is that the entire experience is voice-first.

Here's how I built it.

The idea

I wanted Bhasha Academy to feel less like talking to a chatbot and more like talking to a patient tutor.

The learner should be able to simply say:

"Mujhe English practice karni hai."

And the agent should understand the Hinglish, respond naturally, and continue the conversation.

For math, the learner can say:

"Samar, 25 percent of 200 kitna hai?"

The system can then switch to a math specialist and continue from the same conversation.

The main goals were:

Natural voice conversations
Hinglish support
Indian-accented voices
Personalized learning
Persistent memory
Specialist agents
Human escalation when required
Web and phone/SIP support

How the architecture works

The system is built using several services, with LiveKit sitting at the center of the real-time voice experience.

User
  ↓
Browser / SIP
  ↓
LiveKit
  ↓
Speech-to-Text
  ↓
Deepgram Nova-3
  ↓
Gemini
  ↓
Agent / Tools
  ↓
Murf Falcon TTS
  ↓
LiveKit
  ↓
User

There are also a few supporting components:

SQLite
  ├── Student profiles
  ├── Learning progress
  ├── Call analytics
  └── Escalation tickets

Discord Webhook
  └── Human teacher notifications

Next.js
  └── Admin dashboard

Speech-to-Text

I use Deepgram Nova-3 for speech recognition.

The important part here is multilingual support.

A learner might say:

"Yesterday I went market, but mujhe wahan kuch samajh nahi aaya."

The system needs to understand both languages without forcing the user to speak only English or only Hindi.

LLM

For the conversational layer, I use Google Gemini 3.5 Flash Lite.

The model is responsible for:

Understanding the conversation
Generating responses
Following the tutor personality
Calling tools
Managing agent handoffs
Working with the learner's profile

Text-to-Speech

For voice generation, I use Murf Falcon TTS through LiveKit.

Bhasha Academy currently uses two voices:

Anisha — general English and language practice
Samar — math practice

Both are designed to sound natural for Indian users.

Real-time voice

LiveKit Agents SDK handles the real-time communication layer.

It takes care of things like:

WebRTC audio
Voice activity detection
Turn detection
Streaming audio
SIP integration
Agent sessions

This makes it possible to have a real conversation instead of the traditional:

Record → Upload → Wait → Get Response → Play

Instead, audio can be streamed continuously.

One of the biggest problems: Hinglish pronunciation

One of the first problems I noticed was surprisingly simple.

The AI understood Hinglish perfectly, but the voice didn't always pronounce it naturally.

For example, if the model generated:

Bahut achha, let's try again!

the TTS system could interpret "Bahut achha" using English pronunciation.

The result sounded robotic.

The solution

I changed the system prompt so that Hindi words should be written in Devanagari.

Instead of:

Bahut achha, let's try again!

the model generates:

बहुत अच्छा, let's try again!

This made a huge difference.

Murf Falcon can handle mixed scripts, so it can naturally switch between Hindi and English.

For example:

बहुत बढ़िया! Let's try another word.

This is a small prompt change, but it had a big impact on the voice experience.

Giving the AI memory

A tutor should remember its students.

Bhasha Academy uses SQLite to store learner profiles.

When someone calls for the first time, the agent doesn't automatically save their information.

Instead, it asks for permission.

For example:

"Would you like me to remember your name and learning progress for your next session?"

If the learner agrees, the system can store information such as:

Name
Current English level
Topics practiced
Common mistakes
City
Learning progress

There is also a forget_caller tool.

A learner can ask the system to forget them, and their stored profile can be removed.

This was important to me because personalization shouldn't come at the cost of user control.

Multi-agent conversations

Another interesting part of the project is the specialist handoff system.

Instead of making one huge agent handle everything, I created separate agents.

Anisha

Anisha is the general language tutor.

She handles:

English conversations
Vocabulary
Grammar
Pronunciation
General learning

Samar

Samar is the math specialist.

He handles:

Basic arithmetic
Percentages
Word problems
Math practice

If the learner says:

"Can we do some maths?"

Anisha can transfer the conversation to Samar.

The important part is that Samar shouldn't start from zero.

He should already know who the learner is and what was discussed.

The handoff looks roughly like this:

@function_tool
async def transfer_to_math_specialist(
    self,
    context: RunContext
) -> tuple[Agent, str]:

    math_agent = MathPracticeAgent(
        chat_ctx=self.chat_ctx.copy(
            exclude_instructions=True
        )
    )

    return (
        math_agent,
        "Transferring you to Samar, our maths practice specialist."
    )

The key idea is passing a copy of the existing ChatContext.

This preserves the conversation while allowing the new agent to have its own instructions and personality.

Phone calls and voicemail detection

Bhasha Academy isn't limited to the browser.

Using LiveKit SIP integrations, the system can also make outbound calls.

This opens up the possibility of scheduled lessons.

For example:

Scheduled lesson
      ↓
System calls student
      ↓
Student answers
      ↓
AI tutor starts lesson

But there is another problem with outbound calling: voicemail.

There is no reason to keep expensive AI services running when the call is answered by an answering machine.

So I added voicemail detection.

If the system detects a typical voicemail greeting, it can:

Leave a short message.
Call the hang_up tool.
End the session.

Human teachers are still important

AI shouldn't try to solve every problem.

If a learner repeatedly struggles or becomes frustrated, Bhasha Academy can ask whether they want to speak with a human teacher.

If the learner agrees, the system creates an escalation ticket.

The ticket contains information such as:

Reference ID
Student Name
Urgency
Reason

A Discord webhook then sends the notification to the teacher/admin channel.

For example:

payload = {
    "embeds": [{
        "title": f"🚨 Human Help Request Raised ({urgency.upper()})",
        "color": 15158332 if urgency == "emergency" else 3447003,
        "fields": [
            {
                "name": "Reference ID",
                "value": ref_id,
                "inline": True
            },
            {
                "name": "Student Name",
                "value": name,
                "inline": True
            },
            {
                "name": "Urgency",
                "value": urgency,
                "inline": True
            },
            {
                "name": "Reason",
                "value": reason,
                "inline": False
            }
        ]
    }]
}

The idea is simple:

AI handles the routine conversations. Humans step in when the learner needs more help.

Analytics dashboard

Every call is also logged.

The system tracks things like:

Call status
Call duration
Web vs SIP
Latency
Completion rate

The data is stored in SQLite and exposed through a Next.js dashboard.

This makes it easier to understand how the system is performing instead of relying only on individual conversations.

The biggest lessons I learned

1. Voice AI is more than just an LLM

Getting a good text response is only one part of the problem.

A voice agent also needs:

Good speech recognition
Fast turn detection
Natural TTS
Low latency
Interruption handling
Context management

Small improvements in any of these areas can make the experience feel much more natural.

2. Language and script matter

Hinglish isn't simply English with Hindi words.

The way those words are written can directly affect how the TTS system pronounces them.

Switching Hindi words to Devanagari was one of the simplest and most effective improvements I made.

3. Context makes multi-agent systems usable

Agent handoffs sound simple:

Agent A → Agent B

But without preserving context, it becomes:

Agent A → Agent B → "What's your name?"

Passing the conversation context makes the handoff feel like one continuous conversation.

4. AI should know when to ask for help

A good tutor isn't necessarily one that answers everything.

Sometimes the best action is:

"Would you like me to connect you with a teacher?"

That human fallback makes the system more useful and trustworthy.

How to run it yourself

The project is based on the open-source Murf LiveKit Starter repository.

Repository:

https://github.com/bharatbushan03/murf-livekit-starter

Requirements

You'll need:

Python 3.10+
uv
Node.js
pnpm
LiveKit account
Murf API key
Deepgram API key
Google API key
Optional Discord webhook

Clone the repository

git clone https://github.com/bharatbushan03/murf-livekit-starter.git
cd murf-livekit-starter

Install the backend:

cd backend

uv sync

uv run python src/agent.py download-files

Install the frontend:

cd ../frontend

pnpm install

Environment variables

Configure your environment variables with the required API credentials:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET

MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

DISCORD_WEBHOOK_URL

The Discord webhook is optional.

Start the backend

cd backend

uv run python src/agent.py dev

Then start the frontend in another terminal:

cd frontend

pnpm dev

Open:

http://localhost:3000

Click Start Learning with Anisha and start talking.

What's next?

There are still several things I want to improve.

Better VAD for noisy environments

Real users aren't always sitting in a quiet room.

A student might be practicing from:

A classroom
A bus
A busy home
A shared workspace

I want to tune the voice activity detection system to work better in noisy environments.

Interactive math scorecards

Another feature I'm planning is real-time math progress.

While Samar is teaching, the frontend could show something like:

Math Practice

Questions: 8
Correct: 6
Needs Practice: 2

Topic:
Percentages

This would make the voice interaction feel more connected to the visual interface.

Final thoughts

Building Bhasha Academy taught me that voice AI can be much more than a voice chatbot.

When you combine:

Real-time voice
Hinglish support
Indian-accented TTS
Persistent memory
Multi-agent routing
SIP calling
Human escalation
Analytics

you can start building something that feels closer to a real tutor.

There are still many things to improve, but the core idea is simple:

Give learners a patient tutor they can talk to whenever they want, without being afraid of making mistakes.

That's what I'm trying to build with Bhasha Academy.

If you're building something similar with LiveKit, Murf, Deepgram, or Gemini, I'd love to hear what you're working on.

DEV Community: Bharat Lashotra

Building Bhasha Academy: A Multi-Agent Hinglish Voice Tutor with Murf Falcon & LiveKit

Building Bhasha Academy: An AI Voice Tutor for English and Math

The idea

How the architecture works

Speech-to-Text

LLM

Text-to-Speech

Real-time voice

One of the biggest problems: Hinglish pronunciation

The solution

Giving the AI memory

Multi-agent conversations

Anisha

Samar

Phone calls and voicemail detection

Human teachers are still important

Analytics dashboard

The biggest lessons I learned

1. Voice AI is more than just an LLM

2. Language and script matter

3. Context makes multi-agent systems usable

4. AI should know when to ask for help

How to run it yourself

Requirements

Clone the repository

Environment variables

Start the backend

What's next?

Better VAD for noisy environments

Interactive math scorecards

Final thoughts