DEV Community

Bharat Lashotra
Bharat Lashotra

Posted on

Building Bhasha Academy: A Multi-Agent Hinglish Voice Tutor with Murf Falcon & LiveKit

#agents #ai #llm #software

Building Bhasha Academy: An AI Voice Tutor for English and Math

Language is meant to be spoken, not just read.

For millions of students and job seekers in India, being confident in English and basic math can open the door to better career opportunities. But many learners don't have access to a personal tutor. Even when they do, the fear of making mistakes in front of someone can make practicing uncomfortable.

So I built Bhasha Academy, an AI-powered voice tutor designed to give learners a private, judgment-free space to practice.

It can have conversations in Hinglish, help users practice English, solve basic math problems, remember learning progress, and even connect the learner with a human teacher when needed.

The interesting part is that the entire experience is voice-first.

Here's how I built it.

The idea

I wanted Bhasha Academy to feel less like talking to a chatbot and more like talking to a patient tutor.

The learner should be able to simply say:

"Mujhe English practice karni hai."

And the agent should understand the Hinglish, respond naturally, and continue the conversation.

For math, the learner can say:

"Samar, 25 percent of 200 kitna hai?"

The system can then switch to a math specialist and continue from the same conversation.

The main goals were:

  • Natural voice conversations
  • Hinglish support
  • Indian-accented voices
  • Personalized learning
  • Persistent memory
  • Specialist agents
  • Human escalation when required
  • Web and phone/SIP support

How the architecture works

The system is built using several services, with LiveKit sitting at the center of the real-time voice experience.

User
  ↓
Browser / SIP
  ↓
LiveKit
  ↓
Speech-to-Text
  ↓
Deepgram Nova-3
  ↓
Gemini
  ↓
Agent / Tools
  ↓
Murf Falcon TTS
  ↓
LiveKit
  ↓
User
Enter fullscreen mode Exit fullscreen mode

There are also a few supporting components:

SQLite
  ├── Student profiles
  ├── Learning progress
  ├── Call analytics
  └── Escalation tickets

Discord Webhook
  └── Human teacher notifications

Next.js
  └── Admin dashboard
Enter fullscreen mode Exit fullscreen mode

Speech-to-Text

I use Deepgram Nova-3 for speech recognition.

The important part here is multilingual support.

A learner might say:

"Yesterday I went market, but mujhe wahan kuch samajh nahi aaya."

The system needs to understand both languages without forcing the user to speak only English or only Hindi.

LLM

For the conversational layer, I use Google Gemini 3.5 Flash Lite.

The model is responsible for:

  • Understanding the conversation
  • Generating responses
  • Following the tutor personality
  • Calling tools
  • Managing agent handoffs
  • Working with the learner's profile

Text-to-Speech

For voice generation, I use Murf Falcon TTS through LiveKit.

Bhasha Academy currently uses two voices:

  • Anisha — general English and language practice
  • Samar — math practice

Both are designed to sound natural for Indian users.

Real-time voice

LiveKit Agents SDK handles the real-time communication layer.

It takes care of things like:

  • WebRTC audio
  • Voice activity detection
  • Turn detection
  • Streaming audio
  • SIP integration
  • Agent sessions

This makes it possible to have a real conversation instead of the traditional:

Record → Upload → Wait → Get Response → Play
Enter fullscreen mode Exit fullscreen mode

Instead, audio can be streamed continuously.

One of the biggest problems: Hinglish pronunciation

One of the first problems I noticed was surprisingly simple.

The AI understood Hinglish perfectly, but the voice didn't always pronounce it naturally.

For example, if the model generated:

Bahut achha, let's try again!
Enter fullscreen mode Exit fullscreen mode

the TTS system could interpret "Bahut achha" using English pronunciation.

The result sounded robotic.

The solution

I changed the system prompt so that Hindi words should be written in Devanagari.

Instead of:

Bahut achha, let's try again!
Enter fullscreen mode Exit fullscreen mode

the model generates:

बहुत अच्छा, let's try again!
Enter fullscreen mode Exit fullscreen mode

This made a huge difference.

Murf Falcon can handle mixed scripts, so it can naturally switch between Hindi and English.

For example:

बहुत बढ़िया! Let's try another word.
Enter fullscreen mode Exit fullscreen mode

This is a small prompt change, but it had a big impact on the voice experience.

Giving the AI memory

A tutor should remember its students.

Bhasha Academy uses SQLite to store learner profiles.

When someone calls for the first time, the agent doesn't automatically save their information.

Instead, it asks for permission.

For example:

"Would you like me to remember your name and learning progress for your next session?"

If the learner agrees, the system can store information such as:

  • Name
  • Current English level
  • Topics practiced
  • Common mistakes
  • City
  • Learning progress

There is also a forget_caller tool.

A learner can ask the system to forget them, and their stored profile can be removed.

This was important to me because personalization shouldn't come at the cost of user control.

Multi-agent conversations

Another interesting part of the project is the specialist handoff system.

Instead of making one huge agent handle everything, I created separate agents.

Anisha

Anisha is the general language tutor.

She handles:

  • English conversations
  • Vocabulary
  • Grammar
  • Pronunciation
  • General learning

Samar

Samar is the math specialist.

He handles:

  • Basic arithmetic
  • Percentages
  • Word problems
  • Math practice

If the learner says:

"Can we do some maths?"

Anisha can transfer the conversation to Samar.

The important part is that Samar shouldn't start from zero.

He should already know who the learner is and what was discussed.

The handoff looks roughly like this:

@function_tool
async def transfer_to_math_specialist(
    self,
    context: RunContext
) -> tuple[Agent, str]:

    math_agent = MathPracticeAgent(
        chat_ctx=self.chat_ctx.copy(
            exclude_instructions=True
        )
    )

    return (
        math_agent,
        "Transferring you to Samar, our maths practice specialist."
    )
Enter fullscreen mode Exit fullscreen mode

The key idea is passing a copy of the existing ChatContext.

This preserves the conversation while allowing the new agent to have its own instructions and personality.

Phone calls and voicemail detection

Bhasha Academy isn't limited to the browser.

Using LiveKit SIP integrations, the system can also make outbound calls.

This opens up the possibility of scheduled lessons.

For example:

Scheduled lesson
      ↓
System calls student
      ↓
Student answers
      ↓
AI tutor starts lesson
Enter fullscreen mode Exit fullscreen mode

But there is another problem with outbound calling: voicemail.

There is no reason to keep expensive AI services running when the call is answered by an answering machine.

So I added voicemail detection.

If the system detects a typical voicemail greeting, it can:

  1. Leave a short message.
  2. Call the hang_up tool.
  3. End the session.

Human teachers are still important

AI shouldn't try to solve every problem.

If a learner repeatedly struggles or becomes frustrated, Bhasha Academy can ask whether they want to speak with a human teacher.

If the learner agrees, the system creates an escalation ticket.

The ticket contains information such as:

Reference ID
Student Name
Urgency
Reason
Enter fullscreen mode Exit fullscreen mode

A Discord webhook then sends the notification to the teacher/admin channel.

For example:

payload = {
    "embeds": [{
        "title": f"🚨 Human Help Request Raised ({urgency.upper()})",
        "color": 15158332 if urgency == "emergency" else 3447003,
        "fields": [
            {
                "name": "Reference ID",
                "value": ref_id,
                "inline": True
            },
            {
                "name": "Student Name",
                "value": name,
                "inline": True
            },
            {
                "name": "Urgency",
                "value": urgency,
                "inline": True
            },
            {
                "name": "Reason",
                "value": reason,
                "inline": False
            }
        ]
    }]
}
Enter fullscreen mode Exit fullscreen mode

The idea is simple:

AI handles the routine conversations. Humans step in when the learner needs more help.

Analytics dashboard

Every call is also logged.

The system tracks things like:

  • Call status
  • Call duration
  • Web vs SIP
  • Latency
  • Completion rate

The data is stored in SQLite and exposed through a Next.js dashboard.

This makes it easier to understand how the system is performing instead of relying only on individual conversations.

The biggest lessons I learned

1. Voice AI is more than just an LLM

Getting a good text response is only one part of the problem.

A voice agent also needs:

  • Good speech recognition
  • Fast turn detection
  • Natural TTS
  • Low latency
  • Interruption handling
  • Context management

Small improvements in any of these areas can make the experience feel much more natural.

2. Language and script matter

Hinglish isn't simply English with Hindi words.

The way those words are written can directly affect how the TTS system pronounces them.

Switching Hindi words to Devanagari was one of the simplest and most effective improvements I made.

3. Context makes multi-agent systems usable

Agent handoffs sound simple:

Agent A → Agent B
Enter fullscreen mode Exit fullscreen mode

But without preserving context, it becomes:

Agent A → Agent B → "What's your name?"
Enter fullscreen mode Exit fullscreen mode

Passing the conversation context makes the handoff feel like one continuous conversation.

4. AI should know when to ask for help

A good tutor isn't necessarily one that answers everything.

Sometimes the best action is:

"Would you like me to connect you with a teacher?"

That human fallback makes the system more useful and trustworthy.

How to run it yourself

The project is based on the open-source Murf LiveKit Starter repository.

Repository:

https://github.com/bharatbushan03/murf-livekit-starter

Requirements

You'll need:

  • Python 3.10+
  • uv
  • Node.js
  • pnpm
  • LiveKit account
  • Murf API key
  • Deepgram API key
  • Google API key
  • Optional Discord webhook

Clone the repository 

git clone https://github.com/bharatbushan03/murf-livekit-starter.git
cd murf-livekit-starter
Enter fullscreen mode Exit fullscreen mode

Install the backend:

cd backend

uv sync

uv run python src/agent.py download-files
Enter fullscreen mode Exit fullscreen mode

Install the frontend:

cd ../frontend

pnpm install
Enter fullscreen mode Exit fullscreen mode

Environment variables

Configure your environment variables with the required API credentials:

LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET

MURF_API_KEY
DEEPGRAM_API_KEY
GOOGLE_API_KEY

DISCORD_WEBHOOK_URL
Enter fullscreen mode Exit fullscreen mode

The Discord webhook is optional.

Start the backend 

cd backend

uv run python src/agent.py dev
Enter fullscreen mode Exit fullscreen mode

Then start the frontend in another terminal:

cd frontend

pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

Click Start Learning with Anisha and start talking.

What's next?

There are still several things I want to improve.

Better VAD for noisy environments

Real users aren't always sitting in a quiet room.

A student might be practicing from:

  • A classroom
  • A bus
  • A busy home
  • A shared workspace

I want to tune the voice activity detection system to work better in noisy environments.

Interactive math scorecards

Another feature I'm planning is real-time math progress.

While Samar is teaching, the frontend could show something like:

Math Practice

Questions: 8
Correct: 6
Needs Practice: 2

Topic:
Percentages
Enter fullscreen mode Exit fullscreen mode

This would make the voice interaction feel more connected to the visual interface.

Final thoughts

Building Bhasha Academy taught me that voice AI can be much more than a voice chatbot.

When you combine:

  • Real-time voice
  • Hinglish support
  • Indian-accented TTS
  • Persistent memory
  • Multi-agent routing
  • SIP calling
  • Human escalation
  • Analytics

you can start building something that feels closer to a real tutor.

There are still many things to improve, but the core idea is simple:

Give learners a patient tutor they can talk to whenever they want, without being afraid of making mistakes.

That's what I'm trying to build with Bhasha Academy.

If you're building something similar with LiveKit, Murf, Deepgram, or Gemini, I'd love to hear what you're working on.

Top comments (0)