What if learning didn't require typing?
What if a student could simply speak to an AI tutor in Hindi, English, or Hinglish, ask questions naturally, practice problems, and even get connected to a specialist when needed?
That was the idea behind Bharat Buddy, the voice agent I built during 10 Days of Voice Agents — VoiceForBharat Edition.
Over these 10 days, I went from a basic voice assistant to an agent that can remember users, use tools, make outbound calls, escalate problems to humans, track call analytics, and hand conversations over to specialist agents.
This post is about what I built, what went wrong, what I learned, and how someone else can start building their own voice agent.
🚀 What is Bharat Buddy?
Bharat Buddy is an AI-powered voice tutor designed for students.
The goal was to make learning feel more like talking to a helpful teacher rather than interacting with a traditional chatbot.
A student can speak naturally:
"Mujhe percentage samajh nahi aa raha."
or:
"Can you explain this maths problem?"
or even:
"Bhai, mujhe Hinglish mein samjha."
Bharat Buddy can respond conversationally and adapt to the user's language.
The project focuses on:
🇮🇳 Hindi
🇬🇧 English
🗣️ Hinglish
🎙️ Voice-first interaction
📚 Learning assistance
🧠 User memory
🧮 Maths practice
👨💻 Specialist handoffs
🧑💼 Human escalation
📊 Call analytics
The idea was simple:
Make AI learning more natural, accessible and conversational.
🎯 Why Voice?
Most AI applications still expect users to:
Type → Wait → Read → Type again
But speaking is much more natural.
For a student, saying:
"What is 20 percent of 500?"
is much easier than typing a detailed prompt.
Voice also makes the interaction feel more human.
Instead of staring at a chatbot, you can actually talk to your tutor.
That's what I wanted Bharat Buddy to feel like.
🏗️ How Bharat Buddy Works
The basic architecture looks like this:
USER
│
│ Voice
▼
┌──────────────────┐
│ Speech-to-Text │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ LLM │
│ Bharat Buddy │
└────────┬─────────┘
│
┌─────────┼──────────┐
│ │ │
▼ ▼ ▼
Memory Tools Specialist
Agent
│ │ │
└─────────┼──────────┘
│
▼
┌──────────────────┐
│ Text-to-Speech │
│ Murf Falcon │
└────────┬─────────┘
│
▼
USER
The project uses LiveKit for real-time voice communication, an LLM for reasoning, and Murf Falcon for voice generation.
Murf AI
🎙️ Murf Falcon
One of the most important parts of the project was making the AI actually sound conversational.
I used Murf Falcon, the fastest TTS API mentioned in the challenge, to generate the agent's voice.
This made the interaction much more natural than simply displaying text.
The agent can speak responses such as:
"Namaste! I'm Bharat Buddy. What would you like to learn today?"
and continue the conversation naturally.
The voice layer became a major part of the experience rather than just an additional feature.
🧠 Memory
A useful assistant shouldn't completely forget you after every conversation.
So I added memory capabilities to Bharat Buddy.
This allows the system to maintain useful user context and make future conversations more meaningful.
The important lesson here was that memory should be useful, not excessive.
A voice agent doesn't need to remember every single sentence.
It should remember information that actually improves future interactions.
🛠️ Tools
Another important part of the project was giving the agent access to tools.
Instead of making the LLM pretend it knows everything, tools allow the agent to perform specific actions.
For example:
User
↓
Bharat Buddy
↓
Does this require a tool?
↓
Yes → Call tool
↓
Get result
↓
Explain result to user
This makes the agent more reliable and gives it capabilities beyond simple conversation.
📞 Outbound Calls
Bharat Buddy also gained the ability to make outbound calls.
This was one of the more interesting parts of the challenge because the agent moved beyond:
"User calls AI"
to:
"AI can initiate a call."
This opened up possibilities such as:
Follow-ups
Reminders
Student assistance
Notifications
Future learning workflows
🧑💼 Human Escalation
One of the biggest lessons I learned was:
An AI agent should know when it should stop being the expert.
So I added a human escalation system.
For example, if the user needs help that the agent cannot safely handle, the agent can create a request for a human.
The workflow became:
User needs human help
↓
Agent identifies situation
↓
Agent asks for permission
↓
User agrees
↓
Create escalation request
↓
Generate reference ID
↓
Tell user what happens next
The escalation request contains only useful information rather than dumping the entire conversation.
This was especially important from a privacy perspective.
📊 Call Analytics Dashboard
On Day 8, I built a simple analytics system.
The dashboard tracks:
Total Calls
How many calls have been recorded.
Successful Calls
Calls where the defined success condition was achieved.
Failed Calls
Calls where the success condition wasn't reached.
The data is stored using SQLite and exposed through an API that the dashboard can consume.
For example:
{
"total": 7,
"successful": 4,
"failed": 0
}
The important part is that these aren't hardcoded numbers.
They come from actual agent activity.
🧮 Maths Practice Specialist
One of my favourite features was adding a specialist agent.
Instead of making Bharat Buddy an expert at everything, I created a separate:
Maths Practice Specialist
Its job is focused specifically on:
Arithmetic
Percentages
Fractions
Ratios
Algebra
Basic geometry
Step-by-step maths explanations
The flow looks like:
Student
↓
Bharat Buddy
↓
Maths question?
↓
Yes
↓
"I'll connect you with our Maths Practice Specialist."
↓
Maths Specialist
↓
Continues the same conversation
The user doesn't have to explain the entire problem again.
This taught me an important architectural concept:
A good AI system doesn't necessarily need one giant agent.
Multiple focused agents can often be easier to control.
🌐 Hindi + English + Hinglish
Another important part of Bharat Buddy was language flexibility.
The agent is designed to respond according to the user's language:
English → English
Hindi → Hindi
Hinglish → Hinglish
For example:
User:
"Percentage ka concept simple language mein samjhao."
Bharat Buddy can respond naturally in Hinglish rather than suddenly switching to formal English.
For an India-focused voice assistant, this makes a huge difference.
🎨 Frontend
The frontend provides the interface through which users interact with the voice agent.
The goal wasn't to create an unnecessarily complicated UI.
It focuses on showing the agent's state and making the voice interaction obvious.
The overall experience is:
Open Agent
↓
Connect
↓
Speak
↓
Agent Processes
↓
Agent Responds
😵 The Hardest Part
Honestly, the project didn't work perfectly on the first try.
One of the most frustrating problems happened while implementing the specialist handoff.
I initially received an error related to the tool schema:
invalid JSON schema for tool
handoff_to_maths_specialist
After fixing that, another problem appeared:
TypeError:
missing a required argument: 'reason'
The model was trying to call the handoff function, but the function definition and the arguments expected by the Python function didn't match.
The important lesson:
When using function calling, the tool schema and the actual Python function signature must agree.
For example, if the function expects:
async def handoff_to_maths_specialist(reason: str):
then the model needs to provide a reason argument.
This sounds small, but it can completely break the voice pipeline.
🔐 Another Important Lesson: Secrets
While building the project, I also learned how important it is to keep API credentials outside the source code.
API keys should be stored in environment variables such as:
.env.local
and should never be committed to GitHub.
My .gitignore contains rules for environment files and database files so that sensitive/local data isn't accidentally uploaded.
This is especially important when publishing an AI project publicly.
🧰 Tech Stack
The main technologies used in Bharat Buddy include:
Technology Purpose
Python Backend / agent logic
LiveKit Real-time voice communication
Murf Falcon Text-to-Speech
LLM Reasoning and conversation
SQLite Local data storage
HTML/CSS/JS Dashboard/frontend
Git/GitHub Version control
🚀 How to Run the Project
If you want to experiment with the project, start by cloning the repository:
Bharat Buddy — GitHub Repository
🇮🇳 Bharat Buddy — AI Voice Agent
A multilingual AI voice assistant built during 10 Days of Voice Agents — VoiceForBharat Edition, powered by Murf Falcon and LiveKit.
Bharat Buddy is a real-time AI voice agent designed to make learning and everyday assistance more natural through voice.
It can understand and respond in English, Hindi and Hinglish, remember returning users, use tools, make outbound calls, escalate problems to humans, track call analytics and hand conversations to specialist agents.
🎯 The Problem
Many users are more comfortable speaking than typing, especially when using technology in their everyday language.
Bharat Buddy explores how a voice-first AI assistant can make digital learning and assistance more accessible by allowing users to simply speak naturally instead of typing everything.
The project focuses on:
- Natural voice conversations
- Indian languages and Hinglish
- Learning assistance
- Human escalation when AI needs help
- Specialist agents for focused…
Then create your environment and install the project dependencies.
For example:
git clone https://github.com/Rishabhpandey2007/murf-livekit-starter.git
cd murf-livekit-starter
cd backend
uv sync
Create your local environment file:
.env.local
Add the required API credentials there.
Do not put your real API keys directly into Python files or commit .env.local to GitHub.
Then start the agent according to the project instructions.
Once the agent is running, open the browser client, connect to the voice session, and start speaking.
🧪 Testing the Agent
I tested different paths during development.
Normal conversation
User:
Explain percentage.
Bharat Buddy:
Explains percentage.
Specialist conversation
User:
I want to practice maths percentage problems.
Bharat Buddy:
I'll connect you with our Maths Practice Specialist.
Maths Specialist:
Hi! I'm Bharat Buddy's Maths Practice Specialist...
Human escalation
User needs human help
↓
Agent asks permission
↓
User approves
↓
Escalation created
↓
Reference ID returned
Analytics
Call completed
↓
Outcome stored
↓
Dashboard updated
📈 What I Learned in 10 Days
Before this challenge, I mostly thought about AI assistants as:
Input → LLM → Output
After building Bharat Buddy, I realized a production-style voice agent is much more than that.
You need to think about:
Real-time audio
Speech recognition
TTS latency
Prompt design
Tool calling
Memory
Safety
Human escalation
Specialist routing
Analytics
Privacy
Error handling
The LLM is only one component.
The real engineering challenge is making all the components work together reliably.
🔮 What I Would Build Next
Bharat Buddy is still a work in progress.
Some things I'd like to add next:
📱 Better mobile experience
Make the voice experience easier to use on low-end smartphones.
📚 Study Material Integration
Allow students to upload notes and ask questions about them.
📊 Better Analytics
Track:
Success rate
Call duration
Most common questions
Failure reasons
Specialist handoffs
👨🏫 More Specialist Agents
For example:
Bharat Buddy
│
├── Maths Specialist
├── English Specialist
├── Coding Specialist
└── Exam Preparation Specialist
🌍 More Indian Languages
Eventually, I'd like Bharat Buddy to support more Indian languages and make voice-based learning accessible to more students.
💡 Final Thoughts
The biggest thing I learned from this challenge is that building a voice agent isn't just about making an AI talk.
It's about designing a system that knows:
when to answer,
when to use a tool,
when to remember,
when to call a specialist,
when to ask a human,
and most importantly,
when not to pretend it knows something.
Bharat Buddy started as an idea for a voice tutor.
After 10 days, it became a much bigger experiment in building real-time, conversational AI systems.
I'm still learning, but this challenge gave me a much better understanding of what it takes to turn an AI idea into a working voice product.
🔗 Project
GitHub:
Bharat Buddy Repository
Built during:
10 Days of Voice Agents — VoiceForBharat Edition
Voice: Murf Falcon
Tech: Python + LiveKit + LLM + Murf Falcon + SQLite
🙌 Thanks
A big thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition challenge.
Building something every day for 10 days, debugging it, breaking it, fixing it, and finally documenting the journey was a genuinely valuable experience.
10 days. One voice agent. A lot of debugging. 🚀
Top comments (0)