Building Bharat Buddy: My 10-Day Voice AI Journey with Murf Falcon

Rishabh Pandey — Sat, 15 Aug 2026 05:53:01 +0000

What if learning didn't require typing?

What if a student could simply speak to an AI tutor in Hindi, English, or Hinglish, ask questions naturally, practice problems, and even get connected to a specialist when needed?

That was the idea behind Bharat Buddy, the voice agent I built during 10 Days of Voice Agents — VoiceForBharat Edition.

Over these 10 days, I went from a basic voice assistant to an agent that can remember users, use tools, make outbound calls, escalate problems to humans, track call analytics, and hand conversations over to specialist agents.

This post is about what I built, what went wrong, what I learned, and how someone else can start building their own voice agent.

🚀 What is Bharat Buddy?

Bharat Buddy is an AI-powered voice tutor designed for students.

The goal was to make learning feel more like talking to a helpful teacher rather than interacting with a traditional chatbot.

A student can speak naturally:

"Mujhe percentage samajh nahi aa raha."

or:

"Can you explain this maths problem?"

or even:

"Bhai, mujhe Hinglish mein samjha."

Bharat Buddy can respond conversationally and adapt to the user's language.

The project focuses on:

🇮🇳 Hindi
🇬🇧 English
🗣️ Hinglish
🎙️ Voice-first interaction
📚 Learning assistance
🧠 User memory
🧮 Maths practice
👨‍💻 Specialist handoffs
🧑‍💼 Human escalation
📊 Call analytics

The idea was simple:

Make AI learning more natural, accessible and conversational.

🎯 Why Voice?

Most AI applications still expect users to:

Type → Wait → Read → Type again

But speaking is much more natural.

For a student, saying:

"What is 20 percent of 500?"

is much easier than typing a detailed prompt.

Voice also makes the interaction feel more human.

Instead of staring at a chatbot, you can actually talk to your tutor.

That's what I wanted Bharat Buddy to feel like.

🏗️ How Bharat Buddy Works

The basic architecture looks like this:

            USER
              │
              │ Voice
              ▼
    ┌──────────────────┐
    │   Speech-to-Text │
    └────────┬─────────┘
             │
             ▼
    ┌──────────────────┐
    │       LLM        │
    │  Bharat Buddy    │
    └────────┬─────────┘
             │
   ┌─────────┼──────────┐
   │         │          │
   ▼         ▼          ▼
Memory     Tools    Specialist
                      Agent
   │         │          │
   └─────────┼──────────┘
             │
             ▼
    ┌──────────────────┐
    │   Text-to-Speech │
    │   Murf Falcon    │
    └────────┬─────────┘
             │
             ▼
           USER

The project uses LiveKit for real-time voice communication, an LLM for reasoning, and Murf Falcon for voice generation.

Murf AI

🎙️ Murf Falcon

One of the most important parts of the project was making the AI actually sound conversational.

I used Murf Falcon, the fastest TTS API mentioned in the challenge, to generate the agent's voice.

This made the interaction much more natural than simply displaying text.

The agent can speak responses such as:

"Namaste! I'm Bharat Buddy. What would you like to learn today?"

and continue the conversation naturally.

The voice layer became a major part of the experience rather than just an additional feature.

🧠 Memory

A useful assistant shouldn't completely forget you after every conversation.

So I added memory capabilities to Bharat Buddy.

This allows the system to maintain useful user context and make future conversations more meaningful.

The important lesson here was that memory should be useful, not excessive.

A voice agent doesn't need to remember every single sentence.

It should remember information that actually improves future interactions.

🛠️ Tools

Another important part of the project was giving the agent access to tools.

Instead of making the LLM pretend it knows everything, tools allow the agent to perform specific actions.

For example:

User
↓
Bharat Buddy
↓
Does this require a tool?
↓
Yes → Call tool
↓
Get result
↓
Explain result to user

This makes the agent more reliable and gives it capabilities beyond simple conversation.

📞 Outbound Calls

Bharat Buddy also gained the ability to make outbound calls.

This was one of the more interesting parts of the challenge because the agent moved beyond:

"User calls AI"

to:

"AI can initiate a call."

This opened up possibilities such as:

Follow-ups
Reminders
Student assistance
Notifications
Future learning workflows
🧑‍💼 Human Escalation

One of the biggest lessons I learned was:

An AI agent should know when it should stop being the expert.

So I added a human escalation system.

For example, if the user needs help that the agent cannot safely handle, the agent can create a request for a human.

The workflow became:

User needs human help
↓
Agent identifies situation
↓
Agent asks for permission
↓
User agrees
↓
Create escalation request
↓
Generate reference ID
↓
Tell user what happens next

The escalation request contains only useful information rather than dumping the entire conversation.

This was especially important from a privacy perspective.

📊 Call Analytics Dashboard

On Day 8, I built a simple analytics system.

The dashboard tracks:

Total Calls

How many calls have been recorded.

Successful Calls

Calls where the defined success condition was achieved.

Failed Calls

Calls where the success condition wasn't reached.

The data is stored using SQLite and exposed through an API that the dashboard can consume.

For example:

{
"total": 7,
"successful": 4,
"failed": 0
}

The important part is that these aren't hardcoded numbers.

They come from actual agent activity.

🧮 Maths Practice Specialist

One of my favourite features was adding a specialist agent.

Instead of making Bharat Buddy an expert at everything, I created a separate:

Maths Practice Specialist

Its job is focused specifically on:

Arithmetic
Percentages
Fractions
Ratios
Algebra
Basic geometry
Step-by-step maths explanations

The flow looks like:

Student
↓
Bharat Buddy
↓
Maths question?
↓
Yes
↓
"I'll connect you with our Maths Practice Specialist."
↓
Maths Specialist
↓
Continues the same conversation

The user doesn't have to explain the entire problem again.

This taught me an important architectural concept:

A good AI system doesn't necessarily need one giant agent.

Multiple focused agents can often be easier to control.

🌐 Hindi + English + Hinglish

Another important part of Bharat Buddy was language flexibility.

The agent is designed to respond according to the user's language:

English → English

Hindi → Hindi

Hinglish → Hinglish

For example:

User:

"Percentage ka concept simple language mein samjhao."

Bharat Buddy can respond naturally in Hinglish rather than suddenly switching to formal English.

For an India-focused voice assistant, this makes a huge difference.

🎨 Frontend

The frontend provides the interface through which users interact with the voice agent.

The goal wasn't to create an unnecessarily complicated UI.

It focuses on showing the agent's state and making the voice interaction obvious.

The overall experience is:

Open Agent
↓
Connect
↓
Speak
↓
Agent Processes
↓
Agent Responds
😵 The Hardest Part

Honestly, the project didn't work perfectly on the first try.

One of the most frustrating problems happened while implementing the specialist handoff.

I initially received an error related to the tool schema:

invalid JSON schema for tool
handoff_to_maths_specialist

After fixing that, another problem appeared:

TypeError:
missing a required argument: 'reason'

The model was trying to call the handoff function, but the function definition and the arguments expected by the Python function didn't match.

The important lesson:

When using function calling, the tool schema and the actual Python function signature must agree.

For example, if the function expects:

async def handoff_to_maths_specialist(reason: str):

then the model needs to provide a reason argument.

This sounds small, but it can completely break the voice pipeline.

🔐 Another Important Lesson: Secrets

While building the project, I also learned how important it is to keep API credentials outside the source code.

API keys should be stored in environment variables such as:

.env.local

and should never be committed to GitHub.

My .gitignore contains rules for environment files and database files so that sensitive/local data isn't accidentally uploaded.

This is especially important when publishing an AI project publicly.

🧰 Tech Stack

The main technologies used in Bharat Buddy include:

Technology Purpose
Python Backend / agent logic
LiveKit Real-time voice communication
Murf Falcon Text-to-Speech
LLM Reasoning and conversation
SQLite Local data storage
HTML/CSS/JS Dashboard/frontend
Git/GitHub Version control
🚀 How to Run the Project

If you want to experiment with the project, start by cloning the repository:

Bharat Buddy — GitHub Repository

Rishabhpandey2007 / murf-livekit-starter

🇮🇳 Bharat Buddy — AI Voice Agent

A multilingual AI voice assistant built during 10 Days of Voice Agents — VoiceForBharat Edition, powered by Murf Falcon and LiveKit.

Bharat Buddy is a real-time AI voice agent designed to make learning and everyday assistance more natural through voice.

It can understand and respond in English, Hindi and Hinglish, remember returning users, use tools, make outbound calls, escalate problems to humans, track call analytics and hand conversations to specialist agents.

🎯 The Problem

Many users are more comfortable speaking than typing, especially when using technology in their everyday language.

Bharat Buddy explores how a voice-first AI assistant can make digital learning and assistance more accessible by allowing users to simply speak naturally instead of typing everything.

The project focuses on:

Natural voice conversations
Indian languages and Hinglish
Learning assistance
Human escalation when AI needs help
Specialist agents for focused…

View on GitHub

Then create your environment and install the project dependencies.

For example:

git clone https://github.com/Rishabhpandey2007/murf-livekit-starter.git

cd murf-livekit-starter

cd backend

uv sync

Create your local environment file:

.env.local

Add the required API credentials there.

Do not put your real API keys directly into Python files or commit .env.local to GitHub.

Then start the agent according to the project instructions.

Once the agent is running, open the browser client, connect to the voice session, and start speaking.

🧪 Testing the Agent

I tested different paths during development.

Normal conversation
User:
Explain percentage.

Bharat Buddy:
Explains percentage.
Specialist conversation
User:
I want to practice maths percentage problems.

Bharat Buddy:
I'll connect you with our Maths Practice Specialist.

Maths Specialist:
Hi! I'm Bharat Buddy's Maths Practice Specialist...
Human escalation
User needs human help
↓
Agent asks permission
↓
User approves
↓
Escalation created
↓
Reference ID returned
Analytics
Call completed
↓
Outcome stored
↓
Dashboard updated
📈 What I Learned in 10 Days

Before this challenge, I mostly thought about AI assistants as:

Input → LLM → Output

After building Bharat Buddy, I realized a production-style voice agent is much more than that.

You need to think about:

Real-time audio
Speech recognition
TTS latency
Prompt design
Tool calling
Memory
Safety
Human escalation
Specialist routing
Analytics
Privacy
Error handling

The LLM is only one component.

The real engineering challenge is making all the components work together reliably.

🔮 What I Would Build Next

Bharat Buddy is still a work in progress.

Some things I'd like to add next:

📱 Better mobile experience

Make the voice experience easier to use on low-end smartphones.

📚 Study Material Integration

Allow students to upload notes and ask questions about them.

📊 Better Analytics

Track:

Success rate
Call duration
Most common questions
Failure reasons
Specialist handoffs
👨‍🏫 More Specialist Agents

For example:

Bharat Buddy
│
├── Maths Specialist
├── English Specialist
├── Coding Specialist
└── Exam Preparation Specialist
🌍 More Indian Languages

Eventually, I'd like Bharat Buddy to support more Indian languages and make voice-based learning accessible to more students.

💡 Final Thoughts

The biggest thing I learned from this challenge is that building a voice agent isn't just about making an AI talk.

It's about designing a system that knows:

when to answer,

when to use a tool,

when to remember,

when to call a specialist,

when to ask a human,

and most importantly,

when not to pretend it knows something.

Bharat Buddy started as an idea for a voice tutor.

After 10 days, it became a much bigger experiment in building real-time, conversational AI systems.

I'm still learning, but this challenge gave me a much better understanding of what it takes to turn an AI idea into a working voice product.

🔗 Project

GitHub:
Bharat Buddy Repository

Built during:
10 Days of Voice Agents — VoiceForBharat Edition

Voice: Murf Falcon

Tech: Python + LiveKit + LLM + Murf Falcon + SQLite

🙌 Thanks

A big thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition challenge.

Building something every day for 10 days, debugging it, breaking it, fixing it, and finally documenting the journey was a genuinely valuable experience.

10 days. One voice agent. A lot of debugging. 🚀

VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #GenerativeAI #AI #Python #LiveKit #ConversationalAI #BuildInPublic

DEV Community: Rishabh Pandey

Building Bharat Buddy: My 10-Day Voice AI Journey with Murf Falcon

Rishabhpandey2007 / murf-livekit-starter

🇮🇳 Bharat Buddy — AI Voice Agent

🎯 The Problem

VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #GenerativeAI #AI #Python #LiveKit #ConversationalAI #BuildInPublic