From a Simple Voice Bot to a Real Voice Agent: My 10-Day VoiceForBharat Journey 🎙️🇮🇳

Mahto Mohit — Sat, 15 Aug 2026 11:52:23 +0000

From a Simple Voice Bot to a Real Voice Agent: My 10-Day VoiceForBharat Journey 🎙️🇮🇳

Ten days ago, I started the 10 Days of Voice Agents — VoiceForBharat Edition challenge with a very simple idea.

I wanted to build a voice agent that could actually have a useful conversation with a person.

At the beginning, it was basically:

User speaks → AI understands → AI replies.

But over the next 10 days, I kept adding one feature at a time.

The agent learned to remember users, use tools, make outbound calls, ask for human help, track call history and finally hand conversations over to a specialist agent.

I chose the Financial Services track, and this challenge turned out to be much more than just making an AI speak.

There was a lot of debugging, a lot of testing, and quite a few moments where something that was working suddenly stopped working. 😅

But that was also the best part of the experience.

FinGuard — my voice agent built for the Financial Services track.

💡 Why I Chose Financial Services

Financial services can be confusing for many people.

There are government schemes, eligibility requirements, documents, deadlines and different rules to understand.

A person shouldn't always have to search through multiple websites or read long documents just to get a basic answer.

That's where I wanted voice AI to help.

Instead of typing:

"What documents do I need for this government scheme?"

a user could simply ask the question by voice.

The goal of my project was to make financial assistance feel more like a conversation instead of another complicated form.

I also wanted the agent to know its limits.

For financial use cases, giving a confident but incorrect answer can be worse than saying:

"I'm not able to verify that information right now."

That became an important design principle throughout the project.

🎙️ What I Built

My voice agent uses several technologies working together:

LiveKit — real-time voice communication
Deepgram — speech-to-text
Gemini — language model
Murf Falcon — text-to-speech
Python — backend and agent logic
SQLite — persistent memory
Telephony/Linphone — outbound calling

The basic flow looks like this:

The voice pipeline connecting LiveKit, Deepgram, Gemini, memory, tools, specialist agents and Murf Falcon.

The interesting part isn't any single component.

The interesting part is how all of them work together during a real conversation.

🧠 Giving the Agent Memory

One of the first problems I faced was that the agent had no memory.

A user could have a conversation, end the call and come back later.

The agent would basically start from zero again.

So I added SQLite-based persistent memory.

I created a way to store useful information such as:

User ID
Name
Language preference
Relevant facts
Last interaction

Now the agent can recognize a returning user and continue the conversation with some context from the previous interaction.

For example, instead of treating the user like a completely new person every time, the agent can understand that it has interacted with them before.

But there was an important rule.

The agent shouldn't just save everything automatically.

It needs to ask the user before remembering new information.

Something like:

"Would you like me to remember this for your next conversation?"

If the user says no, that information shouldn't be saved.

This made the memory feature much more meaningful because it wasn't just about storing data.

It was also about consent.

SQLite-based memory and call data used by FinGuard.

🔧 Giving the Agent Tools

After memory, the next question was:

What if the agent needs information that isn't already in the conversation?

That's where tools came in.

For the Financial Services track, I added financial/scheme-related functionality that the agent can call when it needs specific information.

For example, a user might ask:

"Can you check if I qualify for this government scheme?"

Instead of simply generating an answer, the agent can collect the required information and use the appropriate tool.

This was one of the biggest differences between a normal chatbot and the agent I was building.

The agent wasn't only generating text.

It was deciding:

"Do I need to use a tool for this?"

I also paid attention to the failure case.

If the tool or data source isn't available, the agent shouldn't silently fail or make up an answer.

It should tell the user that the information couldn't be retrieved.

For a financial application, I think this is extremely important.

📞 Making Outbound Calls

Then came one of the most exciting parts of the challenge.

Until this point, the user was starting the conversation.

For Day 6, I worked on making the agent call the user.

I used a telephony setup with Linphone for outbound calling.

The use case I chose was related to financial scheme reminders.

For example, imagine a user has already been identified as potentially eligible for a scheme and an important deadline is approaching.

Instead of waiting for the user to remember, the voice agent can proactively make a call.

The flow becomes:

System
   ↓
Trigger
   ↓
Voice Agent
   ↓
Outbound Call
   ↓
User

But outbound calls introduced a completely different voice experience.

When someone receives an unexpected call, the agent has to explain itself quickly.

It needs to tell the user:

Who is calling
Why it is calling
What the call is about
How the user can stop the interaction

That made me realize that voice UX is just as important as the AI model.

🙋 When AI Should Ask for Human Help

Another important feature I worked on was human escalation.

While building the agent, I realized that a good AI agent shouldn't try to answer every possible question.

There are situations where the right thing to do is to involve a human.

So instead of forcing the AI to continue a conversation when it isn't confident or the situation requires human support, the agent can ask for human help.

This is especially important in financial services.

I would rather have my agent say:

"I'll connect you with someone who can help with this."

than confidently provide incorrect information.

For me, this was an important lesson:

A smarter agent isn't the one that answers everything.

Sometimes, it's the one that knows when to stop.

Human escalation requests are tracked when the AI should not handle a situation alone.

📊 Keeping Track of Calls

Once the agent could handle more complex conversations and phone calls, I wanted to know what was actually happening during those interactions.

So I worked on call history and analytics.

The system can keep track of previous call interactions so they can be reviewed later.

This is useful because building an AI agent isn't only about making the conversation work.

You also need to understand what happened during the conversation.

Things like:

Previous calls
Call outcomes
User interactions
Conversation history

can help when improving the system later.

It also gave me a better idea of how a real voice AI application would be monitored.

Call analytics showing previous voice interactions and their outcomes.

🤝 One Agent Doesn't Need to Do Everything

For Day 9, I added one of my favorite features:

Government Scheme Specialist Agent

Initially, I had one main agent handling everything.

But that's not necessarily the best architecture.

A real support system might have different specialists for different problems.

So I created a separate specialist whose job is focused on:

Government schemes
Scheme eligibility
Required documents
Scheme-related questions
Basic scheme guidance

The main agent remains the primary agent.

If the user asks a normal financial question, the main agent answers it.

But if the user needs detailed government scheme assistance, the main agent can hand the conversation over.

For example:

User:

"Can you check if I am eligible for this government scheme?"

Main Agent:

"I'll connect you with our government scheme specialist who can help you with the eligibility details."

Then the specialist takes over.

The important part is that the user doesn't have to explain the whole problem again.

The relevant context is passed to the specialist.

This made the system feel much more like a team of AI agents instead of one huge prompt trying to do everything.

🔐 Guardrails and Safety

Because I chose the Financial Services track, I had to think carefully about what information the agent should and shouldn't ask for.

The agent should never ask users for things such as:

OTPs
Passwords
PINs
Bank credentials
Unnecessary account numbers
Sensitive information that isn't needed for the task

It also shouldn't make claims like:

"You are officially approved for this scheme."

Instead, it should explain that any eligibility result is based on the information available and that final approval belongs to the relevant authority.

This was one of the biggest differences between building a normal demo chatbot and building something for a real-world financial use case.

FinGuard's financial safety guardrails and user-facing security guidance.

🎙️ Why Murf Falcon Was Important

For a voice agent, the response isn't just text.

The user has to listen to it.

That means the voice needs to feel natural enough that the conversation doesn't feel like someone is reading a generated paragraph.

I used Murf Falcon for the text-to-speech part of my project.

The agent generates a response, and Murf Falcon converts that response into speech so the user can hear it in real time.

The challenge also helped me understand how much the voice experience depends on the complete pipeline.

Even if the LLM response is good, the experience can still feel bad if:

Speech recognition is slow
The response takes too long
TTS isn't configured correctly
Audio doesn't reach the user

So voice AI is really a combination of several systems working together.

🧩 The Main Architecture

At a high level, the system can be thought of like this:

                    USER
                     │
                     ▼
              ┌─────────────┐
              │   LiveKit   │
              │ Voice Layer │
              └──────┬──────┘
                     │
                     ▼
              ┌─────────────┐
              │  Deepgram   │
              │     STT     │
              └──────┬──────┘
                     │
                     ▼
              ┌─────────────┐
              │   Gemini    │
              │ Main Agent  │
              └──────┬──────┘
                     │
          ┌──────────┼───────────┐
          │          │           │
          ▼          ▼           ▼
       SQLite      Tools     Specialist
       Memory                  Agent
          │          │           │
          └──────────┼───────────┘
                     │
                     ▼
              ┌─────────────┐
              │ Murf Falcon │
              │     TTS     │
              └──────┬──────┘
                     │
                     ▼
                    USER

For outbound calls, the telephony layer connects the agent to the user's phone.

The Main Architecture

😅 What Was Actually Difficult?

This challenge wasn't:

Write code → everything works → done.

I wish it was. 😅

There were quite a few issues during development.

I faced problems with things like:

LiveKit configuration
Voice/TTS configuration
Dependencies
Agent connections
Telephony
Outbound calling
Backend/frontend communication

One problem I faced was related to the TTS voice configuration.

The voice I initially tried wasn't supported with the model configuration I was using.

There were also times when the agent itself was running, but the expected voice response wasn't coming through.

That's where debugging became important.

Instead of looking at the entire application as one big problem, I started checking each part separately:

Is LiveKit connected?
        ↓
Is STT receiving audio?
        ↓
Is the LLM generating a response?
        ↓
Is TTS receiving the response?
        ↓
Is audio coming back?

Breaking the system into smaller pieces made debugging much easier.

I learned that with real-time AI applications, knowing how to debug the pipeline is just as important as knowing how to build it.

🛠️ How Someone Can Start Building a Voice Agent

If you want to build something similar, I would strongly recommend starting small.

Don't try to build memory, tools, telephony and multiple agents on day one.

Start with the basic voice pipeline:

Speech
   ↓
STT
   ↓
LLM
   ↓
TTS
   ↓
Speech

Once that works, add features one by one.

1. Clone the project

My project is publicly available on GitHub:

GitHub Repository:
https://github.com/mohit2003116/ai-voice-agent

You can clone it using:

git clone https://github.com/mohit2003116/ai-voice-agent.git
cd ai-voice-agent

The backend is inside the backend directory:

cd backend

2. Install dependencies

The project uses uv.

uv sync

This installs the dependencies defined by the project.

3. Configure API keys

The project needs API credentials for the services being used.

Keep them in environment variables rather than writing them directly into the code.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

DEEPGRAM_API_KEY=your_deepgram_api_key
MURF_API_KEY=your_murf_api_key
GOOGLE_API_KEY=your_google_api_key

The values above are only examples.

Never publish your real API keys on GitHub.

Also don't publish:

Private phone numbers
Caller information
Personal user data
Private credentials

4. Run the agent

Once the environment variables and dependencies are configured, start the backend using the project's configured agent startup command.

For development, my agent can be run through the Python/uv environment.

Then connect the frontend or configured telephony setup and start testing.

5. Test the conversation

Start with something simple.

For example:

"Hello, can you help me understand a government scheme?"

Then gradually test:

"Can you remember my name?"

Then:

"Can you check my eligibility?"

And finally test the specialist handoff:

"I need detailed help with a government scheme."

Testing one feature at a time makes it much easier to find where something is going wrong.

📈 What I Would Improve Next

Even though the challenge is completed, I don't consider the project finished.

There are still many things I want to improve.

Some of them are:

Better latency
More Indian language support
More reliable financial data sources
Better call analytics
Better monitoring and error handling
More specialist agents
A more production-ready telephony setup
Better privacy and consent management

I would also like to experiment with having multiple specialist agents working together.

For example:

Main Agent
    │
    ├── Government Scheme Specialist
    │
    ├── Financial Education Specialist
    │
    └── Document Assistance Specialist

That could make the system much more flexible.

🎯 What I Learned From the Challenge

When I started, I thought the biggest challenge would be:

"How do I make an AI talk?"

After 10 days, I realized that's actually just the beginning.

The harder questions are:

When should the agent remember something?

When should it use a tool?

When should it make an outbound call?

When should it ask for human help?

When should it hand the conversation to another agent?

And maybe the most important one:

When should it admit that it doesn't know something?

That's what changed my thinking during this challenge.

A voice agent isn't just an LLM with a microphone.

It's a complete system where speech recognition, reasoning, tools, memory, voice generation, telephony and safety all have to work together.

🚀 Final Thoughts

Looking back at where I started 10 days ago and where the project is now, I'm genuinely happy with the progress.

It started as a simple voice conversation.

Then it learned to remember.

Then it learned to use tools.

Then it learned to make calls.

Then it learned when to ask a human.

Then it learned how to hand a conversation to a specialist.

And along the way, I learned a lot about building and debugging real-time AI systems.

The challenge may be over, but I'm definitely not done building.

🔗 Project Links

GitHub Repository:

mohit2003116 / ai-voice-agent

Voice Agent Starter — Powered by Murf Falcon

Build a production voice AI agent in 5 minutes. Powered by the fastest TTS on the market - swap the system prompt to build anything from customer support to language tutors.

Why Murf Falcon

55ms model latency - fastest production TTS
130ms time-to-first-audio across 10+ global regions
$0.01/1000 characters - up to 10x cheaper than alternatives
150+ voices across 35+ languages
99.38% pronunciation accuracy

Architecture

flowchart LR
    A[🎙️ User speaks] -->|audio| B[Deepgram STT]
    B -->|text| C[LLM]
    C -->|response text| D[Murf Falcon TTS]
    D -->|audio| E[LiveKit]
    E -->|stream| F[🔊 User hears]

    style A fill:#444441,stroke:#888780,color:#fff
    style B fill:#185FA5,stroke:#85B7EB,color:#fff
    style C fill:#534AB7,stroke:#AFA9EC,color:#fff
    style D fill:#0F6E56,stroke:#5DCAA5,color:#fff
    style E fill:#D85A30,stroke:#F0997B,color:#fff
    style F fill:#444441,stroke:#888780,color:#fff

Quickstart

Prerequisites

Python 3.10+

uv - fast Python package manager

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Node.js 18+
pnpm…

View on GitHub

LinkedIn Demo / Journey:

#10daysofaivoiceagents #murffalcon #voiceforbharat #voiceai #aiagents #livekit #ai #buildinpublic #murfai | Mohit kumar Mahto

🚀 10 Days of Voice Agents — Completed!! 10 days ago, I started this challenge with a very simple idea -- I wanted to build a voice agent that could do more than just answer questions. I chose the Financial Services track, and honestly, these 10 days were full of learning, debugging, and a lot of things that didn't work on the first try. 😅 During these 10 days, I added different features to my voice agent step by step: 🧠 Memory I added SQLite so the agent can remember users and continue conversations instead of forgetting everything after a call. 🔧 Tools The agent can use financial tools when it needs specific information instead of simply making up an answer. 📞 Outbound Calls I worked on making the agent call a user instead of always waiting for the user to call it. 🙋 Human Escalation The agent can understand when a situation needs human help instead of trying to answer everything itself. 📊 Call History I also added call history so I can see and track previous interactions. 🤝 Specialist Agent One of my favourite parts was adding a separate Government Scheme Specialist. Now the main agent can hand over a scheme-related conversation to the specialist without asking the user to explain everything again. And of course, the voice side was powered using Murf Falcon, with LiveKit, Deepgram, Gemini and other tools working together behind the scenes.The hardest part for me was honestly not writing the first version. It was debugging everything when something went wrong especially voice configuration, telephony and getting all the different components to work together properly. There were times when something work then suddenly something broke. 😅 But that is probably what I learned the most from this challenge. A voice agent is not just about making an AI talk. It has to know when to remember, when to use a tool, when to call, when to ask a human for help and when to hand the conversation to another agent. I'm really happy with how much the project changed in just 10 days. Big thanks to Murf AI for giving me the opportunity to build and learn through the 10 Days of Voice Agents — VoiceForBharat Edition. This challenge is over, but I'm definitely not done building. 🚀 #10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AIAgents #LiveKit #ai #BuildInPublic #murfai

linkedin.com

Challenge:
10 Days of Voice Agents — VoiceForBharat Edition

🙌 Thank You

A big thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition.

The 10 days were a mix of coding, debugging, experimenting and learning.

Most importantly, it gave me a chance to build something from scratch and understand what it actually takes to turn a simple AI conversation into a complete voice agent.

Built with LiveKit, Deepgram, Gemini, SQLite and Murf Falcon. 🎙️🚀
@murfai

10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AIAgents #GenerativeAI #LiveKit #BuildInPublic

DEV Community: Mahto Mohit

From a Simple Voice Bot to a Real Voice Agent: My 10-Day VoiceForBharat Journey 🎙️🇮🇳

From a Simple Voice Bot to a Real Voice Agent: My 10-Day VoiceForBharat Journey 🎙️🇮🇳

💡 Why I Chose Financial Services

🎙️ What I Built

🧠 Giving the Agent Memory

But there was an important rule.

🔧 Giving the Agent Tools

📞 Making Outbound Calls

🙋 When AI Should Ask for Human Help

📊 Keeping Track of Calls

🤝 One Agent Doesn't Need to Do Everything

Government Scheme Specialist Agent

🔐 Guardrails and Safety

🎙️ Why Murf Falcon Was Important

🧩 The Main Architecture

The Main Architecture

😅 What Was Actually Difficult?

🛠️ How Someone Can Start Building a Voice Agent

1. Clone the project

2. Install dependencies

3. Configure API keys

4. Run the agent

5. Test the conversation

📈 What I Would Improve Next

🎯 What I Learned From the Challenge

🚀 Final Thoughts

🔗 Project Links

mohit2003116 / ai-voice-agent

Voice Agent Starter — Powered by Murf Falcon

Why Murf Falcon

Architecture

Quickstart

Prerequisites

#10daysofaivoiceagents #murffalcon #voiceforbharat #voiceai #aiagents #livekit #ai #buildinpublic #murfai | Mohit kumar Mahto

🙌 Thank You

10DaysofAIVoiceAgents #MurfFalcon #VoiceForBharat #VoiceAI #AIAgents #GenerativeAI #LiveKit #BuildInPublic