Building BharatMoney Voice AI: My 10-Day Journey Building a Voice Agent for Financial Services

Building BharatMoney Voice AI: My 10-Day Journey Building a Voice Agent for Financial Services

Over the last 10 days, I took part in 10 Days of Voice Agents — VoiceForBharat Edition, where I built and improved a voice agent from the ground up.

My project is called BharatMoney Voice AI, a voice-based financial information assistant designed for users who want simple and accessible information about financial services and government schemes.

The project uses LiveKit Agents for real-time voice interaction and Murf Falcon for fast text-to-speech.

Repository:

1. The Problem

I chose the Financial Services track.

Many people need information about financial services and government schemes but may find complicated websites, technical language, or typing-based interfaces difficult to use.

I wanted to explore a simpler interface:

Just talk to the agent.

BharatMoney Voice AI can communicate in English, Hindi, and Hinglish, allowing users to ask questions naturally.

For example:

"PMJDY ke baare mein batao."

Instead of navigating through multiple pages, the user can simply ask the question using their voice.

1. What BharatMoney Voice AI Can Do

During the challenge, I gradually added several capabilities.

🎙️ Voice conversation

The agent listens to the user through speech-to-text, processes the request using an LLM, and responds using text-to-speech.

The core architecture is:

User Voice → STT → LLM → Tools/Logic → TTS → User

The real-time communication layer is handled by LiveKit.

1. Murf Falcon

One of the important parts of the project is the voice output.

I used Murf Falcon, the fast TTS API, to generate the agent's voice.

The goal was to make the conversation feel more natural and responsive rather than like a traditional text-to-speech application.

1. Safety Guardrails

Because BharatMoney deals with financial information, safety was extremely important.

The agent was instructed never to request sensitive information such as:

OTP
UPI PIN
PIN
CVV
Passwords
Full card numbers
Full bank account numbers

The agent also does not claim to be a bank or government authority.

It does not perform banking transactions or guarantee financial outcomes.

This was one of the most important design decisions in the project.

1. Memory for Returning Users

I also added persistent user memory.

This allows the application to remember useful information about returning users instead of treating every conversation as completely new.

The memory is stored locally in the project's database.

The important principle was:

Remember useful context, but don't store unnecessary sensitive information.

1. Real Government Scheme Information

Another important feature was connecting the agent to real PMJDY information.

Instead of simply making the LLM guess an answer, the agent can retrieve information from the official PMJDY source when appropriate.

This makes the system more useful for questions such as:

"PMJDY ke benefits kya hain?"

The agent can then explain the information conversationally.

1. Outbound Voice Calls

I also built an outbound voice capability using LiveKit SIP.

The agent could initiate a phone call and speak with the user.

This was an important step because it moved the project beyond a browser-only chatbot.

The outbound agent included:

SIP calling
Multilingual speech recognition
Gemini LLM
Murf Falcon TTS
Voice activity detection
Multilingual turn detection
Call-ending logic

1. Human Escalation

One of the most important features I built was human escalation.

A voice agent should not try to solve every problem by itself.

For example, if a caller reports possible UPI fraud, the agent should recognize that this is something that may require human assistance.

The agent therefore:

Detects the situation.
Explains that human help is needed.
Asks the caller for permission before sharing information.
Creates an escalation request.
Generates a reference ID.
Displays the request on a human-help dashboard.

For example:

Reference ID: ESC-EFA450D2

The dashboard allows a human to see the request and its urgency.

Human escalation dashboard showing a real request created after the agent identified a possible UPI fraud case.

Importantly, sensitive information such as OTPs, PINs and passwords is not included.

1. Call Analytics Dashboard

After building human escalation, I wanted to know how the agent was actually performing.

So I built a Call Analytics Dashboard.

!After building human escalation, I wanted to know how the agent was actually performing.

So I built a Call Analytics Dashboard.

Real call analytics from BharatMoney Voice AI showing 4 total calls, 4 successful calls, and 0 failed calls.

It tracks:

Total Calls

How many calls have been recorded.

Successful Calls

Calls that reached the defined success condition.

Failed Calls

Calls that ended without reaching the success condition.

The dashboard uses real call data, rather than hardcoded numbers.

During testing, the dashboard increased as real conversations were completed.

My final test reached:

Total Calls: 4
Successful Calls: 4
Failed Calls: 0

The dashboard deliberately avoids displaying sensitive caller information or full conversation transcripts.

1. Specialist Agent Handoff

On Day 9, I learned that one agent should not try to be an expert at everything.

So I created a separate:

Government Scheme Specialist Agent

The main agent handles general financial questions.

When the user asks for detailed government-scheme information, the main agent can hand the conversation to the specialist.

For example:

User:

"Mujhe PMJDY ke baare mein detail mein jaana hai."

Main Agent:

"Main aapko Government Scheme Specialist se connect karti hoon."

Then the specialist takes over and continues the PMJDY conversation.

The important part is that the conversation context is preserved, so the user does not have to explain everything again.

1. Challenges I Faced

The project definitely did not work perfectly on the first attempt.

One issue I faced was running the Python agent directly with:

uv run python src/agent.py dev

This produced:

ModuleNotFoundError: No module named 'src'

I solved it by running the module correctly:

uv run python -m src.agent dev

That allowed the LiveKit worker to register successfully.

Another challenge was making sure the call analytics were actually being recorded.

Initially, I could not find the analytics file.

After restarting the updated backend and making a fresh call, the system created:

backend/analytics.json

The file contained real call outcomes, which I then connected to the dashboard.

The specialist handoff also took some time during testing. The transfer was not instant, but the important thing was that the handoff successfully completed and the specialist continued answering the PMJDY question.

That taught me an important lesson:

A working end-to-end flow is more important than making every component perfect on the first attempt.

1. How You Can Build Your Own Voice Agent

If you want to build a similar project, the basic architecture is:

             ┌──────────────────┐
             │   User Voice     │
             └────────┬─────────┘
                      ↓
             ┌──────────────────┐
             │ Speech-to-Text   │
             └────────┬─────────┘
                      ↓
             ┌──────────────────┐
             │      LLM         │
             └────────┬─────────┘
                      ↓
         ┌────────────┼────────────┐
         ↓            ↓            ↓
      Memory        Tools      Specialist
         │            │            │
         └────────────┼────────────┘
                      ↓
             ┌──────────────────┐
             │   Text-to-Speech │
             │   Murf Falcon    │
             └────────┬─────────┘
                      ↓
             ┌──────────────────┐
             │   User Voice     │
             └──────────────────┘

The major components are:

Speech-to-Text: converts the user's voice into text.
LLM: understands the request and decides what to do.
Tools: provide external data or actions.
Memory: preserves useful user context.
Specialist Agents: handle focused tasks.
Text-to-Speech: converts the response back into natural speech.
Real-time transport: connects the user's audio to the agent.

1. Setting Up the Project

Clone the repository:

git clone https://github.com/khanadil84/murf-livekit-starter.git
cd murf-livekit-starter

The project contains:

backend/
frontend/

Install the backend dependencies using the project's uv setup.

Then configure your environment variables in:

backend/.env.local

Never publish your .env.local file or API keys to GitHub.

Keep secrets in environment variables and make sure they are included in .gitignore.

For the frontend, install the dependencies and start the development server.

The backend can be started with:

uv run python -m src.agent dev

The frontend can be started with:

npm run dev

Then open the local frontend in your browser and start a conversation.

1. What I Learned

This challenge taught me that building a voice agent is much more than connecting an LLM to a microphone.

A useful production-oriented voice agent needs:

Voice + Intelligence + Tools + Memory + Safety + Human Support + Observability

The biggest lesson for me was:

Don't build an AI that tries to do everything. Build an AI that knows what it can do, knows when it needs help, and knows when to hand the conversation to the right system or human.

1. What I Would Improve Next

There are still many things I would like to improve.

Some future improvements include:

Better call latency
More specialist agents
More government scheme integrations
Better analytics and visualizations
More robust error handling
Better production database support
More comprehensive testing
Better multilingual evaluation
Improved outbound-call reliability
Human-agent notification integrations

1. Final Thoughts

Ten days ago, this was just an idea.

Now BharatMoney Voice AI can:

🎙️ Talk with users
🧠 Remember useful context
🌐 Use real information
📞 Make outbound calls
🛡️ Follow financial safety guardrails
🙋 Escalate to humans
📊 Track call outcomes
🤖 Hand conversations to specialist agents

Building it was challenging, but that's exactly what made the experience valuable.

I'm grateful for the opportunity to participate in:

10 Days of Voice Agents — VoiceForBharat Edition

And a special thanks to Murf AI for providing this opportunity to learn, build, experiment, and understand what's possible with voice AI.