Building a Financial Services Voice Agent for Bharat — My 10 Days of Voice Agents Journey
Introduction
Over the last 10 days, I participated in the 10 Days of Voice Agents — VoiceForBharat Edition, where I built and improved a real-time voice agent step by step.
My project focused on financial services, where users often need help understanding eligibility, required documents, and basic financial information.
Instead of forcing users to navigate complicated forms or applications, I wanted to build an agent that could have a natural conversation with them, understand their requirements, provide useful information, and connect them with a human or specialist when needed.
For the voice experience, I used Murf Falcon, which helped me create a fast and natural Indian voice experience.
1. The Problem I Wanted to Solve
Financial services can sometimes be difficult for users because the process may involve:
- Eligibility checks
- Document requirements
- Repeated questions
- Complex information
- Human support
- Different types of customer requests
A voice-based assistant can make this process easier.
Users can simply speak with the agent instead of filling out long forms or searching through different pages.
My target users are people who want quick and simple assistance while checking financial eligibility or understanding what documents they need.
Voice is especially useful because users can interact naturally, including through Indian languages or code-mixed conversations.
2. What My Voice Agent Does
My financial services voice agent can:
- Talk naturally with users
- Understand user requests
- Follow defined objectives and safety guardrails
- Handle Indian and code-mixed conversations
- Perform useful tool-based actions
- Remember returning users
- Make outbound calls
- Escalate conversations to a human
- Transfer conversations to specialist agents
- Track call outcomes
- Display call analytics through a dashboard
The goal was not just to create a chatbot that talks.
I wanted to build a complete voice workflow where the agent can understand the conversation, take appropriate actions, and know when it should stop and involve a human.
3. How the System Works
The basic architecture of the project looks like this:
User Voice → Speech-to-Text → LLM/Agent → Tools & Memory → Text-to-Speech → User
The real-time communication layer connects these components so that the conversation feels natural.
The main components are:
Speech-to-Text
The user's voice is converted into text so that the agent can understand what the user is saying.
LLM
The language model processes the user's request, decides what the agent should say, and determines whether it needs to use a tool or transfer the conversation.
Tools
Tools allow the agent to perform useful actions instead of only generating text.
For example, the agent can use tools for financial eligibility checks or document-related information.
Text-to-Speech
The response generated by the agent is converted back into speech.
For my project, I used Murf Falcon for the voice output.
Real-Time Transport
A real-time communication layer handles the audio communication between the user and the voice agent.
4. Important Features I Built
Indian Voice with Murf Falcon
One of the important parts of my project was creating a voice experience suitable for Indian users.
I used Murf Falcon as the TTS engine to generate a fast and natural voice response.
The goal was to make the agent feel less like a robotic system and more like a real assistant.
Personality and Guardrails
I defined the agent's personality, objectives, and safety rules.
The agent should provide useful information but should not make unsafe assumptions or expose sensitive information.
This became especially important for a financial services use case.
Indian Languages and Code-Mixed Conversations
Users do not always communicate in formal English.
They may use Hindi, English, or a mixture of both.
Designing the agent to handle this type of conversation makes the system more practical for Indian users.
Agent State Frontend
I also created a frontend that shows the current state of the voice agent.
This helps visualize what is happening during a call, such as when the agent is listening, speaking, processing, or using a tool.
Memory
The agent can remember useful information about returning users.
This helps avoid asking the same questions repeatedly and makes future conversations more personalized.
Tools
The agent can use tools to fetch or calculate useful information.
This makes the agent more than just a conversational AI system.
Outbound Calls
I also explored outbound calling, where the agent can initiate a phone conversation instead of only waiting for the user to call.
Human Escalation
A voice agent should not try to solve everything.
When a request requires human assistance, the agent can escalate the conversation to a human.
Specialist Handoffs
I also implemented the idea of transferring conversations between specialized agents.
For example, a general agent can understand the user's problem and then hand the conversation to a specialist when the request requires a specific type of support.
Call Analytics Dashboard
One of the final features I built was a call analytics dashboard.
It tracks important information such as:
- Total calls
- Successful calls
- Failed calls
- Real call outcomes
- Success rate
For my project, a successful call means that the user completes the financial eligibility check or receives the required document information.
I also made sure that sensitive information such as OTPs and private customer data is not exposed in analytics.
5. The Most Difficult Parts
The hardest part was not simply making the agent speak.
The real challenge was making different parts of the system work together reliably.
For example, a voice agent needs to handle:
Audio → Understanding → Decision → Tool → Response → Audio
Any delay or incorrect decision can make the conversation feel unnatural.
Another challenge was handling sensitive financial information.
Initially, it is easy to focus only on whether the agent gives the correct answer. But in a real financial application, we also need to think about what information should never be stored or displayed.
I solved this by adding stronger guardrails and making sure sensitive information such as OTPs and private customer information was not included in analytics.
I also learned that human escalation is not a failure.
Sometimes the best decision for an AI agent is to recognize its limitation and transfer the conversation to a human or specialist.
6. How You Can Build Your Own Voice Agent
If you want to build a similar project, start with four basic components:
- Speech-to-Text — converts the user's voice into text.
- LLM — understands the request and decides what to do.
- Text-to-Speech — converts the response into natural speech.
- Real-Time Transport — moves audio between the user and the agent.
A simple development flow is:
User
↓
Microphone
↓
Speech-to-Text
↓
LLM / Voice Agent
↓
Tools / Memory / APIs
↓
Text-to-Speech
↓
User
For the project setup, first clone the public repository and install the required dependencies.
Then create your environment configuration file and add your API keys there.
Never hard-code API keys inside your source code or publish them on GitHub.
A typical environment file can look like:
LLM_API_KEY=your_key_here
MURF_API_KEY=your_key_here
OTHER_API_KEY=your_key_here
Keep the .env file inside .gitignore.
After configuring the required services, start the voice agent and connect to it through the frontend or supported real-time client.
Then test simple conversations first before adding advanced features such as tools, memory, outbound calls, and handoffs.
Project Repository
GitHub: [ADD YOUR PUBLIC GITHUB REPOSITORY LINK]
Demo
Demo: [ADD YOUR DEMO LINK]
7. What I Learned
This challenge taught me that building a voice agent is much more than connecting an LLM to a TTS API.
A useful voice agent needs:
- Good conversation design
- Low latency
- Clear objectives
- Safety guardrails
- Reliable tools
- Memory
- Error handling
- Human escalation
- Analytics
- Privacy considerations
The biggest lesson for me was that a production-ready voice agent should know not only what to say, but also what action to take and when to involve a human.
8. What I Would Improve Next
There are still many things I would like to improve.
My next improvements would include:
- Better multilingual support
- More financial tools
- Better personalization
- Improved analytics
- More accurate call outcome detection
- Better latency optimization
- More specialist agents
- Stronger privacy and security
- More real-world testing
I would also like to test the agent with a larger number of real conversations and use those results to improve the system.
9. Final Thoughts
The 10 Days of Voice Agents — VoiceForBharat Edition was a great learning experience.
In 10 days, I went from understanding the basics of voice agents to building a complete financial services voice workflow with tools, memory, outbound calling, human escalation, specialist handoffs, and analytics.
The biggest takeaway is simple:
Voice AI is not just about making an AI speak. It is about creating a system that can listen, understand, act, and know when to ask for human help.
I am excited to continue improving this project and explore more real-world applications of voice AI for India.
Built with Murf Falcon and a lot of learning. 🚀
Useful Resources
- Murf LiveKit Starter: https://github.com/rajakmadhav07/murf-livekit-starter
- Murf Falcon 2 Documentation: https://murf.ai/api/docs/text-to-speech-models/falcon-2
- LiveKit Voice AI Quickstart: https://docs.livekit.io/agents/start/voice-ai/
Top comments (0)