Building a Financial Services Voice Agent for Bharat — My 10 Days of Voice Agents Journey

Madhav Rajak — Sat, 15 Aug 2026 13:39:43 +0000

Building a Financial Services Voice Agent for Bharat — My 10 Days of Voice Agents Journey

Introduction

Over the last 10 days, I participated in the 10 Days of Voice Agents — VoiceForBharat Edition, where I built and improved a real-time voice agent step by step.

My project focused on financial services, where users often need help understanding eligibility, required documents, and basic financial information.

Instead of forcing users to navigate complicated forms or applications, I wanted to build an agent that could have a natural conversation with them, understand their requirements, provide useful information, and connect them with a human or specialist when needed.

For the voice experience, I used Murf Falcon, which helped me create a fast and natural Indian voice experience.

1. The Problem I Wanted to Solve

Financial services can sometimes be difficult for users because the process may involve:

Eligibility checks
Document requirements
Repeated questions
Complex information
Human support
Different types of customer requests

A voice-based assistant can make this process easier.

Users can simply speak with the agent instead of filling out long forms or searching through different pages.

My target users are people who want quick and simple assistance while checking financial eligibility or understanding what documents they need.

Voice is especially useful because users can interact naturally, including through Indian languages or code-mixed conversations.

2. What My Voice Agent Does

My financial services voice agent can:

Talk naturally with users
Understand user requests
Follow defined objectives and safety guardrails
Handle Indian and code-mixed conversations
Perform useful tool-based actions
Remember returning users
Make outbound calls
Escalate conversations to a human
Transfer conversations to specialist agents
Track call outcomes
Display call analytics through a dashboard

The goal was not just to create a chatbot that talks.

I wanted to build a complete voice workflow where the agent can understand the conversation, take appropriate actions, and know when it should stop and involve a human.

3. How the System Works

The basic architecture of the project looks like this:

User Voice → Speech-to-Text → LLM/Agent → Tools & Memory → Text-to-Speech → User

The real-time communication layer connects these components so that the conversation feels natural.

The main components are:

Speech-to-Text

The user's voice is converted into text so that the agent can understand what the user is saying.

LLM

The language model processes the user's request, decides what the agent should say, and determines whether it needs to use a tool or transfer the conversation.

Tools

Tools allow the agent to perform useful actions instead of only generating text.

For example, the agent can use tools for financial eligibility checks or document-related information.

Text-to-Speech

The response generated by the agent is converted back into speech.

For my project, I used Murf Falcon for the voice output.

Real-Time Transport

A real-time communication layer handles the audio communication between the user and the voice agent.

4. Important Features I Built

Indian Voice with Murf Falcon

One of the important parts of my project was creating a voice experience suitable for Indian users.

I used Murf Falcon as the TTS engine to generate a fast and natural voice response.

The goal was to make the agent feel less like a robotic system and more like a real assistant.

Personality and Guardrails

I defined the agent's personality, objectives, and safety rules.

The agent should provide useful information but should not make unsafe assumptions or expose sensitive information.

This became especially important for a financial services use case.

Indian Languages and Code-Mixed Conversations

Users do not always communicate in formal English.

They may use Hindi, English, or a mixture of both.

Designing the agent to handle this type of conversation makes the system more practical for Indian users.

Agent State Frontend

I also created a frontend that shows the current state of the voice agent.

This helps visualize what is happening during a call, such as when the agent is listening, speaking, processing, or using a tool.

Memory

The agent can remember useful information about returning users.

This helps avoid asking the same questions repeatedly and makes future conversations more personalized.

Tools

The agent can use tools to fetch or calculate useful information.

This makes the agent more than just a conversational AI system.

Outbound Calls

I also explored outbound calling, where the agent can initiate a phone conversation instead of only waiting for the user to call.

Human Escalation

A voice agent should not try to solve everything.

When a request requires human assistance, the agent can escalate the conversation to a human.

Specialist Handoffs

I also implemented the idea of transferring conversations between specialized agents.

For example, a general agent can understand the user's problem and then hand the conversation to a specialist when the request requires a specific type of support.

Call Analytics Dashboard

One of the final features I built was a call analytics dashboard.

It tracks important information such as:

Total calls
Successful calls
Failed calls
Real call outcomes
Success rate

For my project, a successful call means that the user completes the financial eligibility check or receives the required document information.

I also made sure that sensitive information such as OTPs and private customer data is not exposed in analytics.

5. The Most Difficult Parts

The hardest part was not simply making the agent speak.

The real challenge was making different parts of the system work together reliably.

For example, a voice agent needs to handle:

Audio → Understanding → Decision → Tool → Response → Audio

Any delay or incorrect decision can make the conversation feel unnatural.

Another challenge was handling sensitive financial information.

Initially, it is easy to focus only on whether the agent gives the correct answer. But in a real financial application, we also need to think about what information should never be stored or displayed.

I solved this by adding stronger guardrails and making sure sensitive information such as OTPs and private customer information was not included in analytics.

I also learned that human escalation is not a failure.

Sometimes the best decision for an AI agent is to recognize its limitation and transfer the conversation to a human or specialist.

6. How You Can Build Your Own Voice Agent

If you want to build a similar project, start with four basic components:

Speech-to-Text — converts the user's voice into text.
LLM — understands the request and decides what to do.
Text-to-Speech — converts the response into natural speech.
Real-Time Transport — moves audio between the user and the agent.

A simple development flow is:

User
  ↓
Microphone
  ↓
Speech-to-Text
  ↓
LLM / Voice Agent
  ↓
Tools / Memory / APIs
  ↓
Text-to-Speech
  ↓
User

For the project setup, first clone the public repository and install the required dependencies.

Then create your environment configuration file and add your API keys there.

Never hard-code API keys inside your source code or publish them on GitHub.

A typical environment file can look like:

LLM_API_KEY=your_key_here
MURF_API_KEY=your_key_here
OTHER_API_KEY=your_key_here

Keep the .env file inside .gitignore.

After configuring the required services, start the voice agent and connect to it through the frontend or supported real-time client.

Then test simple conversations first before adding advanced features such as tools, memory, outbound calls, and handoffs.

Project Repository

GitHub: [ADD YOUR PUBLIC GITHUB REPOSITORY LINK]

Demo

Demo: [ADD YOUR DEMO LINK]

7. What I Learned

This challenge taught me that building a voice agent is much more than connecting an LLM to a TTS API.

A useful voice agent needs:

Good conversation design
Low latency
Clear objectives
Safety guardrails
Reliable tools
Memory
Error handling
Human escalation
Analytics
Privacy considerations

The biggest lesson for me was that a production-ready voice agent should know not only what to say, but also what action to take and when to involve a human.

8. What I Would Improve Next

There are still many things I would like to improve.

My next improvements would include:

Better multilingual support
More financial tools
Better personalization
Improved analytics
More accurate call outcome detection
Better latency optimization
More specialist agents
Stronger privacy and security
More real-world testing

I would also like to test the agent with a larger number of real conversations and use those results to improve the system.

9. Final Thoughts

The 10 Days of Voice Agents — VoiceForBharat Edition was a great learning experience.

In 10 days, I went from understanding the basics of voice agents to building a complete financial services voice workflow with tools, memory, outbound calling, human escalation, specialist handoffs, and analytics.

The biggest takeaway is simple:

Voice AI is not just about making an AI speak. It is about creating a system that can listen, understand, act, and know when to ask for human help.

I am excited to continue improving this project and explore more real-world applications of voice AI for India.

Built with Murf Falcon and a lot of learning. 🚀

Useful Resources

Murf LiveKit Starter: https://github.com/rajakmadhav07/murf-livekit-starter
Murf Falcon 2 Documentation: https://murf.ai/api/docs/text-to-speech-models/falcon-2
LiveKit Voice AI Quickstart: https://docs.livekit.io/agents/start/voice-ai/

DEV Community: Madhav Rajak

Building a Financial Services Voice Agent for Bharat — My 10 Days of Voice Agents Journey