FINORA AI: Building a Voice-First Financial Assistant for Bharat with Murf Falcon 🎙️🇮🇳
What if getting help with financial services was as simple as having a conversation?
Introduction
When I started the 10 Days of Voice Agents — VoiceForBharat Edition, I wanted to build something more than a basic voice chatbot.
I wanted to build an assistant that could actually help users with financial questions, remember useful context, communicate naturally, know when it needed human support, and hand specialized questions to another agent.
That idea became FINORA AI.
FINORA AI is a voice-first financial services assistant designed for Bharat. The main voice agent, Dia, interacts with users through voice and supports English and Hindi.
As the challenge progressed, FINORA AI evolved from a simple voice agent into a complete voice-AI system with memory, voice workflows, outbound calling, human escalation, call analytics, and a dedicated Government Scheme Specialist named Krishna.
The goal was simple:
Make financial assistance less complicated and more conversational — just a voice away.
In this blog, I'll walk through what I built, how the system works, the challenges I faced, and what I learned during the ten-day journey.
🌱 The Problem
Financial services can often feel complicated.
Users may have to:
- Search through different websites
- Understand financial terminology
- Navigate multiple pages
- Find the right government scheme
- Understand eligibility requirements
- Figure out where to get help
The information may already exist, but finding and understanding the right information can be the difficult part.
I wanted to make that experience more natural.
Instead of:
Search → Navigate → Read → Understand
I wanted:
Ask → Talk → Understand → Get Help
That's where voice becomes useful.
🎙️ Meet FINORA AI
FINORA AI is a voice-first financial services assistant.
The main agent is Dia.
Dia handles general financial conversations and acts as the primary entry point for the user.
FINORA AI can:
- Handle voice conversations
- Support English and Hindi
- Remember relevant conversational context
- Handle financial-service related questions
- Make outbound calls
- Escalate sensitive situations to humans
- Hand government-scheme questions to Krishna
- Track call outcomes
- Track agent handoffs
- Display analytics through a dashboard
The idea wasn't to make Dia sound like a robot reading information.
I wanted the interaction to feel more like talking to an assistant:
"Sure, let me help you with that."
rather than:
"According to the available information..."
Caption: FINORA AI's minimalistic interface, designed to keep voice interaction simple and easy to understand.
The website follows the same philosophy as the voice experience:
Minimalistic. Clean. Easy to understand.
🎯 Why Voice?
Voice removes a layer of complexity.
Users don't have to figure out exactly what to type or where to click.
They can simply say:
"What documents do I need?"
or:
"Can you explain this government scheme?"
They can also speak naturally in Hindi.
For FINORA AI, voice isn't just another feature.
Voice is the primary interface.
Caption: A live voice conversation with Dia through the FINORA AI interface.
🏗️ How FINORA AI Works
The overall architecture looks like this:
flowchart TD
U[User] --> L[LiveKit]
L --> STT[Speech to Text]
STT --> D[Dia - Main Agent]
D --> LLM[LLM Reasoning]
LLM --> M[Memory / Relevant Context]
D --> K[Krishna - Government Scheme Specialist]
D --> H[Human Escalation]
K --> GS[Government Scheme Guidance]
D --> TTS[Murf Falcon]
K --> TTS
TTS --> L
L --> U
L --> A[Call Analytics]
A --> DB[(Analytics Database)]
DB --> DASH[FINORA Dashboard]
At a high level, the voice flow is:
User Speech
↓
LiveKit
↓
Speech-to-Text
↓
Dia
↓
LLM Reasoning
↓
Response / Specialist / Human Escalation
↓
Murf Falcon
↓
User
LiveKit handles the real-time communication layer.
The agents handle the conversation and decision-making.
Murf Falcon provides the voice output.
The analytics system records call-level information so the dashboard can show how the system is performing.
Caption: High-level architecture of FINORA AI, showing the voice pipeline, Dia, Krishna, human escalation, Murf Falcon, and call analytics.
🔊 Murf Falcon — Giving FINORA AI a Voice
A voice assistant isn't only about what it says.
How it sounds matters too.
For FINORA AI, I used Murf Falcon, the fastest TTS API highlighted in the challenge.
The goal was to make the voice experience:
- Natural
- Clear
- Fast
- Friendly
- Conversational
Instead of feeling like a traditional IVR system, I wanted the interaction to feel like a real conversation.
This was one of the most interesting parts of the project because voice changes how you design the entire agent.
A text chatbot can give a long answer.
A voice assistant needs to think about:
What should I say, and how should I say it?
🇮🇳 English + Hindi Voice Support
FINORA AI supports English and Hindi across both agents — Dia and Krishna.
The language doesn't stop when the conversation is handed from one agent to another.
If a user starts speaking in English, Dia and Krishna can continue in English.
If the user starts speaking in Hindi, the conversation can continue in Hindi even after the handoff.
English
User:
"What documents do I need for PM-KISAN?"
Dia:
"This is a government-scheme question, so I'll connect you with Krishna, our government-scheme specialist."
Krishna:
"Hi, I understand you're asking about the documents required for PM-KISAN. Let me help you with that."
Hindi
User:
"PM-KISAN के लिए कौन-कौन से documents चाहिए?"
Dia:
"यह सरकारी योजना से जुड़ा सवाल है, इसलिए मैं आपको कृष्ण, हमारे सरकारी योजना विशेषज्ञ से जोड़ता हूँ।"
Krishna:
"नमस्ते, मुझे समझ आया कि आप PM-KISAN के लिए जरूरी दस्तावेजों के बारे में पूछ रहे हैं। मैं आपकी मदद करता हूँ।"
This was important to me because building for Bharat isn't just about adding Indian languages — the entire conversation should continue naturally in the user's preferred language, even when another specialist agent takes over.
🧠 Memory and Context
Another important part of FINORA AI is memory.
A useful assistant shouldn't force returning users to repeat relevant information unnecessarily.
FINORA AI can retain useful conversational context where appropriate, helping the interaction feel more continuous.
However, because FINORA is related to financial services, I designed the system with clear privacy boundaries.
Sensitive information such as:
- OTPs
- PINs
- Passwords
- CVVs
- Authentication secrets
should not be stored as ordinary conversational memory.
The goal is to remember useful context, not sensitive credentials.
🛠️ Agent Capabilities and Workflows
Rather than trying to make one agent responsible for everything, FINORA AI separates responsibilities.
Dia is the main financial assistant.
Krishna focuses specifically on government schemes.
And when something requires a human, the system can use the existing human escalation workflow.
This creates a simple separation:
General Financial Assistance
↓
DIA
Government Scheme Questions
↓
KRISHNA
Sensitive / Human-Only Issues
↓
HUMAN SUPPORT
This separation became one of the most important architectural decisions in the project.
📞 Outbound Calling
FINORA AI also supports outbound calling workflows.
This changes the interaction from:
"Ask me something."
to:
"Let me help you with something you need."
Building outbound calls made me think more carefully about how an agent starts a conversation.
An outbound voice agent needs to:
- Introduce itself
- Explain why it is calling
- Establish context quickly
- Avoid sounding overly scripted
- Give the user a clear next step
🧑💼 When AI Should Ask for Human Help
One of the most important things I learned during the challenge was that AI shouldn't try to solve everything.
For financial services, this becomes even more important.
For example, imagine a user says:
"I see a transaction that I didn't make."
Dia shouldn't pretend it can investigate or resolve the issue itself.
Instead, the system recognizes that human assistance may be required.
The flow becomes:
User reports possible fraud
↓
Dia identifies the situation
↓
Dia explains why human help is needed
↓
Ask for user permission
↓
User agrees
↓
Create escalation
↓
Generate reference ID
↓
Send request to human support
The user receives a reference ID and a clear explanation of what happens next.
Caption: FINORA AI creating a human-support request after receiving the user's permission.
🔐 Privacy by Design
Since FINORA AI is related to financial services, privacy cannot be an afterthought.
The escalation workflow is designed to avoid sharing unnecessary sensitive information.
It should never send:
- Passwords
- OTPs
- PINs
- CVVs
- Full account numbers
- Authentication secrets
Instead, the human receives a short summary containing only useful information.
For example:
Reference: FIN-2026-0001
Reason:
Possible Fraud
Summary:
Caller reports an unauthorized transaction.
Urgency:
High
Language:
Hindi
Preferred Follow-up:
Phone
The principle is simple:
Give the human enough information to help without giving them information they don't need.
🤝 Meet Krishna — Government Scheme Specialist
As FINORA AI grew, I realized that Dia shouldn't have to be an expert in everything.
So I created a separate specialist agent:
Krishna
Krishna is FINORA AI's Government Scheme Specialist.
His job is focused specifically on government financial and welfare schemes.
For example:
- PM-KISAN
- PM Jan Dhan Yojana
- PM SVANidhi
- Mudra-related schemes
- Pension schemes
- Insurance schemes
- Scholarship schemes
- Other supported government schemes
The key idea is that Krishna has a smaller and more focused responsibility than Dia.
Caption: Krishna, FINORA AI's dedicated Government Scheme Specialist.
🔄 Real Agent-to-Agent Handoff
This is one of the features I am most excited about.
The system doesn't simply ask Dia to generate a response about government schemes.
Instead, Dia can hand the conversation over to Krishna, a separate specialist agent.
For example:
User:
"What documents are required for PM-KISAN?"
Dia responds:
"This is a government-scheme question, so I'll connect you with Krishna, our government-scheme specialist. You won't need to repeat your question."
Then Krishna takes over:
"Hi, I'm Krishna, the government-scheme specialist. I understand you're asking about the documents required for PM-KISAN. Let me help you with that."
The important part is that the user doesn't have to explain the entire problem again.
The relevant context is passed during the handoff.
flowchart LR
U[User] --> D[Dia]
D -->|Government Scheme Question| K[Krishna]
K -->|Continues Same Conversation| U
D -->|Sensitive Issue| H[Human Support]
This makes FINORA AI a multi-agent voice system, rather than just a single chatbot with many prompts.
Caption: Dia handing the conversation to Krishna without requiring the user to repeat their question.
📊 Building the Call Analytics Dashboard
After adding multiple capabilities, I wanted to answer a simple question:
Is the agent actually working?
That's why I built a call analytics dashboard.
The dashboard tracks real calls and displays:
Total Calls
The total number of calls handled.
Successful Calls
Calls that reached the defined success condition.
Failed Calls
Calls that ended without reaching the intended outcome.
I also added agent handoff information.
For example:
Total Calls: 25
Successful Calls: 21
Failed Calls: 4
Agent Handoffs: 7
Handoffs to Krishna: 7
Replace these example numbers with your actual dashboard values before publishing.
These numbers come from actual call activity rather than being hardcoded.
[INSERT IMAGE: FINORA AI CALL ANALYTICS DASHBOARD]
Caption: FINORA AI's call analytics dashboard showing real call outcomes and specialist handoffs.
📈 Handoff Analytics
An important design decision was that a specialist handoff should not create a second call.
For example:
CALL-001
│
├── Dia
│
└── Krishna
This is still:
ONE CALL
with the handoff recorded as part of that call.
This keeps the analytics accurate.
The dashboard can therefore show the complete journey of a call without artificially inflating the number of calls.
Caption: Call analytics recording a Dia-to-Krishna handoff as part of the same conversation and call.
🎨 Designing a Simple User Experience
While building all these features, I didn't want the website to become complicated.
The FINORA AI interface is intentionally:
- Minimalistic
- Clean
- Easy to understand
- Voice-focused
The main navigation includes:
- Voice Assistant
- Finance Dashboard
- Call Analytics
- Settings
- Help
The user should be able to open the website and immediately understand what to do.
Talk to Dia. Get help.
The technology stays in the background.
🧩 The Challenges I Faced
Building FINORA AI over ten days wasn't a simple process where everything worked on the first attempt.
There were several challenges.
1. Building a Natural Voice Experience
Connecting real-time voice communication, the agent, and TTS required understanding how all the components interact.
A small configuration problem could mean the agent connected successfully but didn't respond correctly.
I had to debug the voice pipeline step by step instead of assuming the issue was always with the LLM.
2. Deciding When AI Should Stop
One of the biggest design questions was:
When should AI answer, and when should AI stop?
Too many escalations make the experience frustrating.
Too few can make the system unsafe.
So I created clear conditions for human escalation and specialist handoff.
3. Keeping Analytics Real
Displaying numbers on a dashboard is easy.
Making those numbers represent actual calls is much harder.
The system needs to record the call outcome and handoff information from the actual voice session.
This helped me understand the difference between building a UI and building a system that produces meaningful data.
4. Multi-Agent Handoff
The biggest challenge with Krishna was making the handoff feel like one continuous conversation.
Krishna needs to understand:
- What the user asked
- Why Dia handed the conversation over
- What language the user is using
- What context is relevant
The user shouldn't have to start again with:
"Hello, I was asking..."
The handoff should feel natural.
🚀 How You Can Build Your Own Voice Agent
If you want to build a voice agent yourself, there are four major pieces you need.
1. Speech-to-Text
Converts the user's speech into text.
User Voice
↓
Speech-to-Text
↓
Text
2. LLM / Agent
The agent understands the request and decides what to do.
Text
↓
LLM
↓
Decision
3. Text-to-Speech
The response is converted back into voice.
For FINORA AI, I use Murf Falcon.
Agent Response
↓
Murf Falcon
↓
Natural Voice
4. Real-Time Transport
A real-time communication layer connects the user and the voice agent.
FINORA AI uses LiveKit.
The simplified pipeline is:
User
↓
Microphone
↓
LiveKit
↓
Speech-to-Text
↓
Dia / Krishna
↓
LLM
↓
Murf Falcon
↓
LiveKit
↓
User
⚙️ Running the Project
Start by cloning the project:
git clone https://github.com/Sujayhs07/murf-livekit-starter/tree/DAY10
cd murf-livekit-starter
Create a .env file for your API keys and configuration.
For example:
MURF_API_KEY=your_murf_api_key
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
Add any additional credentials required by your implementation.
Never commit your .env file to GitHub.
Add it to .gitignore:
.env
Then install the dependencies and start the LiveKit agent according to the project's setup instructions.
Once the browser client connects, you can start a conversation with Dia.
🧪 Testing FINORA AI
I tested different conversation paths.
Test 1 — Normal Question
User:
"What is compound interest?"
Expected:
User → Dia
No specialist handoff.
Test 2 — Government Scheme
User:
"What documents do I need for PM-KISAN?"
Expected:
User
↓
Dia
↓
Krishna
The user should not have to repeat the question.
Test 3 — Hindi Government Scheme
User:
"PM-KISAN के लिए कौन-कौन से documents चाहिए?"
Expected:
User
↓
Dia — Hindi
↓
Krishna — Hindi
The language should remain consistent across the handoff.
Test 4 — Possible Fraud
User:
"Someone made a transaction from my account."
Expected:
User
↓
Dia
↓
Permission
↓
Human Escalation
It should not be routed to Krishna.
Test 5 — Analytics
After the call ends, the dashboard should reflect the real result:
Total Calls ↑
Successful Calls ↑
Failed Calls ↑
Handoffs to Krishna ↑
Caption: A snapshot of FINORA AI's end-to-end voice-agent workflow.
📚 What I Learned
The biggest lesson I learned from this challenge is that building a voice agent isn't really about making an AI talk.
The interesting engineering begins after that.
You have to think about:
- What should the agent know?
- What should it remember?
- When should it use a tool?
- When should it stop?
- When should a human take over?
- When should another agent take over?
- How should context move between agents?
- How do we measure success?
- How do we protect user information?
These questions completely changed how I think about voice AI.
🔮 What I'd Build Next
FINORA AI still has a lot of room to grow.
Some things I'd like to work on next are:
- More Indian languages
- More specialist agents
- Better multilingual conversations
- Improved personalization
- More advanced analytics
- More government-scheme coverage
- Better accessibility
- Smarter follow-up workflows
- A richer human-support dashboard
- More robust real-world integrations
I'd especially like to expand the multi-agent architecture so FINORA AI can route users to different specialists depending on what they need.
❤️ What 10 Days Taught Me
When I started the challenge, I thought the hardest part would be getting an AI agent to speak.
By Day 10, I realized:
Voice is only the beginning.
The real challenge is building an agent that knows how to behave.
An agent that can say:
"I can help with that."
But also knows when to say:
"Let me connect you with Krishna, our specialist."
And sometimes:
"This needs human assistance."
That's what I wanted FINORA AI to become.
Not just an AI that talks.
An AI that knows what to do next.
🔗 Project Links
GitHub Repository
https://github.com/Sujayhs07/murf-livekit-starter/tree/DAY10
Challenge
10 Days of Voice Agents — VoiceForBharat Edition
🙌 Final Thoughts
Building FINORA AI during the 10 Days of Voice Agents — VoiceForBharat Edition has been one of the most hands-on AI projects I've worked on.
Over these ten days, I explored real-time voice communication, TTS, agent design, memory, outbound calls, human escalation, call analytics, and multi-agent handoffs.
More importantly, I learned that building a useful AI system isn't only about adding more features.
It's about knowing when each feature should be used.
Dia doesn't need to know everything.
Krishna doesn't need to handle everything.
And AI doesn't need to handle everything either.
Sometimes the best decision an AI can make is to ask another specialist or a human for help.
A good voice agent isn't the one that tries to do everything. It's the one that knows what it can do, knows what it can't, and knows what to do next.
Thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build, experiment, break things, fix them, and learn along the way.
🛠️ Built With
- 🎙️ Murf Falcon
- ⚡ LiveKit
- 🧠 LLM-powered Agents
- 🗣️ English + Hindi
- 🤝 Human Escalation
- 🔄 Multi-Agent Handoff
- 📊 Call Analytics
- 🎨 Minimalistic Web Interface
#VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #GenerativeAI #AI #FinTech@MURFAI
Top comments (0)