FINSATHI

FinSaathi: Building a Voice-First AI Financial Assistant with Murf Falcon

Over the last 10 days, I built FinSaathi, an AI-powered voice financial assistant as part of Murf AI's 10 Days of Voice Agents – VoiceForBharat Edition.

The goal wasn't just to make an AI that can answer questions. I wanted to build a voice agent that could remember users, use tools, make phone calls, recognize its limits, involve humans when needed, measure its performance, and hand conversations to specialized agents.

Here's what I built and what I learned along the way.

The Problem

Financial information can be confusing, especially when users have to navigate different websites, applications, eligibility requirements, and financial terminology.

FinSaathi is designed as a conversational interface where users can simply talk about their financial questions.

Instead of navigating through menus or typing complicated queries, a user can say:

"I want help managing my monthly expenses."

or:

"Am I eligible for this government scheme?"

and have a natural conversation with the assistant.

Voice is particularly useful here because the interaction becomes much closer to talking to a person rather than operating another application.

What is FinSaathi?

FinSaathi is an AI-powered voice financial assistant designed to help users with everyday financial questions through natural voice conversations.

It can help with areas such as:

Budgeting
Savings
UPI and banking basics
Credit scores
Loans
Government financial schemes
Scheme eligibility
General financial guidance

But the interesting part is that FinSaathi doesn't stop at answering questions.

During the challenge, I gradually added capabilities that made it behave more like a complete voice-agent system.

How the System Works

The basic voice pipeline looks like this:

User
↓
Voice Input
↓
Deepgram Speech-to-Text
↓
Gemini LLM
↓
Memory / Tools / Specialist / Human Escalation
↓
Murf Falcon Text-to-Speech
↓
Voice Response

For outbound calls, the flow extends through LiveKit SIP:

FinSaathi
↓
LiveKit SIP
↓
Linphone
↓
User

The project uses Murf Falcon for natural voice generation, Deepgram for speech recognition, Gemini for conversational intelligence, and LiveKit for real-time communication and telephony.

What I Built During the Challenge

1. Voice Conversations

The first goal was to make FinSaathi capable of having a natural voice conversation instead of just returning text.

The user speaks, the speech is transcribed, Gemini processes the request, and Murf Falcon generates the response.

The result is a conversational voice experience rather than a traditional chatbot.

1. Memory

FinSaathi can remember relevant information about returning users.

This means the assistant doesn't necessarily need to start from zero every time a user interacts with it.

The important lesson here was that memory needs to be useful and intentional. An AI shouldn't simply store everything a user says.

1. Real Tools and Data

The agent was also given tools that allow it to perform tasks instead of simply generating answers.

For example, FinSaathi can use a financial scheme eligibility tool when a user needs help determining whether they may qualify for a scheme.

This changed the architecture from:

User → LLM → Answer

to:

User → LLM → Decide whether a tool is needed → Tool → LLM → Answer

1. Outbound Phone Calls

One of the most interesting parts of the challenge was Day 6.

Instead of waiting for the user to open the browser and start a conversation, I connected FinSaathi with LiveKit SIP and Linphone so that FinSaathi could actually make an outbound call.

The flow became:

FinSaathi
↓
LiveKit SIP
↓
Linphone rings
↓
User answers
↓
FinSaathi speaks

Once the call is answered, FinSaathi automatically introduces itself:

"Hi, this is FinSaathi, your AI Financial Assistant..."

The conversation then continues in real time.

This was also where I faced one of my biggest debugging problems.

A Real Problem I Faced

During the outbound-call implementation, the phone call was working perfectly.

Linphone was ringing.

I could answer the call.

But...

Nobody was speaking.

The call connected, but there was complete silence.

Initially, it looked like a telephony problem, but the actual issue was in the outbound agent flow.

The agent session was starting without properly initiating the first assistant response.

The important fix was to correctly handle the SIP participant joining and explicitly trigger the initial greeting after the agent session was ready.

The final flow became:

SIP participant joins
↓
AgentSession starts
↓
Initial greeting is explicitly generated
↓
Murf TTS
↓
LiveKit audio output
↓
Linphone

This was probably one of my biggest lessons from the challenge:

A successful API connection doesn't necessarily mean the complete real-time pipeline is working.

You have to trace the actual flow of data and audio.

1. Human Escalation

An AI assistant shouldn't try to solve everything itself.

For Day 7, I taught FinSaathi when it should involve a human.

For example, if a user says:

"I think someone made an unauthorized UPI transaction."

FinSaathi shouldn't pretend it can investigate or reverse the transaction.

Instead, it explains that human assistance is appropriate and asks for permission before creating a support request.

The flow is:

Possible fraud detected
↓
Explain why human help is needed
↓
Ask for permission
↓
User says YES
↓
Create escalation
↓
Generate reference ID
↓
Give user the next step

I also made sure sensitive information such as OTPs, PINs, CVVs, passwords, and account credentials isn't included in the escalation summary.

1. Call Analytics

For Day 8, I wanted to answer a simple question:

How do I know whether my voice agent is actually performing well?

So I defined what a successful FinSaathi conversation means and added call outcome tracking.

The dashboard tracks:

Total calls
Successful calls
Failed calls

The important part is that these values come from actual conversations, rather than being hardcoded demo numbers.

This was a useful shift from simply saying:

"My AI agent works."

to asking:

"How well is my AI agent actually working?"

1. Specialist Agent Handoff

For Day 9, I added a separate Government Scheme Specialist.

The main FinSaathi agent handles general financial questions.

The specialist focuses specifically on:

Government schemes
Eligibility
Benefits
Required documents
Application-related guidance

For example, if I ask:

"Am I eligible for PM-KISAN?"

FinSaathi can say:

"I'll connect you to my Government Scheme Specialist. You won't need to repeat your question."

The specialist then takes over with the context of the original question.

The user doesn't have to explain everything again.

This creates a simple multi-agent architecture:

                ┌── General Financial Question
                │

User → FinSaathi ───┤
│
└── Government Scheme Question
↓
Government Scheme Specialist
The Technology Behind FinSaathi

The project combines several components:

Component Technology
Speech-to-Text Deepgram
LLM Gemini
Text-to-Speech Murf Falcon
Real-time voice LiveKit
Telephony LiveKit SIP
SIP client/testing Linphone
Agent framework LiveKit Agents
Memory Project database/memory layer
Analytics Call outcome database + dashboard

The interesting part isn't any individual API.

It's how these pieces work together to create a complete voice-agent workflow.

How to Run the Project

Clone the repository:

git clone YOUR_GITHUB_REPOSITORY
cd murf-livekit-starter

Set up the backend:

cd backend
uv sync

Add your API credentials to your local environment file.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

Never commit these values to GitHub.

Then start the agent:

uv run python -m src.agent dev

For the browser-based experience, start the frontend according to the project's frontend setup.

For outbound testing, the SIP configuration and Linphone account need to be configured separately.

What I Learned

The biggest lesson from these 10 days is that building a voice agent isn't just about connecting an LLM to a TTS API.

The difficult parts are everything around the conversation.

You have to think about:

What should the agent remember?
When should it use a tool?
What happens when an API fails?
When should it stop instead of guessing?
When should a human take over?
How do you measure whether a conversation was successful?
When should another specialist agent take over?
What happens when real-time audio doesn't behave as expected?

The outbound-call debugging experience especially taught me that distributed real-time systems can fail silently. A call can connect successfully while the actual audio pipeline is broken.

What's Next for FinSaathi?

There is still a lot I'd like to improve.

Some of the next steps would be:

More reliable real-time financial data
More Indian language support
More specialist agents
Better financial scheme coverage
Improved human-support workflows
More detailed call analytics
Better production-grade privacy and authentication
More robust handling of failed calls and unavailable services
Final Thoughts

Over these 10 days, FinSaathi went from a basic voice assistant to a much more complete system.

It can now:

Talk → Remember → Use Tools → Make Calls → Escalate → Measure → Handoff

And the biggest thing I learned is that a good voice agent isn't one that tries to do everything.

It's one that knows what it can do, what it shouldn't do, and when it needs help.

This project was built as part of:

10 Days of Voice Agents — VoiceForBharat Edition

with Murf Falcon powering the voice experience.
GitHub

https://github.com/gitsofshailendrayadav/murf-livekit-starter.git