Shazil Parwez

Posted on Aug 15

Building FinSathi: My 10-Day Journey from a Voice Agent to a Multi-Agent Financial Assistant

#ai #javascript #python #voiceai

Building FinSaathi: My 10-Day Journey from a Voice Agent to a Multi-Agent Financial Assistant

What happens when you spend 10 days teaching an AI voice agent not just to talk, but to actually remember, use tools, make calls, ask humans for help, analyze its performance, and delegate work to another agent?

That was my challenge.

As part of the 10 Days of Voice Agents — VoiceForBharat Edition, I built FinSaathi, a voice-based Financial Services assistant designed around one simple idea:

💡 Financial information should be easier to access, easier to understand, and safer to interact with.

What started as a voice conversation gradually became a complete agent system with memory, tools, telephony, human escalation, analytics, and specialist-agent handoffs.

This is the story of how I built it.

💡 What Is FinSaathi?

FinSaathi is a voice-based Financial Services assistant designed to help users understand financial schemes, digital banking safety, and related financial-literacy topics through natural conversations.

The target user doesn't need to type long questions or navigate complicated interfaces.

They can simply speak.

For example:

🗣️ "Mujhe PMJJBY ke baare mein thoda bataiye."

Or:

🗣️ "UPI fraud se safe kaise rahun?"

FinSaathi responds conversationally and can continue the interaction in Hindi, English, or Hinglish, depending on how the user communicates.

Why Voice?

Financial information can sometimes feel intimidating, especially when users have to navigate forms, websites, or unfamiliar terminology.

Voice changes that interaction.

Instead of:

Search → Read → Understand → Repeat

the experience becomes:

Speak → Ask → Listen → Continue

That makes voice particularly interesting for accessibility, conversational guidance, and users who are more comfortable communicating verbally.

🏗️ The Architecture

At a high level, the system looks like this:

                     ┌──────────────────┐
                     │      User        │
                     │  Voice / Browser │
                     └────────┬─────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │  Speech-to-Text  │
                     └────────┬─────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │    FinSaathi     │
                     │    Main Agent    │
                     └────────┬─────────┘
                              │
                ┌─────────────┼───────────────┐
                │             │               │
                ▼             ▼               ▼
             Memory         Tools       Human Escalation
                │             │               │
                │             │               │
                └─────────────┼───────────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │ Specialist Agent │
                     │   SchemeSathi    │
                     └────────┬─────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │  Text-to-Speech  │
                     │   Murf Falcon    │
                     └────────┬─────────┘
                              │
                              ▼
                     ┌──────────────────┐
                     │   User hears     │
                     │    response      │
                     └──────────────────┘

🎙️ Why Murf Falcon?

A voice agent needs more than an accurate text response.

It needs to respond quickly enough that the conversation still feels like a conversation.

For this project, I used Murf Falcon as the TTS layer because it is designed for real-time voice-agent interactions.

The important part for me wasn't simply generating speech.

It was reducing the gap between:

"The AI generated an answer."

and:

"The AI is actually talking to me."

That difference becomes especially noticeable during interruptions, follow-up questions, and multi-turn conversations.

🧠 The Agent's Personality and Guardrails

One of the first things I learned is that a voice agent needs more than a model.

It needs boundaries.

FinSaathi was designed with a clear identity and purpose, along with financial-safety guardrails.

🔐 Sensitive Information FinSaathi Must Never Request

OTPs
PINs
UPI PINs
Passwords
CVVs
Full bank account numbers
Full card numbers

It must also never promise:

❌ Guaranteed scheme approval
❌ Guaranteed enrollment
❌ Guaranteed financial outcomes

If a user asks about an account-specific issue or an application status, FinSaathi doesn't pretend to have access to information it doesn't have.

Instead, it directs the user toward the appropriate official channel.

💡 A good AI agent isn't just defined by what it can do. It is also defined by what it refuses to do.

🇮🇳 Hindi, English and Hinglish

India isn't a single-language conversational environment.

A user might start in Hindi, insert an English technical term, and switch back to Hindi in the same sentence.

For example:

"Mujhe PMJJBY ki eligibility check karni hai."

Or:

"UPI payment karte time safe kaise reh sakta hoon?"

FinSaathi was designed to handle this kind of code-mixed conversation naturally.

The goal wasn't to force users into a particular language.

The goal was to let the user communicate naturally.

🧠 Giving FinSaathi Memory

On Day 4, FinSaathi learned something extremely important:

Users shouldn't have to introduce themselves every time.

I added persistent user memory using a database.

The stored information is intentionally limited to useful conversational context.

A simplified record looks like:

{
  "user_id": "user_123",
  "name": "Example User",
  "language_preference": "Hinglish",
  "facts": {
    "preferred_scheme": "PMJJBY"
  },
  "last_interaction": "timestamp"
}

The actual implementation should only store information that is useful and appropriate for future interactions.

I also added an important rule:

🔐 Ask before saving personal information.

If the user doesn't want something remembered, the system shouldn't save it.

This made memory feel less like hidden tracking and more like an explicit conversational feature.

🔧 Giving the Agent Tools

Another major shift happened when FinSaathi stopped relying only on the model's internal knowledge.

On Day 5, I added domain-specific tools.

The basic architecture became:

User asks a question
        ↓
FinSaathi understands intent
        ↓
Does this require external data?
        ↓
       Yes
        ↓
     Tool Call
        ↓
 Real / Available Data
        ↓
FinSaathi explains the result

This distinction is important.

An AI model can generate an answer.

A tool can retrieve or calculate something.

Those are not the same thing.

I also designed failure handling so that if a data source is unavailable, the agent doesn't silently fail or invent an answer.

Instead, it explains that the information could not be retrieved.

📞 Making FinSaathi Call Users

Day 6 introduced one of the most interesting pieces of the project:

Outbound Voice Calls

Instead of waiting for the user to open the browser and start a conversation, FinSaathi could initiate a call.

I integrated LiveKit's SIP capabilities with a telephony setup using Linphone.

The basic flow became:

Frontend
   ↓
Outbound Call API
   ↓
LiveKit Room
   ↓
FinSaathi Agent
   ↓
SIP Participant
   ↓
Phone

This changed the experience from:

"Come talk to my AI."

to:

"My AI can proactively reach out to me."

That opened up an entirely different set of possibilities for reminders and proactive assistance.

👤 Knowing When to Ask a Human

One of the most important design decisions came on Day 7.

An AI agent shouldn't try to solve everything.

For Financial Services, situations involving possible fraud or decisions beyond the agent's authority should trigger human assistance.

So I introduced a human-help workflow.

The agent:

Identifies a situation requiring human help.
Explains what information it wants to share.
Asks the user for permission.
Creates a human-help request.
Gives the user a reference ID.
Explains what happens next.

🚫 Escalation Data Must Stay Safe

The escalation summary should not contain:

Passwords
OTPs
PINs
Account numbers
Unnecessary private information

This created a simple principle:

🛡️ AI should know when it has reached the edge of its authority.

📊 Measuring the Agent

On Day 8, I added a Call Analytics Dashboard.

Instead of simply saying:

"The agent works."

I wanted to measure what actually happened.

The dashboard tracks:

Metric	Description
Total Calls	Total number of calls
Successful Calls	Calls completed successfully
Failed Calls	Calls that failed
Success Rate	Percentage of successful calls

The values come from actual call outcomes rather than hardcoded numbers.

For FinSaathi, a successful call can mean that the user successfully completes an eligibility interaction or receives the required information.

This creates a feedback loop:

Voice Call
    ↓
Call Outcome
    ↓
Database
    ↓
Analytics
    ↓
Dashboard

That was an important shift from building an AI demo to thinking about an AI system.

🤝 Teaching One Agent to Delegate

Day 9 was probably the most interesting architectural step.

I didn't want FinSaathi to pretend it was an expert in everything.

So I introduced a second agent:

SchemeSathi

SchemeSathi is a Government Scheme Specialist focused specifically on detailed government-scheme questions.

Now the system looks like:

                         FinSaathi
                             │
                  ┌──────────┴──────────┐
                  │                     │
          Normal Question       Specialist Question
                  │                     │
                  ▼                     ▼
             FinSaathi             SchemeSathi
                                     Specialist

Example: Normal Question

"UPI fraud se safe kaise rahun?"

FinSaathi handles it directly.

Example: Specialist Question

"PMJJBY ki detailed eligibility aur required documents kya hain?"

FinSaathi recognizes that the question needs specialized assistance.

It tells the user:

"Main aapko Government Scheme Specialist se connect karta hoon. Aapko apni query dobara explain nahi karni padegi."

Then SchemeSathi takes over.

The specialist already receives the relevant context.

So it can say:

"Aapne PMJJBY ki eligibility aur required documents ke baare mein poocha tha. Main wahi se continue karta hoon."

The user doesn't have to repeat the question.

This is where the project started feeling less like a chatbot and more like a multi-agent system.

🧩 The Biggest Lessons From Building FinSaathi

The biggest lesson wasn't about one specific API.

It was learning how many moving pieces are required to make a voice agent actually useful.

A voice agent is not simply:

LLM + TTS

It is closer to:

Voice
  +
Real-time transport
  +
LLM
  +
Prompting
  +
Tools
  +
Memory
  +
Safety
  +
Error handling
  +
Telephony
  +
Human escalation
  +
Analytics
  +
Specialist routing

Every new capability introduced another failure point.

And that taught me to think more like a systems engineer.

🧯 The Difficult Parts

The challenge wasn't completely smooth.

And honestly, that's one of the most useful parts of building it.

1. Outbound Calling

Getting the outbound voice flow working required more than simply adding a "Call" button.

There was a chain involving:

Frontend
   ↓
Backend
   ↓
LiveKit
   ↓
SIP Trunk
   ↓
Telephony Client
   ↓
Agent

A problem anywhere in that chain could result in a call that connected but didn't behave as expected.

Debugging the call path taught me to inspect the system layer by layer instead of assuming the model itself was the problem.

2. Tool Reliability

Tools introduce another failure mode.

The model might correctly decide that it needs external information, but the external source can still fail.

That meant the agent needed explicit failure behavior.

Instead of:

"I don't know."

or worse:

Inventing an answer.

the agent should communicate that the requested data could not currently be retrieved.

3. Context Transfer

The specialist handoff introduced another subtle problem.

It isn't enough to create a second agent.

The second agent needs to understand:

"Why am I here?"

Without context transfer, the user ends up explaining the same thing twice.

That defeats one of the main advantages of delegation.

So I treated context as part of the handoff itself rather than simply switching agents.

🛠️ How to Build Your Own Voice Agent

If you want to build something similar, don't start by trying to build everything at once.

Start with the smallest possible voice loop.

1. Understand the Core Architecture

At minimum, you need:

Speech-to-Text
      ↓
     LLM
      ↓
Text-to-Speech
      ↓
Real-Time Audio Transport

Then gradually add:

🧠 Memory
🔧 Tools
🛡️ Guardrails
📞 Telephony
👤 Human Escalation
📊 Analytics
🤝 Specialist Agents

This incremental approach makes debugging much easier.

🚀 2. Create the Project

A typical structure can look like:

voice-agent/
│
├── backend/
│   └── src/
│       ├── agent.py
│       ├── prompt.py
│       ├── tools/
│       ├── database/
│       └── outbound_call.py
│
├── frontend/
│   ├── app/
│   ├── components/
│   └── ...
│
├── .env.local
├── .gitignore
└── README.md

Keep the agent logic separate from the frontend.

That makes it much easier to test and extend later.

🔐 3. Keep API Keys Out of Your Code

Never write secrets directly inside Python or JavaScript files.

Use environment variables.

For example:

MURF_API_KEY=your_murf_api_key
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

Then access them from your application environment.

Your .env files should be included in .gitignore.

For example:

.env
.env.local
.env.*.local

And commit a safe example file instead:

MURF_API_KEY=your_api_key_here
LIVEKIT_API_KEY=your_api_key_here
LIVEKIT_API_SECRET=your_api_secret_here

Never publish:

🔑 API keys
🔐 SIP credentials
📞 Phone numbers
👤 Caller information
🗄️ Database credentials
🔒 Private conversation data

🎙️ 4. Connect Your Agent

Once your environment variables are configured:

Start the backend.
Start the frontend.
Start/connect the real-time transport.
Open the voice interface.
Grant microphone permission.
Start a conversation.
Verify that speech reaches the agent.
Verify that the response comes back as audio.

The first milestone should simply be:

I can speak
      ↓
The agent understands
      ↓
The agent speaks back

Don't add memory or telephony before this works reliably.

🧠 5. Add Tools and Memory

Once the basic voice loop works:

Add memory for information that genuinely improves future conversations.

Then add tools for information that the model should not simply guess.

A useful rule is:

💡 If the answer can change or needs verification, consider using a tool instead of relying entirely on the model.

🛡️ 6. Add Guardrails Before Going Further

This is especially important for sensitive domains.

Define clearly:

What the agent CAN do

Explain
Guide
Retrieve information
Ask clarifying questions
Summarize
Escalate

What the agent CANNOT do

❌ Request sensitive credentials
❌ Guarantee financial outcomes
❌ Pretend to access private accounts
❌ Invent unavailable information
❌ Make decisions outside its authority

Writing these boundaries before adding advanced capabilities makes the rest of the system much safer.

📞 7. Add Telephony Carefully

Once browser-based conversations work, telephony can be added.

The architecture becomes:

User Phone
    ↓
SIP / Telephony
    ↓
Real-Time Communication Layer
    ↓
Voice Agent
    ↓
TTS
    ↓
Phone

Start with a number you control and test the entire call lifecycle.

Don't immediately build complicated outbound workflows.

First prove:

Agent can call
      ↓
User answers
      ↓
Agent speaks
      ↓
Call ends

Then add business logic.

🤝 8. Add Human and Specialist Escalation

Once the agent can operate independently, teach it when to stop.

There are two very different forms of escalation.

👤 Human Escalation

AI → Human

Use this when the situation requires human authority or intervention.

🤖 Specialist Handoff

Main Agent → Specialist Agent

Use this when another AI agent has narrower expertise.

Keeping these concepts separate makes the architecture much cleaner.

📊 9. Measure What Actually Happens

Finally, build analytics around real events.

Don't hardcode:

Total Calls = 100
Successful Calls = 85

Instead:

Call
 ↓
Outcome
 ↓
Database
 ↓
Dashboard

That turns a demo into something you can actually evaluate.

🔮 What's Next for FinSaathi?

The 10-day challenge is over, but FinSaathi doesn't have to be.

There are several directions I'd like to explore next:

More financial-literacy workflows
Better multilingual support
More verified financial-data tools
Improved specialist routing
More robust evaluation
Better observability
Production-grade authentication
Stronger privacy controls
More sophisticated analytics
Deployment at scale

The goal isn't to make FinSaathi answer everything.

The goal is to make it useful, trustworthy, and responsible.

🎓 What I Learned

If I had to summarize the entire 10-day challenge in a few lessons, they would be:

1. Voice Changes the UX

A voice agent isn't just a chatbot with audio.

Turn-taking, latency, interruptions, tone, and language all matter.

2. Tools Make Agents More Useful

An LLM can reason about a request, but tools allow the system to interact with the real world.

3. Memory Needs Boundaries

Remembering everything isn't necessarily a feature.

Good memory means remembering the right things, with user awareness and consent.

4. Safety Should Be Designed, Not Added Later

Especially in Financial Services, guardrails aren't optional.

5. Agents Need to Know Their Limits

Sometimes the best answer isn't an AI-generated answer.

Sometimes it's:

"I need a human."

And sometimes it's:

"I'll connect you to a specialist."

6. A Working System Is More Than a Working Model

The model is only one part of the product.

The real engineering challenge is connecting all the pieces reliably.

🚀 Final Thoughts

Ten days ago, I started with the idea of building a voice agent.

Today, FinSaathi can:

🎙️ Hold voice conversations
🇮🇳 Communicate naturally in Hindi/Hinglish
🛡️ Follow financial safety guardrails
🧠 Remember returning users
🔧 Use domain-specific tools
📞 Make outbound calls
👤 Escalate to human support
📊 Track call outcomes
🤝 Hand specialized conversations to another AI agent

And that's probably the biggest thing I learned:

Building an AI agent isn't about giving a model more capabilities. It's about designing a system that knows what to do, when to do it, and when not to do it.

I'm incredibly glad I took on the 10 Days of Voice Agents — VoiceForBharat Edition challenge.

It pushed me beyond simply building an AI model and into thinking about product design, real-time systems, safety, reliability, and agent architecture.

Huge thanks to the team behind the challenge and to Murf AI for creating an environment where builders could experiment with voice agents.

And this is only the beginning. 🚀

💻 GitHub Repository

https://github.com/ShazilParwez/vox-nova/tree/day10

🌐 Live Demo

Day 4 [https://www.linkedin.com/posts/shazil-parwez_voiceforbharat-activity-7492141057862799361-P98H?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAFGqLokBOy5Fxt_Vd6OGVOwKhGPH8G_8tQw]

Day 6 [https://www.linkedin.com/posts/shazil-parwez_voiceforbharat-10daysofvoiceagents-murfai-activity-7492960482925064194-1yV3?utm_source=social_share_send&utm_medium=member_desktop_web&rcm=ACoAAFGqLokBOy5Fxt_Vd6OGVOwKhGPH8G_8tQw]