Tanush

Posted on Aug 15

Building MoneyBuddy: A Multilingual Financial AI Voice Agent for Bharat with Murf Falcon🎙️

#ai #software #design #coding

What started as a simple voice agent became something much bigger over the last ten days.

I built MoneyBuddy, a multilingual AI voice agent designed for the Financial Services track of the 10 Days of Voice Agents — VoiceForBharat Edition.

The goal was simple:

Make financial guidance easier to access through a natural voice conversation.

Instead of forcing users to navigate complicated forms, search through scheme websites, or understand financial terminology on their own, MoneyBuddy lets them simply talk.

It can remember returning callers, check government-scheme information, make outbound calls, escalate difficult situations to humans, track call outcomes, and even hand conversations to a specialist agent.

And the entire journey pushed me far beyond simply making an LLM talk.

The Problem: Financial Information Is Often Hard to Access

Government financial schemes can provide meaningful support, but discovering whether a scheme is relevant, understanding eligibility, finding required documents, and knowing what to do next can be difficult.

For many users, especially those more comfortable speaking than typing, voice can make that interaction much more natural.

That became the idea behind MoneyBuddy:

A voice-first financial companion that explains, guides, and knows when it should stop and ask for help.

MoneyBuddy focuses on areas such as government schemes, financial literacy, eligibility guidance, and fraud awareness.

It is intentionally not a banking transaction system.

It does not ask users for OTPs, PINs, passwords, CVVs, full card numbers, or sensitive account credentials.

What I Built

Over the challenge, MoneyBuddy evolved through several layers:

Indian voice support using Murf Falcon
Financial-services personality and guardrails
English, Hindi, and Hinglish conversations
A dedicated financial-services frontend
Persistent caller memory using SQLite
Function-based memory lookup and saving
Government scheme data tools
Context chaining between memory and tools
Outbound phone calls
Human escalation
Call analytics
Specialist-agent handoffs
Deterministic frontend agent-state tracking

The important lesson was that a useful voice agent is not just:

Speech → LLM → Speech

It becomes a real application when you add state, tools, memory, safety, observability, and failure handling around that loop.

How MoneyBuddy Works

At the core, MoneyBuddy uses four major components:

Speech-to-text — Deepgram converts the caller's speech into text.
LLM — Gemini handles reasoning, conversation, tool selection, and responses.
Text-to-speech — Murf Falcon converts responses into natural speech.
Real-time transport — LiveKit handles the real-time audio connection.

For phone calls, the system also connects through LiveKit SIP.

Architecture

flowchart LR
    U["User / Caller"]
    LK["LiveKit<br/>Real-Time Audio"]
    STT["Deepgram STT"]
    LLM["Gemini LLM"]
    TTS["Murf Falcon TTS"]
    MB["MoneyBuddy"]
    DB["SQLite<br/>Caller Memory"]
    DATA["Scheme Dataset"]
    HUMAN["Human Support"]
    SPEC["Government Scheme<br/>Specialist"]
    SIP["LiveKit SIP<br/>Outbound Calling"]
    DASH["Analytics Dashboard"]

    U -->|Speech| LK
    LK --> STT
    STT --> MB
    MB --> LLM

    LLM --> DB
    LLM --> DATA
    LLM --> SPEC
    LLM --> HUMAN

    LLM --> TTS
    TTS --> LK
    LK -->|Voice response| U

    MB --> DASH

    MB --> SIP
    SIP --> U

The interesting part is that the LLM is not responsible for everything.

Memory is handled through tools.

Scheme information comes from a grounded dataset.

Human escalation is handled through a dedicated tool.

Specialist routing is handled through another tool.

That separation made the system much easier to reason about.

1. Giving MoneyBuddy a Voice

The first challenge was getting a voice agent working end-to-end.

I chose an Indian voice because the target experience is for Indian users.

MoneyBuddy uses Murf Falcon for text-to-speech.

The voice choice was important because a financial assistant should sound approachable and trustworthy rather than like a generic robotic IVR.

Murf Falcon also became an important part of the latency optimization work because voice agents are extremely sensitive to delays between a user finishing their sentence and hearing the response.

Murf AI on LinkedIn

2. Personality, Guardrails, and Multilingual Conversations

Once the basic voice loop worked, MoneyBuddy needed a job and boundaries.

I defined clear objectives around:

Government-scheme guidance
Financial literacy
Fraud awareness

Then I added strict financial safety rules.

MoneyBuddy must never request:

OTPs
PINs
Passwords
CVVs
Full card numbers
Sensitive banking credentials

It also cannot promise guaranteed scheme approval or financial outcomes.

Voice-first prompting matters

A voice response cannot be written like a webpage.

Long paragraphs, bullet lists, brackets, and complicated sentences sound unnatural when spoken.

So I introduced speech-specific rules:

Short sentences
Natural pauses
Simple language
No raw JSON
No markdown-style responses
No unnecessary technical terminology

I also added support for Hindi and Hinglish.

For example, a user can naturally switch between languages instead of having to select a rigid language mode before speaking.

3. Building a Frontend for Voice

The frontend was redesigned around the actual states of a voice conversation.

Instead of simply showing a microphone button, MoneyBuddy clearly communicates whether it is:

Ready
Connecting
Listening
Speaking
Call ended

It also provides:

Live transcript
Microphone permission guidance
Language selection
Financial safety messaging
Reconnection/loading states
Call-ended controls
Analytics access
Human-support access

Frontend screenshot

[INSERT IMAGE HERE — MoneyBuddy frontend showing the main conversation interface]

The interface is intentionally simple.

The user should know what to do without having to understand how LiveKit, STT, LLMs, or TTS work underneath.

4. Giving MoneyBuddy Memory

A voice assistant becomes much more useful when returning callers don't have to repeat everything.

I added SQLite-based persistent memory.

A caller record can contain information such as:

user_id
name
language_preference
facts
last_interaction

But the LLM does not simply receive the entire database through its prompt.

Instead, MoneyBuddy has functions for memory operations.

That distinction was important.

The model has to decide when it needs information and call the appropriate tool.

For example:

@function_tool
async def lookup_caller(user_id: str):
    """Look up a caller's saved profile and relevant financial facts."""
    return get_caller(user_id)

I also added sanitization before storing financial information so sensitive credentials are not persisted as ordinary caller facts.

And importantly:

MoneyBuddy asks for permission before saving information.

5. Giving the Agent Real Domain Knowledge

Memory alone isn't enough.

The next step was giving MoneyBuddy access to actual financial-domain information.

For the challenge, I used a grounded local dataset containing Indian government schemes such as:

PM Kisan
PM Suraksha Bima Yojana
PM Jeevan Jyoti Bima Yojana
Atal Pension Yojana
PM Mudra Yojana

MoneyBuddy can call a function to retrieve relevant scheme information rather than generating eligibility details from memory.

This also gave me an important design principle:

When the answer depends on structured domain data, use a tool instead of hoping the LLM remembers the right answer.

The dataset also includes recency information so the agent can communicate when its information was last updated.

6. Making MoneyBuddy Call Users

On Day 6, MoneyBuddy stopped waiting for the user to initiate every conversation.

I added outbound calling through LiveKit SIP.

The use case was a financial one:

A reminder for someone who had already been identified as eligible for a government scheme with an approaching deadline.

Outbound conversations need a different opening from normal inbound conversations.

The caller didn't ask to speak with the agent.

So MoneyBuddy needs to establish three things immediately:

Who is calling.
Why they are calling.
How the user can stop the call.

This made the outbound experience much more respectful and transparent.

7. Knowing When AI Should Ask a Human

One of the biggest lessons from this challenge was that a good AI agent should know its limits.

For MoneyBuddy, I added human escalation for situations such as:

Possible financial fraud
Problems requiring a decision the AI cannot make

The escalation system creates a short summary containing only useful information.

It can include:

What happened
What the agent already checked
Urgency
Preferred language
Preferred follow-up method

Before sharing the information, MoneyBuddy asks the caller for permission.

Sensitive credentials such as OTPs, PINs, passwords, and account numbers are excluded.

The caller also receives a reference ID and a clear explanation of what happens next.

8. Measuring Calls Instead of Guessing

A voice application needs observability.

MoneyBuddy records call outcomes and exposes them through an analytics dashboard.

The dashboard tracks:

Total calls
Successful calls
Failed calls
Success rate
Call history
Failure categories
Track-specific outcomes
Latency

Dashboard screenshot

[INSERT IMAGE HERE — MoneyBuddy Analytics Dashboard showing real call metrics]

My measured results

The final published values here should come directly from the real MoneyBuddy dashboard, not from automated test fixtures.

Total calls: Use the actual dashboard value
Successful calls: Use the actual dashboard value
Failed calls: Use the actual dashboard value
Success rate: Use the actual dashboard percentage
Latency: Use the actual measured latency from the dashboard/test call

I deliberately don't substitute automated test numbers for real user-call measurements.

9. The Multi-Agent Step

The final major technical step was turning MoneyBuddy into a multi-agent system.

MoneyBuddy remains the main financial assistant.

When the conversation requires deeper government-scheme knowledge, it can hand the conversation to a dedicated:

Government Scheme Specialist

The specialist has its own instructions, role, voice, and boundaries.

Handoff flow

flowchart TD
    A["User asks financial question"] --> B["MoneyBuddy"]
    B --> C{"Does this need<br/>specialist knowledge?"}

    C -->|No| D["MoneyBuddy continues"]
    C -->|Yes| E["Handoff Tool"]

    E --> F["Government Scheme Specialist"]
    F --> G["Continue same conversation"]
    G --> H{"Task complete?"}

    H -->|No| F
    H -->|Yes| I["Return to MoneyBuddy"]
    H -->|User changes topic| I

The important part is that the user should not have to repeat the entire problem.

The specialist receives the relevant conversational context and continues from there.

10. Solving Handoff Latency

The specialist handoff created another voice-specific problem.

If the system waited for the specialist's LLM response before starting speech, the user could experience an awkward silence.

I changed the handoff flow so the specialist can immediately speak its introduction while the rest of the response is being generated.

Conceptually:

await session.say(
    "Hi, I'm the Government Scheme Specialist. "
    "I'll help you with that."
)

session.tts.update_options(voice="Nikhil")

The goal was to separate:

"The specialist has taken over"

from:

"The specialist has finished generating the complete answer."

That small architectural change makes a voice handoff feel much more immediate.

11. Deterministic Agent Identity

Another problem appeared when the frontend tried to determine which agent was active by looking at transcript text.

That approach is fragile.

If the transcript happened to contain words like "specialist", the UI could incorrectly display the specialist state.

So I changed the architecture to use explicit LiveKit participant metadata.

The backend maintains an explicit state such as:

active_agent = moneybuddy

or:

active_agent = specialist

The frontend reads that state directly.

This is a broader lesson:

UI state should come from application state, not guesses extracted from conversation text.

12. One Design Decision I Changed

One of the biggest design decisions I changed during the challenge was how the agent handled memory.

The initial temptation was to load stored caller information directly into the system prompt.

That is easy to implement, but it creates several problems.

The model receives information it may not need, the prompt becomes larger, and the separation between application data and model instructions becomes weaker.

I changed this so MoneyBuddy starts without blindly injecting stored caller facts.

Instead:

Need information → call lookup tool → use returned data.

That made the memory architecture cleaner and much easier to control.

13. The Hard Parts

The hardest part of this project wasn't getting an LLM to answer questions.

It was making the entire system behave reliably as a real-time voice application.

Some of the biggest problems were:

Multilingual voice handling

Hindi and Hinglish are not simply English text translated into Hindi.

The STT, language detection, prompt instructions, and TTS configuration all need to work together.

The agent also needs to produce the correct writing system so the speech pipeline receives appropriate text.

Voice formatting

Text that looks perfectly fine in a chat window can sound terrible when spoken.

I had to continuously simplify responses, remove formatting artifacts, and keep sentences short.

Handoff latency

Switching agents introduces another potential delay.

The solution was to start the specialist's spoken introduction immediately rather than waiting for the complete LLM generation.

Agent identity

Transcript-based UI detection caused unreliable specialist labels.

Explicit LiveKit metadata solved that problem.

Keeping previous days stable

This was probably the most important engineering discipline of the challenge.

Every new feature had to coexist with everything already built.

I maintained separate day branches and repeatedly ran the existing test suites before pushing changes.

That helped prevent a Day 9 feature from accidentally breaking Day 4 memory or Day 8 analytics.

14. Running MoneyBuddy Yourself

The complete Day 10 repository is available here:

MoneyBuddy — Day 10 GitHub Repository

The project is organized around a backend voice agent and a Next.js frontend.

The main components are:

murf-livekit-starter/
├── backend/
│   ├── src/
│   │   ├── agent.py
│   │   ├── prompt.py
│   │   ├── db.py
│   │   ├── schemes_data.json
│   │   └── outbound.py
│   └── tests/
│
└── frontend/
    ├── components/
    ├── app/
    └── styles/

Basic setup

Clone the repository and install the backend dependencies using the project's package manager.

Then install the frontend dependencies.

Create your environment configuration files and add your own API credentials.

For example:

MURF_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
GOOGLE_API_KEY=your_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

Never commit real API keys to GitHub.

Start the LiveKit/backend services and then start the Next.js frontend.

Open the application in your browser, allow microphone access, and start a conversation.

15. Troubleshooting

The agent connects but doesn't speak

Check the backend logs first.

Then verify:

LiveKit credentials
STT configuration
LLM configuration
Murf API credentials
TTS voice configuration

A voice agent is a pipeline. One broken component can make the entire experience appear silent.

Hindi is understood but sounds wrong

Check both sides of the pipeline.

The STT must support multilingual input, and the TTS configuration must use the appropriate Indian voice configuration.

The LLM should also return Hindi in Devanagari rather than romanized Hindi when that is the intended output.

The specialist handoff feels slow

Don't wait for the entire specialist response before starting audio.

Give the specialist an immediate spoken handoff introduction, then allow the model to continue generating the detailed response.

The frontend shows the wrong active agent

Don't infer agent identity from transcript text.

Use explicit application state or LiveKit participant metadata.

16. What I Would Improve Next

If I continued building MoneyBuddy beyond the challenge, I would focus on production readiness.

Some of the next areas would be:

Stronger evaluation of multilingual conversations
More comprehensive government-scheme data sources
Better tool observability
More robust telephony retry handling
Stronger authentication and privacy controls
Better human-support workflows
More detailed latency breakdowns
Production-grade monitoring
Larger-scale testing with real conversations

The goal would be to move from a challenge project toward a system that could be responsibly used in a real financial-support environment.

What Ten Days Taught Me

The biggest lesson from this challenge is that building a voice agent isn't primarily about choosing the biggest model.

It's about everything around the model.

You need:

Good speech recognition.
Fast text-to-speech.
Clear instructions.
Strong guardrails.
Useful tools.
Persistent state.
Reliable transport.
Observability.
Human escalation.
And graceful failure handling.

The LLM is only one part of the system.

And once voice becomes the interface, latency and formatting become product decisions, not just engineering details.

MoneyBuddy started as an agent that could hear me and talk back.

Ten days later, it can remember users, access domain data, make outbound calls, escalate to humans, measure conversations, and hand complex questions to a specialist.

That progression was the most valuable part of the challenge.

Final Links

GitHub

MoneyBuddy — Day 10 Repository

Challenge

10 Days of Voice Agents — VoiceForBharat Edition

Built with

LiveKit • Deepgram • Gemini • Murf Falcon • Python • Next.js • SQLite

If you're building your own voice agent, my biggest advice is simple:

Don't start by trying to make it do everything.

Make it talk.

Then give it a job.

Then give it boundaries.

Then give it tools.

Then give it memory.

Then make it reliable.

And only after that, start making it smarter.

DEV Community