Why I Built This
India has government financial schemes for almost everyone — loans for small shopkeepers, income support for
farmers, pensions for the unorganized sector — but knowing which scheme you qualify for, and what documents
you need, is its own small ordeal. It usually means a form, a queue, or a relative who “knows someone at the bank.”
That gap is what **MoneyMitra **is for. It's a voice agent you can call or open in a browser and just talk to, in Hindi,
English, or the Hindi-English mix most of us actually speak. Tell it your age, your occupation, your rough income
— it tells you whether you qualify for something like PM Mudra Loan or PM Kisan, and exactly what documents to
bring. No forms. No app to download. Just a conversation.
I built MoneyMitra over 10 Days of Voice Agents — VoiceForBharat Edition, one capability a day, using Murf Falcon for **speech **throughout. This post is the story of what got built, what quietly broke along the way, and what I'd
tell someone starting the same challenge today.
I chose the Financial Services track because the problem — scheme discovery being harder than it should be — is one I've watched happen to people around me.
What MoneyMitra Actually Does
By the end of ten days, MoneyMitra could:
• Hold a natural, code-mixed Hindi-English conversation in a real Indian voice (Murf Falcon)
• Refuse to do things it shouldn't — never ask for an OTP, PIN, or account number; never promise a scheme
approval
• Recognize returning callers and pick up where the last conversation left off
• Look up real eligibility rules and document checklists for five government schemes
• Make outbound calls to remind eligible users about deadlines, with a proper compliance opening
• Recognize when it's out of its depth (a fraud report, a decision it can't make) and hand the caller to a
human, with consent
• Track every call's outcome on a dashboard, so I could actually see whether the agent was working or just
sounding like it was
• Hand off scheme-specific questions to a second, specialist agent — Samer — with his own voice and a
narrower, sharper job
🏗️ How the System Works
The stack is the standard voice-agent shape, wired together with LiveKit:
• STT — Deepgram, in multilingual mode, so it can pick up Hindi and English in the same sentence
• LLM — Google Gemini, doing the reasoning and deciding which tool to call
• TTS — Murf Falcon, giving each agent its Indian voice
• Real-time transport — LiveKit, handling browser WebRTC and SIP for outbound calls
• SQLite — for caller memory, escalation tickets, and call analytics
🚀 Complete Day-by-Day Journey
🔹 Day 1: Real-Time Audio Pipeline & Murf Falcon TTS
Configured LiveKit Agents SDK with browser WebRTC and SIP audio streaming.
Wired Deepgram Nova-3 (model="nova-3", language="multi") for multilingual speech recognition, and Murf Falcon TTS (hi-IN-anisha — female Indian Hindi-English voice) for speech synthesis.
Implemented Silero VAD prewarming via prewarm() so the agent starts listening with low latency instead of a cold-start delay.
Demo video -Day-1 video
🔹 Day 2: System Persona, Tone & Safety Guardrails
Defined the MoneyMitra identity — a clear, professional, empathetic Hindi-English (Hinglish) financial-guidance voice.
Set hard safety guardrails: never ask for an OTP, PIN, or account number; never promise scheme approval.
Verified the agent could hold a code-mixed conversation and stay in character across multiple turns.
Demo video -Day-2 video
🔹 Day 3: Financial-Advisor Frontend
Built a custom UI (financial-avatar.tsx, agent-state-header.tsx, mic-permission-modal.tsx) themed in trust-building emerald green and navy.
Implemented all five agent states clearly: Ready, Connecting, Listening, Speaking, and Call Ended.
Added a mic-permission-denied modal with browser-specific unblock instructions (Chrome, Safari, Edge, Firefox).
Demo Video-Day-3
🔹 Day 4: SQLite Persistent Memory
Built caller memory storage (db.py) with lookup_caller and save_caller_info function tools.
Consent rule: MoneyMitra only writes caller data to storage after explicit user_consent=True — declining means nothing is saved.
Returning callers are greeted by name and the conversation continues from their last recorded topic.
Demo video - Day-4 video
🔹 Day 5: Government Scheme Eligibility Engine
Built** check_scheme_eligibility ***(scheme_data.py)*, evaluating real eligibility and document checklists based on age, annual income, occupation, category, and requested loan amount for:
PM Mudra Loan (Pradhan Mantri Mudra Yojana)
PM Kisan Samman Nidhi
PM SVANidhi
Stand-Up India
Atal Pension Yojana (APY)
Added graceful failure handling — a timed-out or failed lookup produces a spoken explanation, not silence or an invented answer, and every response states the policy date the data is current as of.
*Demo video *- Day-5 video
🔹 Day 6: Outbound SIP Calling & Compliance Opening
Integrated outbound telephony over SIP **(via Linphone, in place of Twilio) **for reminder-style calls.
Mandatory 2-sentence compliance opening: every outbound call states who's calling, why, and gives an immediate opt-out — "Agar aap aage se aisi reminder calls nahi chahte, toh aap abhi 'stop' bolein ya bol skate ki 'call mat karo'".
Demo video -Day-6 video
🔹 Day 7: Human Escalation & Fraud Protocol
Built create_escalation — creates a support ticket for fraud reports or decisions beyond the agent's authority, only after explicit caller consent.
Generates secure ticket codes in the format ECW_XXXXXXXX, with urgency ratings (HIGH for fraud), while excluding passwords, OTPs, PINs, and account numbers from the stored summary.
Added opt_out_caller, triggered by phrases like "stop," "opt out," or "don't call me," recorded for future outbound-call suppression.
Demo video-Day-7 video
🔹 Day 8: Call Analytics Dashboard
Added a call_analytics table and a dashboard tracking real-time Total Calls, Successful Calls, and Failed Calls.
Defined success meaningfully — a completed eligibility check or created escalation, not just "the call didn't crash."
Real bug caught and fixed: calls were showing SUCCESS even when hung up on mid-sentence, because outcome tracking relied on crash/no-crash rather than actual goal completion. Fixed with an objective_achieved flag that only flips true inside tools representing a real outcome.
Demo video -Day-8 video
🔹 Day 9: Multi-Agent Handoff — "Samer"
Built a separate specialist agent,** SchemeSpecialist ***(scheme_specialist.py)*, named Samer, scoped narrowly to scheme eligibility, benefits, and documents.
Added transfer_to_scheme_specialist — announces the handoff out loud and transfers the live conversation without asking the caller to repeat themselves.
Dynamic voice swap: session TTS switches from MoneyMitra's hi-IN-anisha to Samer's distinct voice, so the handoff is audible, not just stated.
Used a shared shared_state dictionary across both agents so Day 8's objective_achieved tracking stays correct regardless of which agent completes the task.
Three real bugs caught and fixed: a constructor-ordering issue that left the specialist silent after handoff, a missing tts= parameter that meant the voice wasn't actually swapping, and a stray generate_reply() call left in the introduction.
Demo video - Day-9 video
🐛 Real-World Challenges & Silent Bugs Solved
🐞 Bug #1: Every call showed up as "successful" — even the ones I hung up on
The Symptom: Building the analytics dashboard, I wired up a try/finally block to record the call outcome when the room disconnected. My first instinct was: If nothing crashed, it's a success. I tested it by calling MoneyMitra, saying nothing useful, and hanging up mid-sentence. The dashboard still logged SUCCESS.
The Root Cause: A failed call doesn't mean something broke on the server—it means the call never reached its goal.
The Fix: Shifted from checking "Did an exception happen?" to "Did the agent actually accomplish something?" Added an objective_achieved state flag that only flips to True inside tools that represent real outcomes (e.g., a completed eligibility check or a created escalation ticket). Cutting the call mid-question now correctly shows up as FAILED.
🐞 Bug #2: The specialist agent went completely silent after handoff
The Symptom: MoneyMitra would say "Connecting you to our specialist..." and then dead air. No introduction, no reply, and no error traceback pointing to the cause.
The Root Cause: In the specialist's init, I was setting self.shared_state = shared_state before calling super().init(...). LiveKit's Agent base class performs internal setup inside its constructor. Assigning attributes before that setup ran put the object into a broken state quietly.
The Fix: Reordered the operations in the specialist's constructor:
# Before — silently broken
def __init__(self, shared_state):
self.shared_state = shared_state
super().__init__(instructions=PROMPT)
# After — fixed
def __init__(self, shared_state):
super().__init__(instructions=PROMPT)
self.shared_state = shared_state
🐞 Bug #3: The specialist sounded exactly like the main agent
The Symptom: Once Samer was actually speaking, he sounded identical to MoneyMitra—same voice, same tone. The entire point of a handoff is that it's audibly obvious something changed.
The Root Cause: I had added a distinct voice ID to the design plan, but omitted the tts= parameter in SchemeSpecialist.Agent.init(). The sub-agent was silently inheriting the session's default main voice (hi-IN-anisha).
The Fix: Explicitly passed the specialist TTS instance in the constructor:
super().__init__(
tts=murf.TTS(voice="hi-IN-samer", style="Conversational"),
# ... other tools & configs
)
🗄️ Database Schemas (SQLite)
Located at backend/data/caller_memory.db:
-- Caller Profile Storage
CREATE TABLE caller_profiles (
user_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
language_preference TEXT,
facts TEXT, -- JSON string (Sanitized)
last_topic TEXT,
last_interaction TEXT
);
-- Human Escalation Support Tickets
CREATE TABLE human_escalations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ticket_code TEXT UNIQUE,
user_id TEXT NOT NULL,
caller_name TEXT NOT NULL,
reason_category TEXT NOT NULL,
what_happened TEXT NOT NULL,
what_agent_checked TEXT,
urgency TEXT,
language TEXT,
preferred_followup TEXT,
status TEXT DEFAULT 'OPEN',
created_at TEXT
);
⚡ How to Run MoneyMitra Locally
1. Clone the Repository
git clone https://github.com/krishnaasoni/MoneyMitra-Voice-Agent.git
cd MoneyMitra-Voice-Agent
2. Create a .env file for your API keys and configuration.
# backend/.env.local
LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
MURF_API_KEY=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=
cd backend
uv sync
uv run dotenv -f .env.local run -- python src/agent.py start
cd frontend
pnpm install
pnpm dev
Never commit your .env file to GitHub.
Add it to .gitignore:
Then install the dependencies and start the LiveKit agent according to the project's setup instructions.
3. Run Startup Script
#powershell
.\start_app.ps1
On macOS / Linux:
#Bash
chmod +x start_app.sh
./start_app.sh
Open Web Apps
Voice Agent UI: http://localhost:3000
Human Escalations Portal: http://localhost:3000/escalations
Analytics Dashboard: http://localhost:3000/analytics
🔮 What I'd Improve Next
*MoneyMitra still has a lot of room to grow.
*
Some things I'd like to work on next are:
1.More Indian languages beyond Hindi-English
2.More specialist agents (beyond Samer)
3.Better multilingual conversation handling
4.Improved personalization for returning callers
More advanced call analytics (failure-type breakdown, success rate, trends over time)
5.Wider government-scheme coverage
6.Better accessibility for low-literacy and low-bandwidth users
7.Smarter follow-up workflows (like automatic callbacks after an escalation is resolved)
8.A richer human-support dashboard
9.More robust real-world integrations (payment status checks, live scheme-portal data instead of a local dataset)
📚 What I Learned
The biggest lesson I learned from this challenge is that building a voice agent isn't really about making an AI talk.
The interesting engineering begins after that.
You have to think about:
What should the agent know?
What should it remember?
When should it use a tool?
When should it stop?
When should a human take over?
When should another agent take over?
How should context move between agents?
How do we measure success?
How do we protect user information?
These questions completely changed how I think about voice AI.
🔗 Repository & Resources
GitHub Repository: git-hub link
Murf Falcon Documentation: Falcon 2 TTS Docs
LiveKit Agents Quickstart: LiveKit Documentation
❤️ What 10 Days Taught Me
When I started the challenge, I thought the hardest part would be getting an AI agent to speak.
By Day 10, I realized:
Voice is only the beginning.
The real challenge is building an agent that knows how to behave.
An agent that can say:
"I can help with that."
But also knows when to say:
"Let me connect you with Krishna, our specialist."
And sometimes:
"This needs human assistance."
That's what I wanted FINORA AI to become.
Not just an AI that talks.
An AI that knows what to do next.
🌟 Acknowledgments
A big thank you to @murf AI for hosting the 10 Days of Voice Agents — #VoiceForBharat Edition challenge!
VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #GenerativeAI #AI #FinTech
@MURFAI
If you enjoyed reading this engineering breakdown, please give it a 💖 or 🦄 on DEV.to!
Top comments (0)