Building FinSaathi: A Voice-First Financial Assistant for Bharat 🇮🇳 10 Days of Voice Agents — VoiceForBharat Edition

Over the last 10 days, I built FinSaathi, a voice-first AI assistant for the Financial Services track of the VoiceForBharat challenge.
The goal was simple: build an assistant that can talk naturally with users, understand financial and government-scheme related queries, remember relevant information, use tools, and know when a human or specialist should take over.
What started as a basic voice agent gradually became a complete system with memory, tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs.
💡 The Problem
Financial and government-scheme processes can involve eligibility requirements, documents, deadlines, and complicated terminology.
For users who are more comfortable speaking than typing, voice can make these interactions much more natural.
For example, a user can simply ask:
"PMJJBY ke liye main eligible hoon?"
Instead of navigating through multiple forms, FinSaathi can understand the request, collect the required information, perform an eligibility check, and explain the result conversationally.
The goal is not to replace banks or human support, but to provide a conversational first layer of assistance and escalate situations when human help is required.
🏗️ Architecture
USER
│
▼
LiveKit
│
▼
Speech-to-Text
│
▼
LLM / Agent
│
┌────────────┼────────────┐
▼ ▼ ▼
Memory Tools Escalation
│ │ │
└────────────┼────────────┘
▼
SQLite DB
│
┌──────┴──────┐
▼ ▼
Human Support Analytics
Dashboard Dashboard
│
▼
Murf Falcon
│
▼
USER
Technology Stack
Component
Technology
Frontend
Next.js / React
AI Agent
LiveKit Agents
Real-time Transport
LiveKit
Text-to-Speech
Murf Falcon
Backend
Python
API
FastAPI
Database
SQLite
Calling
SIP / LiveKit
🎙️ Key Features

1. Indian Voice & Natural Conversations FinSaathi uses Murf Falcon for text-to-speech and supports natural Hindi/Hinglish conversations. The goal was to make the interaction feel more like talking to an assistant rather than interacting with a traditional chatbot.
2. Safety Guardrails Financial conversations require strong safety boundaries. For example, when a user reports an unauthorized transaction, FinSaathi warns them not to share: OTP PIN Password CVV Card details The agent also clearly explains when it does not have direct access to a user's bank account or transaction system.
3. User Memory FinSaathi stores relevant user information in a SQLite database so it can be reused in future conversations. The database logic is separated into its own layer: Agent ↓ database.py ↓ SQLite ↓ users table This made it easier to extend the project without mixing database operations throughout the agent code.
4. Government Scheme Eligibility FinSaathi can use tools to perform government-scheme eligibility checks. User asks about scheme ↓ Collect required information ↓ Eligibility tool ↓ Eligible / Not Eligible ↓ Database ↓ User receives result The eligibility result can also be used to determine whether the user's intended task was successfully completed.
5. Outbound Calling FinSaathi can proactively contact eligible users using outbound calling. Database ↓ Check deadlines / eligible users ↓ Outbound calling logic ↓ SIP / LiveKit ↓ AI Agent ↓ User Setting up and debugging the SIP and LiveKit calling flow was one of the most challenging parts of the project.
6. Human Escalation FinSaathi can recognize situations where human support is more appropriate. For example, for an unauthorized transaction, the agent can ask: "Agar aap chahein, toh main aapki situation ke liye ek human support request create kar sakti hoon." After the user gives consent, the system creates an escalation and generates a unique reference ID. Example: FS-A5323F The request is stored with information such as: Reason: possible_fraud Urgency: HIGH Status: OPEN Preferred Follow-up: phone The human-support dashboard then allows the request to move through: OPEN ↓ IN_PROGRESS ↓ RESOLVED
7. Call Analytics FinSaathi also records the outcome of conversations. For this project, a successful interaction means that the user's intended task was completed, such as completing an eligibility check or receiving the required information. The dashboard displays: Total Calls Successful Calls Failed Calls These values come from actual browser or SIP interactions rather than hardcoded numbers. Actual Call ↓ Determine Outcome ↓ call_analytics ↓ FastAPI ↓ Analytics Dashboard A failed call does not necessarily mean a technical failure. For example, if a user leaves before completing an eligibility check, the interaction can be recorded as unsuccessful.
8. Specialist Agent Handoff Instead of making one agent responsible for every type of question, FinSaathi can hand government-scheme related conversations to a Government Scheme Specialist. Main Agent │ Government scheme? │ YES ↓ Government Scheme Specialist │ ↓ Eligibility / Documents / Benefits / Scheme Information This makes the architecture more modular and allows additional specialist agents to be added in the future. 🧩 Challenges I Faced The project definitely did not work perfectly on the first attempt. One of the biggest challenges was real-time calling and LiveKit/SIP integration. During testing, I encountered errors such as: WinError 64 ConnectionResetError DuplexClosed I also faced an outbound-call issue where the AI would start speaking but the call could terminate before the conversation continued. Debugging this required checking: LiveKit worker lifecycle SIP configuration Network connections Agent processes Call state Real-time transport This taught me that building a voice agent is not only about the LLM. The real-time infrastructure around the agent matters just as much. Another important lesson was separating responsibilities between the agent, database, API, and frontend. For example: Agent → database.py → SQLite while the human-support dashboard uses: Frontend → FastAPI → database.py → SQLite This separation made the system easier to extend and debug.🚀 What I Learned Before this challenge, I thought building a voice agent was mainly: Speech → AI → Speech After these 10 days, I realized that a useful voice agent needs much more: Voice + LLM + Memory + Tools + Safety + Real-time communication + Database + Human escalation + Analytics + Specialist handoffs The biggest lesson for me was: A useful voice agent isn't just an AI that can talk. It's a system that can understand, act, remember, measure its performance, and know when a human or specialist should take over. Building FinSaathi during the 10 Days of Voice Agents — VoiceForBharat Edition gave me hands-on experience with all of these pieces.

Comments

[Alex Shev]

The debugging lesson here is that the system needs to explain what it believed at the decision point. Without that, every failure becomes archaeology.