DEV Community: Nipun Goel

Building FinSaathi: A Voice-First Financial Assistant for Bharat

Nipun Goel — Sat, 15 Aug 2026 07:51:24 +0000

A voice-first financial assistant for Bharat, built during the 10 Days of Voice Agents — VoiceForBharat Edition.

1. The Real-World Problem: Making Financial Assistance More Conversational

Financial services and government schemes can be difficult to navigate. Users may need to understand eligibility criteria, required documents, application information, deadlines, or what to do when something goes wrong.

For many users, especially users who are more comfortable speaking in Hindi, English, or Hinglish than typing long queries, a voice interface can provide a more natural way to access assistance.

This led to the idea behind FinSaathi.

Instead of making the user navigate through multiple forms and pages, FinSaathi allows the user to simply describe what they need.

"PMJJBY ke liye main eligible hoon?"

Or:

"Mere account se ek unauthorized transaction hua hai."

The goal is not to replace banks, government portals, or human support teams. The goal is to create a conversational first layer that can understand intent, provide appropriate assistance, perform supported actions, and know when the conversation should be handed to a human or specialist.

Why Voice Matters

A text interface gives users time to edit and rewrite their questions. A voice conversation is different: the user explains their situation naturally and the system has to understand the intent in real time.

The interaction becomes:

Speak → Understand → Decide → Act → Respond

That simple loop became the foundation of FinSaathi.

2. Responsible Product Positioning: Assistant, Not Financial Authority

FinSaathi is designed as an assistance system, not as a bank employee, financial regulator, or replacement for human support.

What FinSaathi Is

A conversational financial assistance interface.
A voice-first way to access supported information.
A tool-using AI agent.
A system with persistent user data for supported workflows.
A bridge to human support.
A platform that can route specialized requests to specialist agents.

What FinSaathi Is Not

A replacement for a bank.
A replacement for a government portal.
A system that should request passwords, PINs, OTPs, or CVVs.
A system with unlimited access to private financial accounts.
A guaranteed financial decision-maker.

Safety Guardrails

Financial conversations require strong safety boundaries.

FinSaathi explicitly avoids requesting sensitive authentication information such as:

OTPs.
PINs.
Passwords.
CVVs.
Full card details.

When the agent cannot perform a requested action directly, it should communicate that limitation rather than inventing an action.

3. What Was Built: The 10-Day Evolution

Day	Focus	What was added
Day 1	Core Voice Agent	Initial real-time conversational voice experience.
Day 2	Personality & Guardrails	Defined objectives, behaviour, and safety boundaries.
Day 3	Indian Conversations	Hindi, English, and Hinglish interaction patterns.
Day 4	Memory	Persistent user information using the database.
Day 5	Tools & Eligibility	Application-specific tools and scheme eligibility workflows.
Day 6	Outbound Calling	Agent-initiated phone conversations through the calling layer.
Day 7	Human Escalation	Human-support requests with generated reference IDs.
Day 8	Call Analytics	Real call outcome tracking and dashboard metrics.
Day 9	Specialist Handoff	Routing government-scheme workflows to a specialist agent.
Day 10	Documentation	Architecture, implementation details, challenges, and lessons learned.

The important part was that the features were not built as isolated demos. Each new capability had to work with the voice experience that already existed.

4. Technical Architecture

FinSaathi is built as a full-stack voice-agent system.

At a high level, the user speaks through the real-time communication layer. The speech is processed by the voice pipeline, the AI agent determines the next action, and the response is converted back into speech using Murf Falcon.

The agent can also access application-specific tools, memory, human escalation, analytics, and specialist workflows.

End-to-End Pipeline

Real-Time Transport: LiveKit manages the real-time voice connection.
Speech Recognition: User speech is converted into text by the configured speech-to-text layer.
Agent Core: The LLM-powered agent understands the request and determines the appropriate response or action.
Tools: The agent can invoke application-specific workflows such as eligibility checks or escalation creation.
Memory & Database: Relevant information is persisted in SQLite.
Specialist Routing: Specific conversations can be transferred to a specialist agent.
Human Escalation: Situations requiring human support can create a support request with a reference ID.
Analytics: Call outcomes are stored and surfaced through the analytics dashboard.
Speech Synthesis: Murf Falcon converts the generated response into natural speech.

Architecture Diagram

Technology Stack

Layer	Technology
Frontend	Next.js
AI Agent	LiveKit Agents
Real-Time Transport	LiveKit
Text-to-Speech	Murf Falcon
Backend	Python
API Layer	FastAPI
Database	SQLite
Phone Calling	SIP / LiveKit

5. Building the Agent

The first version of FinSaathi was intentionally kept simple.

Instead of trying to build every feature at once, I started with the core voice loop and gradually connected the additional services around it.

The basic voice-agent pipeline can be thought of as four major layers:

Speech-to-Text — converts the user's speech into text.
LLM / Agent — understands the request and decides what to do.
Tools — allow the agent to perform real actions.
Text-to-Speech — converts the final response back into natural speech.

LiveKit provides the real-time communication layer.

Real-Time Voice with LiveKit

The simplified flow is:

User speaks → LiveKit → Speech Recognition → AI Agent → Response → Murf Falcon → User

This makes the interaction feel much closer to a normal conversation.

The AI Agent

The AI agent is the decision-making layer of FinSaathi. Its responsibility is to understand the user's intent and determine whether the request can be answered directly or whether a tool or specialist is required.

For example:

User: "PMJJBY ke liye main eligible hoon?"

The agent can identify this as an eligibility request, collect information, invoke the eligibility workflow, store the result, and explain the outcome.

The overall loop is:

Understand → Decide → Act → Respond

6. Giving the Agent Tools

An LLM can generate useful responses, but it cannot reliably perform application-specific operations without access to tools.

For FinSaathi, tools were introduced for:

Government-scheme information.
Eligibility checks.
Document requirements.
User information.
Human-support escalation.
Call-outcome tracking.

A simplified tool pattern looks like this:

@function_tool
async def check_eligibility(
    scheme: str,
    user_information: dict
):
    result = perform_eligibility_check(
        scheme,
        user_information
    )

    return result

The model decides when the tool is needed, while the actual business logic remains inside the tool.

7. Memory and Persistent Data

A voice assistant becomes more useful when it can remember relevant information across conversations.

FinSaathi uses SQLite to persist information required by the application:

User information.
Eligibility results.
Escalation requests.
Call outcomes.
Analytics data.

The architecture is:

Agent → database.py → SQLite

while dashboard requests follow:

Frontend → FastAPI → database.py → SQLite

This kept database operations in one place instead of scattering SQL logic across components.

Example Database Access Pattern

def get_escalations():
    connection = sqlite3.connect(DB_PATH)
    connection.row_factory = sqlite3.Row

    rows = connection.execute(
        "SELECT * FROM escalations ORDER BY created_at DESC"
    ).fetchall()

    connection.close()

    return [dict(row) for row in rows]

8. Human Escalation: Knowing When AI Should Step Aside

Not every financial situation should be handled entirely by an AI agent.

For a potentially unauthorized transaction, FinSaathi can provide safe guidance and ask whether the user wants human support.

The workflow is:

User reports issue → Safe guidance → User gives consent → Escalation created → Reference ID generated → Human Support Dashboard

A unique reference ID is generated for every escalation.

For example:

FS-A5323F

A stored escalation can contain:

Request ID.
User ID.
Reason.
Summary.
What was checked.
Urgency.
Language.
Preferred follow-up method.
Status.
Created timestamp.

Example Escalation Tool

@function_tool
async def create_escalation(
    reason: str,
    summary: str,
    urgency: str = "normal",
    preferred_followup: str = "phone"
):
    request_id = create_escalation_record(
        reason=reason,
        summary=summary,
        urgency=urgency,
        preferred_followup=preferred_followup
    )

    return request_id

9. Call Analytics

For FinSaathi, a successful call means that the intended financial workflow was completed or that an appropriate resolution or escalation was reached.

A failed call does not necessarily mean that the software crashed. A user may leave before completing an eligibility workflow.

The dashboard records:

Total calls.
Successful calls.
Failed calls.

The architecture is:

Actual Call → Outcome → Database → FastAPI → Analytics Dashboard

The values come from actual browser or SIP interactions rather than hardcoded demo numbers.

Analytics Recording

record_call_outcome(
    call_id=call_id,
    outcome="SUCCESS"
)

10. Outbound Calling

FinSaathi was extended beyond incoming browser conversations.

The outbound workflow allows the system to initiate supported phone conversations.

Application Data → Outbound Logic → SIP / LiveKit → Voice Agent → User

One of the challenging parts was debugging the call lifecycle. There were situations where the AI greeting started but the call terminated before the conversation could continue.

The investigation required checking:

Worker lifecycle.
SIP configuration.
Network connections.
Agent state.
Call state.
Real-time transport.

This taught me that voice debugging requires looking at the complete lifecycle rather than only the AI response.

11. Specialist Agent Handoff

As FinSaathi grew, putting every workflow into one large agent became less attractive.

The solution was to introduce specialist agents.

For government-scheme conversations:

Main Agent → Government Scheme Specialist → Eligibility / Scheme Workflow → Result

The specialist can focus on:

Government schemes.
Eligibility.
Benefits.
Required documents.
Application information.

A handoff should not feel like starting a completely new conversation. The specialist needs enough context from the previous conversation.

Context-Preserving Handoff

@function_tool
async def transfer_to_scheme_specialist():
    copied_context = self.chat_ctx.copy(
        exclude_instructions=True
    )

    specialist = GovernmentSchemeSpecialist(
        chat_ctx=copied_context
    )

    return (
        specialist,
        "Connecting you with our government scheme specialist."
    )

12. Visual Evidence From the Build

A. Main FinSaathi Interface

B. Active Voice Conversation

C. Human Support Dashboard

D. Call Analytics Dashboard

13. Verified Implementation Highlights

Example 1: Tool-Based Eligibility

@function_tool
async def check_eligibility(
    scheme: str,
    user_information: dict
):
    result = perform_eligibility_check(
        scheme,
        user_information
    )

    return result

This keeps the eligibility logic deterministic and testable while allowing the agent to decide when the workflow is needed.

Example 2: Human Escalation

@function_tool
async def create_escalation(
    reason: str,
    summary: str,
    urgency: str,
    preferred_followup: str
):
    request_id = create_escalation_record(
        reason=reason,
        summary=summary,
        urgency=urgency,
        preferred_followup=preferred_followup
    )

    return request_id

Example 3: Analytics

record_call_outcome(
    call_id=call_id,
    outcome="SUCCESS"
)

Example 4: Specialist Handoff

@function_tool
async def transfer_to_scheme_specialist():
    copied_context = self.chat_ctx.copy(
        exclude_instructions=True
    )

    return GovernmentSchemeSpecialist(
        chat_ctx=copied_context
    )

14. Hard Technical Challenges and Lessons Learned

Challenge 1: Outbound Call Lifecycle

An outbound call could start with an AI greeting and then terminate unexpectedly.

The important realization was that the generated response was not necessarily the problem.

I had to investigate the worker lifecycle, SIP configuration, network connectivity, room state, agent state, and call termination events.

Lesson: Voice debugging requires observing the entire real-time lifecycle.

Challenge 2: FastAPI Dependency and ASGI Setup

While building the dashboard API, I encountered a missing FastAPI dependency and later an ASGI loading error where Uvicorn could not find the expected app object.

The debugging process was:

Install dependency → verify import → verify module → verify app object → start Uvicorn → test endpoint

Lesson: Verify each backend layer independently before connecting it to the frontend.

Challenge 3: Database and API Separation

The database module does not need to run as a separate process.

The agent can directly use:

agent.py → database.py → SQLite

The dashboard uses:

Frontend → FastAPI → database.py → SQLite

Understanding this distinction made the architecture much clearer.

Challenge 4: Specialist Handoff

A specialist handoff is not simply a function call. The new agent needs enough context to understand the user's previous conversation while keeping its own instructions separate.

Lesson: Context preservation and instruction isolation are both important in multi-agent systems.

15. Architectural Design Decisions

Decision 1: Shared Database Layer

Choice: Keep database operations inside database.py.

Benefit: Agent tools and API routes can reuse the same data-access functions.

Decision 2: FastAPI for Dashboard APIs

Choice: Use FastAPI as the API layer.

Benefit: The frontend receives structured data while the database remains behind the backend boundary.

Decision 3: Tool-Based Actions

Choice: Use function tools for eligibility, escalation, analytics, and other supported workflows.

Benefit: The agent decides when an action is required while the application remains responsible for performing it.

Decision 4: Specialist Agents

Choice: Route domain-specific conversations to specialist agents.

Benefit: Smaller responsibilities, clearer instructions, and easier future expansion.

16. Practical Step-by-Step Build Guide

Prerequisites

You will need:

Python with uv.
Node.js for the frontend.
A LiveKit project.
A Murf API key.
Credentials for the configured speech-to-text and LLM providers.
SQLite.

Step 1: Clone the Repository

git clone https://github.com/NipunGoel02/murf-ai-project
cd murf-ai-project

Step 2: Configure Environment Variables

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key

Never commit real credentials.

Step 3: Install Backend Dependencies

cd backend
uv sync

Step 4: Start the Agent

Use the LiveKit Agents startup command defined in the repository.

Step 5: Start the Frontend

cd frontend
npm install
pnpm dev

Use the package manager and scripts defined by the repository if they differ.

Step 6: Start the Human-Support API

uv run uvicorn src.escalation_api:app --reload --port 8000

Step 7: Test a Conversation

Open the frontend and test:

A normal financial question.
A government-scheme question.
An eligibility workflow.
A human escalation.
A specialist handoff.
A completed call.
An incomplete call.

17. Security: Keep Secrets and Caller Data Private

A public repository should never contain:

API keys.
API secrets.
SIP credentials.
Private phone numbers.
Caller information.
OTPs.
PINs.
Passwords.
Account numbers.
Private database files.
Full private conversation transcripts.

Use environment variables for secrets.

Before publishing the repository, inspect the files and Git history to ensure that secrets were not accidentally committed.

The public analytics dashboard should also avoid exposing sensitive caller information.

18. Practical Troubleshooting

Issue 1: The Agent Speaks and Then the Call Ends

Check:

LiveKit worker logs.
SIP state.
Room state.
Agent process state.
Network connectivity.
Call termination events.

Issue 2: FastAPI Says `app` Is Missing

Verify that the module exposes:

from fastapi import FastAPI

app = FastAPI()

Then start it using the correct module path:

uv run uvicorn src.escalation_api:app --reload --port 8000

Issue 3: Dashboard Data Does Not Change

Debug the complete path:

Agent → Database → API → Browser

Verify the database record first, then the API response, and finally the frontend request.

Issue 4: Specialist Does Not Have Enough Context

Make sure the handoff transfers relevant conversation context while keeping specialist instructions separate.

19. What I Would Improve Next

1. More Financial Specialists

Future specialists could cover:

Banking support.
Insurance.
Loans.
Credit cards.
Government benefits.

2. Better Observability

The analytics system could track:

Average call duration.
Tool usage.
Handoff rate.
Escalation rate.
Failure reasons.
Per-workflow success rate.

3. More Robust Telephony

The SIP layer could be extended with production carrier integrations and more robust call-state handling.

4. More Indian Languages

The system could be extended to support more Indian languages and regional speech patterns.

5. Better Human Support

Human agents could receive privacy-safe context so users do not have to repeat their entire issue.

20. What I Learned

At first, a voice agent looked like:

Speech → AI → Speech

After these 10 days, I realized that a useful voice agent needs much more:

Voice + LLM + Memory + Tools + Safety + Database + Real-Time Transport + Human Escalation + Analytics + Specialist Handoffs

The biggest lesson was:

A useful voice agent is not just an AI that can talk. It is a system that can understand, act, remember, measure its performance, and know when a human or specialist should take over.

FinSaathi started as a simple voice conversation and gradually became a complete financial assistance workflow.

The most valuable part of the challenge was learning how these pieces fit together into one system.

21. Project Links

GitHub Repository: https://github.com/NipunGoel02/murf-ai-project

Built using Murf Falcon, the fastest TTS API.

22. Closing

Building FinSaathi during the 10 Days of Voice Agents — VoiceForBharat Edition was an opportunity to work with voice AI beyond simple prompt engineering.

I worked with real-time communication, speech, LLM orchestration, tools, persistent data, outbound calling, human escalation, analytics, and specialist-agent routing.

The biggest takeaway was simple:

The hardest part of a voice agent is not making it speak. The hardest part is making everything around the conversation reliable.

FinSaathi is still a prototype, but it represents the kind of voice-first system I want to continue building: conversational, useful, safe, and connected to real-world workflows.

References and Further Reading

LiveKit Agents: https://docs.livekit.io/agents
Murf AI: https://murf.ai/
FastAPI: https://fastapi.tiangolo.com/
Next.js: https://nextjs.org/docs
Python: https://docs.python.org/3/

Building FinSaathi: A Voice-First Financial Assistant for Bharat 🇮🇳 10 Days of Voice Agents — VoiceForBharat Edition

Nipun Goel — Fri, 14 Aug 2026 18:21:12 +0000

Over the last 10 days, I built FinSaathi, a voice-first AI assistant for the Financial Services track of the VoiceForBharat challenge.
The goal was simple: build an assistant that can talk naturally with users, understand financial and government-scheme related queries, remember relevant information, use tools, and know when a human or specialist should take over.
What started as a basic voice agent gradually became a complete system with memory, tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs.
💡 The Problem
Financial and government-scheme processes can involve eligibility requirements, documents, deadlines, and complicated terminology.
For users who are more comfortable speaking than typing, voice can make these interactions much more natural.
For example, a user can simply ask:
"PMJJBY ke liye main eligible hoon?"
Instead of navigating through multiple forms, FinSaathi can understand the request, collect the required information, perform an eligibility check, and explain the result conversationally.
The goal is not to replace banks or human support, but to provide a conversational first layer of assistance and escalate situations when human help is required.
🏗️ Architecture
USER
│
▼
LiveKit
│
▼
Speech-to-Text
│
▼
LLM / Agent
│
┌────────────┼────────────┐
▼ ▼ ▼
Memory Tools Escalation
│ │ │
└────────────┼────────────┘
▼
SQLite DB
│
┌──────┴──────┐
▼ ▼
Human Support Analytics
Dashboard Dashboard
│
▼
Murf Falcon
│
▼
USER
Technology Stack
Component
Technology
Frontend
Next.js / React
AI Agent
LiveKit Agents
Real-time Transport
LiveKit
Text-to-Speech
Murf Falcon
Backend
Python
API
FastAPI
Database
SQLite
Calling
SIP / LiveKit
🎙️ Key Features

Indian Voice & Natural Conversations FinSaathi uses Murf Falcon for text-to-speech and supports natural Hindi/Hinglish conversations. The goal was to make the interaction feel more like talking to an assistant rather than interacting with a traditional chatbot.
Safety Guardrails Financial conversations require strong safety boundaries. For example, when a user reports an unauthorized transaction, FinSaathi warns them not to share: OTP PIN Password CVV Card details The agent also clearly explains when it does not have direct access to a user's bank account or transaction system.
User Memory FinSaathi stores relevant user information in a SQLite database so it can be reused in future conversations. The database logic is separated into its own layer: Agent ↓ database.py ↓ SQLite ↓ users table This made it easier to extend the project without mixing database operations throughout the agent code.
Government Scheme Eligibility FinSaathi can use tools to perform government-scheme eligibility checks. User asks about scheme ↓ Collect required information ↓ Eligibility tool ↓ Eligible / Not Eligible ↓ Database ↓ User receives result The eligibility result can also be used to determine whether the user's intended task was successfully completed.
Outbound Calling FinSaathi can proactively contact eligible users using outbound calling. Database ↓ Check deadlines / eligible users ↓ Outbound calling logic ↓ SIP / LiveKit ↓ AI Agent ↓ User Setting up and debugging the SIP and LiveKit calling flow was one of the most challenging parts of the project.
Human Escalation FinSaathi can recognize situations where human support is more appropriate. For example, for an unauthorized transaction, the agent can ask: "Agar aap chahein, toh main aapki situation ke liye ek human support request create kar sakti hoon." After the user gives consent, the system creates an escalation and generates a unique reference ID. Example: FS-A5323F The request is stored with information such as: Reason: possible_fraud Urgency: HIGH Status: OPEN Preferred Follow-up: phone The human-support dashboard then allows the request to move through: OPEN ↓ IN_PROGRESS ↓ RESOLVED
Call Analytics FinSaathi also records the outcome of conversations. For this project, a successful interaction means that the user's intended task was completed, such as completing an eligibility check or receiving the required information. The dashboard displays: Total Calls Successful Calls Failed Calls These values come from actual browser or SIP interactions rather than hardcoded numbers. Actual Call ↓ Determine Outcome ↓ call_analytics ↓ FastAPI ↓ Analytics Dashboard A failed call does not necessarily mean a technical failure. For example, if a user leaves before completing an eligibility check, the interaction can be recorded as unsuccessful.
Specialist Agent Handoff Instead of making one agent responsible for every type of question, FinSaathi can hand government-scheme related conversations to a Government Scheme Specialist. Main Agent │ Government scheme? │ YES ↓ Government Scheme Specialist │ ↓ Eligibility / Documents / Benefits / Scheme Information This makes the architecture more modular and allows additional specialist agents to be added in the future. 🧩 Challenges I Faced The project definitely did not work perfectly on the first attempt. One of the biggest challenges was real-time calling and LiveKit/SIP integration. During testing, I encountered errors such as: WinError 64 ConnectionResetError DuplexClosed I also faced an outbound-call issue where the AI would start speaking but the call could terminate before the conversation continued. Debugging this required checking: LiveKit worker lifecycle SIP configuration Network connections Agent processes Call state Real-time transport This taught me that building a voice agent is not only about the LLM. The real-time infrastructure around the agent matters just as much. Another important lesson was separating responsibilities between the agent, database, API, and frontend. For example: Agent → database.py → SQLite while the human-support dashboard uses: Frontend → FastAPI → database.py → SQLite This separation made the system easier to extend and debug.🚀 What I Learned Before this challenge, I thought building a voice agent was mainly: Speech → AI → Speech After these 10 days, I realized that a useful voice agent needs much more: Voice + LLM + Memory + Tools + Safety + Real-time communication + Database + Human escalation + Analytics + Specialist handoffs The biggest lesson for me was: A useful voice agent isn't just an AI that can talk. It's a system that can understand, act, remember, measure its performance, and know when a human or specialist should take over. Building FinSaathi during the 10 Days of Voice Agents — VoiceForBharat Edition gave me hands-on experience with all of these pieces.