🎙️ VaaniFinance AI

khushi seth — Sat, 15 Aug 2026 09:28:50 +0000

Building a Voice-First Financial Assistant with Murf Falcon & LiveKit

10 Days of Voice Agents — VoiceForBharat Edition

Author: Khushi Seth
GitHub: https://github.com/khushi-seth/VaaniFinance-AI (update if you rename the repository)

🌟 Introduction

When I started the 10 Days of Voice Agents – VoiceForBharat Edition, my goal was simple:

"Can I build a voice assistant that feels like talking to a real financial advisor instead of a chatbot?"

After 10 days of learning, debugging, and building, I created VaaniFinance AI—a real-time voice assistant that can:

🎙️ Listen to users
🧠 Understand context
💾 Remember previous conversations
🛠️ Use tools
📞 Make outbound phone calls
👨‍💼 Escalate conversations to humans
🔄 Transfer to specialist agents
📊 Track call outcomes

This blog shares my journey, architecture, challenges, and what I learned while building it.

🖼️ Project Preview

📷 Add Screenshot Here

Insert a screenshot of your frontend showing the voice assistant interface.

❓ Why Build a Voice Financial Assistant?

Financial services can often be confusing.

People struggle with:

Understanding schemes
Filling forms
Eligibility questions
Required documents
Technical financial terms

Instead of making users search through long websites, I wanted them to simply ask:

"Can I apply for this scheme?"

"What documents do I need?"

The assistant answers naturally using voice.

🏗️ System Architecture
USER
│
🎤 Speaks naturally
│
▼
Speech-to-Text (STT)
│
▼
LiveKit Voice Agent
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Memory Gemini Tools
(SQLite DB) LLM Financial Functions
│ │ │
└───────────────┼────────────────┘
▼
Murf Falcon Text-to-Speech
│
▼
🔊 Spoken Response

📷 Add Architecture Diagram Here

You can recreate the above flow in Canva, Excalidraw, or draw.io and insert it as an image.

🛠️ Tech Stack
Technology Purpose
LiveKit Real-time voice communication
Murf Falcon Fast, natural text-to-speech
Google Gemini Language model
Deepgram Speech-to-text
Twilio Outbound phone calls
SQLite User memory
Python Backend
Next.js + React Frontend
Tailwind CSS UI
✨ Features
🎙️ 1. Real-Time Voice Conversation

Users talk naturally.

The assistant listens, understands, and replies in real time.

🇮🇳 2. Indian Voice using Murf Falcon

Voice quality matters.

Using Murf Falcon, the responses sound natural and conversational, making interactions more engaging.

🧠 3. Persistent Memory

Instead of forgetting every conversation, the assistant remembers returning users.

Conversation 1

User:
"My name is Khushi."

↓

Conversation 2

Agent:
"Welcome back, Khushi!"
🛠️ 4. Tool Calling

The LLM doesn't rely only on generated answers.

When required, it can call tools to fetch information before responding.

User Question
│
▼
Does it require a tool?
│
┌────┴────┐
│ │
No Yes
│ │
▼ ▼
Answer Tool Call
│
▼
Tool Result
│
▼
Final Voice Response
📞 5. Outbound Calling

I integrated Twilio so the assistant can initiate phone conversations.

User Phone
│
▼
Twilio
│
▼
LiveKit Agent
│
▼
Voice Conversation
👨‍💼 6. Human Escalation

AI shouldn't answer everything.

If a request requires human help, the assistant can escalate.

User
│
▼
Voice Agent
│
▼
Needs Human?
│
├── No → Continue
│
└── Yes → Escalate
🔄 7. Specialist Handoff

Different agents can handle different domains.

Main Agent
│
┌───┼────┐
▼ ▼ ▼
Loan Tax Insurance
Agent Agent Agent
📊 8. Analytics Dashboard

Every conversation is useful.

Tracking outcomes helps understand:

Successful calls
Failed calls
User interactions
Performance
🚧 Challenges I Faced
LiveKit Authentication

The biggest challenge initially was getting LiveKit configured correctly.

Solution

I verified:

Environment variables
API Keys
Agent startup
LiveKit room connection

before debugging anything else.

Twilio Integration

Connecting browser conversations with phone calls introduced authentication and routing issues.

After several iterations, outbound calling worked successfully.

Debugging

The project taught me an important lesson:

Debug one layer at a time.

Instead of changing everything, I tested:

Environment

↓

LiveKit

↓

STT

↓

LLM

↓

Memory

↓

Tools

↓

Murf Falcon

↓

Frontend

This saved hours of debugging.

🚀 How to Run the Project

Clone the repository:

git clone https://github.com/khushi-seth/VaaniFinance-AI.git

Install dependencies.

Create:

backend/.env.local

Example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key

Never upload secrets to GitHub.

Run the backend:

uv run python src/agent.py dev

Run the frontend:

pnpm dev
📈 Future Improvements

I would like to add:

🌐 Better multilingual conversations
⚡ Lower latency
📊 Advanced analytics
🤖 Smarter specialist routing
☁️ Cloud deployment
📱 Mobile support
📚 What I Learned

This challenge completely changed my understanding of AI agents.

Initially I thought:

Speech → LLM → Voice

Now I understand that a production-ready voice agent needs:

Speech

↓

Reasoning

↓

Memory

↓

Tools

↓

Phone Support

↓

Analytics

↓

Human Escalation

↓

Specialist Agents

Building a voice agent isn't only about making AI speak.

It's about creating a system that can understand, reason, remember, and act.

📸 Gallery

Add these screenshots before publishing:

📷 Homepage of your Voice Agent
📷 Voice conversation in progress
📷 Twilio successful call
📷 LiveKit dashboard
📷 Analytics dashboard
📷 Project folder in VS Code
📷 GitHub repository
🔗 Project Links

GitHub

https://github.com/khushi-seth/VaaniFinance-AI

GitHub - khushi-seth/VaaniFinance-AI | Khushi Seth

🚀 10 Days of Voice Agents — VoiceForBharat Edition COMPLETED! 🎙️🇮🇳 I’m excited to share my project VaaniFinance AI — a Voice-First Financial Assistant for Bharat. Over the past 10 days, I built and evolved a voice AI agent using Murf Falcon and LiveKit, with capabilities including: 🎙️ Real-time voice conversations 🧠 Persistent user memory 🛠️ Tool calling 📞 Outbound phone calls with Twilio 👤 Human escalation 🔄 Specialist handoffs 📊 Call outcome tracking 🛡️ Safety guardrails The biggest lesson from this challenge was that building a voice agent isn't just about making AI speak — it's about creating a system that can listen, understand, remember, act, and escalate when needed. I faced plenty of debugging challenges along the way, from LiveKit connections and authentication to Twilio integration and frontend issues. Every problem helped me understand real-world AI agent development better. A huge thank you to Murf AI for organizing 10 Days of Voice Agents — VoiceForBharat Edition and for giving builders the opportunity to explore voice AI with Murf Falcon. 🔗 GitHub Repository: https://lnkd.in/dbR8igAt Murf AI #VoiceForBharat #MurfAI #MurfFalcon #VoiceAI #AIAgents #LiveKit #GenerativeAI #BuildInPublic

linkedin.com

🙏 Thank You

A huge thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition.

This challenge helped me understand how modern voice AI systems are built—from speech recognition to real-time conversations, memory, telephony, and intelligent agent workflows.

The journey doesn't end here—this is just the beginning of building better voice experiences for everyone.

VoiceForBharat #MurfAI #MurfFalcon #LiveKit #VoiceAI #AIAgents

DEV Community: khushi seth

🎙️ VaaniFinance AI

GitHub - khushi-seth/VaaniFinance-AI | Khushi Seth

VoiceForBharat #MurfAI #MurfFalcon #LiveKit #VoiceAI #AIAgents