Building a Voice-First Financial Assistant with Murf Falcon & LiveKit
10 Days of Voice Agents — VoiceForBharat Edition
Author: Khushi Seth
GitHub: https://github.com/khushi-seth/VaaniFinance-AI (update if you rename the repository)
🌟 Introduction
When I started the 10 Days of Voice Agents – VoiceForBharat Edition, my goal was simple:
"Can I build a voice assistant that feels like talking to a real financial advisor instead of a chatbot?"
After 10 days of learning, debugging, and building, I created VaaniFinance AI—a real-time voice assistant that can:
🎙️ Listen to users
🧠 Understand context
💾 Remember previous conversations
🛠️ Use tools
📞 Make outbound phone calls
👨💼 Escalate conversations to humans
🔄 Transfer to specialist agents
📊 Track call outcomes
This blog shares my journey, architecture, challenges, and what I learned while building it.
🖼️ Project Preview
📷 Add Screenshot Here
Insert a screenshot of your frontend showing the voice assistant interface.
❓ Why Build a Voice Financial Assistant?
Financial services can often be confusing.
People struggle with:
Understanding schemes
Filling forms
Eligibility questions
Required documents
Technical financial terms
Instead of making users search through long websites, I wanted them to simply ask:
"Can I apply for this scheme?"
or
"What documents do I need?"
The assistant answers naturally using voice.
🏗️ System Architecture
USER
│
🎤 Speaks naturally
│
▼
Speech-to-Text (STT)
│
▼
LiveKit Voice Agent
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Memory Gemini Tools
(SQLite DB) LLM Financial Functions
│ │ │
└───────────────┼────────────────┘
▼
Murf Falcon Text-to-Speech
│
▼
🔊 Spoken Response
📷 Add Architecture Diagram Here
You can recreate the above flow in Canva, Excalidraw, or draw.io and insert it as an image.
🛠️ Tech Stack
Technology Purpose
LiveKit Real-time voice communication
Murf Falcon Fast, natural text-to-speech
Google Gemini Language model
Deepgram Speech-to-text
Twilio Outbound phone calls
SQLite User memory
Python Backend
Next.js + React Frontend
Tailwind CSS UI
✨ Features
🎙️ 1. Real-Time Voice Conversation
Users talk naturally.
The assistant listens, understands, and replies in real time.
🇮🇳 2. Indian Voice using Murf Falcon
Voice quality matters.
Using Murf Falcon, the responses sound natural and conversational, making interactions more engaging.
🧠 3. Persistent Memory
Instead of forgetting every conversation, the assistant remembers returning users.
Conversation 1
User:
"My name is Khushi."
↓
Conversation 2
Agent:
"Welcome back, Khushi!"
🛠️ 4. Tool Calling
The LLM doesn't rely only on generated answers.
When required, it can call tools to fetch information before responding.
User Question
│
▼
Does it require a tool?
│
┌────┴────┐
│ │
No Yes
│ │
▼ ▼
Answer Tool Call
│
▼
Tool Result
│
▼
Final Voice Response
📞 5. Outbound Calling
I integrated Twilio so the assistant can initiate phone conversations.
User Phone
│
▼
Twilio
│
▼
LiveKit Agent
│
▼
Voice Conversation
👨💼 6. Human Escalation
AI shouldn't answer everything.
If a request requires human help, the assistant can escalate.
User
│
▼
Voice Agent
│
▼
Needs Human?
│
├── No → Continue
│
└── Yes → Escalate
🔄 7. Specialist Handoff
Different agents can handle different domains.
Main Agent
│
┌───┼────┐
▼ ▼ ▼
Loan Tax Insurance
Agent Agent Agent
📊 8. Analytics Dashboard
Every conversation is useful.
Tracking outcomes helps understand:
Successful calls
Failed calls
User interactions
Performance
🚧 Challenges I Faced
LiveKit Authentication
The biggest challenge initially was getting LiveKit configured correctly.
Solution
I verified:
Environment variables
API Keys
Agent startup
LiveKit room connection
before debugging anything else.
Twilio Integration
Connecting browser conversations with phone calls introduced authentication and routing issues.
After several iterations, outbound calling worked successfully.
Debugging
The project taught me an important lesson:
Debug one layer at a time.
Instead of changing everything, I tested:
Environment
↓
LiveKit
↓
STT
↓
LLM
↓
Memory
↓
Tools
↓
Murf Falcon
↓
Frontend
This saved hours of debugging.
🚀 How to Run the Project
Clone the repository:
git clone https://github.com/khushi-seth/VaaniFinance-AI.git
Install dependencies.
Create:
backend/.env.local
Example:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
Never upload secrets to GitHub.
Run the backend:
uv run python src/agent.py dev
Run the frontend:
pnpm dev
📈 Future Improvements
I would like to add:
🌐 Better multilingual conversations
⚡ Lower latency
📊 Advanced analytics
🤖 Smarter specialist routing
☁️ Cloud deployment
📱 Mobile support
📚 What I Learned
This challenge completely changed my understanding of AI agents.
Initially I thought:
Speech → LLM → Voice
Now I understand that a production-ready voice agent needs:
Speech
↓
Reasoning
↓
Memory
↓
Tools
↓
Phone Support
↓
Analytics
↓
Human Escalation
↓
Specialist Agents
Building a voice agent isn't only about making AI speak.
It's about creating a system that can understand, reason, remember, and act.
📸 Gallery
Add these screenshots before publishing:
📷 Homepage of your Voice Agent
📷 Voice conversation in progress
📷 Twilio successful call
📷 LiveKit dashboard
📷 Analytics dashboard
📷 Project folder in VS Code
📷 GitHub repository
🔗 Project Links
GitHub
https://github.com/khushi-seth/VaaniFinance-AI
GitHub - khushi-seth/VaaniFinance-AI | Khushi Seth
🚀 10 Days of Voice Agents — VoiceForBharat Edition COMPLETED! 🎙️🇮🇳 I’m excited to share my project VaaniFinance AI — a Voice-First Financial Assistant for Bharat. Over the past 10 days, I built and evolved a voice AI agent using Murf Falcon and LiveKit, with capabilities including: 🎙️ Real-time voice conversations 🧠 Persistent user memory 🛠️ Tool calling 📞 Outbound phone calls with Twilio 👤 Human escalation 🔄 Specialist handoffs 📊 Call outcome tracking 🛡️ Safety guardrails The biggest lesson from this challenge was that building a voice agent isn't just about making AI speak — it's about creating a system that can listen, understand, remember, act, and escalate when needed. I faced plenty of debugging challenges along the way, from LiveKit connections and authentication to Twilio integration and frontend issues. Every problem helped me understand real-world AI agent development better. A huge thank you to Murf AI for organizing 10 Days of Voice Agents — VoiceForBharat Edition and for giving builders the opportunity to explore voice AI with Murf Falcon. 🔗 GitHub Repository: https://lnkd.in/dbR8igAt Murf AI #VoiceForBharat #MurfAI #MurfFalcon #VoiceAI #AIAgents #LiveKit #GenerativeAI #BuildInPublic
linkedin.com
🙏 Thank You
A huge thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition.
This challenge helped me understand how modern voice AI systems are built—from speech recognition to real-time conversations, memory, telephony, and intelligent agent workflows.
The journey doesn't end here—this is just the beginning of building better voice experiences for everyone.
Top comments (0)