Github Repository Link: https://github.com/ShivaKumar-Jatla/murf-livekit-starter
Introduction
What if accessing information about government financial schemes was as simple as making a phone call?
Many people struggle to understand eligibility requirements, required documents, and application procedures for government schemes. Websites are often difficult to navigate, especially for users who prefer speaking over typing.
To solve this, I built DhanVaani, a multilingual AI voice assistant that helps users explore government financial schemes through natural voice conversations.
I built DhanVaani during the 10 Days of Voice Agents – VoiceForBharat Edition using Murf Falcon, LiveKit, Gemini, and Deepgram.
Why Voice?
Not every user wants to interact with a chatbot.
Voice interactions are:
- Faster
- More natural
- Accessible for first-time internet users
- Helpful for multilingual users
- Useful on mobile devices
Instead of searching multiple government websites, users simply talk to DhanVaani.
The Technology Stack
DhanVaani is built using:
- Speech-to-Text: Deepgram Nova-3
- LLM: Gemini 3.5 Flash Lite
- Text-to-Speech: Murf Falcon
- Realtime Communication: LiveKit
- Backend: Python
- Frontend: React
- Database: SQLite
- Telephony: Linphone + LiveKit SIP
The overall pipeline looks like this:
User Speech
│
▼
Deepgram Speech-to-Text
│
▼
Gemini LLM
│
▼
Function Calling
│
▼
Murf Falcon TTS
│
▼
User Hears Response
Features I Built
Natural Voice Conversations
DhanVaani understands spoken conversations and responds using an Indian English voice powered by Murf Falcon.
The goal was to make conversations sound natural rather than robotic.
Personality & Safety
The assistant has:
- Clear objectives
- Financial-service specific guardrails
- Code-mixed language support
- Honest responses when information is unavailable
It never asks for:
- OTPs
- PINs
- Account numbers
and avoids making promises about government approvals.
Modern Voice Interface
I designed a frontend showing the current conversation state:
- Ready
- Connecting
- Listening
- Speaking
- Call Ended
The interface also provides microphone permission guidance for first-time users.
Persistent Memory
Returning users no longer need to introduce themselves again.
With user consent, DhanVaani remembers:
- Name
- Preferred language
- Previous conversations
- Previously checked schemes
This creates much more natural follow-up conversations.
Real Tool Calling
Instead of relying only on the language model, DhanVaani can use tools to:
- Check government scheme eligibility
- Recommend relevant schemes
- Provide required document lists
This makes responses more reliable and practical.
Outbound Calling
Using LiveKit Telephony and Linphone, DhanVaani can initiate outbound calls instead of waiting for users to contact it.
This opens possibilities such as:
- Scheme reminders
- Deadline notifications
- Follow-up conversations
- Human Escalation
AI should know its limits.
Whenever a situation requires human intervention, DhanVaani:
- Requests user consent
- Creates an escalation request
- Generates a reference ID
- Explains the next steps clearly
- Call Analytics Dashboard
To understand how well the assistant performs, I built an analytics dashboard that tracks:
- Total Calls
- Successful Calls
- Failed Calls
- Success Rate
- Recent Call History
The dashboard is driven by real call data rather than hardcoded values.
Multi-Agent Architecture
The final enhancement was introducing specialist agents.
The main assistant now hands complex government scheme queries to a dedicated Government Scheme Specialist.
The specialist:
- Receives conversation context
- Continues naturally
- Returns control after completing its task
The user never has to repeat information.
- Challenges I Faced
- SIP Audio Issues
One of the biggest challenges was outbound telephony.
Although the outbound call connected successfully, ensuring proper two-way communication required debugging SIP configuration, media routing, and audio handling.
This taught me that building production voice systems involves much more than connecting an LLM to speech APIs.
Prompt Engineering
Another challenge was preventing the assistant from becoming overly verbose.
Voice agents need short, conversational responses rather than long paragraphs designed for reading.
Designing prompts specifically for speech made a noticeable difference.
How You Can Build Your Own Voice Agent
Getting started is surprisingly straightforward.
Clone the repository
git clone
Configure API Keys
Create a .env.local file.
Add your:
- LiveKit credentials
- Murf API key
- Deepgram API key
- Gemini API key
Never commit these keys to GitHub.
- Install dependencies
- uv sync
- Download models
- uv run python src/agent.py download-files
- Start the agent
- uv run python src/agent.py dev
Open the frontend, connect to the agent, and start speaking.
What I'd Improve Next
If I continue developing DhanVaani, I would like to add:
- RAG over official government documents
- Multiple financial specialists
- Better multilingual speech synthesis
- Automatic follow-up calls
- Personalized financial planning
- Admin dashboard for human agents
- Cloud deployment
- Production monitoring
What I Learned
This challenge completely changed how I think about conversational AI.
A real voice assistant is far more than:
Speech-to-Text → LLM → Text-to-Speech.
A production-quality system also needs:
- Safety
- Memory
- Tools
- Human escalation
- Analytics
- Telephony
- Multi-agent collaboration
- Privacy
- Good UX
Each day's challenge built toward something much larger than a chatbot.
Conclusion
Building DhanVaani over these ten days has been an incredible learning experience.
I now have a much deeper understanding of voice AI systems, real-time communication, tool integration, prompt engineering, and production-oriented AI design.
Huge thanks to Murf AI for organizing the VoiceForBharat challenge and providing an opportunity to learn by building.
Top comments (0)