From a Simple Voice Bot to a Production-Style AI Assistant: Building DhanVaani in 10 Days

SHIVA KUMAR JATLA — Sat, 15 Aug 2026 12:52:42 +0000

Github Repository Link: https://github.com/ShivaKumar-Jatla/murf-livekit-starter

Introduction

What if accessing information about government financial schemes was as simple as making a phone call?

Many people struggle to understand eligibility requirements, required documents, and application procedures for government schemes. Websites are often difficult to navigate, especially for users who prefer speaking over typing.

To solve this, I built DhanVaani, a multilingual AI voice assistant that helps users explore government financial schemes through natural voice conversations.

I built DhanVaani during the 10 Days of Voice Agents – VoiceForBharat Edition using Murf Falcon, LiveKit, Gemini, and Deepgram.

Why Voice?

Not every user wants to interact with a chatbot.

Voice interactions are:

Faster
More natural
Accessible for first-time internet users
Helpful for multilingual users
Useful on mobile devices

Instead of searching multiple government websites, users simply talk to DhanVaani.

The Technology Stack

DhanVaani is built using:

Speech-to-Text: Deepgram Nova-3
LLM: Gemini 3.5 Flash Lite
Text-to-Speech: Murf Falcon
Realtime Communication: LiveKit
Backend: Python
Frontend: React
Database: SQLite
Telephony: Linphone + LiveKit SIP

The overall pipeline looks like this:

User Speech
│
▼
Deepgram Speech-to-Text
│
▼
Gemini LLM
│
▼
Function Calling
│
▼
Murf Falcon TTS
│
▼
User Hears Response

Features I Built

Natural Voice Conversations

DhanVaani understands spoken conversations and responds using an Indian English voice powered by Murf Falcon.

The goal was to make conversations sound natural rather than robotic.

Personality & Safety

The assistant has:

Clear objectives
Financial-service specific guardrails
Code-mixed language support
Honest responses when information is unavailable

It never asks for:

OTPs
PINs
Account numbers

and avoids making promises about government approvals.

Modern Voice Interface

I designed a frontend showing the current conversation state:

Ready
Connecting
Listening
Speaking
Call Ended

The interface also provides microphone permission guidance for first-time users.

Persistent Memory

Returning users no longer need to introduce themselves again.

With user consent, DhanVaani remembers:

Name
Preferred language
Previous conversations
Previously checked schemes

This creates much more natural follow-up conversations.

Real Tool Calling

Instead of relying only on the language model, DhanVaani can use tools to:

Check government scheme eligibility
Recommend relevant schemes
Provide required document lists

This makes responses more reliable and practical.

Outbound Calling

Using LiveKit Telephony and Linphone, DhanVaani can initiate outbound calls instead of waiting for users to contact it.

This opens possibilities such as:

Scheme reminders
Deadline notifications
Follow-up conversations
Human Escalation

AI should know its limits.

Whenever a situation requires human intervention, DhanVaani:

Requests user consent
Creates an escalation request
Generates a reference ID
Explains the next steps clearly
Call Analytics Dashboard

To understand how well the assistant performs, I built an analytics dashboard that tracks:

Total Calls
Successful Calls
Failed Calls
Success Rate
Recent Call History

The dashboard is driven by real call data rather than hardcoded values.

Multi-Agent Architecture

The final enhancement was introducing specialist agents.

The main assistant now hands complex government scheme queries to a dedicated Government Scheme Specialist.

The specialist:

Receives conversation context
Continues naturally
Returns control after completing its task

The user never has to repeat information.

Challenges I Faced
SIP Audio Issues

One of the biggest challenges was outbound telephony.

Although the outbound call connected successfully, ensuring proper two-way communication required debugging SIP configuration, media routing, and audio handling.

This taught me that building production voice systems involves much more than connecting an LLM to speech APIs.

Prompt Engineering

Another challenge was preventing the assistant from becoming overly verbose.

Voice agents need short, conversational responses rather than long paragraphs designed for reading.

Designing prompts specifically for speech made a noticeable difference.

How You Can Build Your Own Voice Agent

Getting started is surprisingly straightforward.

Clone the repository
git clone
Configure API Keys

Create a .env.local file.

Add your:

LiveKit credentials
Murf API key
Deepgram API key
Gemini API key

Never commit these keys to GitHub.

Install dependencies
uv sync
Download models
uv run python src/agent.py download-files
Start the agent
uv run python src/agent.py dev

Open the frontend, connect to the agent, and start speaking.

What I'd Improve Next

If I continue developing DhanVaani, I would like to add:

RAG over official government documents
Multiple financial specialists
Better multilingual speech synthesis
Automatic follow-up calls
Personalized financial planning
Admin dashboard for human agents
Cloud deployment
Production monitoring

What I Learned

This challenge completely changed how I think about conversational AI.

A real voice assistant is far more than:

Speech-to-Text → LLM → Text-to-Speech.

A production-quality system also needs:

Safety
Memory
Tools
Human escalation
Analytics
Telephony
Multi-agent collaboration
Privacy
Good UX

Each day's challenge built toward something much larger than a chatbot.

Conclusion

Building DhanVaani over these ten days has been an incredible learning experience.

I now have a much deeper understanding of voice AI systems, real-time communication, tool integration, prompt engineering, and production-oriented AI design.

Huge thanks to Murf AI for organizing the VoiceForBharat challenge and providing an opportunity to learn by building.

DEV Community: SHIVA KUMAR JATLA

From a Simple Voice Bot to a Production-Style AI Assistant: Building DhanVaani in 10 Days