Building a Voice Agent for Government Scheme Access — 10 Days of Voice Agents, VoiceForBharat Edition

Tapnanshu Malhotra — Sat, 15 Aug 2026 13:23:42 +0000

The problem and the users

Millions of people in India are eligible for government welfare schemes — pensions, subsidies, insurance — but never find out, because the process to check eligibility assumes a smartphone, a data plan, and comfort reading government portals in English. A phone call, in the caller's own language, is a much lower bar.

I built a Financial Services voice agent that a caller can dial into, speak naturally (English, Hindi, or code-mixed Hinglish), and either:

get an eligibility check done for a specific government scheme, or
reach a human when the agent hits its limits — a fraud report, or a decision it isn't authorized to make.

What the voice agent does

A caller connects (via LiveKit, browser or SIP), talks to the main assistant, and:

Is recognized if they've called before, or the agent asks for consent to remember them for next time.
Gets help with everyday questions directly from the main agent.
Is hooked up with the Government Scheme Specialist — a separate, narrowly-scoped agent — the moment the conversation turns to scheme eligibility or required documents.
Can be escalated to a human teammate if the issue is fraud, or a decision the agent isn't authorized to make — with the caller's explicit consent on what gets shared.

Every call's outcome (success or failure against a clear definition, not "did an error happen") is logged and shown on a live dashboard.

How the system works

Standard voice-agent pipeline, built on the Murf LiveKit Starter:

STT — Deepgram (nova-3, multilingual)
LLM — Google Gemini
TTS — Murf Falcon, switching voice/locale live between en-IN-anisha and hi-IN-anisha based on whether the caller's speech is detected as Hindi/Hinglish (Devanagari script or common Hindi keywords) or English
Transport — LiveKit real-time rooms, with telephony-specific noise cancellation for SIP callers vs. browser callers
Turn detection — LiveKit's multilingual turn-detection model

The most important features

An Indian voice powered by Murf Falcon, switching language live mid-call so the caller never has to pick a language up front.

Memory for returning callers — with explicit consent asked before anything is saved, and hard rules against ever storing OTPs, PINs, account numbers, or passwords, even if the caller offers them.

Human escalation — the agent tries to help first, states plainly what it will and won't send to a human before escalating, and gives the caller a spoken reference ID. Escalation is reserved for fraud reports and decisions the agent isn't authorized to make — not a catch-all.

Handoff to a specialist agent — the main agent recognizes when a question is really about government scheme eligibility, tells the caller it's connecting them ("I'll connect you to our government scheme specialist"), and hands off the same conversation — the specialist doesn't ask the caller to repeat themselves. If the conversation drifts back to something outside its scope, it hands back just as cleanly.

A call analytics dashboard — every call is logged as in_progress → success | failed against one clear definition (an eligibility check completed, a document list delivered, or an escalation created — not just "the call happened without crashing"). The dashboard shows total/successful/failed calls, served on the same port as the backend, with no transcripts, PII, or call content — only aggregate counts.

Challenges and how I overcame them

How readers can build their own
Start from the base pipeline. You need four pieces: speech-to-text (I used Deepgram), an LLM (Google Gemini), text-to-speech (Murf Falcon), and a real-time transport layer to move audio between the caller and the model (LiveKit).
Clone the starter and set up locally:
bash
git clone https://github.com/Tapnanshu045/murf-livekit-starter
cd murf-livekit-starter/backend
uv sync
Add your API keys in a .env.local file in backend/ — never commit this file. At minimum you'll need keys for Murf, Deepgram, and your LLM provider, plus your LiveKit project credentials.
Run it and connect:
bash
uv run python src/agent.py dev

Then connect through the LiveKit frontend playground or your own frontend, and have a conversation to test it end-to-end.

Layer in one capability at a time — memory, escalation, a dashboard, a specialist handoff — the same way this challenge built them up over ten days, rather than all at once.

Full source: https://github.com/Tapnanshu045/murf-livekit-starter

What I'd improve next

Links and demos
Repo: https://github.com/Tapnanshu045/murf-livekit-starter
Challenge: https://github.com/murf-ai/voice-for-bharat-challenge-2026

DEV Community: Tapnanshu Malhotra

Building a Voice Agent for Government Scheme Access — 10 Days of Voice Agents, VoiceForBharat Edition