Building RupeeGPT: A Multilingual Voice AI Financial Assistant for Bharat

Mayank Sharma — Sat, 15 Aug 2026 16:51:27 +0000

How I Built RupeeGPT: A Voice-First AI Financial Assistant for India in 10 Days

#VoiceForBharat | Built with the fastest TTS API — Murf Falcon | 10 Days of Voice Agents

Ten days ago, I started with a blank repo and a challenge: build a production-ready voice AI agent for Indian users — one that could speak naturally in English, Hindi, and Hinglish; remember returning callers; escalate to humans when things got serious; and hand off conversations to specialist agents without ever making the caller repeat themselves.

What came out the other side is RupeeGPT — a conversational AI financial assistant that helps any Indian user navigate banking, UPI, government welfare schemes, loans, and financial safety. Here's everything I built, what broke, how I fixed it, and how you can build your own.

The Problem: Finance Advice Is Inaccessible to Most Indians

India has over 500 million smartphone users, but financial literacy remains a barrier for hundreds of millions of people — especially in tier-2 and tier-3 cities and rural areas. The information exists: government scheme portals, RBI guidelines, banking apps. But it's buried in bureaucratic language, English-only interfaces, and long PDF documents.

A voice agent changes that. You don't need to read anything. You don't need to know the right portal URL. You just talk.

Who it's for: First-generation bank account holders, rural farmers checking PM Kisan eligibility, street vendors exploring PM SVANidhi loans, anyone who's ever been told to "read the fine print" and couldn't.

Why voice: Voice meets people where they are. It removes the literacy barrier, it's faster than navigating apps, and for many rural users, calling is the most intuitive interface they know.

The Architecture

🎙️ User speaks
    → Deepgram STT (nova-3, multilingual)
    → Gemini LLM (gemini-3.5-flash-lite via LiveKit Inference)
    → Murf Falcon TTS (Anisha — Indian English, en-IN)
    → LiveKit real-time transport
    → 🔊 User hears

The stack:

Backend: Python 3.12, LiveKit Agents SDK, uv for dependency management
Frontend: Next.js 14 (App Router), TypeScript, Tailwind
Memory: MongoDB Atlas (persistent caller profiles)
Transport: LiveKit (WebRTC)
TTS: Murf Falcon — more on why this matters below

Feature 1: An Indian Voice That Actually Sounds Indian

The default for most TTS-backed voice agents is a US English voice. For an Indian user asking about PM Kisan Samman Nidhi, hearing a generic American accent reading scheme names in English phonetics feels jarring and impersonal.

Murf Falcon's Anisha voice — Indian English, en-IN, Conversation style — changes this completely. But there was a subtlety: even with an Indian voice, scheme names like "PM Kisan Samman Nidhi" or "Pradhan Mantri Jan Dhan Yojana" are read with English phonetics when spelled in Roman script.

My fix: a TTS pronunciation layer (tts_hindi.py)

_ENGLISH_TO_HINDI: tuple[tuple[str, str], ...] = (
    ("pm kisan samman nidhi",           "पीएम किसान सम्मान निधि"),
    ("pm jan dhan yojana",              "पीएम जन धन योजना"),
    ("pradhan mantri jan dhan yojana",  "प्रधानमंत्री जन धन योजना"),
    ("pm svanidhi",                     "पीएम स्वनिधि"),
    ("aadhaar",                         "आधार"),
    ("yojana",                          "योजना"),
    # ... more
)

Before any text reaches Murf Falcon, it passes through this whitelist rewriter. Known Hindi/Indian terms are converted to Devanagari, so the voice says "पीएम किसान सम्मान निधि" — exactly as a native speaker would say it on TV — instead of "P M Kisan Samman Nidhi" with English stress patterns.

The rewriter is safe to apply for every language mode: a pure-English sentence with none of these terms passes through byte-for-byte unchanged. I also built a detect_language() function that classifies each user utterance as english, hindi, or hinglish using Devanagari character detection and a curated Hinglish marker word list:

HINGLISH_MARKERS = ("mujhe", "kaise", "kya", "chahiye", "baat", "namaste",
                    "yojana", "sarkari", "paise", "rupaye", "bharat", ...)

The agent mirrors the caller's language — answers in Hindi if they speak Hindi, Hinglish if they code-switch — without ever asking them to repeat.

Feature 2: Personality, Objectives, and Safety Guardrails

The system prompt defines the entire character of RupeeGPT: what it will help with, what it refuses, and how it escalates.

Key guardrails baked into the system prompt:

Never ask for OTPs, PINs, passwords, or Aadhaar/PAN numbers — ever, for any reason
Never guarantee loan approval, scheme eligibility, or returns
Never impersonate bank officials or government employees
Two mandatory escalation triggers: suspected fraud/unauthorized transactions, and official decision overrides (e.g., custom loan limit requests)

For the two escalation scenarios, the agent must:

Stop assisting and explain the situation
Name exactly what information it will share
Get explicit spoken consent before proceeding
Call create_escalation() only after consent

This pattern — ask before acting, require a clear YES — became a design principle throughout the whole project.

Feature 3: Multilingual TTS — English, Hindi, and Hinglish

The TTS node hooks into the LiveKit Agents pipeline using Agent.default.tts_node:

async def tts_node(self, text, model_settings):
    language = self._tts_language()

    async def _tracked():
        async for part in tts_hindi.stream_for_tts(text, language=language):
            yield part

    async for frame in Agent.default.tts_node(self, _tracked(), model_settings):
        yield frame

The stream_for_tts function accumulates the LLM's streaming text output into complete sentences before passing each sentence through the Devanagari rewriter. This is important: if a scheme name like "PM Kisan Samman Nidhi" were split across two streamed chunks, the phrase-level rewriter would miss it.

Feature 4: Persistent Memory for Returning Callers

Every caller gets a persistent browser ID (stored in localStorage and passed as a LiveKit participant attribute). The agent reads this at the start of every session and calls lookup_user() to fetch any saved profile from MongoDB.

But here's the part that took the most iteration: consent architecture.

The agent is not allowed to save any personal fact without:

The caller explicitly sharing the fact
The agent asking whether to remember it (naming the exact fact)
The caller saying a clear YES
The agent calling grant_user_memory_consent() with that exact value
Only then calling save_user_memory() with the same value

# Tools must be called in sequence, only after explicit spoken consent:
# 1. grant_user_memory_consent(name="Rahul", ...)
# 2. save_user_memory(name="Rahul", ...)

The save_user_memory tool actively checks the in-session consent store and blocks saves for anything that wasn't consented to in the current call. Returning callers are greeted naturally: "Namaste Rahul, welcome back. Would you like to continue from PM Jan Dhan Yojana?"

The MongoDB document looks like this:

{
  "user_id": "abc123",
  "name": "Rahul",
  "language_preference": "Hinglish",
  "facts": {
    "schemes_checked": ["PM Jan Dhan Yojana"],
    "eligibility_answers": { "income_bracket": "below 3 lakh", "farmer": true }
  },
  "last_interaction": "2026-08-14T10:30:00Z"
}

Feature 5: Tools That Fetch Real Data

Three function-calling tools give the agent live (or near-live) data:

find_eligible_schemes — Matches the caller's profile (age, state, income, occupation, caste, residence, disability, BPL status) against a local dataset of Indian government welfare schemes. Returns preliminary matches with names, benefits, documents required, and official portal URLs.

get_usd_inr_rate — Live USD/INR exchange rate.

get_lending_rates — Current base lending rates and MCLR data.

The scheme-matching tool uses a careful LLM prompt to avoid hallucination:

It never invents schemes or eligibility criteria
If the result set is empty, it says so clearly
If the tool errors, it says only that it cannot check right now — never speculates

Feature 6: Outbound Phone Calls

Using LiveKit's SIP integration, the agent can place outbound calls to real phone numbers. The session pipeline automatically detects SIP participants and switches the noise cancellation model:

noise_cancellation=lambda params: (
    noise_cancellation.BVCTelephony()
    if params.participant.kind == rtc.ParticipantKind.PARTICIPANT_KIND_SIP
    else noise_cancellation.BVC()
)

BVCTelephony is optimized for the narrowband audio characteristics of SIP/PSTN calls. The call analytics dashboard logs whether each session was a web or sip call.

Feature 7: Human Escalation with a Live Dashboard

When a caller reports suspected fraud or requests a decision override, the agent collects their name, contact number, issue summary, and urgency level — all with explicit consent — then calls create_escalation(). This writes to escalations.json (read by the Next.js frontend) and POSTs to a webhook endpoint.

The Escalation Desk (/demo route) shows open escalations in real time:

Caller name, contact, issue summary, urgency badge
Status: Open / In Progress / Resolved
Reference ID (e.g., ESC-492716) that the agent reads back to the caller

Feature 8: Call Analytics Dashboard

Every call session — web or SIP — is logged on close. The _on_close handler fires when LiveKit closes the room:

def _on_close(ev) -> None:
    call_record = {
        "id": ctx.room.name,
        "created_at": start_time,
        "ended_at": datetime.now(timezone.utc).isoformat(),
        "duration_seconds": round(duration, 2),
        "success": userdata.get("success", False),
        "success_reason": userdata.get("success_reason", ""),
        "call_type": call_type,   # 'web' or 'sip'
        "user_id": userdata.get("user_id", "")
    }
    # Write to calls.json + POST to Next.js API

A call is marked successful when the caller either checks their government scheme eligibility or creates a human escalation.

The /dashboard page shows:

Total calls, success rate, successful calls, failed calls (live-updating every 3s)
Filterable call log table with duration, type badge, timestamp, and outcome

Feature 9: Agent Handoff to a Specialist

This was Day 9, and probably the most elegant feature technically. When a caller needs deep, focused help with government schemes — step-by-step application guidance, documents checklist, portal navigation — the main assistant hands off to a dedicated GovernmentSchemeSpecialist agent:

async def transfer_to_scheme_specialist(self, context: RunContext, ...) -> str:
    specialist = GovernmentSchemeSpecialist(
        chat_ctx=context.session.history.copy()  # full conversation history
    )
    context.session.update_agent(specialist)     # live transition, no interruption
    return "Handoff complete."

The specialist receives the full chat_ctx, so the caller never has to repeat themselves. The specialist introduces itself once, then continues the conversation in-context. It's focused: it only handles government scheme questions, and explicitly declines general banking/UPI questions.

The Hard Parts

1. Streaming text rewriting without splitting phrases

When the LLM streams its reply in chunks, a phrase like "PM Kisan Samman Nidhi" might arrive as "PM Kisan" in one chunk and " Samman Nidhi" in the next. My first implementation fed each chunk directly through the regex rewriter — which meant phrase-boundary splits caused silent failures where terms stayed in Roman script.

Fix: Buffer streamed chunks and only emit text at sentence boundaries (".", "!", "?", "\n"). Since scheme names never cross sentence boundaries, the rewriter always sees the full phrase. Added a safety flush at 512 characters for run-on sentences.

2. The LLM saving memory without consent

Early versions of the memory tools had a subtle problem: the LLM would call save_user_memory in the same turn the caller first mentioned a fact, before any consent was sought. I fixed this with two layers:

Prompt-level: Explicit multi-step instructions in the system prompt
Tool-level: save_user_memory checks an in-session consent dict before writing anything; if the consent key isn't there, it returns a detailed refusal explaining exactly what's missing

This meant the enforcement was in the code, not just in the LLM's instruction-following.

3. Deprecation of `gemini-2.5-flash` mid-challenge

Around Day 7, calls started returning 404 errors. The model gemini-2.5-flash had been deprecated. Migrating to gemini-3.5-flash-lite via the LiveKit Inference plugin fixed it — but it required updating both agent.py and the test harness configuration. Always pin your model versions.

4. Getting consistent Hinglish detection

My first Hinglish marker list was too broad — common words like "hai" appeared in some proper nouns — and too narrow — it missed many common code-switch patterns. I iterated through actual test conversations, adding and removing markers until detect_language() was reliably stable across English, Hindi, and code-switched Hinglish inputs.

How to Build Your Own

The Four Core Components

Component	What it does	Used in this project
STT	Turns speech to text (the ears)	Deepgram nova-3, multilingual
LLM	Generates responses (the brain)	Gemini 3.5 Flash Lite
TTS	Turns text to speech (the voice)	Murf Falcon, Anisha (en-IN)
Transport	Real-time audio	LiveKit (WebRTC)

The key insight: these four components are independent and swappable. You can use any STT, any LLM, any TTS — as long as they're wired through a common agent runtime (LiveKit Agents in this case).

Why Murf Falcon for TTS?

55ms model latency — the agent sounds instant, not laggy
130ms time-to-first-audio — faster than any alternative I benchmarked
150+ voices across 35+ languages including Indian English and Hindi
$0.01/1000 characters — dramatically cheaper than comparable options
Native Indian English voices that sound genuinely natural for Indian users

Quickstart

git clone https://github.com/murf-ai/murf-livekit-starter.git
cd murf-livekit-starter

Set up API keys — never commit these to git:

Create backend/.env.local (copy from backend/.env.example):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key          # murf.ai/api/dashboard
DEEPGRAM_API_KEY=your_deepgram_key  # deepgram.com
GOOGLE_API_KEY=your_google_key      # aistudio.google.com

Create frontend/.env.local (copy from frontend/.env.example):

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret

Install and run:

# Backend (Python)
cd backend
uv sync
uv run python src/agent.py download-files

# Frontend (Node)
cd ../frontend
pnpm install

# Run everything from repo root
chmod +x start_app.sh && ./start_app.sh

Open http://localhost:3000, click Start talking, allow microphone access, and speak.

How to customise

The entire personality lives in one constant at the top of backend/src/agent.py:

SYSTEM_PROMPT = """You are RupeeGPT, a personal AI assistant for Indian users.
...
"""

Change that string and you have a completely different agent. Change the voice:

tts=murf.TTS(voice="Anisha", locale="en-IN", style="Conversation")
# Browse all voices: murf.ai/api/docs/voices-styles/voice-library

Tracing a conversation turn

After ./start_app.sh, check the backend terminal for:

[STT] user said: <transcript> — what Deepgram heard
[LLM] metrics model=gemini-3.5-flash-lite ttft=0.42s — LLM latency
[TTS] metrics ttfb=0.13s — Murf Falcon time-to-first-byte
[CALL LOG] Saved to file ... — session logged to dashboard

Architecture Overview

Browser (Next.js)
    ↕ WebRTC (audio)
  LiveKit Server
    ↕ WebRTC
Python Agent Worker
  ├─ Deepgram STT    (nova-3, multilingual)
  ├─ Gemini LLM      (gemini-3.5-flash-lite)
  ├─ Murf Falcon TTS (Anisha, en-IN)
  ├─ tts_hindi.py    (Devanagari pronunciation rewriter)
  ├─ memory.py       (MongoDB caller profiles)
  ├─ schemes.py      (government scheme matching)
  ├─ telephony/      (SIP outbound calls)
  └─ Function tools:
       lookup_user()
       save_user_memory()
       grant_user_memory_consent()
       find_eligible_schemes()
       get_usd_inr_rate()
       get_lending_rates()
       create_escalation()
       transfer_to_scheme_specialist()

Next.js frontend routes:
  /          Voice agent UI
  /demo      Escalation Desk dashboard
  /dashboard Call Analytics dashboard

What I'd Do Differently

Design strict tool schemas from Day 1. The OpenAI strict schema validator requires every object to declare additionalProperties: false. Retrofitting this was painful. The _pick_arg() helper pattern I built to handle both LLM invocations and test-harness direct calls is something I'd design in from the beginning.
Pin model versions immediately. gemini-2.5-flash deprecating mid-challenge cost me debugging time.
Integration test with real audio early. Unit tests caught logic errors; only real voice sessions caught the chunk-splitting bug in the TTS rewriter.

Repository

🔗 GitHub: github.com/murf-ai/murf-livekit-starter

⚠️ Never publish API keys, phone numbers, caller data, or any private information. Use .env.local (gitignored) for all secrets.

Closing Thoughts

Ten days. One voice agent. Nine features that went from zero to production-ready code:

✅ Indian voice with Murf Falcon (Anisha, en-IN)
✅ Personality, guardrails, and safety rules
✅ English / Hindi / Hinglish support with Devanagari TTS rewriting
✅ Frontend showing agent state
✅ Persistent memory with consent enforcement
✅ Real data tools (government schemes, exchange rates, lending rates)
✅ Outbound phone calls via SIP
✅ Human escalation with a live dashboard
✅ Specialist agent handoff

The most important lesson: voice agents are not just chatbots with audio bolted on. The interaction model is fundamentally different — no markdown, no bullet points, short turns, immediate feedback. You have to design for listening, not reading.

And for Indian users specifically, language flexibility and natural pronunciation are the difference between a tool that feels foreign and one that feels like talking to someone who genuinely gets it.

If you're building in this space, I hope this gives you a useful foundation. The code is open, the architecture is documented, and the patterns — consent-gated memory, language-aware TTS rewriting, live handoffs — are all reusable.

Build something for Bharat. Ship it.

Built during **10 Days of Voice Agents — VoiceForBharat Edition, powered by the fastest TTS API: **Murf Falcon.
Tag @MurfAI | #VoiceForBharat

DEV Community: Mayank Sharma