Building a Voice Agent in 10 Days — My VoiceForBharat Journey

Purvi Tech — Sat, 15 Aug 2026 16:50:36 +0000

This document outlines the complete story, architecture, development log, challenges solved, and setup guide for Jan Sahay over 10 Days of Building Voice Agents.

🏛️ The 4 Core Functional Pillars

**
Rather than building a generic open-ended chatbot, Jan Sahay is structured around four foundational pillars designed specifically for citizen welfare and digital safety:

💰 Financial Literacy: Jargon-free guidance on savings accounts, budgeting fundamentals, formal credit, interest rates, and long-term financial planning.
🛡️** Fraud Prevention*: Real-time coaching on OTP/PIN safety, identifying phishing messages, fake caller identification, and avoiding malicious screen-sharing APKs.
🏛️ **Government Welfare Schemes: Deterministic eligibility calculations and checklists for PMJDY, PMSBY, PMJJBY, Atal Pension Yojana (APY), Sukanya Samriddhi (SSY), PM Mudra Yojana, and PM-KISAN.
📞 **Complaint & Emergency Triage:* Structured step-by-step incident response for cybercrime reporting (1930 / cybercrime.gov.in), bank dispute filing for RBI Zero-Liability protection, and consumer forum escalations.

🏗️ System Architecture & Real-Time Voice Flow

Jan Sahay achieves sub-500ms voice turnaround times using real-time WebRTC audio streaming, streaming transcription, low-latency LLM reasoning, and natural Indian voice synthesis.

1. Real-Time Audio Pipeline Flow

2. Complete Multi-Tier Architecture

🎙️ Meet the Team of AI Specialists

**
Jan Sahay uses a multi-agent federation where a main intake guide dynamically hands off the conversation to dedicated domain specialists, matching their persona with the optimal Murf Falcon voice profile:

1. Anisha: Warm intake, 2-turn scoping, personalized greetings, banking basics.
2. Samar: Golden hour containment, 1930 reporting, bank helpline lookup.
3. Pooja: APY pension calculators, Sukanya Samriddhi (8.2%), PMJDY/PMSBY/PMJJBY.
4. Samar: PMMY Mudra loans (Shishu, Kishore, Tarun), PM SVANidhi vendor loans.
5. Palak: PMFBY 72-hr crop loss reporting, PM-KISAN e-KYC status, Kisan Credit Cards.

📅 10-Day Development Journey & Log

🟢 Day 1 — The Problem: Financial Confusion Is Everywhere

- Key Realization: Financial information is available, but accessibility is broken. Ordinary citizens struggle with banking, digital safety (UPI fraud/phishing), welfare schemes, and dispute recourse.
- Objective: Build an AI assistant that feels less like a search engine and more like a patient, empathetic citizen-support counselor.
- Core Pillars Defined: Simple, Conversational, Multilingual, Trustworthy, and Voice-first.

🟢 Day 2 — Designing the Citizen Experience

Structured the core 4-pillar interaction model (Financial Literacy, Fraud Prevention, Government Schemes, Complaint Assistance).
Defined prompt rules prohibiting hallucinations and outlining empathetic responses for users facing financial distress

🟢 Day 3 — Giving the AI a Memory

- Problem: Stateless chatbots suffer from amnesia. If Ramesh calls today about his daughter's education fund and calls back tomorrow, repeating context causes friction.
- Solution: Built the Caller Memory Database (backend/caller_data.db using SQLite) with schema for user_id, name, language_pref, demographic_facts, and discussed_schemes.
- Privacy First: Integrated explicit verbal consent checks ("May I remember this for your next call?") before storing non-PII demographic facts.

🟢 Day 4 — Building the Government Scheme Knowledge Layer

**
**

Architecture Rule:** LLM handles dialogue $\rightarrow$ Deterministic Rules Engine guarantees official policy truth.
Implemented exact formula calculators for: - PMJDY: Zero balance, RuPay debit card, ₹2L accident insurance cover. - PMSBY: Accidental death/disability cover (Age 18–70, ₹20/year premium, ₹2 Lakh cover). - PMJJBY: Life insurance cover (Age 18–50, ₹436/year premium, ₹2 Lakh cover). - APY: Atal Pension Yojana pension slabs (₹1,000–₹5,000/mo at age 60 based on entry age). - SSY: Sukanya Samriddhi Yojana (Girl child < 10 yrs, current 8.2% interest, tax exemptions).

🟢 Day 5 — Turning Text Into a Real-Time Voice Conversation

Integrated LiveKit WebRTC Agents with Deepgram Nova-2 STT and Murf Falcon TTS.
Enabled streaming bidirectional audio with Silero VAD (Voice Activity Detection) for seamless turn detection and low-latency interruptions.

🟢 Day 6 — Making Jan Sahay Multilingual & Code-Mixed

Configured STT and TTS mappings for English (en-IN), Hindi (hi-IN), and conversational Hinglish.
Enforced single-language response purity per turn to eliminate double-audio slash artifacts ("Hello / नमस्ते").

🟢 Day 7 — Fraud Prevention as a First-Class Feature

Implemented Immediate Action Matrix for cyber safety:
Strict alerts on OTP/PIN confidentiality.
Screen-sharing warning triggers (AnyDesk, TeamViewer).
Unverified QR code and UPI PIN receiving scam alerts.
Golden Hour Protocols to halt fund movement.

🟢 Day 8 — Action-Oriented Complaint Assistance

Developed structured escalation protocols: - Banking Disputes: Card freeze, dispute filing within 3 days for RBI Zero-Liability. - Cybercrime Reporting: Guided walkthrough for cybercrime.gov.in and National Helpline 1930. - Human Escalation: Generates unique escalation reference IDs (e.g., ESC-XXXXX) logged to SQLite for officer review.

🟢 Day 9 — Building the Web Portal & Telemetry Dashboard

Developed a high-performance Next.js web application with a single-click 🎙️ "Start Talking" interface.
Built real-time telemetry tracking turn latency, outcome classifications (resolved/escalated), and active escalation queues at /api/dashboard.
Added telephony SIP outbound calling capability (backend/src/outbound_call.py).

🟢 Day 10 — Open-Source Release & Architecture Documentation

Packaged repository with automated startup scripts (start_app.ps1 / start_app.sh), comprehensive API documentation, and architecture diagrams.

🚧 Challenges Faced & How We Solved Them

**
1. 🌐 Multilingual & Language Switching Inconsistencies

The Problem: In bilingual English/Hindi environments, the agent sometimes randomly replied in Hindi to an English prompt or switched languages mid-conversation without user intent.
How We Solved It:
Implemented session-level language tracking in SQLite memory.
Enforced strict prompt guidelines: detect user language preference, maintain it consistently across turns, and switch languages only when explicitly requested by the citizen.
Required native Devanagari script output for Hindi responses to ensure Murf Falcon TTS synthesizes authentic intonation.

2. 🧠 Hallucination-Free Financial Reasoning
The Problem: Financial queries involving age limits, cutoff dates, premium tiers, and pension returns must be 100% accurate. LLMs can hallucinate calculation schedules.
How We Solved It:
Decoupled conversational empathy from calculations.
Gemini extracts parameters (age, target pension, child age) and calls deterministic Python functions (calculate_apy_contribution, get_scheme_details).
Verified math is returned to Gemini to format as a spoken response.

3. 🔄 Dynamic Voice Profile Switching in WebRTC
The Problem: Tearing down WebRTC rooms when switching from Anisha to Samar or Pooja caused noticeable audio stutters and 2–3s delays.
How We Solved It: Kept the WebRTC audio transport live while dynamically swapping the Murf Falcon voice profile parameters in-flight during tool execution.

4. 🛡️ Accidental PII Memorization
The Problem: Panicked callers reporting fraud frequently blurt out card numbers or PINs.
How We Solved It: Built an automated sanitization regex layer in save_caller_facts and create_escalation that automatically scrubs card numbers, OTPs, PINs, and passwords before database commits.

🔮 What's Next on the Roadmap

📞 Telephony & IVR Toll-Free Access:
SIP Trunking integration with Exotel / Twilio so citizens without smartphones can dial a toll-free number from any basic feature phone.

🗣️ Hyper-Local Regional Dialects:
Expand beyond Hindi/English to regional languages (Bhojpuri, Maithili, Bengali, Marathi, Tamil, Telugu).
Acoustic noise suppression tuned for busy Indian environments (markets, bus stops, railway stations).

📄 DigiLocker & Voice-Guided Application:
Voice OTP authorization for instant eligibility verification via DigiLocker.
Voice-guided form pre-filling for PM-KISAN and Sukanya Samriddhi portals.

📲 Multi-Channel Follow-ups (WhatsApp / SMS):
Automated post-call summary SMS with verified helpline links, scheme forms, and grievance reference IDs.
Instant downloadable PDF checklist generated at the end of the voice call.

🚨 Proactive Outbound Alerts:
Automated voice alerts when subsidy installments (PM-KISAN, APY pensions) are credited.
Geo-targeted voice broadcasts warning citizens about emerging phishing scams in their district.

🛠️ Step-by-Step Quickstart Guide

**
Prerequisites

1. Python 3.10+
2. uv (Fast Python package manager)

  # Windows (PowerShell)
  powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

  # macOS/Linux
  curl -LsSf https://astral.sh/uv/install.sh | sh

Node.js 18+ and pnpm

 npm install -g pnpm

LiveKit Cloud Account: cloud.livekit.io
Murf AI API Key: murf.ai/api/dashboard
Google AI Studio API Key: aistudio.google.com
Deepgram API Key: deepgram.com

Step 1: Clone the Repository & Configure Secrets

git clone https://github.com/your-username/jan-sahay-voice-agent.git
cd jan-sahay-voice-agent/murf-livekit-starter

Create .env.local files in both backend/ and frontend/:

backend/.env.local:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_key
LIVEKIT_API_SECRET=your_livekit_secret
MURF_API_KEY=your_murf_falcon_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_gemini_api_key

frontend/.env.local:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_key
LIVEKIT_API_SECRET=your_livekit_secret

Step 2: Install Backend Dependencies

cd backend
uv sync
uv run python src/agent.py download-files

Step 3: Install Frontend Dependencies

cd ../frontend
pnpm install

Step 4: Run the Application
Option A: One-Click Startup Script (Recommended)

# Windows (PowerShell):
.\start_app.ps1

# Linux / macOS:
chmod +x start_app.sh
./start_app.sh

Option B: Separate Terminals

# Terminal 1: Backend Voice Worker
cd backend
uv run python src/agent.py dev

# Terminal 2: Frontend Web UI & Dashboard
cd frontend
pnpm dev

Open http://localhost:3000 in your browser, click "Start talking", grant microphone access, and begin speaking with Anisha!

📁 Repository Structure

jan-sahay/
├── README.md                           # Main documentation & 10-day build story
└── murf-livekit-starter/
    ├── start_app.ps1                   # Windows 1-click startup script
    ├── start_app.sh                    # Linux/macOS 1-click startup script
    ├── backend/
    │   ├── caller_data.db              # SQLite Caller Memory & Escalations DB
    │   ├── src/
    │   │   ├── agent.py                # Main LiveKit Agent & Multi-Specialist logic
    │   │   ├── db.py                   # SQLite memory CRUD, calculators & PII sanitizer
    │   │   ├── prompt.py               # Anisha persona & system guardrails
    │   │   ├── specialist_prompts.py   # Prompts for Samar, Pooja, and Palak
    │   │   └── outbound_call.py        # Telephony SIP outbound dialer
    │   └── pyproject.toml              # uv Python dependencies
    └── frontend/
        ├── app/
        │   ├── page.tsx                # Next.js Voice Assistant Web Interface
        │   └── api/dashboard/route.ts  # Observability & Escalations API
        └── package.json                # Frontend dependencies (pnpm)

DEV Community: Purvi Tech