Sohel Khilji

Posted on Aug 15

Building Jan Sahay: Real-Time AI Voice Agent

#voiceforbharat #murffalcon #10daysofaivoiceagents #murfaivoiceagentschallenge

Over the past ten days as part of the Murf AI Voice for Bharat Challenge, I built Jan Sahay (जन सहाय)—an AI voice assistant engineered to bring financial literacy, government scheme guidance, and transaction safety directly to Indian citizens through real-time spoken voice interactions.

In a country as linguistically diverse as India, navigating text-heavy web portals, complex scheme eligibility forms, and digital banking procedures can be overwhelming. Jan Sahay bridges this gap by transforming dense public documentation into natural, sub-second spoken conversations.

1. Why Jan Sahay? The Mission & Target Users

For millions of citizens across India, accessing benefits under schemes like PM Kisan, PM Awas Yojana, or Sukanya Samriddhi Yojana involves navigating complex bureaucracy and paperwork. Furthermore, millions of first-time digital payment users face daily risks regarding unauthorized transactions and digital banking fraud.

I chose the Financial Services track because voice-first interaction is the most accessible interface for public literacy. Jan Sahay provides:

Natural Spoken Guidance: Low-latency voice interaction tuned for Indian accents and phrasing.
Instant Scheme Assessment: Direct eligibility evaluation and clear document checklists.
Privacy & Safety Safeguards: Built-in PII scrubbing and consent-backed human escalation workflows.

2. System Architecture: Real-Time Audio Pipeline

Jan Sahay utilizes a decoupled, real-time pipeline connected over WebRTC using LiveKit Agents.

┌─────────────┐       Audio Stream (WebRTC)       ┌──────────────────────────┐
│             ├──────────────────────────────────►│  Deepgram Nova-3 (STT)   │
│ User Client │                                   └────────────┬─────────────┘
│  (Next.js)  │                                                │ Spoken Text
│             │                                   ┌────────────▼─────────────┐
│             │◄──────────────────────────────────┤   Google Gemini (LLM)    │
└─────────────┘       Audio Stream (WebRTC)       │   + Dynamic Tools & VAD  │
                                                  └────────────┬─────────────┘
                                                               │ Synthesized Text
                                                  ┌────────────▼─────────────┐
                                                  │ Murf Falcon 2 (TTS API)  │
                                                  └──────────────────────────┘

Speech-to-Text (STT): Deepgram Nova-3 transcribes caller speech in real-time, handling accent variations and background noise.
LLM & Tool Orchestration: Google Gemini evaluates user intent, executes database tools, scrubs PII, and handles agent state transitions.
Text-to-Speech (TTS): Murf Falcon 2 synthesizes plain text responses into natural Indian-accented speech with sub-second latency.
Transport Layer: LiveKit Agents handles full-duplex WebRTC audio streaming, Voice Activity Detection (VAD), and session state.

3. Key Features Built Across the Challenge

Sub-Second Voice Synthesis (Murf Falcon 2): Leveraging Murf Falcon 2 enabled Jan Sahay to achieve sub-second voice synthesis, making the interaction feel natural and human.
Automated PII Scrubbing (sanitize_text()): To preserve user privacy, the agent automatically filters out sensitive personal identifiers (such as account numbers, PINs, OTPs, identity credentials, and PAN details) before logging caller history to SQLite.
Consent-Backed Human Escalation: When callers report fraud or disputed transactions, Jan Sahay explicitly requests user consent before generating an escalation ticket (ESC-2026-XXXXX) for human support.
Multi-Agent Specialist Handoff: In-depth scheme inquiries trigger a function call (transfer_to_scheme_specialist). The main agent announces the transfer out loud ("I am connecting you to our Government Schemes Specialist..."), and the SchemeSpecialist takes over using LiveKit's AgentSession.update_agent() while preserving session context.
Resilient Session Lifecycle Tracking: Configured LiveKit's add_shutdown_callback to guarantee that call metrics, call outcomes, and analytics flush safely to SQLite even if the call disconnects abruptly.

4. Engineering Challenges & Solutions

Challenge 1: Context Loss During Multi-Agent Handoff

Problem: Swapping execution from the main assistant to the specialized agent initially reset the conversation context, forcing users to repeat themselves.
Solution: Used context.session.update_agent(specialist) to dynamically transition execution while retaining the active AgentSession history.

Challenge 2: Disconnect Cleanup & Analytics Loss

Problem: When callers closed their browser tab mid-conversation, standard cleanup routines failed to trigger, leading to lost call outcome logs.
Solution: Registered LiveKit lifecycle hooks (add_shutdown_callback), ensuring that session metadata and database records flush instantly upon transport state changes.

5. How to Build & Run Jan Sahay Locally

Step 1: Clone Repository & Virtual Environment Setup

git clone https://github.com/0xzerex/murf-livekit-starter.git
cd murf-livekit-starter/backend
python -m venv .venv

# Activate environment (Windows)
.venv\Scripts\activate
# Linux/macOS: source .venv/bin/activate

pip install -r requirements.txt

Step 2: Configure Environment Variables

Create an .env.local file in your root folder:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_falcon_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_gemini_api_key

Step 3: Run the Agent Backend

python src/agent.py dev

6. Code Highlight: Implementing Specialist Handoff

Here is how the main assistant transfers execution to the SchemeSpecialist:

from livekit.agents import RunContext, function_tool
from scheme_specialist import SchemeSpecialist

@function_tool(
    name="transfer_to_scheme_specialist",
    description="Transfer the caller to the Government Schemes Specialist for detailed eligibility guidance."
)
async def transfer_to_scheme_specialist(ctx: RunContext):
    # Announce handoff clearly out loud
    await ctx.session.say("I am connecting you to our Government Schemes Specialist. Please hold on a moment.")

    # Instantiate specialist agent & transfer active session state
    specialist = SchemeSpecialist()
    await ctx.session.update_agent(specialist)
    return "Transferred successfully to Scheme Specialist."

Repository & Links

GitHub Repository: Jan Sahay Source Code (Day 10 Branch)
Challenge: 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI

DEV Community