Jana Sakhyam: Building a Voice That Speaks for the People

10 Days of Voice Agents — VoiceForBharat Edition using Murf AI

1. The Problem and the Users

Ramesh has farmed a small two-acre plot in Uttar Pradesh for over twenty years. He uses a basic feature phone, speaks local Hindi, and left school after 6th grade. When his daughter came home from college and asked if he was receiving his ₹6,000 yearly PM-KISAN stipend, he had no idea. He didn't know if he qualified, what documents he needed, or where to go. Navigating a complex, English-heavy government portal wasn't an option, and losing a day’s farm wages just to stand in line at the block office with no guaranteed answer was too big a risk.

This is the real gap in India—not a lack of government schemes, but a massive barrier to access. Millions like Ramesh miss out on entitlements simply because information isn't available in a format they can actually use. Worse, this same gap makes them prime targets for phone scams asking for OTPs under the guise of "releasing" their funds.

Voice changes everything. By letting callers simply speak and listen in their native language, we strip away the literacy and technology barriers holding them back.

To turn this into reality, I built Jana Sakhyam using the Murf Falcon TTS API—the fastest in the market—so farmers like Ramesh can have instant, natural, zero-latency conversations to check scheme eligibility, get document checklists, and protect themselves from fraud, all with a simple phone call.

2. What the Voice Agent Does

Jana Sakhyam is a phone-based (and browser-based) voice agent, built around three pillars a caller can navigate with a single tap or spoken request:

Government Schemes — eligibility checks, required documents, deadlines, for 5 major schemes (PM-KISAN, PM MUDRA, Atal Pension Yojana, Sukanya Samriddhi, Ayushman Bharat)
Bank Scam Safety — recognising fraud patterns, reporting incidents
Savings & Insurance — general banking literacy

The main persona, Anjali Arora, handles the conversation. If a caller's question goes deep into scheme-specific eligibility or into fraud territory, Anjali hands the conversation to a specialist — Smita for schemes, Kriti for cyber fraud — without the caller repeating themselves.

3. How the System Works

A real-time voice agent rests on four core components:

• 1. Speech-to-Text (STT) — converts the caller's spoken audio into text
• 2. LLM — reasons about intent and decides what to say or which tool to call.
• 3. Text-to-Speech (TTS) — converts the response back into natural audio — powered here by Murf Falcon, the fastest TTS API I tested, which mattered for keeping the conversation feeling real-time instead of laggy.
• 4. Real-time transport — LiveKit handles audio streaming; SIP trunking (via Linphone) enables real inbound and outbound phone calls, not just browser conversations.

How audio moves through the system:

Every response passes through a system prompt with hard safety rules: never ask for PINs or OTPs, never guarantee loan approval, always escalate anything outside the agent's scope to a human.

_4. The Most Important Features
_

• A named, consistent persona. Anjali Arora isn't an unnamed bot — she has a defined role and personality, which made conversations feel noticeably more natural than a generic assistant would.
• Memory for returning callers. SQLite-backed memory lets the agent recognize returning users and greet them by name — only after explicit consent, and only after sensitive data (bank numbers, PINs, OTPs) is automatically stripped before anything is stored.
• A real eligibility engine, not a static FAQ. Structured eligibility rules and document checklists for all 5 schemes, with data-freshness metadata so the agent is transparent about when information was last verified — important for someone like Ramesh, who needs to trust the answer, not just hear one.
• Outbound calling. The agent doesn't just wait for calls — it can proactively reach out to remind eligible users about upcoming deadlines. If Ramesh had signed up, Jana Sakhyam could have called him, instead of waiting for Priya to bring it up.
• Human escalation. When something is too sensitive or out of scope, the agent raises a ticket — PII automatically scrubbed, duplicate-check protection — routed to a real dashboard, with the ability to call the user back once resolved.
• A call analytics dashboard. Tracks call topics and outcomes, so I can actually measure whether the agent is helping rather than assume it.
• Handoffs to specialist agents. The main agent recognizes when a question needs deeper expertise and hands off cleanly :

    @function_tool
    async def handoff_to_scheme_specialist(
        self,
        ctx: RunContext,
        query_reason: str,
        caller_question: Optional[str] = None,
    ) -> str:
        """Hand off the current caller and conversation to the Government Scheme Specialist agent.

        MUST CALL THIS TOOL WHEN:
        1. The user asks questions about Indian government financial schemes (PM-KISAN, PM MUDRA, Atal Pension Yojana, Sukanya Samriddhi, Ayushman Bharat).
        2. The user wants to check eligibility, required documents, or application procedures for any government financial scheme.

        DO NOT CALL THIS TOOL FOR:
        - General cyber safety, UPI PIN advice, online fraud helpline (1930) queries, or banking fraud reports.

        Args:
            query_reason: Summary of why the caller is being transferred to the specialist (e.g. 'Caller asked about PM-Kisan eligibility').
            caller_question: The specific query or question asked by the caller (optional).
        """
        try:
            specialist = GovernmentSchemeSpecialist()
            # Retain session context and switch active agent
            call_state = ctx.session.userdata.get("call_state")
            if call_state:
                call_state["transferred_to_specialist"] = True
                call_state["specialist_reason"] = query_reason
                call_state["caller_question"] = caller_question or ""

            # Update session's current active agent to GovernmentSchemeSpecialist
            if hasattr(ctx.session, "update_agent"):
                res = ctx.session.update_agent(specialist)
                if inspect.isawaitable(res):
                    await res

            logger.info("🔀 Handoff initiated to GovernmentSchemeSpecialist (Smita) | reason=%s", query_reason)

            announcement = "मैं आपको हमारे सरकारी योजना विशेषज्ञ (Government Scheme Specialist) से कनेक्ट कर रही हूँ। कृपया एक पल रुकिए।"

            return json.dumps({
                "status": "handoff_success",
                "spoken_announcement": announcement,
                "instruction": "Speak ONLY the spoken_announcement above to the user. Do NOT greet as Smita yourself. The specialist agent will introduce herself in her own voice on the next turn.",
            }, ensure_ascii=False)
        except Exception as e:
            logger.error(f"Error executing handoff to specialist: {e}")
            return json.dumps({
                "status": "error",
                "spoken_failure_message": "माफ़ कीजिए, विशेषज्ञ एजेंट से कनेक्ट करने में कुछ समस्या आई है।",
                "message": str(e),
            }, ensure_ascii=False)

5. Challenges and How I Overcame Them

• Voice mismatches across multiple agents
  . Once I introduced specialist agents, different agents sometimes defaulted to different TTS voices mid-conversation, breaking the illusion of a coherent handoff. I standardized on a single base voice across all agents, using persona differences (Anjali, Smita, Kriti) instead of different actual audio voices — simpler and far more reliable.

• A framework migration mid-build
  . Partway through, I had to rewrite tool registration — moving from an older @llm.ai_callable / FunctionContext pattern to the newer @function_tool decorator in livekit-agents 1.4+. This meant dropping type annotations and switching to plain docstrings. It cost an afternoon of debugging silent tool-call failures before I found the real cause: a version mismatch between what I'd written against and what was actually installed.

• A design decision I changed
  : I originally planned to give each specialist a distinct TTS voice, thinking it would make handoffs feel more dramatic. In practice it introduced bugs and felt jarring. I switched to one consistent voice, letting the specialist's spoken introduction ("Hi, I'm Smita, I specialize in...") signal the handoff instead — simpler and more robust.

• Troubleshooting tip for other builders
  : if an outbound call fails with a bare dispatch_id error, check that your agent worker process is actually running and registered with LiveKit before dispatching. This error usually means the dispatch API call failed upstream, and the code just surfaces the missing response field instead of the real underlying error.

6. How Readers Can Build and Run It

Prerequisites

• Python 3.10+ with uv installed
• Node.js 18+ with pnpm installed (npm install -g pnpm)
• LiveKit Server CLI (livekit-server --dev)

Step 1: Environment Setup
Create .env.local in both backend/ and frontend/ folders:

# LiveKit Credentials
LIVEKIT_URL=wss://your-livekit-domain.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

# AI Models API Keys
GOOGLE_API_KEY=your_gemini_api_key
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key

Use Google API, Deepgram API, MURF API

Step 2: Start the Services
Run each component in a separate terminal:

Terminal 1 — LiveKit Server

livekit-server --dev

Terminal 2 — Escalation & Call Analytics REST API

cd backend
uv run python src/escalation_api.py

Terminal 3 — Backend Voice Agent Workers (Anjali + Specialists)

cd backend
uv sync
uv run python src/agent.py dev

Terminal 4 — Next.js Frontend & Analytics Dashboard

cd frontend
pnpm install
pnpm dev

Terminal 5 (Optional) — Trigger Outbound Phone Call

cd backend
uv run python src/outbound_call.py

7. Links
🔗 Code repository: https://github.com/Abhrio-Star26/murf-livekit-starter
🎥 Demo videos: Day 8 : https://lnkd.in/p/dKUvuDz6
Day 2 : https://lnkd.in/p/dzrTaXpC
Day 6 : https://lnkd.in/p/dpHHwync
(Caller data, phone numbers, and database files with real call history are excluded from the public repository.)

Check out my LinkedIn profile for More 😊

Note : Built as part of the #VoiceForBharat challenge hosted by Murf AI, using Murf Falcon for text-to-speech throughout.