Building Arogya Seva: How I Built an Ultra-Low Latency Telehealth Voice AI for Bharat in 10 Days

1. Introduction: The Problem & The Mission Across rural and semi-urban India, access to timely healthcare advice is hindered by high patient-to-doctor ratios, geographical distances, and language barriers. Millions of citizens delay seeking medical advice simply because scheduling a clinic visit or understanding preliminary symptom urgency is daunting.

Arogya Seva was created to bridge this gap as part of the #VoiceForBharat challenge (Track: Health Access). It is an empathetic, multilingual, real-time voice assistant designed to interact naturally in Indian English, Hindi (Devanagari script), and regional scripts.

Why Voice? For millions of non-tech-savvy users or individuals in low-literacy regions, typing in an app or filling out complex forms is a friction point. Speaking directly over a phone call or web interface is the most accessible, natural, and human way to receive guidance.

1. System Architecture: How Audio & Data Flow To deliver a conversational voice experience, latency is paramount. A delay of more than 800ms between a user finishing their sentence and hearing a response breaks the illusion of natural conversation.

The system is built on LiveKit Agents SDK with a modular pipeline:

Speech-to-Text (STT): Deepgram Nova-3 transcribes spoken voice in real time.
Brain (LLM): Google Gemini 2.0 Flash processes intent, applies clinical guardrails, and decides on function tool calls.
Text-to-Speech (TTS): Murf Falcon (livekit-murf plugin, voice model en-IN-Anisha) streams ultra-low latency, human-like voice synthesis back to the user.
Real-time Transport: LiveKit WebRTC (web frontend) and SIP Telephony (outbound/inbound phone calls).
Memory & State: SQLite (agent_memory.db) for privacy-first caller persistence and escalation management.
Mermaid diagram

1. Important Features Built Over 10 Days 🚀 Feature 1: Sub-Second Voice Synthesis Powered by Murf Falcon Using Murf Falcon (en-IN-Anisha voice model), the agent achieves lightning-fast time-to-first-byte (TTFB). The voice sounds warm, empathetic, and natural—crucial for building trust with patients discussing health concerns.

🛡️ Feature 2: Strict Guardrails & Native Script Enforcement
Health AI requires absolute safety. Arogya Seva follows strict operational boundaries:

Red-Flag Clinical Emergency Protocol: Immediately flags chest pain, dyspnea, heavy bleeding, or acute trauma, urging callers to dial emergency 108.
Native Script Enforcement: To ensure proper acoustic synthesis and avoid awkward transliteration, responses in Hindi are strictly produced in native Devanagari script (e.g., नमस्ते, आप कैसे हैं?), avoiding romanized "Hinglish".
💻 Feature 3: Dynamic Frontend State & Audio Visualizers
Built with Next.js and LiveKit Agents UI, the frontend displays real-time agent states:

Listening (Visualized with dynamic frequency waveforms)
Thinking (Tool execution state)
Speaking (Fluid audio spectrum representation)
🧠 Feature 4: Privacy-First Memory with Explicit Consent
Returning callers don't need to re-explain their location or age band. However, privacy is paramount:

The agent explicitly asks: "May I save your name and basic health details so I can remember you next time?"
Facts are stored only if explicit consent is given.
Users can say "Forget me" at any time to wipe their records via forget_caller.
🛠️ Feature 5: Real-Domain Health Tools & Tool Chaining
classify_symptom_triage: Categorizes symptoms into Self-Care / Low, Moderate / Consult Nurse, or High Urgent / Red-Flag.
lookup_nearest_phc: Searches Primary Health Centres based on district.
Tool Chaining: Automatically reuses district information saved in caller memory without re-asking the user.
Graceful Failure: If the registry API is unreachable, the agent announces the offline status calmly and provides emergency helpline 104/108 numbers.
📞 Feature 6: Outbound Telephony & Mandatory Opt-Out
For automated health reminders and follow-up calls:

Two-Sentence Mandatory Opening: State WHO is calling, WHY, and HOW to opt out in the first two sentences.
Instant Opt-Out: Saying "stop calling me" or pressing 9 immediately executes opt_out_caller in SQLite and terminates the call.
🆘 Feature 7: Human Escalation & Reference IDs
When situations exceed AI scope:

Agent detects clinical doctor requests or red-flag symptoms.
Agent requests explicit permission to create an escalation ticket.
Upon agreement, create_escalation stores a sanitized summary (no passwords/PINs/Aadhaar) and returns a unique reference ID (e.g., ESC-8492).
📊 Feature 8: Call Analytics & Outcome Tracking
Every call session logs structured metrics into SQLite, including call duration, triage classifications, escalation status, and resolution codes (triage_completed, phc_found, escalated, handed_off).

🔀 Feature 9: Multi-Agent Specialist Handoff
When callers request to schedule, modify, or cancel OPD appointments, the main agent invokes transfer_to_clinic_specialist:

python

@function_tool
async def transfer_to_clinic_specialist(self, context: RunContext, reason: str) -> str:
specialist = ClinicAppointmentSpecialist()
context.session.update_agent(specialist)
return "Handed off conversation to Clinic and Appointment Specialist."
The session dynamically updates to ClinicAppointmentSpecialist, seamlessly swapping persona and toolsets without dropping the audio call!

1. Real Challenges & How We Solved Them Challenge 1: TTS Latency Spikes During Conversational Turns Problem: Default chunking caused 1.5-second pauses before the agent spoke. Root Cause: Large sentence buffers in the LLM-to-TTS pipeline. Solution: Integrated Murf Falcon with streaming tokenization and prewarmed Silero VAD models. This reduced speech synthesis latency to under 300ms! Challenge 2: Accidental Code-Mixed Script Bleed Problem: The LLM would occasionally respond to Hindi input with romanized Hindi ("Aapko kya takleef hai?"), causing the TTS to pronounce Hindi words with English phonetics. Solution: Implemented a mandatory system prompt guardrail enforcing native script generation (e.g., Devanagari for Hindi). Challenge 3: Agent Handoff State Management Problem: When handing off from general health to clinic specialist, tool contexts were losing call metadata. Solution: Leveraged context.session.update_agent(specialist) in LiveKit Agents SDK, ensuring the active WebRTC media room remained untouched while the prompt and function tools switched dynamically.
2. Practical Guide: Build & Run Your Own Voice Agent Want to build your own ultra-fast voice agent? Follow these steps!

Step 1: Prerequisites
Python 3.10+ & uv package manager
Node.js 18+ & pnpm
LiveKit Cloud account (URL, API Key, API Secret)
Murf AI API Key (for Falcon TTS)
Deepgram API Key (for STT)
Google Gemini API Key (for LLM)
Step 2: Clone & Configure Backend
bash

git clone https://github.com/viral-1998/VoiceOfBharat.git
cd VoiceOfBharat/backend

Create environment file (.env.local)

cp .env.example .env.local
Add your API keys to backend/.env.local:

env

LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_google_key
Step 3: Run Backend Agent
bash

uv sync
uv run python src/agent.py download-files # First time model download
uv run python src/agent.py dev # Start live dev server
Step 4: Run Frontend UI
bash

cd ../frontend
pnpm install
pnpm dev
Open http://localhost:3000 in your browser, click Connect, and start speaking to your agent!

1. Code Spotlight: Specialist Handoff Logic Here is the exact Python implementation for handing off a LiveKit session from the main Telehealth assistant to the Appointment Specialist agent:

python

Function tool in Assistant class

@function_tool
async def transfer_to_clinic_specialist(
self,
context: RunContext,
reason: str = "User requested appointment booking",
) -> str:
"""Transfer caller to Clinic & Appointment Specialist agent."""
specialist = ClinicAppointmentSpecialist()
context.session.update_agent(specialist)

call_id = getattr(getattr(context, "session", None), "call_id", "")
if call_id:
    db.mark_call_success(call_id, outcome_summary=f"Handed off: {reason}")

return "Handed off conversation to Clinic and Appointment Specialist."

1. What's Next? Future improvements for Arogya Seva include:

Multi-lingual Voice Cloning: Adding localized voice accents across 10+ Indian regional languages using Murf Falcon's voice library.
WhatsApp Telemetry Notifications: Sending automated SMS/WhatsApp appointment receipts following human escalations.
EHR Integration: Connecting triage outcomes directly with ABDM (Ayushman Bharat Digital Mission) health IDs.

1. Links & Resources 🐙 GitHub Repository: https://github.com/Viral-1998/VoiceOfBharat Repo ⚡ Murf Falcon TTS Docs: Falcon 2 Documentation 🎙️ LiveKit Voice AI Quickstart: LiveKit Agents Guide 🏆 Challenge Details: VoiceForBharat Challenge 2026 Thank you to Murf AI and LiveKit for hosting the 10 Days of AI Voice Agents (#VoiceForBharat Edition)!