DEV Community

Darshan Jethava
Darshan Jethava

Posted on

Building Arogya Seva: An Indian Telehealth & Health Access Voice Assistant

#agentaichallenge #python #ai

1. The Problem & The Users

Accessing timely healthcare advice in rural and semi-urban India often comes with significant hurdles—geographic distances to Primary Health Centres (PHCs), long wait times, limited medical literacy, and language barriers. For millions of citizens, reading complex health portals or typing symptoms into an app is neither intuitive nor accessible.

Voice changes everything.

Voice provides a natural, hands-free, and frictionless interface. Anyone who can speak a phone call can interact with a voice AI assistant.

To solve this, I built Arogya Seva (Arogya = Health, Seva = Service) — an empathetic, real-time voice assistant designed to provide accessible health guidance, preliminary symptom triage, real-time health facility lookups, human doctor escalation, and privacy-first caller memory.

2. What Arogya Seva Does

Arogya Seva acts as the first line of telehealth contact for callers across Bharat. Here is what the assistant handles in real time:

  • Preliminary Symptom Triage: Categorizes health concerns into RED (Clinical Emergency), YELLOW (Urgent Care), or GREEN (Routine / Home Care).
  • Real Health Facility Locator: Queries live National Health Facility Registry data to locate nearby PHCs, Community Health Centres (CHCs), and District Hospitals with bed availability and emergency phone numbers.
  • Smart Tool Chaining: Remembers caller context (like location/district) across conversations so users don't have to repeat themselves.
  • Privacy-First Consent & Memory: Remembers returning callers only after explicit verbal consent; supports a "Forget Me" tool (forget_caller) to erase database records instantly.
  • Human Escalation Workflow: Detects severe emergency red flags or requests for human doctors, asks for caller consent to share data, and generates a trackable Reference ID (e.g. ESC-8492).
  • Compliant Outbound Reminders: Makes outbound follow-up calls adhering to strict 2-sentence opening disclosure rules and single-command opt-out/Do Not Disturb (opt_out_caller).

3. System Architecture & How It Works

The voice pipeline is built on LiveKit Agents SDK, orchestrating real-time audio streaming between specialized AI services:

[ User Microphone / WebRTC ]
            │
            ▼
┌─────────────────────────┐
│ Deepgram Nova-3 (STT)   │ ──► Converts Speech to Text in real time
└───────────┬─────────────┘
            │
            ▼
┌─────────────────────────┐
│  Google Gemini (LLM)    │ ──► Evaluates prompts, decides tool calls
└───────────┬─────────────┘      (triage, PHC lookup, memory, escalation)
            │
            ▼
┌─────────────────────────┐
│   Murf Falcon (TTS)     │ ──► Synthesizes sub-200ms ultra-fast streaming
└───────────┬─────────────┘      Indian English voice ("Anisha")
            │
            ▼
[ User Speaker / WebRTC ]
Enter fullscreen mode Exit fullscreen mode

Key Technical Stack:

  • Speech-to-Text (STT): Deepgram Nova-3 (Multilingual speech recognition)
  • Large Language Model (LLM): Google Gemini (gemini-2.5-flash)
  • Text-to-Speech (TTS): Murf Falcon (en-IN voice: Anisha)
  • Real-Time Transport & VAD: LiveKit Agents SDK, Silero VAD, LiveKit Multilingual Turn Detector, and Noise Cancellation.
  • Database & Storage: SQLite DB for structured caller memory, opt-out registry, and escalation logs.
  • Frontend UI: Next.js with LiveKit Agents UI components (audio visualizer, transcripts, session controls).

4. The Most Important Features

A. Lightning-Fast Indian English Voice (Murf Falcon)

Using Murf Falcon's Anisha voice (en-IN), the agent streams natural Indian English audio back to the user with ultra-low latency. Streaming response chunks ensure the user hears immediate verbal feedback without awkward pauses.

B. Tool Chaining & Facility Registry Lookups

When a returning caller asks for a nearby hospital, Arogya Seva reuses their saved district from SQLite memory and feeds it straight into lookup_nearest_phc:

@function_tool
async def lookup_nearest_phc(context: RunContext, district: str, facility_type: str = "PHC"):
    """Lookup real Primary Health Centres and emergency contacts from registry."""
    # Queries health facility data with live timestamping
    ...
Enter fullscreen mode Exit fullscreen mode

C. Human Escalation with Explicit Consent & Ref ID

For medical emergencies or explicit requests for human doctors, the agent follows strict guardrails:

  1. Asks explicit permission before sending data.
  2. Invokes create_escalation function tool.
  3. Provides a clear Reference ID (e.g. ESC-8492) and instructions to dial 108 if symptoms worsen. 
@function_tool
async def create_escalation(context: RunContext, summary: str, urgency: str):
    """Creates a human escalation request for health officer follow-up."""
    # Generates unique trackable Reference ID: ESC-XXXX
    ...
Enter fullscreen mode Exit fullscreen mode

5. Challenges Faced & Solutions

Challenge 1: Handling API Failures Gracefully Out Loud

  • Problem: External facility APIs or database connections can occasionally time out or fail. In a voice interface, silence or throwing uncaught code exceptions creates confusion and panic for a healthcare caller.
  • Solution: Implemented explicit fallback handling inside tool functions and instructions in the system prompt. If lookup_nearest_phc returns offline status, Arogya Seva calmly announces: "The health facility registry is currently unreachable. Please contact emergency services directly at 108 or 104 immediately."

Challenge 2: Voice Turn-Taking vs. Database Execution Time

  • Problem: Executing database lookups and tool chains mid-conversation risked triggering turn-detector timeouts or interjected user silence.
  • Solution: Fine-tuned Silero VAD parameters alongside LiveKit's Multilingual Turn Detector and pre-warmed models during agent startup (prewarm() phase).

6. How to Build & Run It Yourself

Want to set up your own version of Arogya Seva? Follow these simple steps:

Prerequisites

  • Python 3.10+ with uv installed (pip install uv)
  • Node.js 18+ and pnpm (npm install -g pnpm)
  • API Keys: LiveKit Cloud, Murf AI, Deepgram, and Google Gemini.

Step 1: Clone the Repository 

git clone https://github.com/mr-dj06/bharat.git
cd bharat
Enter fullscreen mode Exit fullscreen mode

Step 2: Configure Environment Variables

Create .env.local inside backend/ and frontend/:

LIVEKIT_URL=wss://your-livekit-project.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_gemini_api_key
Enter fullscreen mode Exit fullscreen mode

Step 3: Run the Agent Backend 

cd backend
uv sync
uv run python src/agent.py download-files   # First time only to download VAD models
uv run python src/agent.py dev
Enter fullscreen mode Exit fullscreen mode

Step 4: Run the Frontend UI 

cd frontend
pnpm install
pnpm dev
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000 in your browser, click Connect, and start talking to Arogya Seva!

7. What's Next? Future Improvements

  1. Native Multilingual Speech: Expanding full speech-to-speech support for Hindi, Tamil, Telugu, and Marathi using Murf Falcon's extended language offerings.
  2. SMS & WhatsApp Sync: Automatically dispatching escalation reference IDs and PHC addresses directly to the user's phone via SMS/WhatsApp.
  3. Voice Biometric Verification: Secure voice authentication for returning callers accessing medical history.

8. Repository Links & Acknowledgments

Thank you to the Murf AI team for hosting #VoiceForBharat! 🇮🇳✨

Top comments (0)