I built a voice health agent for Bharat in 10 days.

Subhranshu Kumar Parhi — Fri, 14 Aug 2026 19:03:02 +0000

title: "I Built a Voice Health Agent for Bharat in 10 Days"
published: true
tags: voiceai, ai, python, livekit
cover_image: https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/f8grt2lufyuahwxe23h8.jpg
Github : https://github.com/subhranshuparh/Voice-Agent.git
LinkedIn: www.linkedin.com/in/subhranshu-parhi-021562350

I Built a Voice Health Agent for Bharat in 10 Days

What if you could call a healthcare assistant, speak naturally in Hindi or English, and get help without typing anything?

That's the idea behind Aarogya Mitra—a voice-first AI health access assistant I built during the **10 Days of Voice Agents—#VoiceForBharat Edition challenge powered by Murf AI.

Over 10 days, I worked on real-time voice interaction, multilingual conversations, healthcare tools, consent-based memory, human escalation, outbound calls, specialist agent handoffs, and call analytics.

This is the story of what I built, the challenges I faced, and what I learned while building it.

🏥 The Problem

India has a huge and diverse healthcare ecosystem, but navigating healthcare services isn't always easy.

For many people, especially in rural and semi-urban areas, accessing information can involve:

Finding the nearest healthcare center
Understanding government health schemes
Knowing which documents are required
Booking appointments
Getting vaccination or medication reminders
Reaching a human when an AI assistant cannot help

For someone with limited digital literacy, typing into a website isn't necessarily the easiest interface.

Voice can be much more natural.

Instead of searching through a portal, someone could simply ask:

"मेरे नज़दीक कौन सा PHC है?"

"Which PHC is nearest to me?"

That became the core idea behind Aarogya Mitra.

🎙️ What Is Aarogya Mitra?

Aarogya Mitra is a full-stack, real-time voice AI health access assistant designed to help citizens navigate public healthcare services through natural conversation.

The goal isn't to replace doctors.

Instead, the agent acts as a health access assistant — helping users find information, navigate services, and connect with human support when necessary.

What can it do?

🏥 Find nearby PHCs and hospitals
💳 Explain Ayushman Bharat / PM-JAY eligibility
📄 Explain required documents
📅 Help with clinic appointments
📞 Make outbound vaccination and medication reminder calls
👨‍⚕️ Escalate complex cases to human supervisors
🧠 Remember returning callers with explicit consent
📊 Track calls through analytics
🛑 Process opt-out requests immediately

The core idea is simple:

Make healthcare access simpler through voice.

🧠 How the System Works

The core pipeline is:

🎙️ User speaks
       ↓
Deepgram Nova-3
Speech → Text
       ↓
Google Gemini
Reasoning + Tool Calls + Guardrails
       ↓
Murf Falcon
Text → Speech
       ↓
LiveKit
Real-time audio transport
       ↓
🔊 User hears the response

Technology Stack

Component	Technology	Purpose
Speech-to-Text	Deepgram Nova-3	Multilingual transcription
LLM	Google Gemini	Reasoning and tool calls
Text-to-Speech	Murf Falcon	Natural Indian voice
Transport	LiveKit	Real-time voice communication
Backend	Python	Agent logic and tools
Frontend	Next.js	Voice interface
Database	SQLite	Memory and analytics

The backend connects the entire pipeline through the LiveKit Agents SDK.

🇮🇳 1. Giving the Agent an Indian Voice

One of the first things I changed was the TTS voice.

I used Murf Falcon with the Anisha voice, an Indian English voice designed for natural conversations.

I also configured different voices for specialist agents:

Anisha → Main Aarogya Mitra agent
Pooja → Appointment Specialist
Samar → Health Scheme Specialist

The goal was simple:

The voice should feel familiar to Indian users.

The TTS configuration looked like this:

tts = murf.TTS(
    voice="Anisha",
    style="Conversation",
    tokenizer=tokenize.basic.SentenceTokenizer(
        min_sentence_len=2
    ),
    text_pacing=True,
)

The low latency of the voice pipeline helped make the conversation feel much closer to a real phone conversation rather than a traditional chatbot.

🛡️ 2. Safety Guardrails

Healthcare is a sensitive domain, so I didn't want the agent to behave like a doctor.

The system prompt defines "Aarogya Mitra" as

"An empathetic and reliable voice health access assistant."

The agent does not:

Diagnose medical conditions
Prescribe medication
Pretend to be a doctor
Invent medical information

For example:

"Main doctor nahi hu aur diagnosis ya prescription dawa nahi bata sakta. Kripya certified doctor se consult karein."

The system also has an emergency escalation flow for serious situations such as chest pain or breathing difficulties.

The biggest lesson here:

In healthcare AI, knowing what not to do is just as important as knowing what to do.

🗣️ 3. Hindi + English + Hinglish

One of the most interesting parts of this project was multilingual interaction.

I configured Deepgram Nova-3 for multilingual speech recognition and used LiveKit's multilingual turn detection.

The agent can handle:

Hindi

"मुझे पास का अस्पताल बताओ"

English

"Where is the nearest hospital?"

Hinglish

"Mera health card kaise apply karu?"

Real conversations in India aren't always purely Hindi or purely English.

People naturally switch between languages.

I also instructed the LLM to generate Hindi using Devanagari script instead of Romanized Hindi.

❌ namaste, aap kaise hain?

✅ नमस्ते, आप कैसे हैं?

This improved the naturalness of the TTS output.

🧠 4. Memory for Returning Callers

I wanted Aarogya Mitra to become more useful when someone returns.

The system stores caller information in SQLite only with explicit consent.

Possible information includes:

Name
District
Language preference
Appointment history
Other permitted profile information

The agent can retrieve a caller using a tool like the following:

@function_tool()
async def lookup_caller(
    self,
    context: RunContext,
    query: str
) -> str:

    profile = db.get_user_profile(query)

    if profile:
        return json.dumps(
            profile,
            ensure_ascii=False
        )

    return "No record found for this caller."

The agent always asks for consent before saving information.

It also supports a "forget me" command that removes stored caller records.

🏥 5. Healthcare Domain Tools

A voice agent becomes much more useful when it can actually perform actions instead of only generating text.

I built domain-specific healthcare tools.

PHC Lookup

lookup_nearest_phc(district)

The tool provides information about:

PHCs
District hospitals
Jan Aushadhi Kendras
OPD timings
Doctor availability
Bed information

The current prototype contains data for:

Patna
Varanasi
Lucknow
Jaipur
Bhopal
Ranchi

Scheme Eligibility

I also created:

check_scheme_eligibility(scheme_name)

This provides information about:

Ayushman Bharat / PM-JAY
Eligibility
Coverage
Required documents
Relevant helpline information

👨‍⚕️ 6. Human Escalation

An AI assistant shouldn't try to handle every situation itself.

When a case requires human assistance, Aarogya Mitra can create a human escalation request.

The flow is:

User needs human assistance
        ↓
Agent explains what will be shared
        ↓
Ask for explicit consent
        ↓
User says YES
        ↓
create_human_help_request()
        ↓
Generate reference ID
        ↓
Give reference ID to caller
        ↓
Optional supervisor notification

For example:

"Kya main aapki details supervisor team ko send kar du?"

If the user agrees, the system generates a reference ID such as the following:

ESC-47823

The important part is that the reference ID comes from the backend tool.

The LLM is never allowed to invent it.

I also added automatic redaction for sensitive information such as the following:

Aadhaar numbers
OTPs
PINs

📞 7. Outbound Health Reminder Calls

Aarogya Mitra also supports outbound health reminder calls.

For example:

uv run python src/outbound_dialer.py \
    --to +919876543210 \
    --name "Ramesh Kumar" \
    --reminder "Polio Booster"

It can also process a CSV campaign:

uv run python src/outbound_dialer.py \
    --csv health_reminders.csv

The outbound agent starts with a scripted introduction and provides an opt-out mechanism.

If someone says:

"Stop calling me."

or:

"Mujhe call mat karo."

The opt-out tool is triggered.

This taught me an important lesson:

Voice automation should always respect the user's ability to say no.

🤖 8. Multi-Agent Handoffs

One of my favorite architectural features was the specialist agent system.

Instead of making one huge agent responsible for everything, I created specialist agents.

                 Aarogya Mitra
                       |
          ┌────────────┴────────────┐
          ↓                         ↓
Appointment Specialist       Scheme Specialist
       Pooja                       Samar

When a caller wants to book an appointment, the Aarogya Mitra transfers the conversation to the appointment specialist.

async def transfer_to_appointment_specialist(
    self,
    context: RunContext
) -> tuple[Agent, str]:

    specialist = AppointmentSpecialistAgent(
        chat_ctx=self.chat_ctx.copy(
            exclude_instructions=True
        ),
        call_id=self.call_id,
    )

    return (
        specialist,
        "Main aapko hamare Clinic aur "
        "Appointment Specialist se connect "
        "kar raha hu."
    )

The conversation context is preserved during the handoff.

The specialist can then use the following:

check_available_slots()
book_clinic_appointment()

This architecture makes the system easier to scale because every specialist has its own:

Prompt
Voice
Tools
Responsibilities

📊 9. Call Analytics

Every call is logged in SQLite.

The system tracks:

Total calls
Successful calls
Failed calls
Call duration
Channel
Actions taken
Failure category

Failure categories include:

user_hungup_early
user_declined_consent
tool_or_api_error
no_action_taken

This provides the foundation for a future real-time analytics dashboard.

🧩 Challenges I Faced

Building the agent wasn't simply about connecting APIs.

Most of the interesting work happened when things didn't work as expected.

Challenge 1 — Hindi TTS Pronunciation

Problem

Initially, the LLM generated Romanized Hindi:

namaste, aap kaise hain?

The voice could pronounce it, but it didn't feel as natural as native Hindi.

Solution

I added an explicit system instruction:

Always write every language in its own native script.

Hindi → Devanagari

Never romanize Hindi.

Lesson

TTS quality isn't only about the voice model. The text you send to the model matters too.

Challenge 2 — The LLM Inventing Reference IDs

Before the escalation tool executed, the LLM could sometimes say:

Your ticket ID is ESC-12345.

while the backend generated the following:

ESC-47823

That creates a serious reliability problem.

Solution

I added:

NEVER INVENT OR SPEAK A REFERENCE ID
BEFORE CALLING THE TOOL.

ALWAYS WAIT FOR THE TOOL OUTPUT.

I also added automated tests to catch this behavior.

Lesson

Never let an LLM invent identifiers that should come from your backend.

Challenge 3 — Multilingual Turn Detection

The default English turn detector sometimes interrupted Hindi sentences before the caller had finished speaking.

Solution

I switched to LiveKit's multilingual turn detection.

This improved sentence-boundary detection for Hindi and English conversations.

Lesson

A good voice agent needs to understand not only what the user says, but when they have finished speaking.

Challenge 4 — Enforcing Consent

I needed to make sure the agent could never create a human escalation request without permission.

Prompt instructions alone weren't enough.

So I used two layers.

Layer 1 — System prompt

Before invoking create_human_help_request,
you MUST ask the caller for permission.

Layer 2 — Tool validation

The tool checks:

user_permission_granted

If permission isn't granted, the tool refuses to create the request.

This is a simple example of defense in depth for AI systems.

🏗️ Voice Agent Architecture

The minimum architecture requires four major components:

Component	Purpose	Technology
STT	Speech → Text	Deepgram Nova-3
LLM	Reasoning + Response	Google Gemini
TTS	Text → Speech	Murf Falcon
Transport	Real-time audio	LiveKit

The complete flow looks like:

                    🎙️ USER
                       |
                       ↓
             Deepgram Nova-3 STT
                  Multilingual
                       |
                       ↓
               Google Gemini
            Reasoning + Tool Calls
                       |
          ┌────────────┼─────────────┐
          ↓            ↓             ↓
       PHC Tool    Scheme Tool   Memory Tool
          |            |             |
          └────────────┼─────────────┘
                       ↓
                 Murf Falcon
                    Anisha
                       |
                       ↓
               LiveKit WebRTC
                   / SIP
                       |
                       ↓
                    🔊 USER

🚀 How to Run the Project

The project started from the open-source Murf LiveKit starter.

git clone https://github.com/murf-ai/murf-livekit-starter.git

cd murf-livekit-starter

You'll need API keys for:

LiveKit
Murf
Deepgram
Google AI

Create:

backend/.env.local

and add:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-key
LIVEKIT_API_SECRET=your-secret

MURF_API_KEY=your-key
DEEPGRAM_API_KEY=your-key
GOOGLE_API_KEY=your-key

Never commit API keys to GitHub.

Then run:

.\start_app.ps1

Open:

http://localhost:3000

Click:

Talk to Aarogya Mitra

Then try:

"मेरे नज़दीक Patna में PHC कहाँ है?"

🔄 The Same Architecture Can Build Other Agents

One thing I really liked about this architecture is how easily the use case can change.

The same foundation could be adapted into the following:

🎓 AI language tutor
🏦 Banking voice assistant
🛒 Shopping assistant
🏨 Hotel booking agent
🎧 Customer support agent
📅 Appointment scheduling assistant

The underlying voice pipeline stays almost the same.

🔮 What's Next?

This project is still just the beginning.

1. Live Healthcare Data

The current healthcare facility data is static.

I'd like to integrate government healthcare APIs and ABDM-related infrastructure for more dynamic information.

2. WhatsApp / SMS Follow-ups

After a call, users could receive:

Appointment IDs
Escalation reference IDs
Reminder information

3. Real Inbound SIP Calling

The next step would be a verified Indian inbound number so users could directly call Aarogya Mitra.

4. Real-Time Analytics

I'd like to build a dashboard showing the following:

Total Calls
Successful Calls
Failed Calls
Success Rate
Average Duration
Escalation Count
Opt-out Count

5. More Indian Languages

I'd also like to experiment with the following:

Bengali
Tamil
Rajasthani

The long-term goal is to make the agent accessible to more people across India.

💡 What I Learned

After 10 days of building this project, these are my biggest takeaways:

1. Voice AI is more than STT + LLM + TTS

A good voice agent needs the following:

Low latency + turn detection + prompting + reliable tools + safety + conversation design.

2. LLMs need guardrails

Don't assume the model will always follow your instructions.

Critical operations should have backend-level validation.

3. Consent should be enforced in code

For sensitive operations, don't rely only on the system prompt.

The tool itself should verify permission.

4. Multilingual AI requires more than translation

Hindi-English conversations have their own rhythm and turn-taking patterns.

STT, turn detection, and TTS all matter.

5. Specialized agents scale better

Instead of one massive agent with dozens of responsibilities, specialist agents can handle specific workflows.

6. The best AI projects solve real problems

The most satisfying part wasn't connecting APIs.

It was thinking about:

How can voice make an existing service easier for someone who struggles with traditional digital interfaces?

That's the part of AI development I want to keep exploring.

📂 Project Structure

backend/
├── src/
│   ├── agent.py
│   ├── db.py
│   ├── health_tools.py
│   ├── escalation_tools.py
│   └── outbound_dialer.py
│
└── tests/
    └── test_memory.py

Main files

agent.py — Main voice agent, tools, guardrails, and specialist handoffs.

db.py — SQLite memory and call analytics.

health_tools.py — PHC lookup and health scheme functionality.

escalation_tools.py—Human escalation, sanitization, and supervisor notification.

outbound_dialer.py — Outbound calls and CSV campaigns.

test_memory.py — Evaluation and memory-related tests.

🔗 Project Repository

Murf LiveKit Starter:

https://github.com/murf-ai/murf-livekit-starter

The project uses:

Murf Falcon TTS
LiveKit Agents
Deepgram Nova-3
Google Gemini
Python
Next.js
SQLite

🎤 Final Thoughts

Building Aarogya Mitra in 10 days taught me that creating a voice agent isn't simply about making an AI talk.

It's about designing an entire conversation system.

The agent needs to know:

When to speak
When to listen
When to use a tool
When to ask permission
When to hand off to a human
When to stop
And, most importantly, when not to answer

I'm excited to continue exploring Voice AI, Agentic AI, multilingual systems, and AI for social impact.

If you're building a voice agent too, I'd love to see what you're creating.

Built as part of the 10 Days of Voice Agents — #VoiceForBharat Edition. 🇮🇳🎙️

Let's build useful AI.

DEV Community: Subhranshu Kumar Parhi

I built a voice health agent for Bharat in 10 days.

I Built a Voice Health Agent for Bharat in 10 Days

🏥 The Problem

🎙️ What Is Aarogya Mitra?

What can it do?

🧠 How the System Works

Technology Stack

🇮🇳 1. Giving the Agent an Indian Voice

🛡️ 2. Safety Guardrails

🗣️ 3. Hindi + English + Hinglish

Hindi

English

Hinglish

🧠 4. Memory for Returning Callers

🏥 5. Healthcare Domain Tools

PHC Lookup

Scheme Eligibility

👨‍⚕️ 6. Human Escalation

📞 7. Outbound Health Reminder Calls

🤖 8. Multi-Agent Handoffs

📊 9. Call Analytics

🧩 Challenges I Faced

Challenge 1 — Hindi TTS Pronunciation

Problem

Solution

Lesson

Challenge 2 — The LLM Inventing Reference IDs

Solution

Lesson

Challenge 3 — Multilingual Turn Detection

Solution

Lesson

Challenge 4 — Enforcing Consent

Layer 1 — System prompt

Layer 2 — Tool validation

🏗️ Voice Agent Architecture

🚀 How to Run the Project

🔄 The Same Architecture Can Build Other Agents

🔮 What's Next?

1. Live Healthcare Data

2. WhatsApp / SMS Follow-ups

3. Real Inbound SIP Calling

4. Real-Time Analytics

5. More Indian Languages

💡 What I Learned

1. Voice AI is more than STT + LLM + TTS

2. LLMs need guardrails

3. Consent should be enforced in code

4. Multilingual AI requires more than translation

5. Specialized agents scale better

6. The best AI projects solve real problems

📂 Project Structure

Main files

🔗 Project Repository

🎤 Final Thoughts