DEV Community

Subhranshu Kumar Parhi
Subhranshu Kumar Parhi

Posted on

I built a voice health agent for Bharat in 10 days.

title: "I Built a Voice Health Agent for Bharat in 10 Days"
published: true
tags: voiceai, ai, python, livekit
cover_image: https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/f8grt2lufyuahwxe23h8.jpg
Github : https://github.com/subhranshuparh/Voice-Agent.git
LinkedIn: www.linkedin.com/in/subhranshu-parhi-021562350

I Built a Voice Health Agent for Bharat in 10 Days

What if you could call a healthcare assistant, speak naturally in Hindi or English, and get help without typing anything?

That's the idea behind Aarogya Mitraโ€”a voice-first AI health access assistant I built during the **10 Days of Voice Agentsโ€”#VoiceForBharat Edition challenge powered by Murf AI.

Over 10 days, I worked on real-time voice interaction, multilingual conversations, healthcare tools, consent-based memory, human escalation, outbound calls, specialist agent handoffs, and call analytics.

This is the story of what I built, the challenges I faced, and what I learned while building it.

๐Ÿฅ The Problem

India has a huge and diverse healthcare ecosystem, but navigating healthcare services isn't always easy.

For many people, especially in rural and semi-urban areas, accessing information can involve:

  • Finding the nearest healthcare center
  • Understanding government health schemes
  • Knowing which documents are required
  • Booking appointments
  • Getting vaccination or medication reminders
  • Reaching a human when an AI assistant cannot help

For someone with limited digital literacy, typing into a website isn't necessarily the easiest interface.

Voice can be much more natural.

Instead of searching through a portal, someone could simply ask:

"เคฎเฅ‡เคฐเฅ‡ เคจเคœเคผเคฆเฅ€เค• เค•เฅŒเคจ เคธเคพ PHC เคนเฅˆ?"

"Which PHC is nearest to me?"

That became the core idea behind Aarogya Mitra.

๐ŸŽ™๏ธ What Is Aarogya Mitra?

Aarogya Mitra is a full-stack, real-time voice AI health access assistant designed to help citizens navigate public healthcare services through natural conversation.

The goal isn't to replace doctors.

Instead, the agent acts as a health access assistant โ€” helping users find information, navigate services, and connect with human support when necessary.

What can it do?

  • ๐Ÿฅ Find nearby PHCs and hospitals
  • ๐Ÿ’ณ Explain Ayushman Bharat / PM-JAY eligibility
  • ๐Ÿ“„ Explain required documents
  • ๐Ÿ“… Help with clinic appointments
  • ๐Ÿ“ž Make outbound vaccination and medication reminder calls
  • ๐Ÿ‘จโ€โš•๏ธ Escalate complex cases to human supervisors
  • ๐Ÿง  Remember returning callers with explicit consent
  • ๐Ÿ“Š Track calls through analytics
  • ๐Ÿ›‘ Process opt-out requests immediately

The core idea is simple:

Make healthcare access simpler through voice.

๐Ÿง  How the System Works

The core pipeline is:

๐ŸŽ™๏ธ User speaks
       โ†“
Deepgram Nova-3
Speech โ†’ Text
       โ†“
Google Gemini
Reasoning + Tool Calls + Guardrails
       โ†“
Murf Falcon
Text โ†’ Speech
       โ†“
LiveKit
Real-time audio transport
       โ†“
๐Ÿ”Š User hears the response
Enter fullscreen mode Exit fullscreen mode

Technology Stack

Component Technology Purpose
Speech-to-Text Deepgram Nova-3 Multilingual transcription
LLM Google Gemini Reasoning and tool calls
Text-to-Speech Murf Falcon Natural Indian voice
Transport LiveKit Real-time voice communication
Backend Python Agent logic and tools
Frontend Next.js Voice interface
Database SQLite Memory and analytics

The backend connects the entire pipeline through the LiveKit Agents SDK.

๐Ÿ‡ฎ๐Ÿ‡ณ 1. Giving the Agent an Indian Voice

One of the first things I changed was the TTS voice.

I used Murf Falcon with the Anisha voice, an Indian English voice designed for natural conversations.

I also configured different voices for specialist agents:

  • Anisha โ†’ Main Aarogya Mitra agent
  • Pooja โ†’ Appointment Specialist
  • Samar โ†’ Health Scheme Specialist

The goal was simple:

The voice should feel familiar to Indian users.

The TTS configuration looked like this:

tts = murf.TTS(
    voice="Anisha",
    style="Conversation",
    tokenizer=tokenize.basic.SentenceTokenizer(
        min_sentence_len=2
    ),
    text_pacing=True,
)
Enter fullscreen mode Exit fullscreen mode

The low latency of the voice pipeline helped make the conversation feel much closer to a real phone conversation rather than a traditional chatbot.

๐Ÿ›ก๏ธ 2. Safety Guardrails

Healthcare is a sensitive domain, so I didn't want the agent to behave like a doctor.

The system prompt defines "Aarogya Mitra" as

"An empathetic and reliable voice health access assistant."

The agent does not:

  • Diagnose medical conditions
  • Prescribe medication
  • Pretend to be a doctor
  • Invent medical information

For example:

"Main doctor nahi hu aur diagnosis ya prescription dawa nahi bata sakta. Kripya certified doctor se consult karein."

The system also has an emergency escalation flow for serious situations such as chest pain or breathing difficulties.

The biggest lesson here:

In healthcare AI, knowing what not to do is just as important as knowing what to do.

๐Ÿ—ฃ๏ธ 3. Hindi + English + Hinglish

One of the most interesting parts of this project was multilingual interaction.

I configured Deepgram Nova-3 for multilingual speech recognition and used LiveKit's multilingual turn detection.

The agent can handle:

Hindi

"เคฎเฅเคเฅ‡ เคชเคพเคธ เค•เคพ เค…เคธเฅเคชเคคเคพเคฒ เคฌเคคเคพเค“"

English

"Where is the nearest hospital?"

Hinglish

"Mera health card kaise apply karu?"

Real conversations in India aren't always purely Hindi or purely English.

People naturally switch between languages.

I also instructed the LLM to generate Hindi using Devanagari script instead of Romanized Hindi.

โŒ namaste, aap kaise hain?

โœ… เคจเคฎเคธเฅเคคเฅ‡, เค†เคช เค•เฅˆเคธเฅ‡ เคนเฅˆเค‚?
Enter fullscreen mode Exit fullscreen mode

This improved the naturalness of the TTS output.

๐Ÿง  4. Memory for Returning Callers

I wanted Aarogya Mitra to become more useful when someone returns.

The system stores caller information in SQLite only with explicit consent.

Possible information includes:

  • Name
  • District
  • Language preference
  • Appointment history
  • Other permitted profile information

The agent can retrieve a caller using a tool like the following:

@function_tool()
async def lookup_caller(
    self,
    context: RunContext,
    query: str
) -> str:

    profile = db.get_user_profile(query)

    if profile:
        return json.dumps(
            profile,
            ensure_ascii=False
        )

    return "No record found for this caller."
Enter fullscreen mode Exit fullscreen mode

The agent always asks for consent before saving information.

It also supports a "forget me" command that removes stored caller records.

๐Ÿฅ 5. Healthcare Domain Tools

A voice agent becomes much more useful when it can actually perform actions instead of only generating text.

I built domain-specific healthcare tools.

PHC Lookup 

lookup_nearest_phc(district)
Enter fullscreen mode Exit fullscreen mode

The tool provides information about:

  • PHCs
  • District hospitals
  • Jan Aushadhi Kendras
  • OPD timings
  • Doctor availability
  • Bed information

The current prototype contains data for:

  • Patna
  • Varanasi
  • Lucknow
  • Jaipur
  • Bhopal
  • Ranchi

Scheme Eligibility

I also created:

check_scheme_eligibility(scheme_name)
Enter fullscreen mode Exit fullscreen mode

This provides information about:

  • Ayushman Bharat / PM-JAY
  • Eligibility
  • Coverage
  • Required documents
  • Relevant helpline information

๐Ÿ‘จโ€โš•๏ธ 6. Human Escalation

An AI assistant shouldn't try to handle every situation itself.

When a case requires human assistance, Aarogya Mitra can create a human escalation request.

The flow is:

User needs human assistance
        โ†“
Agent explains what will be shared
        โ†“
Ask for explicit consent
        โ†“
User says YES
        โ†“
create_human_help_request()
        โ†“
Generate reference ID
        โ†“
Give reference ID to caller
        โ†“
Optional supervisor notification
Enter fullscreen mode Exit fullscreen mode

For example:

"Kya main aapki details supervisor team ko send kar du?"

If the user agrees, the system generates a reference ID such as the following:

ESC-47823
Enter fullscreen mode Exit fullscreen mode

The important part is that the reference ID comes from the backend tool.

The LLM is never allowed to invent it.

I also added automatic redaction for sensitive information such as the following:

  • Aadhaar numbers
  • OTPs
  • PINs

๐Ÿ“ž 7. Outbound Health Reminder Calls

Aarogya Mitra also supports outbound health reminder calls.

For example:

uv run python src/outbound_dialer.py \
    --to +919876543210 \
    --name "Ramesh Kumar" \
    --reminder "Polio Booster"
Enter fullscreen mode Exit fullscreen mode

It can also process a CSV campaign:

uv run python src/outbound_dialer.py \
    --csv health_reminders.csv
Enter fullscreen mode Exit fullscreen mode

The outbound agent starts with a scripted introduction and provides an opt-out mechanism.

If someone says:

"Stop calling me."

or:

"Mujhe call mat karo."

The opt-out tool is triggered.

This taught me an important lesson:

Voice automation should always respect the user's ability to say no.

๐Ÿค– 8. Multi-Agent Handoffs

One of my favorite architectural features was the specialist agent system.

Instead of making one huge agent responsible for everything, I created specialist agents.

                 Aarogya Mitra
                       |
          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
          โ†“                         โ†“
Appointment Specialist       Scheme Specialist
       Pooja                       Samar
Enter fullscreen mode Exit fullscreen mode

When a caller wants to book an appointment, the Aarogya Mitra transfers the conversation to the appointment specialist.

async def transfer_to_appointment_specialist(
    self,
    context: RunContext
) -> tuple[Agent, str]:

    specialist = AppointmentSpecialistAgent(
        chat_ctx=self.chat_ctx.copy(
            exclude_instructions=True
        ),
        call_id=self.call_id,
    )

    return (
        specialist,
        "Main aapko hamare Clinic aur "
        "Appointment Specialist se connect "
        "kar raha hu."
    )
Enter fullscreen mode Exit fullscreen mode

The conversation context is preserved during the handoff.

The specialist can then use the following:

check_available_slots()
book_clinic_appointment()
Enter fullscreen mode Exit fullscreen mode

This architecture makes the system easier to scale because every specialist has its own:

  • Prompt
  • Voice
  • Tools
  • Responsibilities

๐Ÿ“Š 9. Call Analytics

Every call is logged in SQLite.

The system tracks:

  • Total calls
  • Successful calls
  • Failed calls
  • Call duration
  • Channel
  • Actions taken
  • Failure category

Failure categories include:

user_hungup_early
user_declined_consent
tool_or_api_error
no_action_taken
Enter fullscreen mode Exit fullscreen mode

This provides the foundation for a future real-time analytics dashboard.

๐Ÿงฉ Challenges I Faced

Building the agent wasn't simply about connecting APIs.

Most of the interesting work happened when things didn't work as expected.

Challenge 1 โ€” Hindi TTS Pronunciation

Problem

Initially, the LLM generated Romanized Hindi:

namaste, aap kaise hain?
Enter fullscreen mode Exit fullscreen mode

The voice could pronounce it, but it didn't feel as natural as native Hindi.

Solution

I added an explicit system instruction:

Always write every language in its own native script.

Hindi โ†’ Devanagari

Never romanize Hindi.
Enter fullscreen mode Exit fullscreen mode

Lesson

TTS quality isn't only about the voice model. The text you send to the model matters too.

Challenge 2 โ€” The LLM Inventing Reference IDs

Before the escalation tool executed, the LLM could sometimes say:

Your ticket ID is ESC-12345.
Enter fullscreen mode Exit fullscreen mode

while the backend generated the following:

ESC-47823
Enter fullscreen mode Exit fullscreen mode

That creates a serious reliability problem.

Solution

I added:

NEVER INVENT OR SPEAK A REFERENCE ID
BEFORE CALLING THE TOOL.

ALWAYS WAIT FOR THE TOOL OUTPUT.
Enter fullscreen mode Exit fullscreen mode

I also added automated tests to catch this behavior.

Lesson

Never let an LLM invent identifiers that should come from your backend.

Challenge 3 โ€” Multilingual Turn Detection

The default English turn detector sometimes interrupted Hindi sentences before the caller had finished speaking.

Solution

I switched to LiveKit's multilingual turn detection.

This improved sentence-boundary detection for Hindi and English conversations.

Lesson

A good voice agent needs to understand not only what the user says, but when they have finished speaking.

Challenge 4 โ€” Enforcing Consent

I needed to make sure the agent could never create a human escalation request without permission.

Prompt instructions alone weren't enough.

So I used two layers.

Layer 1 โ€” System prompt 

Before invoking create_human_help_request,
you MUST ask the caller for permission.
Enter fullscreen mode Exit fullscreen mode

Layer 2 โ€” Tool validation

The tool checks:

user_permission_granted
Enter fullscreen mode Exit fullscreen mode

If permission isn't granted, the tool refuses to create the request.

This is a simple example of defense in depth for AI systems.

๐Ÿ—๏ธ Voice Agent Architecture

The minimum architecture requires four major components:

Component Purpose Technology
STT Speech โ†’ Text Deepgram Nova-3
LLM Reasoning + Response Google Gemini
TTS Text โ†’ Speech Murf Falcon
Transport Real-time audio LiveKit

The complete flow looks like:

                    ๐ŸŽ™๏ธ USER
                       |
                       โ†“
             Deepgram Nova-3 STT
                  Multilingual
                       |
                       โ†“
               Google Gemini
            Reasoning + Tool Calls
                       |
          โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
          โ†“            โ†“             โ†“
       PHC Tool    Scheme Tool   Memory Tool
          |            |             |
          โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                       โ†“
                 Murf Falcon
                    Anisha
                       |
                       โ†“
               LiveKit WebRTC
                   / SIP
                       |
                       โ†“
                    ๐Ÿ”Š USER
Enter fullscreen mode Exit fullscreen mode

๐Ÿš€ How to Run the Project

The project started from the open-source Murf LiveKit starter.

git clone https://github.com/murf-ai/murf-livekit-starter.git

cd murf-livekit-starter
Enter fullscreen mode Exit fullscreen mode

You'll need API keys for:

LiveKit
Murf
Deepgram
Google AI
Enter fullscreen mode Exit fullscreen mode

Create:

backend/.env.local
Enter fullscreen mode Exit fullscreen mode

and add:

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-key
LIVEKIT_API_SECRET=your-secret

MURF_API_KEY=your-key
DEEPGRAM_API_KEY=your-key
GOOGLE_API_KEY=your-key
Enter fullscreen mode Exit fullscreen mode

Never commit API keys to GitHub.

Then run:

.\start_app.ps1
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:3000
Enter fullscreen mode Exit fullscreen mode

Click:

Talk to Aarogya Mitra

Then try:

"เคฎเฅ‡เคฐเฅ‡ เคจเคœเคผเคฆเฅ€เค• Patna เคฎเฅ‡เค‚ PHC เค•เคนเคพเค เคนเฅˆ?"

๐Ÿ”„ The Same Architecture Can Build Other Agents

One thing I really liked about this architecture is how easily the use case can change.

The same foundation could be adapted into the following:

  • ๐ŸŽ“ AI language tutor
  • ๐Ÿฆ Banking voice assistant
  • ๐Ÿ›’ Shopping assistant
  • ๐Ÿจ Hotel booking agent
  • ๐ŸŽง Customer support agent
  • ๐Ÿ“… Appointment scheduling assistant

The underlying voice pipeline stays almost the same.

๐Ÿ”ฎ What's Next?

This project is still just the beginning.

1. Live Healthcare Data

The current healthcare facility data is static.

I'd like to integrate government healthcare APIs and ABDM-related infrastructure for more dynamic information.

2. WhatsApp / SMS Follow-ups

After a call, users could receive:

  • Appointment IDs
  • Escalation reference IDs
  • Reminder information

3. Real Inbound SIP Calling

The next step would be a verified Indian inbound number so users could directly call Aarogya Mitra.

4. Real-Time Analytics

I'd like to build a dashboard showing the following:

Total Calls
Successful Calls
Failed Calls
Success Rate
Average Duration
Escalation Count
Opt-out Count
Enter fullscreen mode Exit fullscreen mode

5. More Indian Languages

I'd also like to experiment with the following:

  • Bengali
  • Tamil
  • Rajasthani

The long-term goal is to make the agent accessible to more people across India.

๐Ÿ’ก What I Learned

After 10 days of building this project, these are my biggest takeaways:

1. Voice AI is more than STT + LLM + TTS

A good voice agent needs the following:

Low latency + turn detection + prompting + reliable tools + safety + conversation design.

2. LLMs need guardrails

Don't assume the model will always follow your instructions.

Critical operations should have backend-level validation.

3. Consent should be enforced in code

For sensitive operations, don't rely only on the system prompt.

The tool itself should verify permission.

4. Multilingual AI requires more than translation

Hindi-English conversations have their own rhythm and turn-taking patterns.

STT, turn detection, and TTS all matter.

5. Specialized agents scale better

Instead of one massive agent with dozens of responsibilities, specialist agents can handle specific workflows.

6. The best AI projects solve real problems

The most satisfying part wasn't connecting APIs.

It was thinking about:

How can voice make an existing service easier for someone who struggles with traditional digital interfaces?

That's the part of AI development I want to keep exploring.

๐Ÿ“‚ Project Structure 

backend/
โ”œโ”€โ”€ src/
โ”‚   โ”œโ”€โ”€ agent.py
โ”‚   โ”œโ”€โ”€ db.py
โ”‚   โ”œโ”€โ”€ health_tools.py
โ”‚   โ”œโ”€โ”€ escalation_tools.py
โ”‚   โ””โ”€โ”€ outbound_dialer.py
โ”‚
โ””โ”€โ”€ tests/
    โ””โ”€โ”€ test_memory.py
Enter fullscreen mode Exit fullscreen mode

Main files

agent.py โ€” Main voice agent, tools, guardrails, and specialist handoffs.

db.py โ€” SQLite memory and call analytics.

health_tools.py โ€” PHC lookup and health scheme functionality.

escalation_tools.pyโ€”Human escalation, sanitization, and supervisor notification.

outbound_dialer.py โ€” Outbound calls and CSV campaigns.

test_memory.py โ€” Evaluation and memory-related tests.

๐Ÿ”— Project Repository

Murf LiveKit Starter:

https://github.com/murf-ai/murf-livekit-starter

The project uses:

  • Murf Falcon TTS
  • LiveKit Agents
  • Deepgram Nova-3
  • Google Gemini
  • Python
  • Next.js
  • SQLite

๐ŸŽค Final Thoughts

Building Aarogya Mitra in 10 days taught me that creating a voice agent isn't simply about making an AI talk.

It's about designing an entire conversation system.

The agent needs to know:

  • When to speak
  • When to listen
  • When to use a tool
  • When to ask permission
  • When to hand off to a human
  • When to stop
  • And, most importantly, when not to answer

I'm excited to continue exploring Voice AI, Agentic AI, multilingual systems, and AI for social impact.

If you're building a voice agent too, I'd love to see what you're creating.

Built as part of the 10 Days of Voice Agents โ€” #VoiceForBharat Edition. ๐Ÿ‡ฎ๐Ÿ‡ณ๐ŸŽ™๏ธ

Let's build useful AI.

Top comments (0)