Kumar-nm

Posted on Aug 15

Building Arogya: A Voice-First Healthcare Agent with Murf Falcon #10DaysofAIVoiceAgents, #MurfFalcon and #VoiceForBharat

#10daysofaivoiceagents #murffalcon #voiceforbharat #10daysofvoiceagents

Building Arogya: A Voice-First Healthcare Agent with Murf Falcon 🩺🎙️

Over the last 10 days, I built Arogya Health Access as part of the 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI.

What started as a basic voice assistant gradually evolved into a healthcare-focused voice agent with multilingual conversations, memory, real-time healthcare tools, outbound calls, human escalation, analytics, and specialist agent handoffs.

The biggest lesson for me was that building a useful voice agent is not just about making it talk.

It is about giving the agent the right context, tools, boundaries, memory, failure handling, and escalation paths.

1. The Problem

Healthcare information is often difficult to access when users have to:

Type everything into an application
Navigate multiple healthcare services
Search for facilities manually
Communicate in a language they are less comfortable typing
Know whether a request should be handled by an AI system or a human

I wanted to explore whether a voice-first healthcare assistant could make some of these interactions simpler.

That led to Arogya Health Access.

Arogya is not designed to replace doctors or provide medical diagnosis.

Instead, it focuses on healthcare access and assistance — helping users find healthcare facilities, interact naturally through voice, receive reminders, and reach human support when necessary.

Voice is especially useful for users who are more comfortable speaking than typing and for conversations involving Indian languages and code-mixed communication.

2. Meet Arogya

Arogya is a real-time voice agent that can:

🎙️ Have natural voice conversations
🌐 Understand multilingual and code-mixed conversations
🧠 Remember information for returning users with consent
🔧 Retrieve live healthcare facility information
📍 Use location information for nearby facility searches
📞 Make outbound healthcare reminder calls
🤝 Escalate sensitive requests to human support
📊 Track support requests through a dashboard
🏥 Hand facility and appointment queries to a specialist agent
🔄 Transfer conversations between agents while preserving context

The project was built incrementally over the 10-day challenge.

3. How the Voice Agent Works

At the core of Arogya is a real-time voice pipeline.

User speaks
    ↓
LiveKit
    ↓
Deepgram Speech-to-Text
    ↓
Arogya Agent + Gemini
    ↓
Tools / Memory / Escalation / Specialist Handoff
    ↓
Murf Falcon Text-to-Speech
    ↓
LiveKit
    ↓
User hears the response

The main components are:

Speech-to-Text

Deepgram converts the user's speech into text so the agent can understand the conversation.

LLM

Google Gemini acts as the reasoning layer for the agent.

It decides how to respond and, when appropriate, whether it should use a tool, escalate the request, or hand the conversation to a specialist.

Text-to-Speech

Murf Falcon converts the agent's response back into natural speech.

This was particularly important because the experience depends heavily on how quickly the user receives audio after speaking.

Real-Time Transport

LiveKit handles the real-time audio communication between the user and the voice agent.

4. What I Built

Rather than keeping Arogya as a simple question-answering voice bot, I gradually added capabilities that made it behave more like a complete voice-agent system.

4.1 Multilingual & Code-Mixed Conversations 🌐

Arogya can detect the user's language and respond in the same language.

I also added instructions to prevent unnecessary language switching or translation when the user is already communicating naturally.

This also allows conversations involving mixed-language speech.

For a voice-first healthcare assistant, this was an important part of making the system more accessible.

4.2 A Custom Voice Interface 🎨

The frontend was designed specifically around the healthcare use case instead of using a generic voice-agent interface.

The interface includes:

Healthcare-focused landing page
Voice conversation interface
Real-time transcripts
Connection states
Active conversation states
Multi-language interface
Visual display of structured tool results

The frontend was also designed to remain extensible so individual UI elements can be enhanced with additional interactions later.

4.3 Memory & Personalisation 🧠

Arogya can remember useful information for returning users.

I implemented:

User identification
Persistent memory
Conversation history
Temporary session memory
Returning-user personalisation
Post-call memory consent
Save / Discard memory controls

An important part of this feature was consent.

The system does not simply persist everything from a conversation. The user needs to explicitly allow the information to be saved.

This made the memory feature more appropriate for a healthcare-oriented application.

4.4 Healthcare Tools 🔧

One of the biggest changes was teaching Arogya to use tools instead of relying entirely on the LLM.

I built an MCP healthcare server that connects Arogya to live healthcare facility data.

The system can retrieve:

Nearby hospitals
Clinics
PHCs
Other healthcare facilities
Distances
Addresses when available
Public/government facility information when available
Google Maps links

The flow is roughly:

User asks for nearby healthcare facilities
                ↓
Arogya determines that a tool is required
                ↓
Location information is used
                ↓
MCP healthcare server
                ↓
OpenStreetMap / Overpass
                ↓
Facility results
                ↓
Results returned to Arogya
                ↓
Voice response + frontend display

I also added fallback handling because external data sources can fail.

This was an important lesson:

A tool-using agent needs failure paths just as much as successful paths.

4.5 Outbound Healthcare Calls 📞

Arogya is not limited to incoming conversations.

I added outbound calling using LiveKit Telephony.

The system supports:

Scheduled healthcare reminders
Dynamic reminder purposes
Two-way conversations
User-controlled call termination
Call opt-out handling
Multilingual speech

The calling flow uses:

Arogya
   ↓
LiveKit Telephony
   ↓
SIP
   ↓
Linphone
   ↓
User Phone

This changed the interaction model from:

"The user calls the assistant."

to:

"The assistant can proactively reach the user."

4.6 Human Escalation 🤝

One of the most important design decisions was teaching Arogya when not to answer.

A healthcare agent should not attempt to diagnose every request.

I therefore added human escalation for situations such as:

Red-flag symptoms
Diagnosis requests
Requests that require human assistance

The escalation flow also asks the user for permission before sharing relevant information.

The system generates:

A unique escalation reference ID
A concise human-readable summary
A support request
A dashboard entry

Sensitive information such as OTPs, PINs, passwords and account numbers is not included in the information shared with human support.

4.7 Human Support Dashboard 📊

The escalation system also needed somewhere for support requests to go.

So I added a Human Support Dashboard.

It allows support requests to move through states such as:

Open
  ↓
In Progress
  ↓
Resolved

This made the escalation feature more than just an API call.

There is now a workflow around the human handoff.

4.8 Specialist Agent Handoff 🏥

As Arogya gained more capabilities, putting everything inside one agent became less practical.

So I introduced a dedicated Clinic & Appointment Specialist Agent.

The specialist focuses on:

Hospitals
Clinics
Doctors
Healthcare facilities
Appointments
Facility comparisons
Healthcare specialties

The main Arogya agent can decide when a request should be handled by the specialist.

The important part is that the user does not need to repeat the entire conversation.

Context can be passed during the handoff.

The specialist can also hand the conversation back to Arogya when the topic changes back to general healthcare.

This gave me a first look at how a multi-agent voice system can be structured.

5. Overall Architecture

The final architecture connects the real-time voice pipeline with memory, external tools, human escalation, telephony, and specialist agents.

                         ┌─────────────────────┐
                         │       User 🎙️       │
                         └──────────┬──────────┘
                                    │
                                    ▼
                         ┌─────────────────────┐
                         │      LiveKit        │
                         │  Real-Time Audio    │
                         └──────────┬──────────┘
                                    │
                                    ▼
                         ┌─────────────────────┐
                         │     Deepgram STT    │
                         └──────────┬──────────┘
                                    │
                                    ▼
                  ┌─────────────────────────────────┐
                  │       Arogya Main Agent         │
                  │          + Gemini LLM            │
                  └───────┬─────────┬─────────┬─────┘
                          │         │         │
              ┌───────────┘         │         └─────────────┐
              ▼                     ▼                       ▼
       ┌────────────┐       ┌──────────────┐      ┌─────────────────┐
       │   Memory   │       │ Healthcare   │      │   Specialist    │
       │   SQLite   │       │    Tools     │      │     Agent       │
       └────────────┘       └──────┬───────┘      └─────────────────┘
                                   │
                                   ▼
                         ┌─────────────────────┐
                         │ MCP Healthcare      │
                         │ Server              │
                         └──────┬───────┬──────┘
                                │       │
                                ▼       ▼
                         OpenStreetMap  Foursquare

                  ┌─────────────────────────────┐
                  │     Human Escalation        │
                  │   Support Dashboard         │
                  └─────────────────────────────┘

                  ┌─────────────────────────────┐
                  │     LiveKit Telephony       │
                  │        SIP / Linphone       │
                  └─────────────────────────────┘

                                    │
                                    ▼
                         ┌─────────────────────┐
                         │     Murf Falcon     │
                         │        TTS          │
                         └──────────┬──────────┘
                                    │
                                    ▼
                         ┌─────────────────────┐
                         │       LiveKit       │
                         └──────────┬──────────┘
                                    │
                                    ▼
                         ┌─────────────────────┐
                         │       User 🎙️       │
                         └─────────────────────┘

The interesting part is that the voice pipeline is only the foundation.

The agent then connects to:

Memory
External tools
Human escalation
Specialist agents
Telephony
Frontend components

That is what turned the project from a basic voice chatbot into a more complete voice-agent system.

6. The Technology Stack

Backend

Python
LiveKit Agents
LiveKit Telephony
Google Gemini
Deepgram
Murf Falcon
MCP
SQLite
Silero

Frontend

Next.js
React
TypeScript
Tailwind CSS
LiveKit Components

Healthcare Data

OpenStreetMap
Overpass API
Nominatim
Foursquare

Communication

LiveKit
SIP
Linphone

7. The Hard Parts

The project definitely did not work perfectly on the first attempt.

Here are some of the problems that taught me the most.

Language Consistency

When multilingual support was added, simply telling the model to "support multiple languages" wasn't enough.

The agent could switch languages unnecessarily or translate when it wasn't needed.

I solved this by explicitly detecting the user's language and updating the agent's instructions to maintain that language throughout the conversation.

External Tool Reliability

Healthcare data should not be hallucinated by an LLM.

That meant connecting the agent to external data sources.

But external APIs can fail.

So I added fallback handling and a secondary Overpass endpoint.

This taught me that agent tools need to be designed with failure scenarios in mind.

Knowing When to Escalate

A general-purpose LLM may try to answer almost anything.

That is not appropriate for healthcare.

I had to introduce explicit boundaries around situations such as diagnosis requests and red-flag symptoms.

Instead of trying to make the AI more confident, the better solution was to make it know when to stop and ask for human help.

Growing Agent Complexity

Initially, it was easy to keep everything inside one agent.

As more tools and capabilities were added, the agent became harder to reason about.

Introducing a specialist agent gave different responsibilities a clearer boundary.

This was one of the biggest architectural lessons from the challenge.

8. How to Run Arogya

The complete source code is available on GitHub:

🔗 Arogya Health Access – GitHub

Prerequisites

You need:

Python 3.10+
Node.js 18+
uv
pnpm
A LiveKit project
Murf API key
Deepgram API key
Google Gemini API key

Clone the Repository

git clone https://github.com/Kumar-nm/murf-livekit-starter.git
cd murf-livekit-starter

Install Backend Dependencies

cd backend
uv sync
uv run python src/agent.py download-files

Install Frontend Dependencies

cd ../frontend
pnpm install

9. Environment Variables

Never commit API keys to GitHub.

Create your environment files from the provided .env.example files.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

Replace the placeholder values with your own credentials.

Do not publish:

API keys
API secrets
SIP credentials
Phone numbers
Caller data
Private healthcare information

10. Run the Agent

Terminal 1 — LiveKit

livekit-server --dev

Terminal 2 — Backend

cd backend
uv run python src/agent.py dev

Terminal 3 — Frontend

cd frontend
pnpm dev

Then open:

http://localhost:3000

Allow microphone access and start a conversation.

A simple first test is:

"Hello Arogya, I need help finding a healthcare facility near me."

You should be able to speak with the agent and see the conversation reflected in the frontend.

11. What I Learned

The biggest lesson from these 10 days is that a voice agent is much more than:

Speech → LLM → Speech

A useful production-oriented agent needs much more:

Speech
  ↓
Understanding
  ↓
Reasoning
  ↓
Memory
  ↓
Tools
  ↓
Guardrails
  ↓
Escalation
  ↓
Specialist Handoff
  ↓
Voice Response

I also learned that:

Latency matters a lot more in voice than in text
Tool failures need graceful handling
Memory should be intentional and consent-aware
Healthcare agents need strong boundaries
Voice and visual interfaces should complement each other
Multi-agent systems can help keep responsibilities manageable
The best agent is not necessarily the one that answers everything

12. What I Would Improve Next

There is still a lot I would like to improve.

Some of the next steps would be:

Expand healthcare data coverage across more regions
Add more Indian languages
Improve real-world latency measurement
Add stronger automated voice-agent evaluation
Improve specialist routing
Add more healthcare facility and appointment integrations
Improve call-quality analytics
Add stronger authentication and authorization
Perform broader safety testing
Improve production observability

The challenge gave me a strong foundation, but there is still a lot of room to turn the prototype into a more production-ready system.

13. The 10-Day Journey

Day 1
Voice Agent Foundation
        ↓
Day 2
Multilingual Voice
        ↓
Day 3
Custom Healthcare UI
        ↓
Day 4
Memory & Personalisation
        ↓
Day 5
Healthcare Tools + MCP
        ↓
Day 6
Outbound Calls
        ↓
Day 7
Human Escalation
        ↓
Day 8
Multilingual + Low Latency
        ↓
Day 9
Specialist Agent + Handoff
        ↓
Day 10
Share the Journey

Each day added another layer to the same system instead of creating a completely separate project.

14. Final Thoughts

Building Arogya over these 10 days changed the way I think about voice agents.

At the beginning, the goal was simply to make a healthcare assistant that could talk.

By the end, the goal had become much bigger:

Can the agent understand the user, remember useful context, use real tools, know its limitations, involve humans when necessary, and delegate work to another agent?

That is where voice agents become much more interesting.

The challenge also gave me the opportunity to work with Murf Falcon, which was an important part of keeping the voice experience responsive.

A huge thanks to the team behind the 10 Days of Voice Agents — VoiceForBharat Edition for the challenge.