Building HealthSaathi: A Voice Health Agent with Murf Falcon & LiveKit

Building HealthSaathi AI: A Multilingual Voice Health Agent with Murf Falcon & LiveKit

10 Days of Voice Agents — VoiceForBharat Edition

What if people could simply speak naturally with an AI health assistant instead of typing their symptoms into a chatbot?

During the 10 Days of Voice Agents — VoiceForBharat Edition, I built HealthSaathi AI, a real-time voice health assistant designed around natural conversations in English, Hindi, and Hinglish.

The project combines real-time voice AI, multilingual interaction, safety guardrails, human escalation, and analytics into one application.

🩺 What is HealthSaathi AI?

HealthSaathi AI is a voice-first health assistant that allows users to speak about:

Symptoms
General health questions
Wellness concerns
Follow-up needs
Other non-emergency health-related questions

Instead of forcing users to type, the system allows them to talk naturally with an AI voice agent.

The goal was not to build an AI doctor.

The goal was to explore how voice AI can make health-related digital interactions more accessible and natural, while keeping appropriate safety boundaries and providing a path to human assistance when needed.

⚠️ HealthSaathi is a technical demonstration, not a replacement for professional medical advice or emergency medical services.

🎙️ Why Voice?

Typing isn't always the most natural way to explain a problem.

Voice makes it possible to communicate using the same interaction pattern people already use every day: speaking.

This becomes particularly interesting when supporting:

Hindi-speaking users
Hinglish conversations
Users who are more comfortable speaking than typing
Users who want a conversational interface
Voice-based follow-ups and assistance

For this project, I wanted the experience to feel less like filling out a form and more like having a conversation.

🖥️ The HealthSaathi Interface

The application starts with a simple voice-first interface.

Users can immediately see that the agent supports English, Hindi, and Hinglish, then start a consultation using the microphone.

📸 Screenshot 1 — HealthSaathi Voice Assistant

The interface also provides access to:

Analytics Dashboard
Human Help
Voice consultation
Privacy/security indicators

The focus was intentionally kept simple: start a conversation without unnecessary complexity.

🏗️ How the Voice Agent Works

At a high level, the architecture looks like this:

             User
               │
               │ Voice
               ▼
      ┌─────────────────┐
      │    LiveKit      │
      │ Real-time Audio │
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │ Speech-to-Text  │
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │   LLM / Agent   │
      │                 │
      │ Guardrails      │
      │ Memory          │
      │ Tools           │
      │ Escalation      │
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │   Murf Falcon   │
      │  Text-to-Speech │
      └────────┬────────┘
               │
               ▼
          User hears
           response

The core idea is a continuous pipeline:

Speech → STT → Agent → LLM reasoning/tools → Murf Falcon → Speech

LiveKit handles the real-time communication layer while Murf Falcon provides the voice output.

⚡ Murf Falcon for Voice Generation

One of the important parts of the project was using Murf Falcon for text-to-speech.

The goal was to make the assistant's responses feel natural enough for a real conversation rather than sounding like traditional robotic TTS.

The project uses the voice agent stack to connect:

Real-time audio
Agent logic
LLM responses
Murf Falcon TTS

This made voice the primary interaction method instead of simply adding voice as an extra feature.

🌐 English, Hindi & Hinglish

India has a huge variety of languages and communication styles.

For HealthSaathi, I specifically designed the experience around:

English

Hindi

Hinglish

Code-mixed conversations are particularly interesting because users don't always switch languages in a clean way.

A real conversation can naturally move between Hindi and English.

That makes multilingual voice agents an interesting engineering challenge.

🛡️ Safety Guardrails

Healthcare is a high-stakes domain, so simply making an AI that can answer questions isn't enough.

HealthSaathi was designed with safety boundaries around the conversation.

The agent should avoid behaving like a doctor and should recognize situations where continuing a normal AI conversation isn't appropriate.

One of the important design principles was:

AI should assist the user, not replace professional medical care.

For situations requiring human attention, the system can create an escalation request.

👨‍⚕️ Human Help & Escalation

This became one of the most important features of the project.

Instead of expecting the AI to handle everything, HealthSaathi provides a Human Help Dashboard.

The dashboard can display escalation requests with information such as:

Reference ID
Urgency
Status
Language
Follow-up method
Summary
Creation timestamp

For example:

{
"reference_id": "HS-DEMO-1012",
"urgency": "EMERGENCY",
"language": "Hindi",
"follow_up_method": "call",
"status": "OPEN"
}
📸 Screenshot 2 — Human Help Dashboard

This reflects an important principle when building AI systems:

The best AI system isn't necessarily the one that handles everything itself.

Sometimes the correct behavior is to hand the conversation to a human.

📊 Call Analytics Dashboard

I also built an analytics dashboard to understand how the voice agent performs.

The dashboard tracks information such as:

Total calls
Successful calls
Failed calls
Success rate
Recent consultations
Call duration
Channel
Call outcome

During testing, the dashboard showed:

Total Calls 14
Successful Calls 13
Failed Calls 1
Success Rate 92.9%

🔄 The Interesting Part: Connecting Everything

Building an individual voice component isn't particularly useful by itself.

The real challenge was connecting the different pieces into one workflow:

User speaks
↓
Real-time audio transport
↓
Speech recognition
↓
Voice agent
↓
Safety / guardrails
↓
LLM response
↓
Murf Falcon
↓
Audio response
↓
Analytics
↓
Human escalation when required

Each component has its own failure modes.

The challenge was making them work together while keeping the user experience simple.

🧩 Challenges I Faced

1. Real-time voice is different from a normal chatbot

A text chatbot can tolerate a small delay.

Voice cannot.

When someone is talking to an AI, even small delays can make the conversation feel unnatural.

This meant paying attention to:

Audio streaming
Response latency
TTS generation
Agent state
Connection handling

1. Multilingual conversations

Supporting English is relatively straightforward.

But real users don't necessarily speak in one language.

For example:

"Doctor, mujhe since yesterday headache ho raha hai."

This kind of Hinglish interaction makes voice-agent behavior more interesting because the system has to understand the meaning rather than simply matching a language.

1. Human escalation

Another challenge was deciding what information should be passed to the human side.

The escalation record needs enough context to be useful without exposing unnecessary private information.

This resulted in a structured escalation model containing fields such as:

{
"reference_id": "HS-DEMO-1012",
"urgency": "EMERGENCY",
"language": "Hindi",
"follow_up_method": "call",
"status": "OPEN"
}

For a real deployment, this would require significantly stronger privacy, authentication, auditing, access control, and clinical workflows.

🛠️ Technology Stack

The project is built around:

Python — Voice agent/backend logic
LiveKit — Real-time voice communication
Murf Falcon — Text-to-speech
LLM — Conversation intelligence
Next.js / React — Frontend
TypeScript — Frontend development
Tailwind CSS — UI styling
JSON-based storage — Development/demo data

The architecture is intentionally modular so individual components can evolve independently.

🚀 How to Run the Project

If you want to experiment with the project, the general workflow is:

1. Clone the repository git clone YOUR_GITHUB_REPOSITORY_URL cd your-project
2. Install dependencies

For the frontend:

cd frontend
pnpm install

For the Python agent, install the project's Python dependencies according to the repository instructions.

1. Configure environment variables

Create the appropriate .env files locally.

For example:

MURF_API_KEY=your_key_here
LIVEKIT_API_KEY=your_key_here
LIVEKIT_API_SECRET=your_secret_here
LIVEKIT_URL=your_livekit_url

Never commit these values to GitHub.

Use .env.local, .env, or your deployment platform's secret manager and keep them in .gitignore.

1. Start the frontend cd frontend pnpm dev

Then open:

http://localhost:3000

1. Start the voice agent

Run the Python LiveKit agent using the commands defined in the repository.

Once the agent and frontend are running, start a consultation and test the conversation.

🔐 Security & Privacy

Because this project deals with health-related conversations, security is especially important.

For a public repository:

Never commit API keys
Never publish real phone numbers
Never publish real caller IDs
Never publish private conversation transcripts
Never publish real patient information
Use .env files for secrets
Use demo/anonymized data for screenshots

The screenshots and demo records shown in this article use demo-style data for presentation purposes.

🔮 What I Would Improve Next

There is still a lot I would like to add.

Some of the next improvements would be:

More Indian language support
Better multilingual/code-mixed detection
Persistent user memory
More advanced agent handoffs
Production-grade authentication
Role-based access for human operators
Secure database storage
Better observability and tracing
Real outbound calling workflows
More detailed analytics
Production deployment
Stronger privacy and security controls

The biggest next step would be moving from a development/demo architecture toward a properly secured production architecture.

🎯 What I Learned

The biggest lesson from this challenge wasn't simply how to connect an LLM to a TTS API.

It was learning that a useful voice agent requires much more:

Voice + intelligence + tools + state + safety + observability + human fallback

A voice agent becomes much more interesting when it can do something useful beyond simply answering questions.

🙌 10 Days of Voice Agents — VoiceForBharat Edition

This project was built as part of 10 Days of Voice Agents — VoiceForBharat Edition.

Over the challenge, I explored how to build voice agents that can:

Have natural conversations
Use Indian voices
Work with multilingual interactions
Follow safety guardrails
Remember context
Use tools
Handle calls
Escalate to humans
Track outcomes
Hand conversations to specialized agents

Building HealthSaathi helped me understand how all of these pieces fit together into a real application.

🔗 Project & Resources

GitHub: https://github.com/arpit005c/HealthSaathi

Murf Falcon: Murf AI

LiveKit: LiveKit

Voice AI Quickstart: LiveKit Voice AI

🚀 Final Thoughts

Voice AI is moving beyond simple voice assistants.

The interesting opportunity is building systems that can listen, understand, act, measure, and know when to involve a human.

HealthSaathi is my exploration of that idea.

And this 10-day challenge gave me the opportunity to go from a basic voice agent to a much more complete system.

Built with Python, LiveKit, Murf Falcon, React, and a lot of experimentation.