🩺 Aarogya Sahayak — Building a Voice-First Health Assistant in 10 Days | #VoiceForBharat

Abhishek — Sat, 15 Aug 2026 09:17:31 +0000

🌍 Multilingual from Day One

Aarogya Sahayak currently supports English 🇬🇧, Hindi 🇮🇳 and Hinglish 🗣️, making voice interaction more natural and accessible.

🩺 Meet Aarogya Sahayak: 👇

Over the last 10 days, I built Aarogya Sahayak, a voice-first health-access assistant designed to help users get basic health-related guidance, understand their situation, find the right next step, and connect with human help when required.

The goal is simple:

Make basic health access more natural through voice.

Aarogya Sahayak can communicate through voice, use AI reasoning, remember relevant context, use tools, connect through SIP, and escalate conversations to human specialists when needed.

It is designed as an assistant, not a replacement for a doctor.

🎯 Why Did I Choose the Health Track?

When I started, I was considering three tracks:

💰 Finance
🌾 Farming
🩺 Health

I chose Health because:

Health is Wealth. ❤️

Health affects everyone, and people don't always need a doctor immediately for every basic question.

I wanted to build something where voice AI could provide basic guidance, useful information and the right next step from home.

That's what made the Health track the most meaningful choice for me.

🚀 My 10-Day Journey:

🟢 Day 1 — Building the Voice Foundation 🎙️

I started with the basic voice pipeline and connected the core components required for a real-time conversational voice agent.

The goal was simple: make the agent talk.

🟡 Day 2 — Giving the Agent a Personality 🧠

I defined the agent's role, personality and guardrails so that Aarogya Sahayak would stay within its intended health-assistance scope.

🔵 Day 3 — Personalising the Health Frontend 💻

I personalised the frontend for the Health track and added clear agent states:

🟢 Ready
🔄 Connecting
🎧 Listening
🗣️ Speaking
🔴 Call Ended

This made the experience feel more like a real product.

🟣 Day 4 — Adding Memory 🧠💾

I introduced persistent memory so the assistant could retain relevant conversational context instead of treating every interaction as completely new.

Used SQLite to store and persist user-related information.
Connected the stored data with the voice agent for more personalized conversations.
This helped the assistant reuse relevant information across interactions and reduce repetitive questions.

### 🟠 Day 5 — Giving the Agent Tools 🛠️

I added tools/function calls so the agent could perform useful actions instead of only generating responses.

One of the main tools was a government health-facility lookup, using government data with a local fallback dataset.

The tool covers 20 Indian states/UTs and helps users find relevant government health facilities based on their location.

The architecture moved from:

User → Question → AI Answer

to:

User → Agent → Understand → Decide → Tool → Result → User

🔴 Day 6 — Connecting Voice Calls with SIP 📞

I connected the agent with SIP to move beyond browser-based voice interaction and work with telephony.

I used Linphone to test the SIP call flow.

🟤 Day 7 — Human Escalation 🤝

I added a human escalation flow so that when a situation requires human assistance, the AI can hand the conversation over instead of continuing beyond its limits.

📊 Day 8 — Call Analytics Dashboard

I started building a Call Analytics Dashboard to monitor calls and understand their outcomes.

The dashboard helped me track what was happening inside the voice system instead of simply looking at individual conversations.

🧑‍⚕️ Day 9 — Specialist Handoffs

I focused on specialist handoffs so that conversations requiring additional human support could move toward the appropriate specialist flow.

🚀 Day 10 — Final Blog & Sharing the Journey

Day 10 was about documenting the complete journey and turning everything I learned during these 10 days into a final blog post.

The goal was not just to share what I built, but also to explain the process, challenges and learnings so that someone else can get inspired and start building their own voice agent.

By the end, the vision was to bring all the individual capabilities together — voice, AI reasoning, memory, tools, SIP, escalation, specialist handoffs and analytics — into one complete voice-first health-access experience: Aarogya Sahayak.

🏗️ Technology Stack — What Each Part Does

Technology	What It Does
⚡ LiveKit	Handles real-time audio communication between the user and the voice agent.
🤖 LiveKit Agents	Builds and orchestrates voice-agent logic, sessions, tools and interactions.
🎙️ Deepgram	Speech-to-Text (STT) — converts user speech into text.
🧠 Gemini	LLM/reasoning layer that understands requests and decides how the agent should respond or act.
🔊 Murf Falcon	Text-to-Speech (TTS) — converts responses into natural voice. It is also highlighted as a fast TTS API in the challenge.
📞 SIP	Connects the voice agent with telephony infrastructure.
📱 Linphone	Used for testing SIP calls and telephony flow.
🧠 Memory	Stores relevant conversation context for more personalised interactions.
🛠️ Tools / Function Calling	Allows the agent to perform actions, not just generate responses.
🤝 Escalation / Handoff	Transfers conversations to human support or specialists when required.
📊 Analytics Dashboard	Monitors calls, outcomes and system performance.
💻 Next.js / React	Used to build the frontend interface for the application.

🔄 Complete Architecture

At a high level:

👤 User
↓
🎙️ Voice Input
↓
⚡ LiveKit
↓
📝 Deepgram — STT
↓
🧠 Gemini — Reasoning
↓
🛠️ Tools + Memory + Escalation
↓
🔊 Murf Falcon — TTS
↓
🎧 Voice Response

For telephony:

📞 SIP / Linphone → LiveKit → Agent → Gemini → Murf Falcon → Voice Response

👨‍💻 How Can Someone Build Something Similar?

⚙️ Quick Setup

Don't try to build everything at once.

Start with the smallest voice pipeline:

Voice Input → STT → LLM → TTS → Voice Output

Then add memory, tools, SIP and escalation one by one.

🔐 Environment Variables

All API keys and sensitive configuration are stored locally in .env.local files and are never hard-coded into the source code or exposed publicly.

For example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
MURF_API_KEY=your_murf_api_key
DATA_GOV_IN_API_KEY=your_data_gov_api_key

Make sure .env.local is included in .gitignore before pushing the project to GitHub.

🚀 Run the Project

Backend:

cd backend
uv sync
uv run python src/agent.py dev

Frontend:

cd frontend
pnpm install
pnpm dev

Then open:

http://localhost:3000

Click Start talking, allow microphone access, and ask a question to verify that the agent responds with voice.

The basic flow is:

🎙️ Voice → ⚡ LiveKit → 📝 Deepgram → 🧠 Gemini → 🔊 Murf Falcon → 🎧 Voice

📞 Test Outbound SIP

cd backend
uv run python src/outbound_call.py

SIP flow:

📞 SIP / Linphone → ⚡ LiveKit → 🤖 Agent → 🧠 Gemini → 🔊 Murf Falcon → 🎧 Voice Response

🧪 Run Tests

cd backend
uv run pytest

💻 Start with a Simple Agent

A simple agent can start like this:

class HealthAssistant(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are Aarogya Sahayak, a helpful health-access assistant."
        )

A simple escalation tool:

@function_tool
async def escalate_to_human(reason: str):
    return f"Escalating because: {reason}"

And the basic voice pipeline:

session = AgentSession(
    llm=gemini,
    stt=deepgram,
    tts=murf_falcon
)

The important part is understanding what each layer is responsible for.

Don't start with:

"How do I build a huge AI health platform?"

Start with:

"How do I make one reliable voice conversation work?"

Then build from there. 🚀

A simple agent can start like this:

python
class HealthAssistant(Agent):
    def __init__(self):
        super().__init__(
            instructions="You are Aarogya Sahayak, a helpful health-access assistant."
        )

A simple escalation tool:

@function_tool
async def escalate_to_human(reason: str):
    return f"Escalating because: {reason}"

And the basic voice pipeline:

session = AgentSession(
    llm=gemini,
    stt=deepgram,
    tts=murf_falcon
)

The important part is understanding what each layer is responsible for.

Don't start with:

"How do I build a huge AI health platform?"

Start with:

"How do I make one reliable voice conversation work?"

Then build from there. 🚀

😅 Challenges I Faced

🔥 1. Understanding Where to Start

There were many APIs, services and architectural decisions.

The biggest challenge wasn't writing code — it was understanding how all the pieces should fit together.

Breaking the project into smaller layers made it much easier.

📞 2. SIP Connected but No Voice Came Back

The SIP call was connecting, but the expected voice response wasn't coming back.

I had to debug the SIP configuration and understand how audio was moving between the telephony layer and the agent.

Getting the complete call flow working was one of the most satisfying parts of the project.

📊 3. Dashboard Showing Incorrect Call Status

The calls were actually successful, but the dashboard was incorrectly categorising some of them as failed.

The issue was in how the call data was being processed and mapped.

This taught me:

Building the feature is only half the work. Making the data flow correctly is equally important.

🔮 What's Next for Aarogya Sahayak?

The 10-day challenge gave me a strong foundation, but there is still a lot I want to build.

📊 1. Advanced Dashboard

More detailed call trends, response-time metrics, escalation statistics, language analytics and call insights.

📱 2. Dedicated Mobile App

Make Aarogya Sahayak accessible through a simple mobile experience.

🌍 3. More Indian Languages

Expand beyond Hindi, English and Hinglish to languages such as Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada and Malayalam.

🧠 4. Better Personalisation

Improve memory and contextual interactions while maintaining safety and privacy.

🤝 5. Smarter Specialist Routing

Make specialist handoffs more intelligent and route conversations toward the most relevant human support.

❤️ What I Learned

The biggest lesson was learning to think about an AI agent as a complete system, not just an LLM.

I started by thinking about:

LiveKit + Gemini + Deepgram + Murf + SIP

But by the end, I was thinking about:

User → Conversation → Intelligence → Action → Escalation → Analytics

A useful agent needs more than AI:

🧠 Intelligence
🎙️ Voice
🛠️ Tools
💾 Memory
📞 Communication
🤝 Human Handoff
📊 Observability
🎨 Good UX

🚀 Final Thoughts

Aarogya Sahayak started as an idea and became a working voice-first health-access assistant in 10 days.

There were broken calls, missing audio, incorrect dashboard data and plenty of debugging.

But every problem helped me understand the system better.

This project taught me that you don't need to know everything before you start.

Build → Break → Debug → Learn → Improve.

This is Aarogya Sahayak — my attempt to make basic health access more simple, natural and accessible through voice. 🩺🎙️🇮🇳

And this is only the beginning.

🙌 Built as Part of 10 Days of Voice Agents

A huge thank you to everyone behind the 10 Days of Voice Agents challenge for creating an environment where we could build, break, debug and ship every day.

Special thanks to Murf AI for making voice interaction faster and more natural with the Murf Falcon API. 🔊⚡

🔗 GitHub: [ https://github.com/Abhishek-M-2025/murf-livekit-starter ]

🏷️ Hashtags

10DaysOfVoiceAgents #VoiceForBharat #VoiceAI #AIAgents #GenerativeAI #HealthTech #MurfAI #MurfFalcon #LiveKit #Gemini #Deepgram #SIP #Python #NextJS #React #BuildInPublic #AarogyaSahayak

DEV Community: Abhishek