๐ Multilingual from Day One
Aarogya Sahayak currently supports English ๐ฌ๐ง, Hindi ๐ฎ๐ณ and Hinglish ๐ฃ๏ธ, making voice interaction more natural and accessible.
๐ฉบ Meet Aarogya Sahayak: ๐
Over the last 10 days, I built Aarogya Sahayak, a voice-first health-access assistant designed to help users get basic health-related guidance, understand their situation, find the right next step, and connect with human help when required.
The goal is simple:
Make basic health access more natural through voice.
Aarogya Sahayak can communicate through voice, use AI reasoning, remember relevant context, use tools, connect through SIP, and escalate conversations to human specialists when needed.
It is designed as an assistant, not a replacement for a doctor.
๐ฏ Why Did I Choose the Health Track?
When I started, I was considering three tracks:
๐ฐ Finance
๐พ Farming
๐ฉบ Health
I chose Health because:
Health is Wealth. โค๏ธ
Health affects everyone, and people don't always need a doctor immediately for every basic question.
I wanted to build something where voice AI could provide basic guidance, useful information and the right next step from home.
That's what made the Health track the most meaningful choice for me.
๐ My 10-Day Journey:
๐ข Day 1 โ Building the Voice Foundation ๐๏ธ
I started with the basic voice pipeline and connected the core components required for a real-time conversational voice agent.
The goal was simple: make the agent talk.
๐ก Day 2 โ Giving the Agent a Personality ๐ง
I defined the agent's role, personality and guardrails so that Aarogya Sahayak would stay within its intended health-assistance scope.
๐ต Day 3 โ Personalising the Health Frontend ๐ป
I personalised the frontend for the Health track and added clear agent states:
๐ข Ready
๐ Connecting
๐ง Listening
๐ฃ๏ธ Speaking
๐ด Call Ended
This made the experience feel more like a real product.
๐ฃ Day 4 โ Adding Memory ๐ง ๐พ
I introduced persistent memory so the assistant could retain relevant conversational context instead of treating every interaction as completely new.
- Used SQLite to store and persist user-related information.
- Connected the stored data with the voice agent for more personalized conversations.
- This helped the assistant reuse relevant information across interactions and reduce repetitive questions.
### ๐ Day 5 โ Giving the Agent Tools ๐ ๏ธ
I added tools/function calls so the agent could perform useful actions instead of only generating responses.
One of the main tools was a government health-facility lookup, using government data with a local fallback dataset.
The tool covers 20 Indian states/UTs and helps users find relevant government health facilities based on their location.
The architecture moved from:
User โ Question โ AI Answer
to:
User โ Agent โ Understand โ Decide โ Tool โ Result โ User
๐ด Day 6 โ Connecting Voice Calls with SIP ๐
I connected the agent with SIP to move beyond browser-based voice interaction and work with telephony.
I used Linphone to test the SIP call flow.
๐ค Day 7 โ Human Escalation ๐ค
I added a human escalation flow so that when a situation requires human assistance, the AI can hand the conversation over instead of continuing beyond its limits.
๐ Day 8 โ Call Analytics Dashboard
I started building a Call Analytics Dashboard to monitor calls and understand their outcomes.
The dashboard helped me track what was happening inside the voice system instead of simply looking at individual conversations.
๐งโโ๏ธ Day 9 โ Specialist Handoffs
I focused on specialist handoffs so that conversations requiring additional human support could move toward the appropriate specialist flow.
๐ Day 10 โ Final Blog & Sharing the Journey
Day 10 was about documenting the complete journey and turning everything I learned during these 10 days into a final blog post.
The goal was not just to share what I built, but also to explain the process, challenges and learnings so that someone else can get inspired and start building their own voice agent.
By the end, the vision was to bring all the individual capabilities together โ voice, AI reasoning, memory, tools, SIP, escalation, specialist handoffs and analytics โ into one complete voice-first health-access experience: Aarogya Sahayak.
๐๏ธ Technology Stack โ What Each Part Does
| Technology | What It Does |
|---|---|
| โก LiveKit | Handles real-time audio communication between the user and the voice agent. |
| ๐ค LiveKit Agents | Builds and orchestrates voice-agent logic, sessions, tools and interactions. |
| ๐๏ธ Deepgram | Speech-to-Text (STT) โ converts user speech into text. |
| ๐ง Gemini | LLM/reasoning layer that understands requests and decides how the agent should respond or act. |
| ๐ Murf Falcon | Text-to-Speech (TTS) โ converts responses into natural voice. It is also highlighted as a fast TTS API in the challenge. |
| ๐ SIP | Connects the voice agent with telephony infrastructure. |
| ๐ฑ Linphone | Used for testing SIP calls and telephony flow. |
| ๐ง Memory | Stores relevant conversation context for more personalised interactions. |
| ๐ ๏ธ Tools / Function Calling | Allows the agent to perform actions, not just generate responses. |
| ๐ค Escalation / Handoff | Transfers conversations to human support or specialists when required. |
| ๐ Analytics Dashboard | Monitors calls, outcomes and system performance. |
| ๐ป Next.js / React | Used to build the frontend interface for the application. |
๐ Complete Architecture
At a high level:
๐ค User
โ
๐๏ธ Voice Input
โ
โก LiveKit
โ
๐ Deepgram โ STT
โ
๐ง Gemini โ Reasoning
โ
๐ ๏ธ Tools + Memory + Escalation
โ
๐ Murf Falcon โ TTS
โ
๐ง Voice Response
For telephony:
๐ SIP / Linphone โ LiveKit โ Agent โ Gemini โ Murf Falcon โ Voice Response
๐จโ๐ป How Can Someone Build Something Similar?
โ๏ธ Quick Setup
Don't try to build everything at once.
Start with the smallest voice pipeline:
Voice Input โ STT โ LLM โ TTS โ Voice Output
Then add memory, tools, SIP and escalation one by one.
๐ Environment Variables
All API keys and sensitive configuration are stored locally in .env.local files and are never hard-coded into the source code or exposed publicly.
For example:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
MURF_API_KEY=your_murf_api_key
DATA_GOV_IN_API_KEY=your_data_gov_api_key
Make sure .env.local is included in .gitignore before pushing the project to GitHub.
๐ Run the Project
Backend:
cd backend
uv sync
uv run python src/agent.py dev
Frontend:
cd frontend
pnpm install
pnpm dev
Then open:
http://localhost:3000
Click Start talking, allow microphone access, and ask a question to verify that the agent responds with voice.
The basic flow is:
๐๏ธ Voice โ โก LiveKit โ ๐ Deepgram โ ๐ง Gemini โ ๐ Murf Falcon โ ๐ง Voice
๐ Test Outbound SIP
cd backend
uv run python src/outbound_call.py
SIP flow:
๐ SIP / Linphone โ โก LiveKit โ ๐ค Agent โ ๐ง Gemini โ ๐ Murf Falcon โ ๐ง Voice Response
๐งช Run Tests
cd backend
uv run pytest
๐ป Start with a Simple Agent
A simple agent can start like this:
class HealthAssistant(Agent):
def __init__(self):
super().__init__(
instructions="You are Aarogya Sahayak, a helpful health-access assistant."
)
A simple escalation tool:
@function_tool
async def escalate_to_human(reason: str):
return f"Escalating because: {reason}"
And the basic voice pipeline:
session = AgentSession(
llm=gemini,
stt=deepgram,
tts=murf_falcon
)
The important part is understanding what each layer is responsible for.
Don't start with:
"How do I build a huge AI health platform?"
Start with:
"How do I make one reliable voice conversation work?"
Then build from there. ๐
A simple agent can start like this:
python
class HealthAssistant(Agent):
def __init__(self):
super().__init__(
instructions="You are Aarogya Sahayak, a helpful health-access assistant."
)
A simple escalation tool:
@function_tool
async def escalate_to_human(reason: str):
return f"Escalating because: {reason}"
And the basic voice pipeline:
session = AgentSession(
llm=gemini,
stt=deepgram,
tts=murf_falcon
)
The important part is understanding what each layer is responsible for.
Don't start with:
"How do I build a huge AI health platform?"
Start with:
"How do I make one reliable voice conversation work?"
Then build from there. ๐
๐ Challenges I Faced
๐ฅ 1. Understanding Where to Start
There were many APIs, services and architectural decisions.
The biggest challenge wasn't writing code โ it was understanding how all the pieces should fit together.
Breaking the project into smaller layers made it much easier.
๐ 2. SIP Connected but No Voice Came Back
The SIP call was connecting, but the expected voice response wasn't coming back.
I had to debug the SIP configuration and understand how audio was moving between the telephony layer and the agent.
Getting the complete call flow working was one of the most satisfying parts of the project.
๐ 3. Dashboard Showing Incorrect Call Status
The calls were actually successful, but the dashboard was incorrectly categorising some of them as failed.
The issue was in how the call data was being processed and mapped.
This taught me:
Building the feature is only half the work. Making the data flow correctly is equally important.
๐ฎ What's Next for Aarogya Sahayak?
The 10-day challenge gave me a strong foundation, but there is still a lot I want to build.
๐ 1. Advanced Dashboard
More detailed call trends, response-time metrics, escalation statistics, language analytics and call insights.
๐ฑ 2. Dedicated Mobile App
Make Aarogya Sahayak accessible through a simple mobile experience.
๐ 3. More Indian Languages
Expand beyond Hindi, English and Hinglish to languages such as Marathi, Bengali, Tamil, Telugu, Gujarati, Kannada and Malayalam.
๐ง 4. Better Personalisation
Improve memory and contextual interactions while maintaining safety and privacy.
๐ค 5. Smarter Specialist Routing
Make specialist handoffs more intelligent and route conversations toward the most relevant human support.
โค๏ธ What I Learned
The biggest lesson was learning to think about an AI agent as a complete system, not just an LLM.
I started by thinking about:
LiveKit + Gemini + Deepgram + Murf + SIP
But by the end, I was thinking about:
User โ Conversation โ Intelligence โ Action โ Escalation โ Analytics
A useful agent needs more than AI:
๐ง Intelligence
๐๏ธ Voice
๐ ๏ธ Tools
๐พ Memory
๐ Communication
๐ค Human Handoff
๐ Observability
๐จ Good UX
๐ Final Thoughts
Aarogya Sahayak started as an idea and became a working voice-first health-access assistant in 10 days.
There were broken calls, missing audio, incorrect dashboard data and plenty of debugging.
But every problem helped me understand the system better.
This project taught me that you don't need to know everything before you start.
Build โ Break โ Debug โ Learn โ Improve.
This is Aarogya Sahayak โ my attempt to make basic health access more simple, natural and accessible through voice. ๐ฉบ๐๏ธ๐ฎ๐ณ
And this is only the beginning.
๐ Built as Part of 10 Days of Voice Agents
A huge thank you to everyone behind the 10 Days of Voice Agents challenge for creating an environment where we could build, break, debug and ship every day.
Special thanks to Murf AI for making voice interaction faster and more natural with the Murf Falcon API. ๐โก
๐ GitHub: [ https://github.com/Abhishek-M-2025/murf-livekit-starter ]
Top comments (0)