Aarogyam: Building a Voice-First AI Healthcare Companion

The Problem: Healthcare Needs More Than a Chatbot

Healthcare questions are often difficult to communicate through text alone. A user may not know the right medical terminology, may be more comfortable speaking than typing, or may simply need guidance in a familiar language.

That led me to a simple question:

What if accessing basic healthcare assistance felt more like having a conversation than filling out a form?

This was the idea behind Aarogyam, my voice-first AI healthcare companion built during the 10 Days of Voice Agents — VoiceForBharat Edition.

The goal was not to build an AI that pretends to be a doctor. Instead, I wanted to build an assistant that could provide useful first-level guidance, access relevant healthcare information, remember returning users, and—most importantly—recognise when a situation should be handled by a human.

Voice became an important part of that experience because it makes interaction more natural and accessible. Aarogyam can handle multilingual conversations so users can communicate in the language they are most comfortable with, including code-mixed conversations where supported.

The project gradually grew from a voice conversation into a complete system with memory, real-world tools, human escalation, outbound calling, call analytics, and specialist-agent handoffs.

And that evolution became the most interesting part of the project.

Meet Aarogyam

Aarogyam is designed as a voice-first healthcare companion that sits between a user's everyday health questions and the services or human support they may need next.

Instead of treating every conversation as a simple question-and-answer exchange, Aarogyam can decide which capability is appropriate for the situation.

A user can:

• Ask general health and wellness questions through voice.
• Communicate in their preferred supported language.
• Get information about nearby healthcare facilities.
• Return to the system later and benefit from remembered preferences.
• Request human healthcare assistance when the situation requires it.
• Continue a conversation with a specialised clinic and appointment agent.
• Use voice interactions through browser-based sessions and outbound calling.
• View anonymised call outcomes through the analytics dashboard.

The system is deliberately designed with boundaries. Aarogyam is not intended to replace a doctor or make unsupported medical decisions. When a request falls outside its role, it can guide the user toward an appropriate human or specialist instead of attempting to answer everything itself.

That distinction became one of the central design principles of the project:

The goal is not to make one AI agent capable of doing everything. The goal is to build a system that knows which capability—or which person—should handle the next step.

Why Voice?

Voice changes the interaction model completely.

Instead of navigating menus, typing long descriptions, or knowing exactly what information to search for, users can simply explain what they need conversationally.

Behind that conversation is a pipeline connecting speech recognition, an LLM, tools, memory, specialist agents, and text-to-speech into a real-time interaction.

And that brings us to how Aarogyam actually works.

How Aarogyam Works

Aarogyam is built as a real-time voice pipeline rather than a traditional text chatbot.

At a high level, the conversation moves through these components:

User speaks
    ↓
LiveKit
    ↓
Speech-to-Text
    ↓
Aarogyam Main Agent
    ↓
Gemini LLM
    ↓
Memory / Healthcare Tools / Specialist Handoff
    ↓
Murf Falcon TTS
    ↓
User hears the response

LiveKit handles the real-time communication layer, while the speech-to-text system converts the user's voice into text that the agent can understand. The LLM then determines how to respond or whether another capability is required.

When the agent needs additional information, it can use tools instead of relying on assumptions. For example, healthcare-related requests can trigger a facility lookup, while a request that requires human attention can create an escalation after obtaining the user's consent.

The response is then converted back into speech using Murf Falcon, allowing the entire interaction to remain conversational rather than forcing the user back into a text-based interface.

For speech generation, I used Murf Falcon—the fastest TTS API featured in the challenge—to keep Aarogyam's responses natural and responsive.

This architecture also made it possible to add capabilities incrementally without turning the main agent into one giant piece of logic.

The system eventually evolved into four layers:

• Conversation layer — voice interaction, multilingual support and the main agent.
• Capability layer — memory, healthcare lookup and other tools.
• Safety layer — consent-based human escalation and boundaries around what the AI should handle.
• Orchestration layer — outbound calling, analytics and specialist-agent handoffs.

This separation became increasingly important as Aarogyam grew. Every new feature didn't need to become another responsibility of the main agent.

Technology Stack

Layer Technology
Real-time voice LiveKit
Speech-to-Text Deepgram
LLM Google Gemini
Text-to-Speech Murf Falcon
Backend Python
Frontend Next.js / React
Data & memory SQLite
Voice communication Browser + SIP/telephony
Specialist orchestration LiveKit Agent Handoffs

The Features That Made Aarogyam More Than a Voice Bot

The first version of Aarogyam was simple: listen, process, respond.

But as I kept building, I realised that a useful voice agent needs much more than a good conversation loop. It needs context, tools, boundaries, and a way to recover when it reaches the edge of its capabilities.

🧠 Memory for Returning Users

Aarogyam can retain useful user context so returning conversations don't always have to start from zero.

The goal isn't to remember everything. It is to retain useful, non-sensitive context that can make future interactions more natural while avoiding unnecessary exposure of private information.

🏥 Real-World Healthcare Tools

Instead of asking the LLM to simply generate an answer, Aarogyam can use external tools when information needs to come from a real source.

For example, when a user asks about healthcare facilities, the agent can perform a healthcare lookup rather than inventing a clinic or hospital.

This became an important lesson in building AI systems:

When the answer depends on real-world data, the model shouldn't have to guess.

🤝 Knowing When to Ask a Human

One of the most important features I built was human escalation.

Aarogyam can recognise situations where continuing to answer independently is not appropriate. Before creating a human-support request, it explains what information it wants to share and asks the user for permission.

If permission is given, it creates a concise support request containing useful context, urgency and a reference ID—without unnecessarily sending sensitive information or the entire conversation.

If the user declines, no request is created.

This turned escalation from a hidden backend action into an explicit part of the user experience.

📞 From Browser Conversations to Outbound Calls

The project also explored voice interactions beyond the browser.

With outbound telephony support, Aarogyam can initiate voice calls through the configured telephony/SIP infrastructure.

This was one of the more challenging parts of the build because browser-based voice and phone-based voice introduce very different infrastructure requirements.

It also taught me an important engineering lesson:

A feature can be technically correct and still depend heavily on external provider limitations.

📊 Making the Agent Accountable

Aarogyam's analytics dashboard records outcomes from real conversations instead of displaying hardcoded demo numbers.

It tracks:

• Total calls
• Successful calls
• Failed calls
• Success rate
• Language
• Channel
• Recent conversation metadata

The idea was simple: if the agent is supposed to accomplish something useful, I should be able to measure whether it actually did.

For example, during testing, a completed browser conversation appeared as:

1 Total Call → 1 Successful Call → 0 Failed Calls

That small number represented something important: the dashboard was connected to an actual conversation rather than a mock dataset.

👩‍⚕️ One Agent Doesn't Need to Do Everything

The final major addition was specialist-agent handoff.

The main Aarogyam agent handles general healthcare conversations, while a dedicated Clinic & Appointment Specialist focuses on healthcare facilities and appointment-related assistance.

When a user asks for something within the specialist's scope, Aarogyam announces the handoff and transfers the conversation without making the user repeat their request.

The specialist receives the relevant context and can continue from there.

This changed how I thought about multi-agent systems:

Specialisation isn't about adding more agents. It's about giving each agent a clear reason to exist.

What Didn't Go According to Plan

Building Aarogyam over ten days was not a straight line from idea to finished product. Some of the most useful lessons came from things that didn't work the first time.

SIP Was More Complicated Than Expected

Adding outbound calling was one of the most challenging parts of building Aarogyam.

I had to work through the SIP setup from the beginning—understanding how the trunk, outbound configuration, authentication details, destination numbers, and agent connection needed to fit together.

The biggest challenge was getting the Twilio SIP setup working correctly. Finding the right configuration, creating the SIP connection, dealing with the account limitations, and understanding which settings were actually required took considerably more time than I expected.

What made it more difficult was that the browser-based voice agent was already working, so the telephony layer introduced an entirely different set of infrastructure and configuration problems.

Instead of treating this as a reason to abandon the feature, I separated the telephony layer from the core agent so the browser-based experience could continue working independently.

This taught me an important lesson:

Integrating a voice agent is not only about the AI model. The surrounding communication infrastructure can be just as challenging as the AI itself.

The Voice UI Wasn't Right the First Time

Another unexpected problem was purely frontend.

The voice experience initially opened as a large standalone interface and then conflicted with the dashboard flow. It looked functional, but it didn't feel like a coherent product.

I reworked the experience into a dedicated flow:

Dashboard
↓
Talk to Aarogyam
↓
Introduction / Confirmation
↓
"Yes, I want to talk"
↓
Active Voice Session

This also made it possible to separate starting a conversation from actually connecting to the voice session.

Specialist Handoffs Needed More Than a Transfer

The first handoff implementation worked, but testing exposed another problem: a specialist could hand the conversation back to the main agent, only for the same topic to trigger another handoff.

The solution wasn't simply to add more routing rules. I had to think about conversation ownership and context—which agent should remain responsible for the current topic and when a hand-back actually makes sense.

This reinforced something I hadn't expected when I started:

Multi-agent systems are less about moving a conversation between agents and more about deciding who should own the conversation at each moment.

Real Data Is Messier Than Demo Data

Healthcare lookup also taught me not to treat external data as guaranteed.

Sometimes a facility could not be found, location names could be ambiguous, or information such as ratings and reviews simply wasn't available.

Instead of filling those gaps with generated information, Aarogyam was designed to acknowledge when it couldn't verify something.

For a healthcare system, “I don't have verified information” is a much better answer than a convincing wrong answer.

What I Learned From Building a Voice Agent

The biggest lesson from this challenge was that voice AI is not just an LLM that happens to speak.

A text chatbot can often get away with thinking in terms of messages and responses. A voice agent has to deal with timing, interruptions, session state, speech recognition, audio output, tool execution, routing, and what happens when something fails.

A few lessons stood out to me.

1. Good voice UX starts before the first sentence

The user experience isn't only about how natural the generated voice sounds. Connection states, listening states, interruptions, loading states, and ending a conversation all matter.

A technically working agent can still feel frustrating if the user doesn't know whether it heard them or is still processing.

2. Tools should extend the model, not replace its judgment

Giving an agent access to tools doesn't automatically make it reliable.

The agent still needs clear instructions about when a tool should be used, what information it can trust, and what it should do when the tool doesn't have an answer.

3. Safety needs to be part of the architecture

For healthcare, safety cannot be something added at the end.

Human escalation, consent, limitations, and avoiding unsupported information became part of Aarogyam's actual conversation flow rather than just documentation around it.

4. More agents don't necessarily mean a better system

The specialist-agent experiment taught me that delegation works best when responsibilities are clearly separated.

A focused specialist can be more useful than continuously expanding one general-purpose agent.

5. Build, test, break, repeat

Perhaps the most practical lesson was the simplest one:

The system I imagined at the beginning was not the system I ended up building.

Real testing exposed UI problems, routing loops, external-service limitations, incomplete data, and edge cases that were impossible to see from the initial design.

That is probably what I value most about the ten-day process: every working feature came with a better understanding of what the system actually needed.

How You Can Build Your Own Voice Agent

If you want to build something similar, you don't need to start with a complex multi-agent architecture. Start with the smallest working voice loop and add capabilities one at a time.

The basic pipeline is:

Microphone
    ↓
Speech-to-Text
    ↓
LLM
    ↓
Tools / Logic
    ↓
Text-to-Speech
    ↓
Speaker

For real-time communication, a transport layer such as LiveKit connects the user's audio session with the agent.

1. Set Up the Project

Start with a LiveKit Agents project and configure the required environment variables.

Keep API credentials in a local .env file:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_google_key
MURF_API_KEY=your_murf_key

Never commit the real .env file to GitHub.

Instead, provide a safe .env.example containing variable names without actual secrets.

2. Build the First Conversation

The first milestone should be very small:

User speaks → STT → LLM → Murf Falcon → user hears response.

Once that works reliably, add tools rather than trying to build the complete application at once.

3. Add Tools Carefully

For example, a healthcare lookup tool can be exposed to the agent with a clear description explaining:

• When it should be used
• What information it requires
• What it returns
• What it should do when no result is available

This keeps the model from trying to invent information that should come from an external source.

4. Add Memory and Specialised Capabilities

Once the basic agent works, capabilities such as memory, human escalation and specialist handoffs can be added around the core conversation loop.

The important part is keeping each responsibility clear rather than putting every piece of logic into one large agent.

5. Test Before You Add More

For every new capability, test both:

The path where it should activate
and
the path where it should not activate.

For example:

General wellness question
↓
Main agent

Clinic request
↓
Clinic specialist

This simple approach catches routing problems much earlier.

Running Aarogyam

The exact commands depend on the project setup, but the general process is:

install dependencies
configure .env
start the backend/agent
start the frontend
open the browser-based voice session

For Aarogyam specifically, the complete setup instructions and source code are available in the public repository:

GitHub: https://github.com/chetna-29/murf-livekit-starter

If someone wants to reproduce the project, the repository is the best place to follow the current installation and environment configuration rather than copying credentials or configuration from a blog post.

What I Would Build Next

Aarogyam is functional, but I don't consider the 10-day challenge the finish line.

The next version would focus on making the system more useful, measurable, and personalised without making the main agent unnecessarily complicated.

🧑‍⚕️ More Specialist Agents

The current Clinic & Appointment Specialist could become the first of several focused agents.

For example:

• Clinic & Appointment Specialist — facilities and appointments
• Wellness Specialist — everyday health and lifestyle guidance
• Medication Information Specialist — safe, factual medication information with clear limitations

The main agent would act as an orchestrator, handing conversations to the appropriate specialist only when necessary.

🌍 Broader Language Support

I'd like to expand multilingual support further so that users can interact with Aarogyam in more regional languages and switch languages naturally during a conversation.

The goal isn't simply translation. The interaction should feel natural in the language the user actually prefers.

🎙️ A More Expressive Voice Interface

The next version of the frontend would make the AI feel more present during a conversation.

Instead of a static voice indicator, Aarogyam could have a consistent animated AI doctor that changes its expression and state while:

• Listening
• Thinking
• Speaking
• Processing information
• Completing a request

This would make the voice interaction easier to understand visually without distracting from the conversation.

📈 Deeper Observability

The current analytics dashboard tracks basic call outcomes. I'd like to extend it with metrics such as:

• Average call duration
• Success rate over time
• Tool usage
• Handoff frequency
• Failure reasons
• Response latency
• Language distribution

That would make it possible to improve the agent based on evidence rather than intuition.

🔐 Stronger Privacy Controls

Healthcare applications require continuous attention to privacy.

Future versions could add more structured data minimisation, stronger redaction of sensitive information, clearer retention policies, and more granular controls over what is remembered.

The long-term goal is simple:

Make Aarogyam more capable without making it less trustworthy.

Project & Resources

Source Code: GitHub Repository

The repository contains the Aarogyam agent, frontend, backend services, memory, healthcare tools, analytics, and specialist-agent handoff implementation.

Built with: LiveKit, Murf Falcon, Deepgram, Google Gemini, Python, Next.js/React, and SQLite.

Final Thoughts

Ten days ago, Aarogyam was an idea: could a voice interface make basic healthcare assistance feel more natural and accessible?

Ten days later, it became a working system with voice interaction, memory, real-world tools, human escalation, outbound calling, analytics, and specialist-agent handoffs.

But the most valuable part of the challenge wasn't adding features.

It was learning how those features should work together.

Aarogyam doesn't need to know everything. It needs to know what it can reliably handle, when it should use a tool, when another specialist is better suited, and when a human should take over.

That became the principle I kept coming back to while building:

Build AI that knows its capabilities—and its boundaries.

I'm still going to keep improving Aarogyam, but this challenge gave me something more valuable than a finished demo: a much better understanding of what it takes to turn a voice model into an actual product.

10 days. A lot of debugging. A lot of rebuilding. And one voice agent that taught me that sometimes the smartest response is knowing when to hand off.