How I Built Care Voice in 10 Days: A Journey into Voice AI

It Started With a Simple Voice Conversation

When I started the 10 Days of Voice Agents: VoiceForBharat Edition, my goal was simple.
I wanted to build something that could listen to a person, understand what they said, and respond naturally through voice.

I chose the Health Access track and built Care Voice, a healthcare focused voice agent.
The idea was not to build something that replaces doctors or makes medical decisions. Instead, I wanted to explore how voice could make basic healthcare interactions more natural, while giving the agent clear boundaries for situations where human support is needed.

What started as a simple conversation gradually became a system that could remember users, use tools, make phone calls, create escalation requests, track call outcomes, and hand conversations to a specialist.
That transformation is what these ten days were really about.

Care Voice landing page, introducing the voice-based healthcare assistant and its core capabilities.

What Makes Care Voice More Than a Chatbot?

A chatbot mainly gives you an answer.
I wanted Care Voice to be able to do something after understanding the user.

Over the challenge, I added:

• Memory for relevant user context
• Symptom triage and safety guardrails
• Consent based healthcare escalation
• Outbound medicine reminder calls
• Call outcome tracking and analytics
• A Clinic and Appointment Specialist
• Agent handoff for appointment related conversations

Each feature solved a different problem.

• Memory helped the agent maintain context.
• Tools allowed it to perform actions instead of only generating responses.
• Escalation gave it a way to involve human support.
• Analytics helped me understand whether a conversation actually achieved its goal.
• And specialist handoff taught the agent that it does not need to handle everything itself.
• LiveKit handles the real time voice communication between the user and the agent.

The main pieces of my system were:

Deepgram: speech recognition
Gemini: understanding the conversation and deciding what the agent should do
Murf Falcon: the fastest TTS API, powering Care Voice's spoken responses.
LiveKit: real time voice communication and telephony
Python + FastAPI: backend workflows and APIs
Next.js: frontend interface

The important thing is that these components are not separate features sitting around independently. They form one conversation loop.

Care Voice voice pipeline, from the user's speech to an AI-generated spoken response.

The Features That Changed the Project

1. Memory
A conversation becomes much more useful when the agent does not treat every interaction as completely new.
I added memory so Care Voice could retain relevant context for returning users.
This also taught me an important design lesson: memory should be intentional. An agent should not blindly store everything a user says.
For a healthcare system especially, deciding what should and should not be remembered matters.

2. Symptom Triage and Escalation
Healthcare was also where I had to think seriously about boundaries.
Care Voice can perform basic symptom triage, but it should not pretend to be a doctor.
When a conversation indicates that human attention may be required, the agent can ask for the user's consent and create an escalation request with a unique reference ID.

This creates a simple but important fallback:
AI can help
↓
Situation needs human attention
↓
Ask for consent
↓
Create escalation request
↓
Human follow-up

The goal is not for AI to replace human support.The goal is for AI to know when human support is the better next step.

Care Voice Call Analytics Dashboard, showing call outcomes and performance metrics from real voice interactions.

3. Outbound Healthcare Calls
On Day 6, Care Voice moved beyond the browser.
I worked on outbound calling using LiveKit Telephony, allowing the agent to make real phone calls for healthcare reminder use cases such as medication reminders.
This was one of the points where the project stopped feeling like a normal web application.

A text response can be delayed by a second and nobody cares.
During a phone conversation, timing, audio quality, connection stability, and call termination suddenly become very important.
That was a completely different debugging experience.
Making the Agent Measurable

By Day 8, I had another question.
How do I know whether a call was actually successful?
A call ending does not necessarily mean the user's goal was achieved.

So I built a Call Analytics Dashboard.
It tracks:

• Total calls
• Successful calls
• Failed calls
• Success rate
• Privacy safe call history

The dashboard uses real call data from the agent instead of hardcoded numbers.
This changed how I evaluated the project.
I was no longer asking only:
"Can my agent talk?"
I was asking:
"Did the conversation actually accomplish what it was supposed to?"

Call Analytics Dashboard

One Agent Does Not Need to Do Everything

For Day 9, I built a Clinic and Appointment Specialist.
Suppose a user says:
"I want to book an appointment with a doctor."
Instead of making the main healthcare agent handle everything, it can hand the conversation to a specialist.
User
↓
Care Voice
↓
Appointment request detected
↓
Clinic & Appointment Specialist
↓
Find clinic / schedule appointment

The important part is that the user does not need to restart the conversation.
This was my introduction to a bigger idea behind agent systems:
A good agent does not have to know everything. It needs to know what it can handle and when another specialist is better suited for the task.

What Did Not Go Smoothly

This challenge definitely did not go from Day 1 to Day 10 in a straight line.
I dealt with LiveKit connection issues, disconnected sessions, telephony problems, frontend and backend coordination issues, and agent changes that sometimes broke something that had already been working.

There were plenty of moments where debugging meant reading a wall of terminal logs and trying to figure out which small piece had decided to stop cooperating.

But that was probably the most useful part.
I learned that building a voice agent is not just about choosing an LLM.
You are connecting speech recognition, reasoning, text to speech, real time communication, frontend code, backend services, tools, and data.

If one important piece fails, the user does not care which component caused it.
They just experience a broken conversation.
That is where the real engineering begins.

How You Can Build Your Own Voice Agent

You do not need to start with a healthcare agent.
The same basic architecture can work for education, customer support, productivity, local commerce, or many other use cases.

1. Start with the voice pipeline

You need four basic pieces:
Speech-to-Text
↓
LLM
↓
Text-to-Speech
↓
Real-Time Voice Transport
For my project, I used Deepgram + Gemini + Murf Falcon + LiveKit.

2. Give the agent one clear purpose

Do not start with:
"Build an AI that can do everything."
Start with:
"Build an agent that helps users book appointments."
A clear objective makes everything else easier.

3. Add tools

Once conversation works, give the agent useful actions.
For example:
find_clinic()
book_appointment()
create_escalation()
The exact tools depend on your use case.

4. Add memory carefully

Store only information that is actually useful.
For sensitive applications, think carefully about consent, privacy, retention, and deletion before adding memory.

5. Add guardrails

Define what your agent can do and cannot do.
For healthcare, this is especially important.

6. Add a fallback

Ask yourself:
"What happens when my agent cannot safely handle something?"
That answer might be a human escalation, a specialist agent, or simply asking the user to seek appropriate professional help.

Running the Project

• The general setup looks like this:
• > git clone
• > cd
• Install the backend dependencies and configure your environment variables.
• Your .env should contain your API credentials for services such as LiveKit, Murf, Deepgram, and Gemini.
• Never commit .env files or API keys to GitHub.
• Then start the backend and frontend using the commands provided in the repository.

For my project, the backend runs the LiveKit agent and the frontend provides the browser based voice interface.

Public Care Voice GitHub repository containing the backend, frontend, call analytics, escalation workflow, and project documentation.

What I Learned in 10 Days

The biggest lesson was that voice AI is not just about making an AI talk.
It is about designing what happens around the conversation.

I learned to think about:

• What should the agent remember?
• What actions should it be allowed to take?
• When should it ask for consent?
• What happens when something goes wrong?
• When should a human take over?
• How do we measure whether the interaction was successful?
• When should another specialist handle the conversation?

Those questions turned a simple voice demo into an actual agent system.

What's Next for Care Voice?

• There is still a lot I would improve.
• I would like to add stronger clinic and appointment integrations, expand Indian language support, improve healthcare information retrieval, strengthen privacy and security, and make the system more robust for real world use.
• Ten days was enough to build a foundation.
• It was not enough to build a finished healthcare product.
• And that is okay.
• The challenge was never just about finishing ten features.
• It was about understanding what it actually takes to build an AI agent.

Final Thoughts

I started these ten days wanting to build a voice assistant.
I ended up building something that could listen, remember, act, call, escalate, measure, and hand off.

The most important thing I learned was simple:
An intelligent agent is not one that tries to do everything. It is one that knows what to do next.
That is what Care Voice became for me.
A small experiment in voice AI that taught me a much bigger lesson about building intelligent systems.

Built with: Murf Falcon, Deepgram, Gemini, LiveKit, Python, FastAPI, and Next.js.

GitHub: Care Voice Repository

Also Published On: https://theprettydev.medium.com/how-i-built-care-voice-in-10-days-a-journey-into-voice-ai-817f31ff2713