I started this challenge with a simple idea: what if accessing basic healthcare information could be as simple as talking to someone?
I didn't want to build another chatbot where a user types a question and gets back a wall of text. I wanted something that felt like a conversation — where someone could just say "Mujhe dermatologist ki appointment book karni hai," or "Mere father ko severe chest pain ho raha hai," and have the system understand the situation, reply in their language, use the right tool, and know when to stop and bring in a human.
That became CareConnect, my AI health-access voice agent, built during 10 Days of Voice Agents – VoiceForBharat Edition. What started as a basic voice loop grew into a system with memory, healthcare tools, outbound calling, human escalation, call analytics, and specialist agent handoffs.
Honestly, the hardest part wasn't making the AI talk. It was making it know what to do when things don't go as expected.
![CareConnect cover image showing the voice assistant interface]
The problem I wanted to solve
Healthcare information is often available but still hard to reach. A user may need to search for a hospital, find the right department, understand a scheme, book an appointment, or figure out what to do next — often in a language they're not fully comfortable typing in.
I wanted CareConnect to collapse that funnel from open → search → read → find → act down to something closer to speak → understand → guide → act.
One boundary shaped almost every decision I made: CareConnect is a healthcare access assistant, not an AI doctor.
Why voice changed the product
Voice removes the step where a user has to figure out which button to press before asking a question — they just speak. CareConnect works across English, Hindi, and Hinglish, and a user can move between them naturally mid-conversation: "Mujhe hospital ka information chahiye," or "Can you help me find a dermatologist?"
The chest-pain example matters most. A generic chatbot might just generate a fluent-sounding answer. A healthcare voice agent has to treat the right action as more important than a confident-sounding response — recognizing an emergency and routing to urgent guidance instead of a normal appointment flow.
How CareConnect works
![CareConnect system architecture diagram]
| Layer | Technology |
|---|---|
| Real-time voice | LiveKit |
| Speech-to-text | Deepgram |
| LLM | Gemini |
| Text-to-speech | Murf Falcon |
| Backend | Python |
| Database | SQLite |
| Frontend | Next.js + TypeScript |
Audio from the browser goes through LiveKit to a backend agent. Deepgram converts speech to text, Gemini handles reasoning and conversation state, and when the agent needs reliable domain data or an action, it calls a tool instead of generating an answer from scratch. Murf Falcon converts the response back into speech and streams it to the user.
That sounds simple written out like this. It wasn't.
The 10-day build
Rather than trying to build everything at once, I treated each day as one architectural upgrade on top of the last.
Day 1 — Make the agent speak. The goal was deliberately narrow: listen, understand, respond. Getting the real-time voice pipeline working became the foundation everything else sat on.
Day 2 — Give it a personality and boundaries. A healthcare agent can't behave like a generic assistant, so I added conversational tone, multilingual support, emergency handling, and a hard rule for diagnosis refusal. The rule sounds obvious — CareConnect should never pretend to be a doctor — but it's much harder to hold when a user asks directly, "Do I have diabetes?" The system has to resist the pull toward a confident-sounding answer it has no business giving.
Day 3 — Build the interface.
![CareConnect frontend screenshot showing the voice call interface]
I wanted CareConnect to feel like a real product, not a raw developer demo, so I built a dedicated frontend around the voice experience — one that made it clear what the agent was doing and later gave me a home for analytics.
Day 4 — Give it memory. I added persistent memory with SQLite, but the goal wasn't to remember everything — it was to remember only what could genuinely improve a future conversation. In a healthcare context, that distinction matters: just because a system can store something doesn't mean it should. Data minimization became a working principle, not an afterthought.
Day 5 — Give the agent tools. This is the day that changed how CareConnect actually worked. Instead of asking the LLM to generate every answer, I added domain-specific tools for symptom triage, facility information, healthcare scheme lookups, medication-related assistance, and general health-access workflows. The lesson: LLMs are good at reasoning, but tools are better for deterministic actions and factual data. If a real data source exists, I'd rather the agent call it than guess.
Day 6 — Make outbound calls.
![CareConnect outbound call feature demonstration]
CareConnect stopped being a browser-only assistant. Outbound calling introduces a different problem entirely: when the user didn't initiate the conversation, the agent needs to immediately establish who's calling, why, and how to opt out. Outbound voice isn't just inbound voice run in reverse — the opening carries far more weight.
Day 7 — Know when to ask a human. This was one of the more important architectural changes. An agent shouldn't try to solve every problem itself, so I built a human escalation flow: detect a potential need for escalation, explain exactly what will be shared, get explicit consent, and only then create an escalation record with a reference ID — storing a short structured summary rather than the full conversation. Human escalation needed to be a controlled system action, not a silent side effect.
Day 8 — Build call analytics.
![CareConnect analytics dashboard showing call metrics]
Once CareConnect could handle real calls, I needed a way to answer "is this actually working?" I built a dashboard tracking total, successful, and failed calls — all generated from real call records in SQLite. A dashboard showing hardcoded numbers isn't useful; I wanted it to reflect what actually happened.
Day 9 — Give it a specialist.
![CareConnect specialist handoff to Clinic & Appointment Specialist]
The final major feature was specialist routing. The main CareConnect agent handles general healthcare access; a dedicated Clinic & Appointment Specialist takes over for appointment requests, rescheduling, cancellations, department selection, and clinic information. When a user says "Mujhe dermatologist ki appointment book karni hai," CareConnect recognizes the intent, announces the handoff, and transfers the relevant context so the user never has to repeat themselves. This is where the project stopped feeling like a single assistant and started feeling like a small multi-agent system.
The safety layer
Healthcare was the reason strong guardrails came first, not last. CareConnect is built to never diagnose, never prescribe, never pretend to be a doctor, and to handle emergencies as a separate path from normal workflows. It escalates when appropriate, but always asks for consent first, and it avoids exposing more personal information than a situation requires.
If a user asks "Do I have diabetes?", the correct response isn't a yes or no — it's an honest explanation that CareConnect can't diagnose, paired with guidance toward the right kind of care. If a user reports a serious emergency symptom, that takes priority over any normal appointment flow. That distinction ended up shaping most of the architecture around it.
The hardest parts
The UI was never the hard part. Getting the different pieces of the system to behave correctly together was.
Voice reliability. Speech-to-text, LLM reasoning, text-to-speech, turn detection, and the LiveKit transport all have to stay in sync. A text app can absorb a small delay without anyone noticing. Voice can't hide it nearly as well.
Outbound calling. Beyond the technical setup, outbound calls demanded a much more intentional opening than I initially expected.
Persistent data. Once memory, escalation records, and analytics all needed the database, it stopped being an afterthought. IDs, persistence, call outcomes, data minimization, and structured records all needed real thought.
Agent handoff. A handoff isn't just relabeling which agent is "active." The specialist needs to know why the user was transferred, what they already asked, what context is relevant, and what it's actually allowed to do. Getting that context transfer right took more iteration than anything else in the build.
Git hygiene across ten days. A small but real lesson: checking git status and git branch before switching, and git status plus git log --oneline -3 before pushing, saved me from accidentally losing or overwriting work more than once.
What I learned
The biggest takeaway from these ten days is that a voice agent is not just an LLM with a microphone. A system that's actually useful needs voice, an LLM, tools, memory, guardrails, human escalation, analytics, and specialist routing — each one solving a different problem. The LLM reasons, tools act reliably, memory gives continuity, guardrails set boundaries, escalation provides a safety net, analytics gives visibility, and specialists provide focused capability. The real engineering challenge is making all of that work together without the user ever feeling the complexity underneath.
Run CareConnect yourself
The project is public on GitHub:
git clone https://github.com/RajeshKumar92828282/murf-livekit-starter.git
cd murf-livekit-starter
Configure your local environment variables:
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key
Keep real credentials in local .env or .env.local files only — never commit API keys, tokens, phone numbers, or private caller information to GitHub. Then follow the repository's setup instructions to start the backend and frontend.
Once it's running, try: "Mujhe dermatologist ki appointment book karni hai." You should see CareConnect recognize the appointment intent, hand off, and continue the conversation through the Clinic & Appointment Specialist without making you repeat yourself.
How I tested it
I tested failure paths as deliberately as I tested success paths:
- Normal conversation — "What are some healthy lifestyle tips?" → the main agent answers directly.
- Appointment request — "Mujhe dermatologist ki appointment book karni hai." → hands off to the specialist.
- Diagnosis request — "Do I have diabetes?" → the agent declines to diagnose.
- Emergency — "Mere father ko severe chest pain ho raha hai." → emergency handling takes priority.
- Human escalation — the agent asks for explicit permission before creating an escalation.
- Analytics — after calls complete, the dashboard reflects the real outcome.
Failure behavior turned out to matter just as much as the happy path.
What I would build next
CareConnect is still a starting point. If I keep building it, I'd focus on real hospital appointment APIs, more Indian languages, stronger healthcare knowledge grounding, production-grade database infrastructure, more specialist agents, deeper hospital and clinic integrations, better analytics, more robust voice failure handling, and improved accessibility for low-bandwidth users. I'd also like specialist routing to get smarter about picking the right specialist from intent alone, without loosening any of the safety boundaries around it.
Final thoughts
Ten days ago, CareConnect was just an idea for a voice agent. Today it can talk, understand, remember, use tools, make calls, ask humans for help, track outcomes, and hand conversations off to specialists.
But the real lesson wasn't learning how to make an AI speak — it was learning how to make an AI understand its limits. A good voice agent doesn't need to answer everything. It needs to know when to answer, when to use a tool, when to ask a question, when to ask for permission, when to hand off, and when to stop.
That's what building CareConnect during 10 Days of Voice Agents – VoiceForBharat Edition, using Murf Falcon for voice, taught me. And this is only the beginning.
Source code: CareConnect GitHub Repository
Built with: LiveKit Agents · Murf Falcon · Deepgram · Gemini · Python · SQLite · Next.js · TypeScript
Challenge: 10 Days of Voice Agents – VoiceForBharat Edition
Top comments (0)