Building JanMitra: A Voice for Better Health Access — My 10-Day VoiceForBharat Journey
What if accessing basic healthcare information was as simple as having a conversation?
That question became the starting point for JanMitra (जनमित्र) — a multilingual healthcare voice assistant I built during the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI.
I didn't want to build another chatbot where users type questions into a box.
I wanted to explore something more natural:
What if people could simply talk to an AI assistant in the language they are comfortable with and get guidance about healthcare access?
Over ten days, JanMitra evolved from a basic voice agent into a complete conversational system with:
- Real-time voice conversations
- Multilingual and code-mixed interaction
- Healthcare safety guardrails
- Consent-based persistent memory
- Healthcare-access tools
- Health-camp information
- Outbound calling
- Human escalation
- Call analytics
- Clinic and appointment specialist handoff
- Conversation-context preservation
And getting there wasn't easy.
I faced silent voice sessions, extremely high response latency, database bottlenecks, interrupted audio, and regression risk every time I added something new.
This article is the story of what I built, what broke, how I fixed it, and what I learned while building a real-time voice AI system.
Meet JanMitra
![JanMitra welcome screen — Aapka Swasthya Saathi]
JanMitra's welcome screen — PM-JAY & Ayushman Bharat guidance, PHC/CHC/hospital discovery, and the "Speak Naturally" language chips (Hindi, Telugu, Tamil, Malayalam, Bengali, Marathi, Kannada).
JanMitra (जनमित्र) represents the idea of a friend or companion for people.
The vision is simple:
Aapka Swasthya Saathi — Your Healthcare Companion
JanMitra is designed to help users access general healthcare information and healthcare services through natural voice conversations.
It is not a doctor and is not designed to replace medical professionals.
Its role is to:
- Provide general healthcare information
- Guide users toward appropriate healthcare services
- Provide health-camp information
- Support multilingual conversations
- Remember returning users with consent
- Escalate when human assistance is needed
- Route appointment-related conversations to a specialist workflow
The goal isn't to replace healthcare professionals with AI.
The goal is to make access to healthcare information easier through conversation.
The welcome screen reflects that directly: JanMitra's name in both scripts, the tagline underneath, quick pointers to PM-JAY & Ayushman Bharat guidance and PHC/CHC/hospital discovery, and a row of language chips — Hindi, Telugu, Tamil, Malayalam, Bengali, Marathi, Kannada — inviting the caller to just speak naturally. One button: JanMitra से बात करें | Start Talking.
Why Build a Voice-First Healthcare Assistant?
Healthcare information can sometimes be difficult to access.
A person may want to know:
- Where can I find healthcare services?
- Is there a health camp available?
- How can I approach a clinic?
- When should I seek professional medical attention?
- Can I get help with an appointment?
- Can I communicate in my preferred Indian language?
For many people, speaking can be more natural than typing.
Instead of:
Open website
↓
Find the required section
↓
Type a question
↓
Read the answer
The experience becomes:
Open JanMitra
↓
Start talking
↓
Get guidance
That difference became the core motivation behind the project.
What I Built
By the end of the challenge, JanMitra had evolved into a multi-component voice AI system.
| Capability | What it does |
|---|---|
| Voice conversation | Enables real-time spoken interaction |
| Multilingual support | Handles multilingual and code-mixed conversations |
| Safety guardrails | Keeps healthcare responses within safe boundaries |
| Memory | Remembers permitted user information |
| Healthcare tools | Provides healthcare-access functionality |
| Health camps | Provides available health-camp information |
| Outbound calling | Supports outbound healthcare communication |
| Human escalation | Transfers situations requiring human assistance |
| Analytics | Tracks call-related outcomes |
| Specialist agent | Handles clinic and appointment enquiries |
| Context preservation | Prevents users from repeating their request |
Each feature introduced a new engineering challenge. That is what made the project interesting.
Technology Stack
JanMitra combines several technologies into one real-time voice pipeline.
| Component | Technology |
|---|---|
| Real-time transport | LiveKit |
| Voice agent runtime | LiveKit Agents |
| Speech-to-Text | Deepgram Nova-3 |
| Language Model | OpenRouter (meta-llama/llama-3.3-70b-instruct) |
| Text-to-Speech | Murf Falcon |
| Voice Activity Detection | Silero |
| Persistent memory | SQLite |
| Backend | Python |
| Frontend | Next.js / React |
| Testing | pytest |
The LLM layer didn't start on OpenRouter. Earlier development used a Google Gemini configuration, and that history is still visible in the repository — test_gemini.py remains part of the test suite. The move to OpenRouter and Llama 3.3 70B was a later decision, not the starting point, and I've kept the earlier test rather than pretending Gemini was never part of the project.
The interesting part isn't any individual API. The challenge is making all of these components work together in real time.
System Architecture
The high-level JanMitra architecture looks like this:
┌──────────────────┐
│ USER │
└────────┬─────────┘
│
Voice Input
│
▼
┌──────────────────┐
│ LiveKit │
│ Real-time Audio │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ Deepgram STT │
│ Nova-3 │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ OpenRouter │
│ LLM │
└────────┬─────────┘
│
┌────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌──────────────┐
│ Memory │ │ Healthcare │ │ Specialist │
│ SQLite │ │ Tools │ │ Agent │
└────────────┘ └────────────┘ └──────────────┘
│ │ │
└────────────────────┼────────────────────┘
│
▼
┌──────────────────┐
│ Murf Falcon │
│ TTS │
└────────┬─────────┘
│
Voice Response
│
▼
┌──────────────────┐
│ USER │
└──────────────────┘
This architecture separates the major responsibilities while keeping the interaction real-time.
The Voice Pipeline
A typical JanMitra conversation follows this sequence:
User speaks
↓
LiveKit receives audio
↓
Deepgram converts speech to text
↓
OpenRouter generates the response
↓
Tools/memory/specialist workflows are invoked when required
↓
Murf Falcon converts the response into speech
↓
LiveKit publishes the audio
↓
User hears JanMitra
This looks simple on paper. In practice, every arrow can become a failure point.
Feature 1 — Natural Voice with Murf Falcon
One of the most important parts of JanMitra is its voice experience.
I didn't want the system to feel like a text chatbot that simply reads answers aloud. I wanted the interaction to feel conversational.
The voice path is:
User Speech
↓
Deepgram STT
↓
LLM Reasoning
↓
Murf Falcon
↓
Spoken Response
Murf Falcon became the text-to-speech layer of JanMitra.
But integrating TTS taught me an important lesson:
A fast LLM alone does not create a fast voice agent.
If the database blocks the event loop, the conversation slows down. If TTS playout is interrupted, the user hears incomplete audio. If the browser doesn't receive the published audio correctly, the generated response never reaches the user.
Voice AI is a complete pipeline. Every layer matters.
![JanMitra mid-conversation — live transcript and status]
JanMitra mid-conversation — live transcript, "Listening to you..." status, and the text fallback input for anyone who'd rather type.
Feature 2 — Multilingual Conversations
India is multilingual. So JanMitra wasn't designed around a single-language interaction.
The system is designed to support multilingual and code-mixed conversations. English, Hindi, and Telugu got the most hands-on testing during development; the welcome screen also surfaces Tamil, Malayalam, Bengali, Marathi, and Kannada as options, though I'd trust the Hindi and Telugu experience most today since that's where most of the real conversational testing happened.
For example, a user can ask:
"Please explain this in Hindi."
Or use a code-mixed request such as:
"Telugu lo cheppandi." ("Please say it in Telugu.")
The objective isn't simply translation. The objective is:
Let the user communicate naturally.
Feature 3 — Consent-Based Persistent Memory
A returning user shouldn't always have to start from zero.
JanMitra includes persistent caller memory using SQLite.
The memory workflow supports:
- Caller identification
- Memory lookup
- Saving permitted information
- Returning-user recognition
- Forget-me functionality
- Explicit consent
The intended flow is:
User provides information
↓
JanMitra asks for permission
↓
User agrees
↓
Information is stored
If the user doesn't provide permission, the information should not be stored. This made memory more than a database feature. It became a privacy-aware conversational feature.
Feature 4 — Healthcare Safety Guardrails
Healthcare is a sensitive domain. An AI assistant should not confidently pretend to be a doctor.
JanMitra is therefore designed to:
- Provide general healthcare information
- Provide supportive guidance where appropriate
- Provide healthcare-access information
- Encourage professional care when necessary
- Avoid pretending to diagnose
- Avoid inappropriate medication prescribing
The guiding principle is:
AI can assist. Healthcare professionals remain responsible for medical care.
When professional evaluation is required, JanMitra should guide the user toward an appropriate healthcare facility rather than attempting to replace medical care.
Feature 5 — Health Camp Information
Healthcare access isn't only about symptoms. Sometimes the most useful question is:
"Where and when can I get healthcare services?"
JanMitra includes health-camp functionality so the assistant can provide available health-camp information rather than relying entirely on generated responses.
This functionality also connects naturally with outbound communication.
Feature 6 — Outbound Calling
JanMitra isn't limited to waiting for users to open the application.
The project also includes an outbound calling workflow. One use case is communicating health-camp information.
Conceptually:
Healthcare Information
↓
JanMitra
↓
Outbound Call
↓
User
This changed how I thought about voice agents. A voice agent doesn't have to be only:
"Ask me something."
It can also become:
"I'll reach you when useful information is available."
Feature 7 — Human Escalation
AI shouldn't try to solve every problem. Especially in healthcare.
JanMitra includes a human escalation workflow. When human assistance is required, the system can create an escalation request with a reference ID rather than forcing the AI to continue beyond its role.
The philosophy is:
AI should help when it can and escalate when it should.
This became one of the most important safety principles in the project.
Feature 8 — Call Analytics Dashboard
Building the voice agent was only half the problem. I also wanted to understand what happened during conversations, so I built a call analytics dashboard that records call-related outcomes and surfaces them on the frontend.
The lesson here wasn't about a specific dashboard bug — it was more general:
A feature isn't finished just because the UI exists.
The data behind the UI has to be trustworthy too, which is part of why analytics ended up going through the same regression testing as everything else.
Feature 9 — Clinic & Appointment Specialist
This became one of my favourite architectural improvements.
A single AI agent shouldn't have to handle every responsibility. So I created a dedicated:
ClinicAppointmentSpecialist
Its responsibility is focused on clinic and appointment-related enquiries. For example:
User:
"I want to book an appointment for a general health check-up."
Main JanMitra:
"I'll connect you to our clinic and appointment specialist."
↓
ClinicAppointmentSpecialist
↓
Continues the conversation
The important part is that the user does not need to repeat the request. The specialist continues with the relevant conversation context.
Feature 10 — Context-Preserving Handoff
A handoff is not useful if the user has to explain everything again.
The intended experience is:
User asks appointment question
↓
Main JanMitra understands request
↓
Handoff announcement
↓
ClinicAppointmentSpecialist
↓
Specialist continues with context
This makes the architecture more modular and provides a foundation for adding additional specialist agents later.
The Handoff Implementation
The main agent uses a dedicated handoff tool:
transfer_to_clinic_specialist
Conceptually, the session transitions to the specialist while the existing conversation context remains available to the session.
The key idea is:
Main Agent
↓
Identify specialist need
↓
Announce handoff
↓
Complete speech playout
↓
Transfer control
↓
Specialist introduction
↓
Continue conversation
The "complete speech playout" step wasn't there from the start — it's the fix for a real bug, covered next.
The user experiences one continuous conversation rather than starting a new conversation from scratch.
The Problems That Almost Broke JanMitra
The final result looks clean. The development process wasn't. Some of the most valuable lessons came from failures.
Challenge 1 — Two Different 45-Second Delays
At two separate points in development, JanMitra ended up taking around 45 seconds to respond. For a voice assistant, that's effectively broken — but the two incidents had different root causes, and it's worth keeping them separate rather than telling it as one bug.
The first time, the cause was an invalid tool schema — a Python tool definition (using a union type for one parameter, plus a couple of zero-argument tools) that generated JSON the LLM provider rejected. Retries stacked up and the delay ballooned. The fix was to simplify the tool signatures to explicit, valid types.
The second time, later on, once more database operations had accumulated (get_user, save_user, delete_user, save_escalation, record_call_analytics), I traced the delay to synchronous SQLite calls running directly on the main asyncio event loop — meaning disk I/O could stall LiveKit's voice-activity detection and turn detection at the same time. A warning made this visible directly:
turn detection transport latency is too high: 4213ms
The fix was to move blocking database operations into background threads:
await asyncio.to_thread(...)
After that fix, turn detection latency dropped to roughly 12ms.
The lesson:
Latency isn't always an LLM problem — and it isn't always the same problem twice, either.
Challenge 2 — The Agent Joined but Didn't Speak
This was one of the most frustrating bugs.
The browser connected. The LiveKit room existed. The agent appeared to join. But there was no voice.
I checked browser volume, microphone, speaker, audio permissions — everything looked fine. So I traced the complete lifecycle:
Browser
↓
LiveKit Room
↓
Agent Joins
↓
Session Starts
↓
Greeting Generation
↓
Murf TTS
↓
Audio Publication
↓
Browser Subscription
↓
Audio Playback
A few distinct issues turned out to be contributing to this at different points: session.start() was being awaited synchronously in a way that blocked the greeting code from running until the call ended; a Deepgram configuration using language="multi" was causing the STT WebSocket connection to be rejected outright, silently dropping incoming speech; and a token route that generated a fresh random room name (voice_assistant_room_${Math.random()}) on every request could cause the client to disconnect from one room and try to join another mid-session.
The lesson:
A successful room connection does not mean a successful voice conversation.
For voice systems, the complete audio lifecycle must be verified — and "silence" can have more than one cause hiding behind it.
Challenge 3 — Handoff Audio Was Being Cut Off
During the specialist handoff, another problem appeared.
The main agent would say:
"I'll connect you to our clinic and appointment specialist."
and then immediately switch control. The announcement could be interrupted mid-sentence.
The correct sequence needed to be:
Main Agent
↓
Speak handoff announcement
↓
Wait for audio playout
↓
Switch active agent
↓
Specialist introduction
The fix was to explicitly wait for playout before switching agent state:
await handle.wait_for_playout()
The lesson:
Don't switch conversational control while the previous agent is still speaking.
Challenge 4 — MongoDB vs SQLite
For persistent memory, I initially explored MongoDB. While working with MongoDB Atlas, I encountered TLS/SSL connectivity issues during the handshake.
Instead of adding more infrastructure, I stepped back and asked:
Do I actually need a remote database for this challenge?
For this project, SQLite was sufficient. So I moved the persistent memory system to SQLite. That simplified local development, debugging, database access, testing, and project setup.
One of my favourite lessons was:
Sometimes good engineering means removing infrastructure instead of adding it.
Challenge 5 — Keeping Everything Working
The hardest part wasn't adding the first feature. It was adding the tenth feature without breaking the first.
The system eventually contained:
Voice
↓
Multilingual Interaction
↓
Memory
↓
Healthcare Tools
↓
Health Camps
↓
Outbound Calling
↓
Human Escalation
↓
Analytics
↓
Specialist Handoff
A change in one component could affect another. That made regression testing increasingly important.
Testing
Before preparing the repository for public release, I updated outdated automated tests to match the current agent interface — the Assistant class had grown a user_id and ctx requirement in its constructor, so older tests were updated with a dummy test user and a mocked JobContext, without changing what the tests actually asserted.
The final local automated test result was:
32 / 32 tests passed
The tests covered areas including agent behaviour, database operations, memory, consent, analytics, escalation, and specialist handoff.
I also manually tested the major voice workflows. The objective wasn't simply:
"The code runs."
It was:
"The complete system still works after everything has been combined."
Public Repository Structure
The final repository was organized into a clean backend/frontend structure:
JanMitra/
│
├── backend/
│ ├── src/
│ │ ├── __init__.py
│ │ ├── agent.py
│ │ ├── database.py
│ │ └── outbound.py
│ │
│ ├── tests/
│ │ ├── test_agent.py
│ │ ├── test_agent_memory_flow.py
│ │ ├── test_analytics.py
│ │ ├── test_database.py
│ │ ├── test_day7.py
│ │ ├── test_day9_handoff.py
│ │ └── test_gemini.py
│ │
│ ├── .env.example
│ ├── pyproject.toml
│ └── README.md
│
├── frontend/
│ ├── app/
│ ├── components/
│ ├── .env.example
│ ├── package.json
│ └── README.md
│
├── .gitignore
├── README.md
└── start_app.*
Security Before Going Public
Because JanMitra operates in a healthcare-related context, repository security was especially important.
Before publishing the project, I verified that:
- No API keys were present in tracked source files
- No
.env.localfile was committed - No database files were committed
- No caller information was committed
- No phone numbers or private credentials were committed
- Environment examples contain placeholders
-
.gitignoreprotects local secrets and generated files
The principle was simple:
Show the engineering without exposing private information.
How to Run JanMitra
The current project is primarily prepared as a local development project rather than a publicly deployed production healthcare service.
1. Clone the repository
git clone https://github.com/allenkisathya9723/murf-livekit-starter.git
cd murf-livekit-starter
2. Configure environment variables
Use the provided examples:
backend/.env.example
frontend/.env.example
Create your own local environment files and add your credentials. Never commit real API keys.
3. Install dependencies
Install the backend dependencies according to the backend project configuration. Then install the frontend dependencies from the frontend directory.
4. Start the application
Start the LiveKit voice agent backend and the Next.js frontend using the project's local development setup.
5. Test the conversation
Open the frontend and start a voice session.
For the specialist workflow, try:
"I want to book an appointment for a general health check-up."
Expected flow:
Main JanMitra
↓
Handoff announcement
↓
ClinicAppointmentSpecialist
↓
Conversation continues
What I Learned
Before this challenge, I thought a voice agent was basically:
Speech → AI → Speech
Now I know it is much more than that.
A reliable voice agent involves speech recognition, language understanding, text-to-speech, real-time transport, turn detection, memory, tool calling, database design, safety, consent, analytics, human escalation, specialist workflows, testing, and security.
Every component matters. A fast LLM doesn't help if your database blocks the event loop. A working TTS API doesn't help if audio playout gets interrupted. A specialist doesn't help if conversation context disappears. And a healthcare assistant isn't trustworthy if it confidently pretends to diagnose someone.
That was probably my biggest lesson from these ten days.
Build → Break → Debug → Test → Improve
If I had to summarize the entire challenge in five words:
Build → Break → Debug → Test → Improve
The most valuable moments weren't always when everything worked. They were when something completely failed and I had to figure out why.
The 45-second responses taught me about asynchronous architecture — twice, in two different ways. The silent agent taught me about real-time audio lifecycles. The handoff bug taught me about speech playout. The database problem taught me to question whether I actually needed additional infrastructure. Regression testing taught me that every new feature has a cost.
That is what made this challenge much more than simply connecting APIs.
What I Would Build Next
JanMitra is still a challenge project, so there is a lot more I would like to explore.
Future improvements include:
- More regional Indian languages tested to the same depth as Hindi and Telugu
- Better healthcare information retrieval
- More robust clinic and appointment discovery
- Better emergency workflows
- Additional specialist agents
- Production-grade telephony
- Improved analytics and observability
- Retrieval-Augmented Generation
- Cloud deployment
- More extensive real-world testing
The long-term goal is to move JanMitra from a challenge project toward something that could genuinely improve access to healthcare information.
Open Source Repository
The complete public source code is available here:
The repository contains the backend, frontend, automated tests, and configuration examples. Private environment files and local databases are intentionally excluded.
Final Thoughts
Building JanMitra over these ten days taught me much more than how to connect AI APIs.
It taught me how to think about conversations, reliability, latency, safety, memory, privacy, debugging, and user experience.
The project started as a voice assistant. It gradually became:
Voice
↓
Multilingual Conversations
↓
Memory
↓
Healthcare Tools
↓
Health-Camp Information
↓
Outbound Calling
↓
Human Escalation
↓
Call Analytics
↓
Clinic & Appointment Specialist
↓
Context-Preserving Handoff
The most valuable part wasn't getting everything working on the first attempt. It was breaking things, investigating why they broke, fixing them, and testing again. That process changed how I think about AI development.
Build. Break. Debug. Test. Improve.
That's what these ten days were really about.
A huge thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build and experiment with real-time voice AI. I'm especially grateful for the opportunity to work with Murf Falcon, which became an important part of JanMitra's voice experience.
JanMitra is not the final version of what I want to build. But it is a strong beginning.
🚀 JanMitra — A Voice for Better Health Access.
Useful Links
- GitHub: https://github.com/allenkisathya9723/murf-livekit-starter
- Murf AI: https://murf.ai/
- LiveKit: https://livekit.io/
- Deepgram: https://deepgram.com/
Top comments (0)