Building JanMitra: My 10-Day Journey Building a Multilingual Healthcare Voice Agent for Bharat

Allenki Sathyendra — Sat, 15 Aug 2026 13:16:37 +0000

Building JanMitra: A Voice for Better Health Access — My 10-Day VoiceForBharat Journey

What if accessing basic healthcare information was as simple as having a conversation?

That question became the starting point for JanMitra (जनमित्र) — a multilingual healthcare voice assistant I built during the 10 Days of Voice Agents — VoiceForBharat Edition challenge by Murf AI.

I didn't want to build another chatbot where users type questions into a box.

I wanted to explore something more natural:

What if people could simply talk to an AI assistant in the language they are comfortable with and get guidance about healthcare access?

Over ten days, JanMitra evolved from a basic voice agent into a complete conversational system with:

Real-time voice conversations
Multilingual and code-mixed interaction
Healthcare safety guardrails
Consent-based persistent memory
Healthcare-access tools
Health-camp information
Outbound calling
Human escalation
Call analytics
Clinic and appointment specialist handoff
Conversation-context preservation

And getting there wasn't easy.

I faced silent voice sessions, extremely high response latency, database bottlenecks, interrupted audio, and regression risk every time I added something new.

This article is the story of what I built, what broke, how I fixed it, and what I learned while building a real-time voice AI system.

Meet JanMitra

![JanMitra welcome screen — Aapka Swasthya Saathi]

JanMitra's welcome screen — PM-JAY & Ayushman Bharat guidance, PHC/CHC/hospital discovery, and the "Speak Naturally" language chips (Hindi, Telugu, Tamil, Malayalam, Bengali, Marathi, Kannada).

JanMitra (जनमित्र) represents the idea of a friend or companion for people.

The vision is simple:

Aapka Swasthya Saathi — Your Healthcare Companion

JanMitra is designed to help users access general healthcare information and healthcare services through natural voice conversations.

It is not a doctor and is not designed to replace medical professionals.

Its role is to:

Provide general healthcare information
Guide users toward appropriate healthcare services
Provide health-camp information
Support multilingual conversations
Remember returning users with consent
Escalate when human assistance is needed
Route appointment-related conversations to a specialist workflow

The goal isn't to replace healthcare professionals with AI.

The goal is to make access to healthcare information easier through conversation.

The welcome screen reflects that directly: JanMitra's name in both scripts, the tagline underneath, quick pointers to PM-JAY & Ayushman Bharat guidance and PHC/CHC/hospital discovery, and a row of language chips — Hindi, Telugu, Tamil, Malayalam, Bengali, Marathi, Kannada — inviting the caller to just speak naturally. One button: JanMitra से बात करें | Start Talking.

Why Build a Voice-First Healthcare Assistant?

Healthcare information can sometimes be difficult to access.

A person may want to know:

Where can I find healthcare services?
Is there a health camp available?
How can I approach a clinic?
When should I seek professional medical attention?
Can I get help with an appointment?
Can I communicate in my preferred Indian language?

For many people, speaking can be more natural than typing.

Instead of:

Open website
    ↓
Find the required section
    ↓
Type a question
    ↓
Read the answer

The experience becomes:

Open JanMitra
    ↓
Start talking
    ↓
Get guidance

That difference became the core motivation behind the project.

What I Built

By the end of the challenge, JanMitra had evolved into a multi-component voice AI system.

Capability	What it does
Voice conversation	Enables real-time spoken interaction
Multilingual support	Handles multilingual and code-mixed conversations
Safety guardrails	Keeps healthcare responses within safe boundaries
Memory	Remembers permitted user information
Healthcare tools	Provides healthcare-access functionality
Health camps	Provides available health-camp information
Outbound calling	Supports outbound healthcare communication
Human escalation	Transfers situations requiring human assistance
Analytics	Tracks call-related outcomes
Specialist agent	Handles clinic and appointment enquiries
Context preservation	Prevents users from repeating their request

Each feature introduced a new engineering challenge. That is what made the project interesting.

Technology Stack

JanMitra combines several technologies into one real-time voice pipeline.

Component	Technology
Real-time transport	LiveKit
Voice agent runtime	LiveKit Agents
Speech-to-Text	Deepgram Nova-3
Language Model	OpenRouter (`meta-llama/llama-3.3-70b-instruct`)
Text-to-Speech	Murf Falcon
Voice Activity Detection	Silero
Persistent memory	SQLite
Backend	Python
Frontend	Next.js / React
Testing	pytest

The LLM layer didn't start on OpenRouter. Earlier development used a Google Gemini configuration, and that history is still visible in the repository — test_gemini.py remains part of the test suite. The move to OpenRouter and Llama 3.3 70B was a later decision, not the starting point, and I've kept the earlier test rather than pretending Gemini was never part of the project.

The interesting part isn't any individual API. The challenge is making all of these components work together in real time.

System Architecture

The high-level JanMitra architecture looks like this:

                         ┌──────────────────┐
                         │       USER       │
                         └────────┬─────────┘
                                  │
                             Voice Input
                                  │
                                  ▼
                         ┌──────────────────┐
                         │     LiveKit      │
                         │ Real-time Audio  │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │  Deepgram STT    │
                         │   Nova-3         │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │   OpenRouter     │
                         │       LLM        │
                         └────────┬─────────┘
                                  │
             ┌────────────────────┼────────────────────┐
             │                    │                    │
             ▼                    ▼                    ▼
      ┌────────────┐       ┌────────────┐      ┌──────────────┐
      │   Memory   │       │ Healthcare │      │  Specialist  │
      │   SQLite   │       │   Tools    │      │    Agent     │
      └────────────┘       └────────────┘      └──────────────┘
             │                    │                    │
             └────────────────────┼────────────────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │   Murf Falcon    │
                         │       TTS        │
                         └────────┬─────────┘
                                  │
                           Voice Response
                                  │
                                  ▼
                         ┌──────────────────┐
                         │       USER       │
                         └──────────────────┘

This architecture separates the major responsibilities while keeping the interaction real-time.

The Voice Pipeline

A typical JanMitra conversation follows this sequence:

User speaks
    ↓
LiveKit receives audio
    ↓
Deepgram converts speech to text
    ↓
OpenRouter generates the response
    ↓
Tools/memory/specialist workflows are invoked when required
    ↓
Murf Falcon converts the response into speech
    ↓
LiveKit publishes the audio
    ↓
User hears JanMitra

This looks simple on paper. In practice, every arrow can become a failure point.

Feature 1 — Natural Voice with Murf Falcon

One of the most important parts of JanMitra is its voice experience.

I didn't want the system to feel like a text chatbot that simply reads answers aloud. I wanted the interaction to feel conversational.

The voice path is:

User Speech
     ↓
Deepgram STT
     ↓
LLM Reasoning
     ↓
Murf Falcon
     ↓
Spoken Response

Murf Falcon became the text-to-speech layer of JanMitra.

But integrating TTS taught me an important lesson:

A fast LLM alone does not create a fast voice agent.

If the database blocks the event loop, the conversation slows down. If TTS playout is interrupted, the user hears incomplete audio. If the browser doesn't receive the published audio correctly, the generated response never reaches the user.

Voice AI is a complete pipeline. Every layer matters.

![JanMitra mid-conversation — live transcript and status]

JanMitra mid-conversation — live transcript, "Listening to you..." status, and the text fallback input for anyone who'd rather type.

Feature 2 — Multilingual Conversations

India is multilingual. So JanMitra wasn't designed around a single-language interaction.

The system is designed to support multilingual and code-mixed conversations. English, Hindi, and Telugu got the most hands-on testing during development; the welcome screen also surfaces Tamil, Malayalam, Bengali, Marathi, and Kannada as options, though I'd trust the Hindi and Telugu experience most today since that's where most of the real conversational testing happened.

For example, a user can ask:

"Please explain this in Hindi."

Or use a code-mixed request such as:

"Telugu lo cheppandi." ("Please say it in Telugu.")

The objective isn't simply translation. The objective is:

Let the user communicate naturally.

Feature 3 — Consent-Based Persistent Memory

A returning user shouldn't always have to start from zero.

JanMitra includes persistent caller memory using SQLite.

The memory workflow supports:

Caller identification
Memory lookup
Saving permitted information
Returning-user recognition
Forget-me functionality
Explicit consent

The intended flow is:

User provides information
          ↓
JanMitra asks for permission
          ↓
User agrees
          ↓
Information is stored

If the user doesn't provide permission, the information should not be stored. This made memory more than a database feature. It became a privacy-aware conversational feature.

Feature 4 — Healthcare Safety Guardrails

Healthcare is a sensitive domain. An AI assistant should not confidently pretend to be a doctor.

JanMitra is therefore designed to:

Provide general healthcare information
Provide supportive guidance where appropriate
Provide healthcare-access information
Encourage professional care when necessary
Avoid pretending to diagnose
Avoid inappropriate medication prescribing

The guiding principle is:

AI can assist. Healthcare professionals remain responsible for medical care.

When professional evaluation is required, JanMitra should guide the user toward an appropriate healthcare facility rather than attempting to replace medical care.

Feature 5 — Health Camp Information

Healthcare access isn't only about symptoms. Sometimes the most useful question is:

"Where and when can I get healthcare services?"

JanMitra includes health-camp functionality so the assistant can provide available health-camp information rather than relying entirely on generated responses.

This functionality also connects naturally with outbound communication.

Feature 6 — Outbound Calling

JanMitra isn't limited to waiting for users to open the application.

The project also includes an outbound calling workflow. One use case is communicating health-camp information.

Conceptually:

Healthcare Information
          ↓
       JanMitra
          ↓
     Outbound Call
          ↓
        User

This changed how I thought about voice agents. A voice agent doesn't have to be only:

"Ask me something."

It can also become:

"I'll reach you when useful information is available."

Feature 7 — Human Escalation

AI shouldn't try to solve every problem. Especially in healthcare.

JanMitra includes a human escalation workflow. When human assistance is required, the system can create an escalation request with a reference ID rather than forcing the AI to continue beyond its role.

The philosophy is:

AI should help when it can and escalate when it should.

This became one of the most important safety principles in the project.

Feature 8 — Call Analytics Dashboard

Building the voice agent was only half the problem. I also wanted to understand what happened during conversations, so I built a call analytics dashboard that records call-related outcomes and surfaces them on the frontend.

The lesson here wasn't about a specific dashboard bug — it was more general:

A feature isn't finished just because the UI exists.

The data behind the UI has to be trustworthy too, which is part of why analytics ended up going through the same regression testing as everything else.

Feature 9 — Clinic & Appointment Specialist

This became one of my favourite architectural improvements.

A single AI agent shouldn't have to handle every responsibility. So I created a dedicated:

ClinicAppointmentSpecialist

Its responsibility is focused on clinic and appointment-related enquiries. For example:

User:
"I want to book an appointment for a general health check-up."

Main JanMitra:
"I'll connect you to our clinic and appointment specialist."

                ↓

ClinicAppointmentSpecialist

                ↓

Continues the conversation

The important part is that the user does not need to repeat the request. The specialist continues with the relevant conversation context.

Feature 10 — Context-Preserving Handoff

A handoff is not useful if the user has to explain everything again.

The intended experience is:

User asks appointment question
            ↓
Main JanMitra understands request
            ↓
Handoff announcement
            ↓
ClinicAppointmentSpecialist
            ↓
Specialist continues with context

This makes the architecture more modular and provides a foundation for adding additional specialist agents later.

The Handoff Implementation

The main agent uses a dedicated handoff tool:

transfer_to_clinic_specialist

Conceptually, the session transitions to the specialist while the existing conversation context remains available to the session.

The key idea is:

Main Agent
    ↓
Identify specialist need
    ↓
Announce handoff
    ↓
Complete speech playout
    ↓
Transfer control
    ↓
Specialist introduction
    ↓
Continue conversation

The "complete speech playout" step wasn't there from the start — it's the fix for a real bug, covered next.

The user experiences one continuous conversation rather than starting a new conversation from scratch.

The Problems That Almost Broke JanMitra

The final result looks clean. The development process wasn't. Some of the most valuable lessons came from failures.

Challenge 1 — Two Different 45-Second Delays

At two separate points in development, JanMitra ended up taking around 45 seconds to respond. For a voice assistant, that's effectively broken — but the two incidents had different root causes, and it's worth keeping them separate rather than telling it as one bug.

The first time, the cause was an invalid tool schema — a Python tool definition (using a union type for one parameter, plus a couple of zero-argument tools) that generated JSON the LLM provider rejected. Retries stacked up and the delay ballooned. The fix was to simplify the tool signatures to explicit, valid types.

The second time, later on, once more database operations had accumulated (get_user, save_user, delete_user, save_escalation, record_call_analytics), I traced the delay to synchronous SQLite calls running directly on the main asyncio event loop — meaning disk I/O could stall LiveKit's voice-activity detection and turn detection at the same time. A warning made this visible directly:

turn detection transport latency is too high: 4213ms

The fix was to move blocking database operations into background threads:

await asyncio.to_thread(...)

After that fix, turn detection latency dropped to roughly 12ms.

The lesson:

Latency isn't always an LLM problem — and it isn't always the same problem twice, either.

Challenge 2 — The Agent Joined but Didn't Speak

This was one of the most frustrating bugs.

The browser connected. The LiveKit room existed. The agent appeared to join. But there was no voice.

I checked browser volume, microphone, speaker, audio permissions — everything looked fine. So I traced the complete lifecycle:

Browser
   ↓
LiveKit Room
   ↓
Agent Joins
   ↓
Session Starts
   ↓
Greeting Generation
   ↓
Murf TTS
   ↓
Audio Publication
   ↓
Browser Subscription
   ↓
Audio Playback

A few distinct issues turned out to be contributing to this at different points: session.start() was being awaited synchronously in a way that blocked the greeting code from running until the call ended; a Deepgram configuration using language="multi" was causing the STT WebSocket connection to be rejected outright, silently dropping incoming speech; and a token route that generated a fresh random room name (voice_assistant_room_${Math.random()}) on every request could cause the client to disconnect from one room and try to join another mid-session.

The lesson:

A successful room connection does not mean a successful voice conversation.

For voice systems, the complete audio lifecycle must be verified — and "silence" can have more than one cause hiding behind it.

Challenge 3 — Handoff Audio Was Being Cut Off

During the specialist handoff, another problem appeared.

The main agent would say:

"I'll connect you to our clinic and appointment specialist."

and then immediately switch control. The announcement could be interrupted mid-sentence.

The correct sequence needed to be:

Main Agent
    ↓
Speak handoff announcement
    ↓
Wait for audio playout
    ↓
Switch active agent
    ↓
Specialist introduction

The fix was to explicitly wait for playout before switching agent state:

await handle.wait_for_playout()

The lesson:

Don't switch conversational control while the previous agent is still speaking.

Challenge 4 — MongoDB vs SQLite

For persistent memory, I initially explored MongoDB. While working with MongoDB Atlas, I encountered TLS/SSL connectivity issues during the handshake.

Instead of adding more infrastructure, I stepped back and asked:

Do I actually need a remote database for this challenge?

For this project, SQLite was sufficient. So I moved the persistent memory system to SQLite. That simplified local development, debugging, database access, testing, and project setup.

One of my favourite lessons was:

Sometimes good engineering means removing infrastructure instead of adding it.

Challenge 5 — Keeping Everything Working

The hardest part wasn't adding the first feature. It was adding the tenth feature without breaking the first.

The system eventually contained:

Voice
  ↓
Multilingual Interaction
  ↓
Memory
  ↓
Healthcare Tools
  ↓
Health Camps
  ↓
Outbound Calling
  ↓
Human Escalation
  ↓
Analytics
  ↓
Specialist Handoff

A change in one component could affect another. That made regression testing increasingly important.

Testing

Before preparing the repository for public release, I updated outdated automated tests to match the current agent interface — the Assistant class had grown a user_id and ctx requirement in its constructor, so older tests were updated with a dummy test user and a mocked JobContext, without changing what the tests actually asserted.

The final local automated test result was:

32 / 32 tests passed

The tests covered areas including agent behaviour, database operations, memory, consent, analytics, escalation, and specialist handoff.

I also manually tested the major voice workflows. The objective wasn't simply:

"The code runs."

It was:

"The complete system still works after everything has been combined."

Public Repository Structure

The final repository was organized into a clean backend/frontend structure:

JanMitra/
│
├── backend/
│   ├── src/
│   │   ├── __init__.py
│   │   ├── agent.py
│   │   ├── database.py
│   │   └── outbound.py
│   │
│   ├── tests/
│   │   ├── test_agent.py
│   │   ├── test_agent_memory_flow.py
│   │   ├── test_analytics.py
│   │   ├── test_database.py
│   │   ├── test_day7.py
│   │   ├── test_day9_handoff.py
│   │   └── test_gemini.py
│   │
│   ├── .env.example
│   ├── pyproject.toml
│   └── README.md
│
├── frontend/
│   ├── app/
│   ├── components/
│   ├── .env.example
│   ├── package.json
│   └── README.md
│
├── .gitignore
├── README.md
└── start_app.*

Security Before Going Public

Because JanMitra operates in a healthcare-related context, repository security was especially important.

Before publishing the project, I verified that:

No API keys were present in tracked source files
No .env.local file was committed
No database files were committed
No caller information was committed
No phone numbers or private credentials were committed
Environment examples contain placeholders
.gitignore protects local secrets and generated files

The principle was simple:

Show the engineering without exposing private information.

How to Run JanMitra

The current project is primarily prepared as a local development project rather than a publicly deployed production healthcare service.

1. Clone the repository

git clone https://github.com/allenkisathya9723/murf-livekit-starter.git
cd murf-livekit-starter

2. Configure environment variables

Use the provided examples:

backend/.env.example
frontend/.env.example

Create your own local environment files and add your credentials. Never commit real API keys.

3. Install dependencies

Install the backend dependencies according to the backend project configuration. Then install the frontend dependencies from the frontend directory.

4. Start the application

Start the LiveKit voice agent backend and the Next.js frontend using the project's local development setup.

5. Test the conversation

Open the frontend and start a voice session.

For the specialist workflow, try:

"I want to book an appointment for a general health check-up."

Expected flow:

Main JanMitra
      ↓
Handoff announcement
      ↓
ClinicAppointmentSpecialist
      ↓
Conversation continues

What I Learned

Before this challenge, I thought a voice agent was basically:

Speech → AI → Speech

Now I know it is much more than that.

A reliable voice agent involves speech recognition, language understanding, text-to-speech, real-time transport, turn detection, memory, tool calling, database design, safety, consent, analytics, human escalation, specialist workflows, testing, and security.

Every component matters. A fast LLM doesn't help if your database blocks the event loop. A working TTS API doesn't help if audio playout gets interrupted. A specialist doesn't help if conversation context disappears. And a healthcare assistant isn't trustworthy if it confidently pretends to diagnose someone.

That was probably my biggest lesson from these ten days.

Build → Break → Debug → Test → Improve

If I had to summarize the entire challenge in five words:

Build → Break → Debug → Test → Improve

The most valuable moments weren't always when everything worked. They were when something completely failed and I had to figure out why.

The 45-second responses taught me about asynchronous architecture — twice, in two different ways. The silent agent taught me about real-time audio lifecycles. The handoff bug taught me about speech playout. The database problem taught me to question whether I actually needed additional infrastructure. Regression testing taught me that every new feature has a cost.

That is what made this challenge much more than simply connecting APIs.

What I Would Build Next

JanMitra is still a challenge project, so there is a lot more I would like to explore.

Future improvements include:

More regional Indian languages tested to the same depth as Hindi and Telugu
Better healthcare information retrieval
More robust clinic and appointment discovery
Better emergency workflows
Additional specialist agents
Production-grade telephony
Improved analytics and observability
Retrieval-Augmented Generation
Cloud deployment
More extensive real-world testing

The long-term goal is to move JanMitra from a challenge project toward something that could genuinely improve access to healthcare information.

Open Source Repository

The complete public source code is available here:

👉 JanMitra GitHub Repository

The repository contains the backend, frontend, automated tests, and configuration examples. Private environment files and local databases are intentionally excluded.

Final Thoughts

Building JanMitra over these ten days taught me much more than how to connect AI APIs.

It taught me how to think about conversations, reliability, latency, safety, memory, privacy, debugging, and user experience.

The project started as a voice assistant. It gradually became:

Voice
  ↓
Multilingual Conversations
  ↓
Memory
  ↓
Healthcare Tools
  ↓
Health-Camp Information
  ↓
Outbound Calling
  ↓
Human Escalation
  ↓
Call Analytics
  ↓
Clinic & Appointment Specialist
  ↓
Context-Preserving Handoff

The most valuable part wasn't getting everything working on the first attempt. It was breaking things, investigating why they broke, fixing them, and testing again. That process changed how I think about AI development.

Build. Break. Debug. Test. Improve.

That's what these ten days were really about.

A huge thank you to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition and giving me the opportunity to build and experiment with real-time voice AI. I'm especially grateful for the opportunity to work with Murf Falcon, which became an important part of JanMitra's voice experience.

JanMitra is not the final version of what I want to build. But it is a strong beginning.

🚀 JanMitra — A Voice for Better Health Access.

Useful Links

GitHub: https://github.com/allenkisathya9723/murf-livekit-starter
Murf AI: https://murf.ai/
LiveKit: https://livekit.io/
Deepgram: https://deepgram.com/

VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #LiveKit #HealthcareAI #GenerativeAI #MultilingualAI

DEV Community: Allenki Sathyendra

Building JanMitra: My 10-Day Journey Building a Multilingual Healthcare Voice Agent for Bharat

Building JanMitra: A Voice for Better Health Access — My 10-Day VoiceForBharat Journey

Meet JanMitra

Why Build a Voice-First Healthcare Assistant?

What I Built

Technology Stack

System Architecture

The Voice Pipeline

Feature 1 — Natural Voice with Murf Falcon

Feature 2 — Multilingual Conversations

Feature 3 — Consent-Based Persistent Memory

Feature 4 — Healthcare Safety Guardrails

Feature 5 — Health Camp Information

Feature 6 — Outbound Calling

Feature 7 — Human Escalation

Feature 8 — Call Analytics Dashboard

Feature 9 — Clinic & Appointment Specialist

Feature 10 — Context-Preserving Handoff

The Handoff Implementation

The Problems That Almost Broke JanMitra

Challenge 1 — Two Different 45-Second Delays

Challenge 2 — The Agent Joined but Didn't Speak

Challenge 3 — Handoff Audio Was Being Cut Off

Challenge 4 — MongoDB vs SQLite

Challenge 5 — Keeping Everything Working

Testing

Public Repository Structure

Security Before Going Public

How to Run JanMitra

1. Clone the repository

2. Configure environment variables

3. Install dependencies

4. Start the application

5. Test the conversation

What I Learned

Build → Break → Debug → Test → Improve

What I Would Build Next

Open Source Repository

Final Thoughts

🚀 JanMitra — A Voice for Better Health Access.

Useful Links

VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #LiveKit #HealthcareAI #GenerativeAI #MultilingualAI