From a Voice Bot to a Responsible Health Access Agent: Building Jana Seva in 10 Days

ABHINAV DASH — Sat, 15 Aug 2026 11:57:56 +0000

From a Voice Bot to a Responsible Health Access Agent: Building Jana Seva in 10 Days

The Problem

Accessing healthcare information is not always as simple as opening a website and reading a page.

For many users, especially in contexts where typing, navigating websites, or understanding technical health terminology is difficult, voice can be a more natural interface.

That became the starting point for Jana Seva — a voice-first Health Access AI assistant built during the 10 Days of Voice Agents — VoiceForBharat Edition.

The goal was not to build an AI doctor.

The goal was to build a system that can help users navigate health-access information, use real data when appropriate, remember useful context, and know when it should stop and involve a human.

Meet Jana Seva

Jana Seva is designed as a real-time voice assistant for public-health access.

A user can talk naturally through the browser and ask about healthcare access, nearby health facilities, environmental health information, or appointment-related help.

The system can also:

use live data tools;
remember consented user context;
make controlled outbound reminder calls;
create human escalation requests;
show call analytics;
hand a conversation to a specialist agent.

The project uses Murf Falcon for the voice output and LiveKit Agents for real-time communication.

Why Voice?

Voice changes the interaction model.

Instead of:

Search → Read → Understand → Navigate → Decide

the experience can become:

Speak → Listen → Respond → Continue

That matters when a user is more comfortable speaking than typing or when a conversational interface is more natural for the task.

For an Indian Health Access use case, the ability to support English, Hindi and Hinglish interaction paths also makes the experience more conversational.

How the System Works

At a high level:

Browser / Phone
      ↓
LiveKit
      ↓
Voice Activity + Turn Detection
      ↓
Deepgram Nova-3
      ↓
Jana Seva Agent
      ↓
Google Gemini
      ↓
Tools / Memory / Escalation / Specialist
      ↓
Murf Falcon
      ↓
LiveKit
      ↓
User

The system separates real-time transport, speech recognition, reasoning, external tools, persistence and voice synthesis.

That separation made it possible to add new capabilities over the ten-day challenge without turning the entire agent into one large block of logic.

The Journey

Day 1 — Making the Agent Speak

The first milestone was the basic real-time voice pipeline.

The browser connects through LiveKit, speech is recognized, the agent reasons about the request, and Murf Falcon generates the response.

This established the foundation.

Day 2 — Defining What the Agent Should Do

The next step was defining the agent's role and boundaries.

The system was designed around Health Access rather than diagnosis.

That distinction became important later because it determined when the agent should answer, when it should use a tool, and when it should escalate.

Day 3 — Safety and Guardrails

Healthcare is not a domain where an AI should confidently improvise.

The agent therefore needed boundaries around diagnosis, prescription-like behavior and emergency situations.

The design principle became:

Be useful without pretending to be a doctor.

Day 4 — Memory

A returning user should not necessarily have to repeat the same basic context every time.

Jana Seva introduced consent-based caller memory so useful context could be reused by later workflows.

This became especially useful once the agent started using tools.

For example, a previously known district could help a health-facility lookup without forcing the user to repeat it.

Day 5 — Real-World Data

This was where the project started feeling less like a demo chatbot.

Jana Seva gained function-calling tools for:

Nearby health facilities

A facility lookup can search for:

PHCs
CHCs
District Hospitals
Jan Aushadhi stores

The system can use OpenStreetMap Nominatim and a local fallback registry.

Environmental health advisory

The agent can retrieve:

AQI
PM2.5
PM10
temperature
respiratory precautions

using Open-Meteo.

The tools also have failure paths.

If an external service fails, the agent should not silently invent a result.

It should communicate the failure and use the available fallback where appropriate.

Day 6 — Outbound Calls

The next challenge was moving beyond inbound browser conversations.

Jana Seva added controlled outbound health-reminder workflows using Twilio and LiveKit SIP.

The use case was intentionally narrow:

vaccination follow-ups;
medication reminders.

The call begins with identification, purpose and an opt-out mechanism.

If the user says they do not want future calls, the agent immediately handles the opt-out path.

Automatic retries are also intentionally conservative.

This was an important lesson:

A voice system that can call people needs stronger boundaries than a system that only waits for users to call it.

Day 7 — Knowing When to Ask a Human

This became one of the most important parts of the project.

The agent should not try to solve every problem.

Two major escalation conditions were:

red-flag symptoms / emergency situations;
requests for diagnosis or medical decisions outside the agent's safe scope.

The workflow is:

Detect
  ↓
Explain
  ↓
Ask permission
  ↓
Summarize
  ↓
Create request
  ↓
Generate reference ID
  ↓
Tell user what happens next

The human receives a concise operational summary rather than an unnecessary full transcript.

The escalation dashboard organizes requests by urgency and status.

This changed my understanding of "smart" agents.

A smart agent is not necessarily the one that answers everything.

Sometimes the smarter action is:

"This is outside what I should handle. Let me get the right human involved."

Day 8 — Measuring the Agent

Once an agent is doing real work, I need to know whether it is actually succeeding.

Jana Seva therefore gained a call analytics dashboard backed by SQLite.

The required metrics are:

Total Calls
Successful Calls
Failed Calls

The system also calculates success rate and shows operational call history.

A successful call means the intended health-access task was safely completed, or a required human escalation was correctly completed with user permission.

A failed call means the intended outcome was not reached.

This distinction matters because a "failed" call does not necessarily mean the software crashed.

A user hanging up before completing the task can also be a failed outcome.

Day 9 — Handing the Conversation to a Specialist

The final major agent capability was specialist routing.

Instead of building one huge agent that tries to know everything, Jana Seva now has a focused:

Clinic & Appointment Specialist

The main agent detects requests related to:

clinic discovery;
department navigation;
appointment assistance.

It announces the handoff first.

Then it transfers useful context so the user does not need to repeat the entire request.

For example:

User:
"I want to find a clinic and get help with an appointment."

        ↓

Main Jana Seva Agent:
"I'll connect you with our Clinic & Appointment Specialist."

        ↓

Clinic & Appointment Specialist:
"I understand you're looking for help with a general health consultation appointment..."

The specialist has its own scope and safety boundaries.

Emergency red flags bypass the specialist and use the human escalation workflow instead.

The Technical Stack

Layer	Technology
Real-time transport	LiveKit
Voice synthesis	Murf Falcon
Speech recognition	Deepgram Nova-3
LLM	Google Gemini
Backend	Python
Frontend	Next.js / React
Database	SQLite
Telephony	Twilio + LiveKit SIP
Facility lookup	OpenStreetMap Nominatim
Environmental data	Open-Meteo

The Hard Parts

1. Designing safe boundaries

The easiest mistake is to make a healthcare agent sound too confident.

The system needed explicit boundaries around diagnosis and emergency situations.

The solution was to make escalation a first-class workflow rather than an afterthought.

2. Handling external data failures

Live APIs can fail.

A voice assistant cannot simply show an error stack trace to a user.

The tool layer therefore needed fallback behavior and a spoken failure path.

The agent should communicate that the live lookup was unavailable rather than pretending that stale or invented information is current.

3. Making handoffs feel natural

A bad multi-agent handoff would sound like:

"Please repeat everything to the next agent."

That defeats the purpose.

The specialist handoff therefore transfers the relevant intent and context so the conversation can continue naturally.

4. Knowing what to measure

A dashboard is only useful if the metrics represent the actual objective.

Instead of measuring only technical uptime, Jana Seva records whether the call reached its intended outcome.

That makes the analytics more useful for evaluating the actual agent experience.

How Someone Else Can Build a Similar Voice Agent

The architecture can be reduced to four core components:

1. Speech-to-text

Convert the user's speech into text.

2. LLM

Reason about the request and decide what should happen.

3. Text-to-speech

Turn the response back into natural speech.

4. Real-time transport

Move audio between the user and the agent with low latency.

LiveKit handles the real-time layer in Jana Seva.

Murf Falcon handles speech synthesis.

Deepgram handles speech recognition.

Google Gemini handles reasoning and tool calling.

Running Jana Seva

Prerequisites

Install:

Python 3.10+
uv
Node.js 18+
pnpm
LiveKit

Then configure the required environment variables locally.

Example:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key
GOOGLE_API_KEY=your_google_api_key

Never commit real credentials.

Install backend dependencies:

cd backend
uv sync
uv run python src/agent.py download-files

Install frontend dependencies:

cd frontend
pnpm install

Start the project using the included startup script or run the LiveKit server, backend agent and frontend in separate terminals.

Then open:

http://localhost:3000

Allow microphone access and start a conversation.

Evidence From the Build

The project includes screenshots showing:

the Jana Seva homepage;
an active voice session;
the human escalation command center;
the call analytics dashboard;
a specialist-agent handoff.

The project also includes demonstration videos from the challenge stages.

What I Would Build Next

If I continued developing Jana Seva, I would focus on:

verified government health-service integrations;
stronger multilingual and code-mixed conversations;
more specialist agents;
improved escalation workflows;
duplicate escalation detection;
resolution callbacks;
production authentication;
stronger safety evaluations;
better observability and latency measurement.

Final Takeaway

The biggest lesson from these ten days was that building a voice agent is not just about making an AI that can talk.

It is about designing a system that knows what it can do.

It needs to know:

when to answer;
when to use a tool;
when to remember;
when to ask permission;
when to stop;
when to involve a human;
and when another specialist should take over.

That is the idea behind Jana Seva.

Healthcare, in your voice.

Links

Source code: https://github.com/lightcode01-oss/murf-livekit-starter/tree/day10

Demo: https://www.linkedin.com/posts/abhinav-dash_voiceforbharat-10daysofai-ai-ugcPost-7494361062327246848-3T22/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAEFeFmgBuzjc34NJOKKeAvlxlTqBPFxgbLw

Built for

10 Days of Voice Agents — VoiceForBharat Edition

VoiceForBharat #VoiceAI #MurfAI #LiveKit #AI #HealthTech #GenerativeAI

DEV Community: ABHINAV DASH

From a Voice Bot to a Responsible Health Access Agent: Building Jana Seva in 10 Days

From a Voice Bot to a Responsible Health Access Agent: Building Jana Seva in 10 Days

The Problem

Meet Jana Seva

Why Voice?

How the System Works

The Journey

Day 1 — Making the Agent Speak

Day 2 — Defining What the Agent Should Do

Day 3 — Safety and Guardrails

Day 4 — Memory

Day 5 — Real-World Data

Nearby health facilities

Environmental health advisory

Day 6 — Outbound Calls

Day 7 — Knowing When to Ask a Human

Day 8 — Measuring the Agent

Day 9 — Handing the Conversation to a Specialist

The Technical Stack

The Hard Parts

1. Designing safe boundaries

2. Handling external data failures

3. Making handoffs feel natural

4. Knowing what to measure

How Someone Else Can Build a Similar Voice Agent

1. Speech-to-text

2. LLM

3. Text-to-speech

4. Real-time transport

Running Jana Seva

Prerequisites

Evidence From the Build

What I Would Build Next

Final Takeaway

Links

Built for

VoiceForBharat #VoiceAI #MurfAI #LiveKit #AI #HealthTech #GenerativeAI