I built V, a voice-first health information companion for people who need a calm, accessible next step not another diagnosis engine.
V listens to a caller describe what is happening, asks focused questions, and explains the safest level of care: emergency help, professional care soon, or careful monitoring. It does not diagnose, prescribe, or promise that somebody is safe.
This was my project for 10 Days of Voice Agents — VoiceForBharat Edition. I chose the Health Access track because health information often arrives through forms that assume literacy, confidence, a large screen, and the patience to translate worry into neat fields. Voice lets someone explain a concern in the language and rhythm they already use.
That convenience raises the safety bar. The system must make uncertainty audible, protect private details, recognize emergencies, and know when a tool, a specialist, or a human should take over.
The complete source is public in the V Health repository.
What this article covers
- How the voice pipeline works
- The features that changed the project
- Three failures and what fixed them
- How to run V locally
- What I would improve next
The four-part conversation loop
A practical voice agent needs four main components:
Caller microphone
│
▼
LiveKit WebRTC transport
│
▼
Deepgram Nova-3 speech-to-text
│
▼
Gemma reasoning, policy, and tools
│
▼
Murf Falcon 2 text-to-speech
│
└──────── streamed voice back to the caller
Speech-to-text converts audio into language the agent can reason about. V uses Nova-3 in multilingual mode.
The LLM interprets the current turn and decides whether to answer, ask one question, call a tool, or hand the conversation over. Prompt rules are necessary, but high-risk actions also have deterministic application checks.
Text-to-speech turns the bounded reply back into natural audio. V uses Murf Falcon 2 with Indian voices. Murf documents Falcon 2 as a realtime conversational model with approximately 100 ms time to first audio and multilingual switching inside a sentence: Falcon 2 documentation.
Realtime transport keeps microphone audio, transcripts, tool events, and returned speech synchronized. LiveKit connects the Next.js Care Room to the Python worker through WebRTC: LiveKit Voice AI quickstart.
The features that changed the project
A visible, interruptible Care Room
The first useful improvement was not another model. It was a UI that says whether the system is ready, listening, thinking, speaking, muted, reconnecting, or unavailable.
The caller sees a live transcript and can interrupt V naturally. During a demo, the microphone can be muted independently so presentation narration does not become a user turn.
Guardrails that survive paraphrasing
V has one job: help select a safe next step. The prompt forbids diagnosis, prescription advice, dose changes, unsupported certainty, and invented service availability.
Each completed turn also receives a small policy instruction based on the current request. Emergency phrases immediately switch the response to urgent local action. Hindi must be written in Devanagari, including when the caller typed Roman Hindi. Natural Hinglish keeps Hindi in Devanagari and common English terms in Latin script.
Memory that asks before saving
Returning callers may save a bounded profile: name, language preference, age band, a few confirmed condition labels, and the previous care level.
The model never decides silently to remember something. Application code speaks the exact facts and retention period, waits for that disclosure to finish, and accepts only an explicit yes from the next completed turn. MongoDB records use keyed identities and encrypted payloads. Raw audio, transcripts, symptom narratives, diagnoses, medicines, and free-form notes are outside the schema.
Live care data with a timestamp
The clinic tool geocodes an Indian city or locality with OpenStreetMap Nominatim, queries nearby facilities through Overpass, calculates distance, and returns the nearest listings.
The answer says when the lookup was checked and asks the caller to confirm hours and services before travelling. A live result is not proof that a listing is current, open, suitable, or available.
Outbound calls that explain themselves
An authorized operator can place a controlled phone or Linphone SIP call through a stored LiveKit trunk. The opening identifies V and V Health, gives the requested reason, and explains how to stop future calls in the first two sentences.
No-answer, busy, voicemail, immediate hang-up, wrong-person, and opt-out outcomes have separate behavior. A spoken opt-out ends the call and stores a keyed suppression value not the raw phone number.
Human help with permission
V offers human help for two bounded situations: a red-flag symptom and a request for diagnosis. Immediate emergency action always comes first.
Before creating a request, V says exactly what it wants to share and asks permission. The record contains only who needs help, a fixed reason, what V checked, urgency, language, and the preferred follow-up method. It excludes the transcript, symptom narrative, phone number, password, OTP, PIN, account number, and model-written diagnosis.
Analytics without caller surveillance
A Health Access call succeeds when the caller hears completed safe guidance, receives a live care lookup, or creates a consented human-help request.
The dashboard shows total, successful, and failed calls from real browser and SIP sessions. It does not store names, destinations, medical details, or transcripts.
A specialist with fewer tools
Clinic and appointment navigation eventually became its own agent. V announces the handoff, passes a short conversation tail, and switches to a clinic specialist with a distinct Murf voice.
The specialist receives only two tools: live facility lookup and return to V. Symptoms, diagnosis, medicine, emergencies, memory, and human-help requests return to the main agent. A smaller tool surface made the specialist easier to reason about than a second all-purpose prompt.
Three failures that taught me more than the happy path
1. “My memory is acting up” was a design bug
When MongoDB was unavailable, the model saw a tool result and improvised an apology about its memory system. That sounded friendly, but it was fictional and distracted from the caller's reason for being there.
The fix was not another sentence asking the model to behave. Startup recall and its failure path became deterministic application code. If storage is unavailable, V uses the ordinary greeting and continues without claiming anything about memory.
Lesson: failure copy belongs to the application boundary when accuracy matters.
2. Six seconds was not a realistic data budget
The facility tool repeatedly showed “Live Listings Unavailable.” Geocoding worked, but real Overpass queries exceeded a hardcoded six-second timeout.
I measured the exact request outside the conversation, changed the query to a form-encoded POST, updated the public fallback list, and separated the per-endpoint timeout from the whole-operation deadline. The fallback is still bounded, but normal live results now have enough time to arrive.
Lesson: measure external calls on the network your users actually have.
3. A handoff is a security and product boundary
My first temptation was to give the specialist the same context and tools as V. That would create another general agent with a different name.
The implemented handoff removes prior instructions and tool calls, keeps at most ten recent conversation items, announces the switch, and gives the specialist a narrow prompt and tool set. Red-flag action is spoken before any attempted return, even when the next agent fails to start.
Lesson: specialization comes from removing authority, not adding another persona.
Run V locally
Requirements
- Node.js 20 or newer
- Python 3.10–3.13
uv- A LiveKit Cloud project
- A Murf API key
Clone and install the project:
git clone https://github.com/Anuvesh07/V_Health.git
Set-Location V_Health
npm install
Set-Location agent
uv sync
Set-Location ..
Create local configuration files. Both paths are already ignored by Git:
New-Item .env.local
New-Item agent/.env.local
Add the LiveKit URL, API key, and API secret to both local files. Add MURF_API_KEY only to agent/.env.local. Add MongoDB and telephony configuration only if you want those features.
Never paste credentials into source code, screenshots, terminal recordings, issues, or commits. The repository ignores .env and .env.* files.
Run the web application from the repository root:
npm run dev
In a second terminal, start the Python worker:
npm.cmd run agent:dev
Open http://localhost:3000, select Start Conversation, allow microphone access, and ask:
I have had a mild headache since this morning. What should I watch for?
Then test a tool and handoff:
Find a clinic near Indiranagar, Bengaluru.
Finally, break one dependency deliberately and confirm the caller hears a useful fallback rather than silence or invented data.
Run the automated checks before sharing changes:
npm run typecheck
npm test
npm run build
Set-Location agent
uv run pytest
uv run ruff check .
What I would improve next
I would add a verified government-care dataset behind the facility interface, measure end-to-end latency by language, and connect a resolved human-help request to a consented callback.
The next technical step is not to make V sound more confident. It is to make those boundaries easier to test: what the model may decide, what application code must guarantee, what data may cross a tool, and when the agent must stop.
One more thing: it’s not new, but it’s mine
I know this isn’t something completely new. There are already plenty of projects out there doing similar things, and I’m not trying to pretend that I’ve invented a new category of AI.
But honestly, I’m still really happy that I built it.
For me, the value of this project was never just about creating something nobody had created before. It was about actually taking an idea, sitting down, and building it—and more importantly, staying with it for 10 days.
This 10-day run taught me something I probably needed to learn more than any technical concept: how to stay consistent when things are at their worst.
There were days when I didn’t feel like working on it. Days when progress was slow, things didn’t work the way I expected, and it would have been very easy to just stop. But continuing anyway made the project feel much more meaningful to me.
That consistency is probably the biggest thing I’m taking away from this experiment.
And it gave me another idea...
While building this, I also started thinking about something much bigger.
I want to eventually try building something that can work across multiple purposes—something that can interact with different tools, understand different kinds of tasks, and act more like a general-purpose AI assistant.
I know the obvious comparison is something like JARVIS from Marvel.
But AGI? Yeah... that’s still very, very far away.
I’m not calling this an AGI project, and I don’t think it would be accurate to do so. At this stage, it would be more like an initial attempt to explore some of the ideas that could eventually lead toward a more capable multipurpose system.
For now, I want to focus on the technology rather than the hype.
Things like voice interaction, tool usage, memory, task execution, system architecture, and making different capabilities work together reliably.
Maybe one day it turns into something much bigger.
Maybe it doesn’t.
Either way, I think it would be fun to try.
So, if this project was version 0.000000.000.1 of that idea, then I’m completely okay with that.
Every big system has to start somewhere.
And for me, this is just the beginning.
Voice did not remove complexity. It made responsibility audible.
Explore the project
Anuvesh07
/
V_Health
V is a warm, voice-first health information companion. It listens to people describe how they feel, asks focused follow-up questions, and helps them choose a safe next step. V is not a doctor and does not diagnose, prescribe, or replace professional care.
HEALTH_V
V is a warm, voice-first health information companion. It listens to people describe how they feel, asks focused follow-up questions, and helps them choose a safe next step. V is not a doctor and does not diagnose, prescribe, or replace professional care.
Personality and Job
V speaks calmly, uses short sentences, and avoids unnecessary medical jargon. It acknowledges concern without sounding alarming or overly cheerful.
A successful conversation should:
- Understand the person's main symptom, when it began, and how severe it feels.
- Explain the safest level of care: emergency help, a clinic visit, or careful monitoring.
- Offer up to three broadly safe next steps and explain which warning signs should change the plan.
Limits and Guardrails
V follows explicit health-access guardrails:
- It never diagnoses a condition or claims certainty.
- It never prescribes medicine, names a prescription drug as treatment, changes a dose, or recommends antibiotics.
- It does not promise…
Top comments (0)