From Rakshika to Doctor Gurleen: Building a Voice AI That Hands You Off, Not On Hold

Aryan patil — Sat, 15 Aug 2026 15:36:44 +0000

Rakshika: An AI Voice Agent That Doesn't Put You on Hold During a Disaster

Built for 10 Days of Voice Agents — VoiceForBharat Edition

The Problem, and Who It's For
What Rakshika Actually Does
How the System Works
Indian Voice, Hindi/Hinglish, and Safety Guardrails
Bringing In a Specialist: Doctor Gurleen
Memory, Escalation, Outbound Calls, and the Dashboard
The 10-Day Build Journey
Troubleshooting Notes for Fellow Builders
Build It Yourself
What's Next
Links

1. The Problem, and Who It's For

During a disaster — a flood, a cyclone, an earthquake — the phone lines that are supposed to help people are exactly the ones that get overwhelmed first. Everyone in the affected area is trying to reach the same emergency number at the same time, in a language and tone that doesn't always match a rushed, overworked human operator on the other end.

Two things tend to go wrong at once: capacity (too many callers, not enough responders) and language (panicked callers naturally code-mix Hindi and English mid-sentence — they don't switch into formal, textbook language just because they're on a phone call).

Rakshika is built for exactly this moment. It's for:

Callers who are more comfortable speaking Hindi or Hinglish than typing into an app
People in semi-urban and rural areas where code-mixed speech is simply how people talk, not a fallback
Anyone calling under stress, who cannot navigate a menu tree ("press 1 for shelter, press 2 for medical...") while something urgent is happening
Human rescue coordinators, who need callers pre-triaged so they can focus their limited time on cases that genuinely need a human

Rakshika isn't meant to replace a real emergency helpline or a real doctor. It's meant to be the layer in front of them — always available, speaks the way the caller speaks, and hands over a clean, already-understood conversation the moment a human or a specialist needs to take over.

1.1 Why Voice, Specifically

A chat app would have been simpler to build and debug. But during an actual emergency, people's hands are full, their attention is scattered, and typing — in any language, under stress — is a much higher-friction ask than speaking. Voice is the interface that demands the least from the person using it, which is exactly why it fits a crisis tool.

2. What Rakshika Actually Does

Rakshika's landing screen — deliberately calm and uncluttered. It's upfront about scope: "For life-threatening emergencies, contact local emergency services directly. Rakshika provides general guidance only."

On a live call, Rakshika can:

Hold a natural, code-mixed Hindi/English conversation, in Devanagari script, using short and calm sentences
Look up live weather, active disaster alerts, and recent earthquake data for the caller's district through the get_emergency_status tool, so her guidance is grounded in real data rather than a guess
Remember returning callers — name, location, household size, mobility needs — through a SQLite-backed memory system, so a caller who reconnects doesn't have to explain everything again
Escalate to a human rescue team, but only after explicitly asking the caller's permission first
Place outbound calls through a SIP/Linphone integration, so the system can proactively reach people instead of only waiting for them to call in
Hand off medical questions to Doctor Gurleen, a dedicated specialist agent, mid-call — without the caller repeating a single word of what they already told Rakshika

Here's a condensed sense of what an actual call sounds like:

Caller: "Rakshika, mere area mein bahot tez baarish ho rahi hai, kya koi alert hai?"
Rakshika: "मैं अभी आपके इलाके की जानकारी देख रही हूँ. कृपया अपना शहर या ज़िला बताएं."
Caller: "Nashik"
Rakshika: "नासिक में अभी कोई गंभीर चेतावनी सक्रिय नहीं है, लेकिन बारिश तेज़ है. कृपया निचले इलाकों से दूर रहें और मुझे बताएं अगर पानी बढ़ने लगे."

The design principle behind all of it: a caller in an emergency should never have to wait, never have to navigate a menu, and never have to explain their situation twice.

3. How the System Works

Rakshika runs on the LiveKit Agents SDK, which handles the real-time audio transport so the agent code itself can focus on reasoning, tool calls, and conversation logic rather than the plumbing of getting audio in and out reliably.

Audio flows left to right through the STT → LLM → TTS pipeline. The LLM decides both tool calls and specialist handoffs; a parallel data channel keeps the dashboard and voice orb in sync in real time.

3.1 Voice Activity Detection — Silero VAD

Turn-taking is one of the most underrated problems in voice AI. If the agent doesn't know precisely when the caller has stopped speaking, it either interrupts them mid-sentence or leaves an awkward pause — both especially jarring when the caller is already stressed. Silero VAD flags speech-start and speech-end boundaries with low latency, which is what makes the pipeline feel like a phone call rather than a walkie-talkie exchange.

3.2 Speech-to-Text — Deepgram Nova-3

Set to Hindi, and critical for handling code-mixed speech correctly — a sentence like "mere ghar mein paani bhar gaya hai, please help jaldi" gets transcribed as spoken, without forcing it into a single language.

3.3 The LLM Layer — Google Gemini 3.5 Flash-Lite

Gemini sits at the center of the pipeline doing three jobs: understanding intent, deciding when to call a tool (weather/alerts lookup, escalation, specialist handoff) via function calling so the model never has to hallucinate live data, and generating the spoken response in natural, code-mixed Hindi following the persona and safety guardrails baked into the system prompt. Flash-Lite was chosen specifically for low latency — in a voice pipeline, every extra hundred milliseconds of "thinking time" is audible as dead air.

3.4 Text-to-Speech — Murf Falcon

Murf Falcon's response times are low enough that Rakshika's replies start playing almost immediately after Gemini finishes generating them, instead of a noticeable "processing" gap. She speaks in the Namrita voice.

3.5 The Frontend — Next.js 15

A custom animated SVG voice orb reflects the live call state — idle, listening, or speaking — via CSS-variable-driven colors, with a smooth transition instead of a hard cut. The same LiveKit room carries a data channel alongside the audio, which the backend uses to push live events (escalations, analytics, which agent is currently active) straight to the frontend with no polling.

3.6 Storage — SQLite

A local database handles two jobs: caller memory (name, location, and prior-call context, so returning callers are recognised instantly) and escalation/outcome tracking (every escalation and every call's success/failure flag, which is what powers the dashboard in Section 6).

3.7 A Simplified Session Setup

A trimmed-down look at how the agent session ties the STT/LLM/TTS/VAD stack together:

from livekit.agents import AgentSession, Agent
from livekit.plugins import deepgram, google, murf, silero

session = AgentSession(
    vad=silero.VAD.load(),
    stt=deepgram.STT(model="nova-3", language="hi"),
    llm=google.LLM(model="gemini-3.5-flash-lite"),
    tts=murf.TTS(voice="Namrita"),
)

await session.start(
    agent=Assistant(instructions=SYSTEM_PROMPT, user_identity=user_identity, ctx=ctx),
    room=ctx.room,
)

Everything downstream — tool calls, escalation creation, the specialist handoff — is layered on top of this session as @function_tool-decorated methods on the agent class.

4. Indian Voice, Hindi/Hinglish, and Safety Guardrails

Getting the language right wasn't a small detail bolted onto an English-first design — it was a first-class requirement from Day 1.

4.1 The Voice

Rakshika speaks in Murf Falcon's Namrita voice, chosen specifically because Falcon's latency is low enough that a reply doesn't feel like it's being "processed" before it plays. In a distress call, that pause is the fastest way to make the line feel untrustworthy.

4.2 Code-Mixing, Not Translation

Callers say things like "mere ghar mein paani bhar gaya hai, please help jaldi" — mixing Hindi and English naturally. Rakshika is instructed to understand and respond to this exactly as spoken, replying primarily in Hindi (Devanagari script) while letting common English words (hospital, ambulance, locality names) stay in English, the way people actually say them.

4.3 Why Script Matters, Not Just Language

The system prompt requires Rakshika to output responses in native Devanagari, not romanized Hindi ("Ghabraiye mat, main aapki madad ke liye yahaan hoon"). Romanization forces the TTS engine to guess at pronunciation instead of reading native-script text with correct phonetics — native Devanagari produces noticeably better pronunciation and intonation.

4.4 Prompt Engineering for Natural Speech

A few rules that made a real difference in how natural Rakshika sounds:

An explicit punctuation rule. The prompt requires English periods, commas, and question marks instead of the Hindi poorna viram (।) at the end of sentences — required for the streaming TTS to pace her speech correctly.
Short sentences. Long, clause-heavy sentences are harder to parse by ear, especially under stress.
Confirm before acting. Before escalating or handing off, Rakshika states what she's about to do first, so the caller isn't caught off guard by a sudden change.
Guardrails on scope. Rakshika always confirms the caller's safety status before secondary requests, and always asks explicit permission before escalating. Doctor Gurleen's guardrails are stricter still — she never names a specific diagnosis or recommends a specific medicine, and defaults immediately to "get in-person help" for anything serious or ambiguous.

5. Bringing In a Specialist: Doctor Gurleen

5.1 Why One Agent Wasn't Enough

By Day 8, Rakshika's single system prompt was carrying a lot — general safety triage, live disaster data, memory management, escalation. Asking it to also carry deep, careful medical guidance on top of all that risked making every one of those responsibilities shallower. Medical guidance specifically needed much stricter, narrower safety rules than general disaster triage — rules that didn't sit naturally in the same prompt as "check the weather" and "log a caller's address."

So Day 9 split the system into two agents:

Rakshika — the primary agent. General safety, live alerts, memory, escalation. Red UI theme.
Doctor Gurleen — a focused specialist, brought in only for injury/medical-symptom questions, with stricter safety rules than Rakshika carries. Blue UI theme.

5.2 How the Handoff Actually Works

@function_tool(
    description=(
        "Hand off the call to the Doctor Specialist. Use this ONLY when the "
        "caller describes an injury, medical symptom, health emergency, or "
        "explicitly asks for medical/first-aid advice. Do NOT use this for "
        "shelter questions, weather/disaster status, or general escalation."
    )
)
async def transfer_to_doctor_specialist(self, context: RunContext):
    await context.session.say(
        "मैं आपको हमारी मेडिकल स्पेशलिस्ट डॉक्टर से जोड़ती हूँ, जो आपकी मदद करेंगी.",
        allow_interruptions=False,
    )
    return (
        DoctorSpecialist(user_identity=self.user_identity, ctx=self.ctx),
        "Transferring caller to doctor specialist",
    )

Two things happen the instant a handoff triggers: the caller hears the transfer line spoken naturally, and the full conversation context carries over automatically to Doctor Gurleen — she never asks the caller to repeat themselves. If she later determines the question is non-medical, handoff_back_to_rakshika sends the caller back the same way, with a short role-context note so Rakshika also picks the thread back up correctly.

Here's what the handoff actually sounds like, mid-call:

Caller: "Mujhe chot lag gayi hai, khoon beh raha hai haath se"
Rakshika: "मैं आपको हमारी मेडिकल स्पेशलिस्ट डॉक्टर से जोड़ती हूँ, जो आपकी मदद करेंगी. कृपया लाइन पर बनी रहें."
(orb: red → connecting → blue)
Doctor Gurleen: "नमस्ते, मैं डॉक्टर गुरलीन हूँ और मैंने आपका हाल सुन लिया है, आप मुझे यह बताएं कि अभी खून बहना बंद हुआ या नहीं?"

Left: Rakshika mid-call, orb glowing red. Right: seconds later, the handoff complete — orb now blue, label switched to "DOCTOR GURLEEN," state pill now reading "LISTENING TO YOU."

5.3 Making the Handoff Feel Instant, Not Jarring

The handoff logic itself — a @function_tool returning a new specialist Agent instance — is a documented LiveKit pattern and wasn't the hard part. The hard part was timing. An early version fired the orb-color change the moment the handoff tool was called, which meant the orb turned blue while Rakshika was still mid-sentence saying "let me connect you." It looked like a glitch, not a clean transfer.

The fix was to sequence the events deliberately: the frontend shows a "Connecting..." state while Rakshika finishes speaking her transfer line, and the orb only commits to blue once Doctor Gurleen's own introduction actually begins — timed with a short delay against when the transfer speech finishes, rather than switching the instant the backend function returns.

async def broadcast_active_agent(ctx, agent_id: str) -> None:
    """agent_id: 'rakshika' | 'doctor' — pushes the orb state over the data channel."""
    payload = json.dumps({"type": "agent_switch", "data": {"agent": agent_id}}).encode("utf-8")
    await ctx.room.local_participant.publish_data(payload, reliable=True)

5.4 What I'd Do Differently

A fixed short delay works, but it's a blunt instrument. A cleaner long-term fix would be listening for a "TTS playback complete" event from the pipeline itself and triggering the orb swap off that, instead of a hand-tuned timer.

6. Memory, Escalation, Outbound Calls, and the Dashboard

6.1 Memory

A SQLite database (db.py) stores caller details — name, location, household size, mobility needs — but only after Rakshika explicitly asks permission to save them. On a caller's next connection, the agent looks up their identity and injects what's known into Rakshika's system prompt before she even speaks, so returning callers aren't starting from zero mid-disaster.

6.2 Human Escalation and the Dashboard

When a case is serious, Rakshika asks permission, then calls create_escalation, which logs the case. A live Next.js admin dashboard reads this and updates instantly, giving a human coordinator a structured case file — who needs help, what happened, and what Rakshika already checked and confirmed — instead of a raw transcript to sift through.

The live escalation feed. Each card shows a reference ID and urgency level, who needs help, what happened, and a note on what Rakshika already verified before escalating — e.g. "Verified caller location as College Road, Nashik and noted a major building fire due to short circuit with people trapped." A human responder acknowledges the request directly from here.

The caller-facing side has its own short status flow, so the person on the call knows help is actually being requested — not just told "someone will call you back":

Left: the moment an escalation triggers, the caller sees a live "Calling Rescue Team" card with the case reference and urgency. Right: once the team is alerted, it flips to "Request Dispatched" — a small piece of UI, but it closes the loop for a caller who has no way of knowing otherwise whether their emergency was actually acted on.

@function_tool()
async def create_escalation(self, context: RunContext, location: str, situation: str, urgency: str):
    ref_id = f"REQ-{random.randint(1000, 9999)}"
    db.log_escalation(ref_id=ref_id, location=location, situation=situation, urgency=urgency)
    payload = json.dumps({"type": "escalation", "data": {"ref_id": ref_id}}).encode("utf-8")
    await self.ctx.room.local_participant.publish_data(payload, reliable=True)
    return f"Escalation {ref_id} created and sent to the response team."

6.3 Outbound Calling

Using linphone_outbound.py, the backend can dial a real phone number over a SIP trunk and bridge that audio into a LiveKit room — the same agent logic that handles inbound calls handles outbound ones. When an outbound call connects, Rakshika doesn't wait for the person to speak first; she immediately opens with a distress-appropriate greeting.

6.4 Call Analytics

Every call gets a success/failure flag based on whether the caller dropped early or got real help, calculated the instant a participant_disconnected event fires — zero delay, no batch job.

The analytics panel at the top of the dashboard: total calls, successful outcomes, failed/dropped calls, and an overall success-rate figure, calculated live as calls come in.

7. The 10-Day Build Journey

Days 1–3 — Getting a voice pipeline working at all. agent.py set up around the LiveKit voice pipeline, Deepgram for Hindi STT, Murf Falcon (Namrita) for TTS, and the first version of Rakshika's persona — calm, short sentences, Hindi/Hinglish.

Day 4 — Memory. db.py and SQLite came in, along with save_caller_info and delete_caller_info, so Rakshika could ask permission to remember a caller and recall them automatically on reconnection.

Day 5 — Human escalation and the admin dashboard. create_escalation was added, with an explicit rule that Rakshika must ask permission first. The dashboard started reading live escalation data.

Day 6 — Outbound calling. Linphone/SIP integration let the backend dial real numbers and bridge audio into the room, plus logic to detect outbound rooms and open with a mandatory greeting instead of waiting.

Day 7 — Dashboard polish. UI refinements to the escalation cards and the live caller-facing "Calling Rescue Team → Request Dispatched" status flow.

Day 8 — Call analytics. Success/failure tracking, tied to the participant_disconnected event for instant sync, plus the analytics panel on the dashboard.

Day 9 — The multi-agent handoff. The hardest day of the build — splitting Rakshika into two agents and getting the Rakshika ↔ Doctor Gurleen handoff to feel seamless (see Section 5.3).

Day 10 — This post.

8. Troubleshooting Notes for Fellow Builders

A few lessons, condensed for anyone building something similar:

If your multi-agent handoff cuts off audio mid-sentence, check whether your agent swap is happening before the TTS has actually finished speaking. Sequencing a "connecting" state and delaying the swap until the handoff line finishes is often the fix.
If your STT is mangling code-mixed speech, check whether the model you're using actually supports multilingual/code-mixed transcription — many STT APIs default to a single-language mode that silently degrades on mixed speech instead of erroring out.
If your TTS pronunciation sounds "off" for Hindi, check whether you're feeding it romanized text. Native Devanagari script almost always produces noticeably better pronunciation than a transliterated string.
If you're using an AI coding agent for part of the build, always have it verify against the current state of your files before applying a change — a suggested patch based on a stale assumption of what a file contains will confidently point you at code that no longer exists.
Test with real, messy speech early. Clean, scripted test sentences make almost any voice pipeline look good. The bugs that matter only show up with genuine code-mixing, interruptions, and background noise.

9. Build It Yourself

Prerequisites

Python 3.10+
Node.js (for the frontend)
API keys for LiveKit, Murf AI, Deepgram, and Google Gemini

Step 1 — Clone the repo

git clone [ADD YOUR PUBLIC GITHUB REPO LINK HERE]
cd rakshika

Step 2 — Set environment variables

Create a .env.local file in both backend/ and frontend/. Never commit this file.

# backend/.env.local
LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=<your_livekit_key>
LIVEKIT_API_SECRET=<your_livekit_secret>
MURF_API_KEY=<your_murf_api_key>
DEEPGRAM_API_KEY=<your_deepgram_key>
GOOGLE_API_KEY=<your_google_gemini_key>
SIP_USERNAME=<your_sip_username>
SIP_PASSWORD=<your_sip_password>
SIP_DOMAIN=<your_sip_domain>

# frontend/.env.local
LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=<your_livekit_key>
LIVEKIT_API_SECRET=<your_livekit_secret>

Step 3 — Run the backend

cd backend
pip install -r requirements.txt
python src/agent.py dev

Step 4 — Run the frontend

cd frontend
npm install
npm run dev

Step 5 — Talk to Rakshika

Open http://localhost:3000, connect your microphone, and start talking — in Hindi, English, or a natural mix of both.

A Quick Sanity-Check Flow

Say a simple greeting and confirm you get a spoken Hindi reply.
Ask about the weather/situation in a city to confirm get_emergency_status fires.
Describe an injury to confirm the handoff to Doctor Gurleen triggers, and the orb turns blue.
Say something indicating a serious emergency to confirm the escalation flow generates a reference ID and shows up on the dashboard.

If all four steps work, your local setup is wired up correctly.

10. What's Next

More specialist agents beyond Doctor Gurleen — a shelter-logistics specialist would let Rakshika stay even more focused
Expand beyond Hindi to other Indian languages
Move the handoff timing from a fixed delay to an event-driven "TTS playback complete" signal, for a more robust sync than a hand-tuned timer
Load-test the outbound calling flow for genuinely high call volume during an active disaster

11. Links

GitHub repository: https://github.com/aryxett/MurfAI
Built with Murf Falcon — the fastest TTS API I used in this build — for 10 Days of Voice Agents — VoiceForBharat Edition.

DEV Community: Aryan patil