Prathamesh khairnar

Posted on Aug 15

Meet Rakshika — The AI That Picks Up When 112 Can't

#voiceforbharat #programming #murfai

The Problem, and Who It's For
What Rakshika Actually Does
How the System Works
Indian Voice + Code-Mixed Language, In Its Own Script
A Specialist Handoff
Outbound Calls, Human Escalation, and the Dashboard
The 10-Day Build Journey
Troubleshooting Notes for Fellow Builders
Build It Yourself
What's Next
Links & Resources

1. The Problem, and Who It's For

Picture a fairly ordinary monsoon night in a mid-sized Indian city. Water is rising faster than anyone expected. Someone's ground floor is flooding, a grandmother is stuck upstairs with a locked wheelchair ramp, and every neighbour is trying to call the same emergency number at the same time.

That number is usually 112, India's unified emergency helpline. It's a genuinely good piece of public infrastructure — but it was designed for a world where disasters are rare, isolated events. It was never designed for the moment when everyone in a five-kilometre radius tries to call it within the same twenty minutes.

When that happens, a few predictable things go wrong:

Lines get overwhelmed. Callers sit on hold during the exact minutes that matter most.
Language becomes a barrier. Not every operator is fluent in every regional dialect, and panicked callers don't naturally switch into formal, textbook Hindi or English — they code-mix, they stutter, they repeat themselves.
Triage is inconsistent. A rushed, overworked operator might not ask the right follow-up questions to figure out whether someone needs an ambulance, a rescue boat, or just directions to a shelter.
There's no persistent memory. If the same person calls back five minutes later because the situation has changed, they start the conversation from zero, all over again.

None of this is because the people running the helpline aren't trying. It's a scaling problem, and scaling problems are exactly what software is good at solving if it's built for the right audience.

Who Rakshika is for:

Elderly callers who are more comfortable speaking than typing, and more comfortable in Hindi than in formal English.
People in semi-urban and rural areas, where code-mixed Hindi-English (or "Hinglish") is simply how people talk day to day — not a compromise, just the default.
Anyone calling under stress, who doesn't have the mental bandwidth to navigate an IVR menu tree ("Press 1 for medical, press 2 for fire, press 3 for...") while water is literally rising around them.
Disaster response coordinators, who need a reliable, always-on first point of contact that pre-triages callers so human responders can focus on the cases that genuinely need a human in the loop.

The goal was never to replace NDRF operators. It was to build the layer in front of them — the layer that never puts anyone on hold, that speaks the way the caller speaks, and that hands over a clean, structured case file the moment a human needs to step in.

That's what Rakshika is.

2. What Rakshika Actually Does

Rakshika isn't a scripted IVR bot reading out FAQs in a flat, pre-recorded voice. She's a real-time conversational triage agent, built to behave the way a calm, well-trained first responder would behave on the other end of the line.

Here's a sense of what a real interaction looks like, condensed:

Caller: "Rakshika, mere ghar mein paani bhar gaya hai, please help jaldi"
Rakshika: "घबराइए मत, मैं आपकी मदद के लिए यहाँ हूँ। क्या आप अभी सुरक्षित हैं? क्या घर में कोई ऊँची जगह है जहाँ आप जा सकते हैं?"
Caller: "Haan, first floor pe ja sakte hai, par meri maa ko chalna mushkil hai"
Rakshika: "ठीक है, मैं आपके लिए नज़दीकी राहत टीम को सूचित कर रही हूँ। क्या मैं आपका पता ले सकती हूँ ताकि रेस्क्यू टीम पहुँच सके?"

Behind that seemingly simple exchange, a lot is happening. On any given call, Rakshika can:

Greet and remember callers. A local SQLite database looks up the caller by phone number the instant they connect, so a returning caller is greeted by name and Rakshika already has context on their earlier call, instead of starting the conversation from zero.
Check live weather and flood alerts for the caller's area using the Open-Meteo API, so her guidance ("it's safer to move to higher ground now" vs. "the rain is expected to ease in the next hour") is grounded in real data, not a guess.
Track nearby earthquakes in real time via the USGS API, within a configurable radius, in case a caller is reporting aftershocks or asking whether it's safe to re-enter a building.
Route callers to the right hospital based on the type of injury they describe — a burn case gets routed differently from a suspected fracture or a neurological emergency — using Google Maps / Nominatim to find the nearest facility with relevant capability.
Escalate to a human rescue team the moment someone indicates they're trapped, injured, or in immediate danger, generating a reference ID and logging a structured case file for responders instead of relying on a human transcribing a panicked phone call in real time.
Hand off to a specialist agent — Aarav — the moment the conversation moves into territory that deserves a dedicated expert, like detailed shelter and relief-camp logistics.

The underlying design principle was simple: a caller should never have to repeat themselves, never wait on hold, and never have to fight through a menu tree while something urgent is happening around them.

2.1 Why Voice, Specifically

It would have been easier, in a lot of ways, to build this as a chat app. Text is simpler to debug, cheaper to run, and doesn't have to deal with audio latency.

But during an actual emergency, people don't type. Their hands are full — carrying a child, holding a torch, bracing against a wall. Many of the people this is built for aren't comfortable typing fast in any language, let alone under stress. Voice is the interface that requires the least from the person using it, which is exactly why it's the right interface for a crisis tool.

2.2 Safety Guardrails

Because this is a disaster-response context, Rakshika's prompt is deliberately constrained:

She never gives medical instructions beyond basic first-aid-style guidance (e.g., "stay above the water line," "if you smell gas, do not use any switches") — anything requiring clinical judgment routes to human escalation.
She always confirms a caller's safety status before moving on to secondary requests (like shelter information), so the highest-priority information is captured first.
She's instructed to stay calm and repeat key instructions when a caller sounds panicked or the transcript shows signs of distress, rather than rushing through a script.

3. How the System Works

Rakshika runs on LiveKit's Agents Framework, which handles real-time audio transport and orchestration end-to-end, so the actual "agent" code can focus purely on reasoning and tool use rather than the plumbing of getting audio in and out reliably.

At a high level, every turn of the conversation flows through five stages:

Caller Audio
    ↓
Silero VAD        (detects when the user starts/stops speaking)
    ↓
Deepgram Nova-3    (STT — transcribes, handles code-mixed Hindi/English)
    ↓
Google Gemini 1.5 Flash   (LLM — reasoning, tool calls, response generation)
    ↓
Murf Falcon        (TTS — converts the response back into natural Hindi speech)
    ↓
Caller hears Rakshika's response

3.1 Voice Activity Detection — Silero VAD

Turn-taking is one of the most underrated problems in voice AI. If the agent doesn't know precisely when the caller has stopped speaking, it either interrupts them mid-sentence or leaves an awkward, unnatural pause before responding — both of which are especially jarring when the person on the other end is already stressed.

Silero VAD handles this by continuously analysing the incoming audio stream and flagging speech-start and speech-end boundaries with very low latency, which is what lets the whole pipeline feel like a real phone call instead of a walkie-talkie exchange.

3.2 Speech-to-Text — Deepgram Nova-3

This was one of the most important model choices in the entire project. Most STT engines are trained primarily on monolingual speech, which means they fall apart the moment a speaker naturally switches between Hindi and English mid-sentence — exactly the way most Indians actually talk.

Deepgram Nova-3 handles this code-mixing gracefully, transcribing a sentence like "mere ghar mein paani bhar gaya hai, please help jaldi" correctly, without trying to force it into a single language or mangling the English words phonetically.

3.3 The LLM Layer — Gemini 1.5 Flash

Gemini 1.5 Flash sits at the center of the pipeline, doing three jobs simultaneously:

Understanding intent — is this a safety-status update, a request for shelter info, a request for hospital routing, or an escalation?
Deciding when to call a tool — function calling is used for every external data lookup (weather, earthquakes, hospitals, escalation creation, handoff), so the model never has to hallucinate live data.
Generating the spoken response — in natural, code-mixed Hindi, following the persona and safety guardrails baked into the system prompt.

Flash was chosen specifically for its low latency relative to larger models — in a voice pipeline, every extra hundred milliseconds of "thinking time" is audible as dead air, and dead air is the fastest way to make an emergency line feel untrustworthy.

3.4 Text-to-Speech — Murf Falcon

Murf Falcon is, by a noticeable margin, the fastest TTS engine I tested during this build. In a voice pipeline, TTS latency is the single biggest lever on whether the conversation feels real — and Falcon's response times are low enough that Rakshika's replies start playing almost immediately after Gemini finishes generating them, instead of there being a noticeable "processing" gap.

3.5 The Frontend — Next.js

The frontend is a Next.js app that renders the live state of a call:

Whose turn it is to speak (caller vs. agent), shown as an animated waveform.
Which agent is currently active — Rakshika or Aarav — reflected instantly through a color theme change (red vs. green).
Any active escalation, shown as a prominent alert card with the generated reference ID.

It communicates with the backend through LiveKit's real-time data channels, so UI updates (like an agent handoff or a new escalation) arrive the moment they happen on the backend, with no polling.

3.6 Storage — SQLite

A local SQLite database handles two jobs:

Caller memory — phone number, name, and a summary of prior calls, so returning callers are recognised instantly.
Escalation and outcome tracking — every escalation gets a row with a reference ID, timestamp, and status, which is what powers the analytics dashboard described in Section 6.

3.7 A Simplified Session Setup

Here's a trimmed-down look at how a LiveKit agent session gets wired up, tying the STT/LLM/TTS/VAD stack together:

from livekit.agents import AgentSession, Agent
from livekit.plugins import deepgram, google, silero

session = AgentSession(
    vad=silero.VAD.load(),
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=google.LLM(model="gemini-1.5-flash"),
    tts=murf.TTS(voice_id="hi-IN-namrita"),
)

await session.start(
    agent=RakshikaAgent(),
    room=ctx.room,
)

Everything downstream — tool calls, the handoff logic, escalation creation — is layered on top of this session as function_tool-decorated methods on the agent class.

4. Indian Voice + Code-Mixed Language, In Its Own Script

A huge part of making this feel real, rather than like a translated bot bolted onto an English-first system, came down to getting the language and voice right — not as an afterthought, but as a first-class design constraint from day one.

4.1 The Voice

Rakshika speaks with Murf Falcon's hi-IN-namrita voice — a natural female Indian voice with warmth in the tone and none of the flat, robotic cadence older TTS engines are known for. The specialist agent, Aarav, uses the distinct male hi-IN-karan voice, so the two are instantly distinguishable by ear alone, even before the UI theme changes color. That mattered more than I expected — in early testing, users could tell who was speaking within the first half-second of the handoff, purely from the voice.

4.2 Code-Mixing, Handled Properly

Callers don't have to pick a language, and they're never asked to. Someone can say:

"मेरे घर में पानी भर गया है, please help जल्दी"

and Rakshika understands it exactly as spoken — no forcing the caller into pure Hindi, no forcing them into pure English. This works because of two deliberate choices:

On the input side: Deepgram Nova-3's multilingual mode transcribes code-mixed speech into a single coherent transcript, correctly preserving both the Hindi and English segments instead of trying to translate one into the other or dropping words it doesn't recognise.
On the output side: the system prompt explicitly instructs Gemini to respond primarily in Hindi (written in Devanagari script, not romanized "Hinglish") while allowing common English technical or place-name terms to stay in English where that's how people would naturally say them — words like "hospital," "ambulance," or a locality name.

A few more example exchanges, to show the range:

Caller: "Earthquake aaya kya abhi?"
Rakshika: "जी हाँ, कुछ मिनट पहले आपके क्षेत्र के पास हल्के झटके दर्ज हुए हैं। कृपया खुली जगह में रहें और इमारतों के पास खड़े न हों।"

Caller: "Mujhe nearest shelter chahiye, meri family ke saath"
Rakshika: "ठीक है, मैं आपको हमारे शेल्टर स्पेशलिस्ट आरव से जोड़ रही हूँ, वो आपको सबसे नज़दीकी राहत शिविर की पूरी जानकारी देंगे।"

4.3 Why Script Matters, Not Just Language

Early prototypes generated responses in romanized Hindi ("Ghabraiye mat, main aapki madad ke liye yahaan hoon") because it was simpler to get working with an English-first prompt. It technically worked, but it read — and more importantly, it sounded — subtly off, because romanization forces the TTS engine to guess at pronunciation instead of reading native Devanagari text with correct phonetics.

Switching the LLM's output to native Devanagari script fixed this immediately. Murf Falcon's pronunciation improved noticeably, intonation felt more natural, and — as a side benefit — the live transcript shown in the frontend UI became genuinely readable for Hindi speakers instead of looking like a half-transliterated mess.

4.4 Prompt Engineering for Natural Speech

A meaningful chunk of the "language" work wasn't about the STT/TTS models at all — it was prompt engineering. A few rules that made a measurable difference:

No markdown, ever. LLMs default to bullet points and bold text when listing information, which sounds absurd when read aloud (more on this in Section 5). The prompt explicitly forbids markdown formatting in any spoken response.
Short sentences. Long, clause-heavy sentences are harder to parse by ear, especially under stress. The prompt nudges the model toward shorter, simpler sentence structures.
Confirm before acting. Before triggering an escalation or handoff, Rakshika is instructed to briefly state what she's about to do ("मैं आपके लिए रेस्क्यू टीम को सूचित कर रही हूँ") so the caller isn't caught off guard by a sudden change.

5. A Specialist Handoff

5.1 Why One Agent Wasn't Enough

Early on, Rakshika tried to do everything herself — general safety triage, live weather and earthquake data, hospital routing, and shelter logistics, all inside one system prompt. It technically worked, but two problems showed up quickly:

The prompt got bloated, and the more responsibilities got crammed into it, the more the model's answers on any one topic got shallower and less reliable.
Shelter logistics specifically needed a different kind of precision — exact bed counts, contact persons, capacity numbers pulled from a structured CSV — that didn't sit naturally next to the more conversational, reassurance-heavy tone needed for general safety triage.

So I split the system into a proper multi-agent architecture:

Rakshika (Primary Triage Agent) — red UI theme, handles general safety, live alerts, and hospital routing.
Aarav (Shelter Specialist Agent) — green UI theme, takes over the moment a caller needs relief-camp details, querying a local shelters.csv for exact bed capacity and contact persons.

5.2 How the Handoff Actually Works

The handoff is fully two-way. If a caller asks Rakshika for shelter details, she hands off to Aarav. If that same caller then asks Aarav about the weather mid-conversation, he hands the call straight back to Rakshika — no dead ends, no "I can't help with that, please call back."

Two things happen simultaneously the instant a handoff triggers:

The frontend swaps its color theme (red ↔ green) and the visible agent name/voice indicator, driven by a LiveKit data-channel message sent the moment the transfer tool fires.
The full conversational context (chat_ctx) is passed along to the new agent, so the caller never has to repeat information they've already given — Aarav already knows the caller's name, location, and what's already been discussed.

5.3 The Bug That Nearly Broke It

On Day 9 of the build, the handoff was a mess. Two separate, ugly failure modes showed up back to back:

Failure mode 1 — the silent/double-speak bug. When Rakshika decided to hand off to Aarav, she was supposed to say a natural line like "मैं आपको हमारे शेल्टर स्पेशलिस्ट से जोड़ रही हूँ।" Instead, one of two things happened: either she went completely silent mid-transfer, or she said the handoff line twice in a row, stepping on her own audio.

Failure mode 2 — the robotic-CSV-reading bug. Once Aarav actually took over, he'd read shelter data straight out of the CSV file as if he were reading raw markdown out loud: "Asterisk Asterisk Rajiv Gandhi Camp Dash Total Capacity Two Hundred..." — technically correct information, delivered in a way that sounded like a broken text-to-speech demo from 2008.

Root cause and fix, part 1 — timing. The root cause of the silence/double-speak bug was that LiveKit's transfer call was abruptly killing the TTS stream mid-sentence before Rakshika had finished speaking her handoff line — the agent swap was happening faster than the audio could physically play out. The fix had two parts: I stopped hardcoding the handoff phrase inside the tool call itself and instead let the LLM generate and speak it naturally as part of its own conversational output, and I added a short asyncio.sleep(4.5) inside the transfer tool's execution to give the TTS engine enough breathing room to finish speaking before the agent context actually swapped underneath it.

Root cause and fix, part 2 — formatting. The robotic-reading bug came down to prompt design, not a model limitation. Aarav's original prompt didn't explicitly forbid markdown, so Gemini defaulted to its natural instinct to format structured data (like a CSV row) as a bullet list with bold headers — which is exactly what got read aloud, punctuation and all. Rewriting Aarav's system prompt to explicitly forbid markdown formatting, and instructing it to convert CSV rows into a single natural, conversational Hindi sentence, fixed the readout completely.

@function_tool()
async def transfer_to_shelter_specialist(self, context: RunContext):
    # 1. Trigger UI theme change on the frontend
    data = json.dumps({"type": "agent_transfer", "to": "shelter"}).encode("utf-8")
    self._room.local_participant.publish_data(data, reliable=True)

    # 2. Give the TTS engine time to finish speaking naturally,
    #    instead of the transfer cutting it off mid-sentence
    await asyncio.sleep(4.5)

    # 3. Hand off full context + control to Aarav
    return (
        ShelterSpecialistAgent(chat_ctx=new_ctx, room=self._room),
        "Transferring to shelter specialist Aarav",
    )

# Aarav's system prompt excerpt — the fix for the robotic-CSV bug
AARAV_SYSTEM_PROMPT = """
You are Aarav, a shelter information specialist.
When reading data from the shelters database:
- NEVER use markdown formatting (no asterisks, no bullet points, no bold text)
- Convert structured data into ONE natural, spoken Hindi sentence
- Speak the way a calm human coordinator would explain it over the phone
"""

5.4 What I'd Do Differently

If I were starting the multi-agent piece over, I'd design the handoff protocol — the explicit "finish speaking, then swap" sequencing — from day one, instead of discovering it as a bug on Day 9. A fixed sleep() duration works, but it's a blunt instrument; a cleaner long-term fix would be listening for a "TTS playback complete" event from the pipeline itself and triggering the swap off that, rather than a hardcoded timer tuned by trial and error.

6. Outbound Calls, Human Escalation, and the Dashboard

Rakshika isn't only a passive line that waits for someone to dial in — the same underlying agent logic is built to work in both directions, and to leave a clean paper trail behind every call.

6.1 Human Escalation

The moment a caller indicates they're trapped, seriously injured, or in immediate danger, Rakshika doesn't try to resolve it herself — she asks for the caller's permission to escalate, then triggers a create_escalation tool. This:

Generates a unique, human-readable reference ID (e.g. REQ-1234) that the caller can quote if they call back.
Writes the case — location, injury type, urgency level, and a summary of the conversation — to the SQLite database.
Pushes a red alert card to the frontend UI in real time, so a human rescue coordinator monitoring the dashboard sees the case appear the instant it's created, without needing to be actively listening to that specific call.

@function_tool()
async def create_escalation(
    self,
    context: RunContext,
    location: str,
    injury_type: str,
    urgency: str,
):
    ref_id = f"REQ-{random.randint(1000, 9999)}"
    db.log_escalation(
        ref_id=ref_id,
        location=location,
        injury_type=injury_type,
        urgency=urgency,
        caller_id=self.caller_id,
    )
    self._room.local_participant.publish_data(
        json.dumps({"type": "escalation", "ref_id": ref_id}).encode("utf-8"),
        reliable=True,
    )
    return f"Escalation {ref_id} created and sent to the response team."

The reference ID matters more than it might seem — during a real disaster, a caller might get disconnected and call back several times as the situation evolves. Being able to say "REQ-1234, meri situation change ho gayi hai" instantly ties the new call back to the existing case, instead of a coordinator having to piece the story together from scratch.

6.2 Outbound Calling

Because the entire system runs on LiveKit's real-time telephony integration, the same agent logic that handles an inbound call can be triggered to place an outbound call — the pipeline (VAD → STT → LLM → TTS) doesn't care which direction the call started in.

In practice, this opens up flows like:

Proactive check-ins on a caller who was flagged as high-risk in an earlier escalation, without waiting for them to call back.
Follow-ups on an already-logged case, confirming whether a rescue team successfully reached a location tied to a specific reference ID.
Batch outreach in a defined area, e.g. verifying whether households in a flood zone have already evacuated, driven off a list rather than one-by-one manual dialing.

This is the part of the system I'm least finished with — the inbound flow is far more battle-tested at this stage — but architecturally, it's a natural extension of the same agent rather than a separate system, which is exactly the point of building on LiveKit's agent framework instead of a bespoke telephony stack.

6.3 The Analytics Dashboard

Sitting on top of the SQLite call logs is a Next.js dashboard that gives a live, human-readable view into what the whole system is doing:

Active calls — which calls are currently in progress, and which agent (Rakshika or Aarav) is handling each one.
Escalation feed — every open and resolved escalation, with its reference ID, location, and urgency, sorted so the most urgent cases are impossible to miss.
Call outcomes over time — a running count of resolved vs. escalated vs. handed-off calls, giving a rescue coordination team a sense of overall load and how the system is being used.

The point of the dashboard is that nothing about this system is a black box. A human coordinator can see, at a glance, exactly what Rakshika and Aarav are doing across every single call, in real time — which matters enormously for a tool that's meant to sit in front of a real emergency response workflow, not replace human oversight of it.

7. The 10-Day Build Journey

Ten days sounds like a short runway for a multi-agent, multilingual voice system with live data integrations — and honestly, it was. Here's roughly how the build progressed, from a simple prompt to the system described above.

Days 1–3: Getting a Voice to Talk at All

The earliest version of this project was intentionally small: get any agent talking on a LiveKit room, with Murf Falcon as the TTS engine, and confirm the STT → LLM → TTS loop actually closes with acceptable latency. Most of the early effort went into wiring up the LiveKit Agents session correctly and picking the right STT model — this is where Deepgram Nova-3's code-mixed handling became a non-negotiable requirement rather than a nice-to-have, once early tests with a monolingual STT model started mangling half of every code-mixed sentence.

Days 4–5: Giving Rakshika a Personality and Guardrails

Once the pipeline worked, the focus shifted to prompt design — giving Rakshika a consistent, calm persona, and writing explicit safety guardrails (no clinical medical advice, always confirm safety status first, stay calm under distress signals in the transcript). This is also where the romanized-Hindi-vs-Devanagari-script issue described in Section 4.3 got discovered and fixed.

Days 6–7: Wiring Up Live Tools

With the persona solid, I added the function-calling tools one at a time: weather/flood alerts via Open-Meteo, earthquake tracking via USGS, and hospital routing via Google Maps/Nominatim. Each tool integration surfaced its own small debugging cycle — mostly around making sure Gemini called the right tool at the right point in the conversation, rather than either over-calling tools for information the caller hadn't asked for, or under-calling them and guessing at data it didn't actually have.

Day 8: Memory and Escalation

SQLite-backed caller memory went in next, followed by the human escalation flow — the create_escalation tool, reference ID generation, and the first version of the frontend alert card. This is also roughly when the first version of the analytics dashboard started taking shape, initially just as a raw table view of the SQLite escalation log.

Day 9: The Multi-Agent Handoff (and Its Bugs)

This was the hardest day of the build by a wide margin. Splitting Rakshika into a two-agent system (Rakshika + Aarav) meant re-architecting how context got passed between agents, and it's where the handoff bugs described in Section 5.3 showed up — the silent/double-speak issue and the robotic CSV read-out. Both got fixed on the same day, but it took most of the day to properly diagnose that the two issues had completely different root causes (a timing race condition vs. a prompt formatting gap) rather than one shared bug.

Day 10: Polish, Outbound Calling, and Writing This Up

The final day went into rounding out the outbound-calling capability, tightening up the dashboard, and — as required by the challenge — writing this post and preparing the LinkedIn submission.

8. Troubleshooting Notes for Fellow Builders

A few lessons, condensed for anyone building something similar:

If your multi-agent handoff cuts off audio mid-sentence, check whether your transfer call is killing the TTS stream synchronously. A short, deliberate delay (or better, waiting on a playback-complete event) before actually swapping the agent context is often the fix.
If your agent reads structured data (CSV, JSON, database rows) like a robot, it's almost never a TTS problem — it's a prompt problem. Explicitly forbid markdown formatting and instruct the model to convert structured data into a single natural sentence before it ever reaches the TTS engine.
If your STT is mangling code-mixed speech, check whether the model you're using actually supports multilingual/code-mixed transcription mode — many STT APIs default to a single-language mode that silently degrades on mixed speech instead of erroring out, which makes the bug much harder to notice.
If your TTS pronunciation sounds "off" for Hindi, check whether you're feeding it romanized text. Native-script text (Devanagari, in this case) almost always produces noticeably better pronunciation than a transliterated string.
Test with real, messy speech early. Clean, scripted test sentences will make almost any voice pipeline look good. The bugs that matter only show up with genuine code-mixing, interruptions, and background noise.

9. Build It Yourself

If you want to run this project yourself, here's how to get started.

Prerequisites

Python 3.10+
Node.js (for the frontend)
API keys for LiveKit, Murf AI, Deepgram, and Google Gemini

Step 1 — Clone the repo

git clone https://github.com/Prathameshkhairnarr/Murf-Ai.git
cd Murf-Ai

Step 2 — Set environment variables

Create a .env.local file in both the backend/ and frontend/ directories. Never commit this file — it holds live API credentials.

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
MURF_API_KEY=your_murf_key
DEEPGRAM_API_KEY=your_deepgram_key
GOOGLE_API_KEY=your_gemini_key

Step 3 — Run the backend

This project uses uv for fast, reliable Python dependency management.

cd backend
uv sync
uv run python src/agent.py dev

Step 4 — Run the frontend

cd frontend
pnpm install
pnpm dev

Step 5 — Talk to Rakshika

Open http://localhost:3000, connect your microphone, and start talking — in Hindi, English, or a natural mix of both.

A Quick Sanity-Check Conversation

If you want a fast way to confirm your local setup is working end-to-end, try this test flow:

Say something simple like "Hello Rakshika" and confirm you get a spoken greeting back.
Ask a weather-related question ("mausam kaisa hai") to confirm the Open-Meteo tool call is firing.
Ask for shelter information to confirm the handoff to Aarav triggers correctly, including the UI color change.
Say something indicating an emergency to confirm the escalation flow generates a reference ID and shows the alert card on the dashboard.

If all four steps work, your local environment is wired up correctly.

10. What's Next

This project is nowhere near finished — ten days was enough to prove the architecture works, not enough to make it production-ready for a real disaster response deployment. The next iteration is focused on:

More Indian languages beyond Hindi — Marathi and Tamil are next on the list, given how regionally concentrated a lot of India's flood and earthquake risk is.
A wider, more realistic shelter and hospital dataset, replacing the local CSV/lookup approach with something closer to a live, continuously-updated government data source.
A battle-tested outbound-calling flow, with proper rate limiting and prioritization logic for proactive check-ins during an active disaster, rather than the current early-stage implementation.
A more robust handoff protocol, replacing the fixed sleep() timing fix from Day 9 with an event-driven "TTS playback complete" signal, as noted in Section 5.4.
Load testing to understand how the system behaves under genuinely high concurrent call volume — the scenario this entire project exists to solve for.

11. Links & Resources

🐙 GitHub Repository: Prathameshkhairnarr/Murf-Ai
🎙️ LiveKit Voice AI Quickstart: Docs
🦅 Murf Falcon Docs: Docs
🧩 Murf LiveKit Starter: GitHub

Building Rakshika was, without question, the most technically demanding thing I've built in this compressed a timeframe — going from a single text prompt to a fully conversational, multi-agent voice system that handles live data, human escalation, and outbound calling in ten days. A huge shoutout to Murf AI and LiveKit for hosting the 10 Days of Voice Agents challenge and for building tooling that made an ambitious scope like this actually achievable in the time given.

Table of Contents