How I built a multilingual AI disaster-response voice agent with memory, real-time weather data, telephony, human escalation, analytics, and specialist handoffs during the 10 Days of Voice Agents — VoiceForBharat Edition.
Introduction
What started as a simple voice agent quickly turned into something much more ambitious.
During the 10 Days of Voice Agents — VoiceForBharat Edition, I built Pooja, a real-time AI-powered disaster-response voice agent designed to help people during situations where information needs to be delivered quickly, naturally, and reliably.
The core idea was simple:
What if someone could call an AI assistant during a disaster, speak naturally in an Indian language, receive verified information, get connected to a human when necessary, and have the system remember important information across conversations?
That question became the foundation for Pooja.
Over the challenge, I progressively evolved the system from a basic conversational voice agent into a more complete architecture involving:
- Real-time voice communication
- Indian English and Hindi code-switching
- Murf Falcon for low-latency text-to-speech
- Gemini as the reasoning layer
- Deepgram for speech recognition
- LiveKit for real-time communication
- Persistent caller memory
- Live weather data
- Outbound phone calls
- Human escalation
- Slack notifications
- Call analytics
- A WebGL-powered frontend
- Multi-agent specialist handoffs
This article explains what I built, why I built it, the problems I encountered, and how the different components fit together.
1. The Problem I Wanted to Solve
During a disaster, people don't necessarily need another complicated application.
They need answers.
Questions such as:
- "Is there a flood warning in my district?"
- "What should I do right now?"
- "Where is the nearest shelter?"
- "I'm trapped. Can someone help me?"
These questions become particularly challenging when users are under stress, using mobile devices, have limited digital literacy, or prefer communicating in their native language.
That's where voice becomes valuable.
Instead of requiring someone to:
- Open a website
- Navigate through menus
- Read alerts
- Search for a shelter
- Fill out a form
they can simply speak to the system.
The goal of Pooja was therefore not to create a generic chatbot.
It was to create a voice-first disaster-response interface.
2. Why Voice?
Voice changes the interaction model completely.
A traditional application expects:
User
↓
Read
↓
Understand
↓
Navigate
↓
Click
↓
Submit
A voice agent can reduce that to:
User
↓
Speak
↓
AI understands
↓
AI responds
This becomes especially useful when the user may be:
- stressed
- moving
- unable to type
- unfamiliar with technology
- more comfortable speaking Hindi or another Indian language
For this reason, multilingual and code-mixed conversations became an important part of Pooja.
- The Core Architecture
At a high level, Pooja uses the following voice pipeline:
USER
│
▼
┌───────────┐
│ LiveKit │
│ Real-Time │
│ Transport │
└─────┬─────┘
│
▼
┌───────────┐
│ Deepgram │
│ STT │
└─────┬─────┘
│
▼
┌───────────┐
│ Gemini │
│ LLM │
└─────┬─────┘
│
┌───────────┼────────────┐
│ │ │
▼ ▼ ▼
Memory Weather Escalation
Tools Tool Tool
│ │ │
▼ ▼ ▼
SQLite Open-Meteo SQLite
│
▼
Slack
│
▼
┌───────────┐
│ Murf │
│ Falcon │
│ TTS │
└─────┬─────┘
│
▼
USER
For real phone calls, I extended the architecture further:
Pooja
↓
LiveKit Cloud
↓
SIP
↓
Twilio
↓
Real Phone Number
This allowed the project to move beyond a browser-only demonstration.
4. Giving the Agent a Personality
One of the first lessons I learned was that connecting an LLM to speech does not automatically create a good voice agent.
The agent needs a clear identity, objectives, boundaries, and communication style.
I structured Pooja's system prompt into operational sections:
- IDENTITY
- OBJECTIVES
- KNOWLEDGE
- LANGUAGE
- GUARDRAILS
- STYLE
For example:
Identity
Pooja is a disaster-response voice assistant.
Objectives
Her job is to:
- provide verified disaster-related information
- understand the caller's situation
- provide safe guidance
- escalate serious emergencies when required
Guardrails
Pooja must never:
- invent emergency information
- claim an evacuation order without verification
- fabricate rescue ETAs
- pretend a shelter is available without reliable data
- provide unsupported medical advice
This separation made the system prompt much easier to reason about and debug.
5. Multilingual and Code-Mixed Conversations
For an India-focused voice agent, supporting only English wasn't enough.
I wanted Pooja to naturally handle conversations such as:
"Mere area mein flood warning hai kya?"
or:
"Can you tell me ki nearest shelter kahan hai?"
The objective wasn't simply translation.
The agent needed to understand code-mixed speech and respond naturally in the appropriate language/register.
The pipeline therefore combines:
Deepgram STT → Gemini → Murf Falcon TTS
with instructions to preserve the user's language preference and conversational style.
This allowed Pooja to move between Indian English and Hindi naturally rather than forcing the user into a rigid language mode.
6. Murf Falcon for Real-Time Voice
The voice is a major part of the experience.
For this project, I used Murf Falcon as the text-to-speech layer through the LiveKit Murf integration.
The objective was low-latency, conversational speech rather than waiting for an entire response to be generated before playback began.
This made interactions feel much closer to a real conversation.
The voice configuration was also aligned with the project's India-focused use case.
7. Building Persistent Memory
A voice assistant that forgets everything after every call quickly becomes frustrating.
So I introduced persistent caller memory using SQLite.
The system stores information such as:
- user_id
- name
- language_preference
- location
- household information
- last interaction
The important architectural decision was:
The LLM doesn't directly manipulate the database.
Instead, it uses tools.
For example:
- lookup_caller()
- save_caller_info()
The flow becomes:
Caller
↓
Pooja
↓
Needs caller information?
↓
lookup_caller()
↓
SQLite
↓
Information returned
↓
Pooja continues conversation
This keeps database access inside controlled application code.
8. Consent and Safety Around Memory
Memory introduces another problem:
What information should an AI actually remember?
I added explicit guardrails around persistent information.
For sensitive information, the agent should not casually save everything it hears.
The system is designed to explain when information is being retained and why it is needed for the disaster-response workflow.
This was an important lesson for me:
Adding memory isn't just a database problem. It's also a trust and privacy problem.
9. Connecting Pooja to Live Data
A disaster-response agent cannot rely entirely on static knowledge.
For Day 5, I connected Pooja to the Open-Meteo API to retrieve live weather information.
I implemented a custom asynchronous tool:
get_district_alert()
The tool retrieves weather metrics such as:
precipitation
wind speed
short-term forecast information
timestamps associated with the data
The LLM decides when this tool is necessary based on the user's request.
For example:
"Is there a flood warning in my district?"
can trigger the weather lookup.
The important part is that the returned API data is transformed into a natural spoken response rather than being read out as raw JSON.
10. Handling API Failure Gracefully
One of the most important lessons from the project was:
A tool failing should never cause the voice agent to silently break or hallucinate an answer.
External APIs can fail.
They can:
- timeout
- return errors
- become unavailable
- hit rate limits
- return incomplete data
So I implemented explicit fallback handling.
If the weather API becomes unavailable, Pooja can say something like:
"I'm currently unable to reach the meteorological live feed, so I can't verify the latest information right now."
Instead of:
"There is no flood warning."
The difference is critical.
The agent must distinguish between:
"I don't know"
and:
"I verified that there isn't one."
- Solving an LLM Tool Loop
One particularly interesting debugging problem happened when the weather tool was introduced.
Under certain ambiguous user inputs, the LLM could repeatedly attempt to call the same tool.
That created an LLM tool loop, eventually contributing to API quota exhaustion.
The root problem wasn't simply the API.
It was the interaction between:
- tool descriptions
- system instructions
- ambiguous user input
- LLM decision-making
I refined the tool descriptions and system prompt to make the tool's intended usage much more explicit.
The lesson:
A function tool's description is part of the agent's control logic.
If the model doesn't understand when a tool should be used, the function implementation itself may be perfectly correct and the system can still behave incorrectly.
*12. Taking Pooja Beyond the Browser
*
A major milestone came on Day 6.
I migrated the system from a local LiveKit server to LiveKit Cloud and connected it to real telephony infrastructure.
The pipeline became:
Pooja
↓
LiveKit Cloud
↓
SIP Outbound Trunk
↓
Twilio
↓
Real Phone
I built an outbound call orchestration flow using the LiveKit Server API.
The system could:
- Create a unique LiveKit room
- Dispatch the agent
- Configure the outbound SIP participant
- Initiate the call
- Connect Pooja to a real phone number
This was the moment when the project stopped being only a browser demo.
Pooja could now actually make a phone call.
13. Context-Aware Outbound Calls
Outbound calls require a slightly different conversational strategy.
A user calling Pooja might start with:
"Hello, what can you help me with?"
But when Pooja initiates the call, she needs to proactively explain why she's calling.
I therefore introduced outbound/inbound context detection and separate greeting behavior.
This allowed the same agent architecture to support both:
- Inbound:
- Caller → Pooja
and:
- Outbound:
- Pooja → Caller
I also added handling for explicit "stop" requests so the agent would respect the caller's decision to end the interaction.
14. Knowing When AI Should Stop
This became one of the most important features of Pooja.
An AI disaster-response system should not try to solve every emergency itself.
For example:
"I'm trapped inside my house and the water is rising."
is fundamentally different from:
"What's the weather forecast?"
So I built a human escalation mechanism.
The architecture became:
Caller
↓
Pooja
↓
Emergency detected
↓
Ask permission
↓
Create escalation ticket
↓
Generate reference ID
↓
Notify human operator
15. Human-in-the-Loop Escalation
I created an escalation tool that generates a persistent ticket in SQLite.
The ticket captures useful operational information such as:
- who needs help
- what happened
- urgency
- relevant location information
- what Pooja already checked
- caller language/follow-up preference The caller receives a reference ID.
This means that when a human responder takes over, the caller doesn't necessarily have to repeat their entire situation.
16. Slack Integration for Human Operators
I also connected the escalation system to Slack using a webhook.
When an escalation is created:
Pooja
↓
SQLite Ticket
↓
Slack Webhook
↓
Human Operator
The operator receives a structured emergency alert containing the relevant information and reference ID.
This created a bridge between the AI system and an actual human workflow.
The objective was never:
"AI replaces the emergency operator."
It was:
"AI handles the first interaction and knows when to bring a human into the loop."
17. Measuring Whether Calls Actually Succeed
After building all these capabilities, another question became important:
How do I know whether Pooja is actually doing its job?
That's what led to the Day 8 analytics layer.
I introduced persistent call outcome tracking.
A call could be considered successful if, for example:
Pooja successfully delivered verified information
Pooja successfully created a human-help request
Otherwise, the call could be recorded as unsuccessful depending on whether the defined objective was achieved.
The backend now records metrics such as:
- Total Calls
- Successful Calls
- Failed Calls
These values are derived from real interactions rather than hardcoded dashboard numbers.
18. Building the Analytics Dashboard
I built a lightweight Python analytics server exposing:
/api/stats
This acts as the bridge between SQLite and the Next.js dashboard.
The frontend then retrieves the actual call statistics dynamically.
I also spent time making the dashboard feel like a proper product rather than a plain admin page.
The dashboard includes:
- WebGL shader background
- Ruby Red visual identity
- Glassmorphism cards
- GSAP skeleton loading
- Staggered entrance animations
- Responsive layout
I also encountered an interesting frontend bug where GSAP cleanup was inadvertently removing React's inline styling.
Fixing that required separating the animation properties from the visual styling properties rather than allowing the animation lifecycle to manipulate both.
19. Introducing a Specialist Agent
By Day 9, Pooja had accumulated quite a lot of responsibilities.
Instead of making the main agent even larger, I introduced a specialist agent.
For the Disaster Response track, I created a:
Shelter Information Specialist
Its job is intentionally narrow:
Find and communicate verified shelter information relevant to the caller.
It can handle questions such as:
"Where is the nearest shelter?"
"Is there a shelter near me?"
"Does the shelter have capacity?"
But it doesn't need to understand everything Pooja understands.
20. Multi-Agent Handoff
The architecture now looks like:
POOJA
Main Agent
│
│
Shelter-related request
│
▼
┌─────────────────┐
│ Shelter │
│ Specialist │
└────────┬────────┘
│
▼
Shelter Lookup
Pooja uses a dedicated handoff tool when the user's request requires specialist knowledge.
The transition is explicit:
"I'll connect you to our shelter information specialist."
The specialist then introduces itself and continues the conversation.
Most importantly, the conversation context is preserved.
The caller doesn't have to explain everything again.
21. The Final Architecture
After the challenge, Pooja's architecture looks approximately like this:
┌─────────────────────┐
│ Caller │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ LiveKit │
│ Real-Time Transport│
└──────────┬──────────┘
│
Speech Input
│
▼
┌─────────────────────┐
│ Deepgram │
│ STT │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Gemini │
│ LLM │
└──────────┬──────────┘
│
┌────────────────────┼────────────────────┐
│ │ │
▼ ▼ ▼
Memory Tool Weather Tool Escalation
│ │ │
▼ ▼ ▼
SQLite Open-Meteo SQLite
│
▼
Slack
│
▼
┌─────────────────────┐
│ Shelter Specialist │
└─────────────────────┘
│
▼
Shelter Data
│
▼
┌─────────────────────┐
│ Murf Falcon │
│ TTS │
└──────────┬──────────┘
│
▼
Caller
And for real phone calls:
Pooja
↓
LiveKit Cloud
↓
SIP
↓
Twilio
↓
Real Phone
22. The Hardest Problems I Encountered
Building this wasn't a straight line from idea to working application.
Some of the most valuable lessons came from things that didn't work immediately.
22.1 Localhost vs Real Telephony
Initially, the agent worked in a local development environment.
Moving to real phone calls introduced an entirely different set of considerations:
- LiveKit Cloud
- SIP
- Twilio
- outbound trunks
- authentication
- room orchestration
- latency
The lesson was that a voice agent is not just an LLM + TTS.
The transport layer matters just as much.
22.2 Tool Loops and LLM Behavior
The weather tool taught me that a correctly implemented function can still cause problems if the LLM doesn't understand when it should call it.
Repeated tool calls resulted in quota exhaustion.
Improving the tool description and system-level instructions helped constrain the behavior.
The lesson:
Prompt engineering and tool engineering are interconnected.
22.3 External API Failures
Real APIs fail.
Instead of allowing failures to propagate into the conversation, I built explicit fallback paths.
That made Pooja behave more like a reliable system and less like a demo that assumes everything is always available.
22.4 Context Preservation During Agent Handoffs
Introducing a specialist created another challenge.
A new agent normally needs its own context.
But the caller shouldn't have to repeat:
"I'm in X district and I need a shelter because..."
every time control changes.
Passing the existing conversation context during the handoff allowed the specialist to continue naturally.
22.5 Frontend Animation and React State
The analytics dashboard also produced a subtle issue.
GSAP cleanup was interacting with React's inline styles.
The animations worked, but the cleanup phase could remove styling that the React component still depended on.
The solution was to separate:
Animation state
from:
Component visual state
This was a good reminder that polished UI work requires understanding the lifecycle of both libraries involved.
23. How to Build Your Own Voice Agent
If you want to build a similar system, the basic architecture is surprisingly approachable.
A real-time voice agent generally needs four major pieces:
Speech-to-Text
↓
LLM
↓
Text-to-Speech
↓
Real-Time Transport
For my project:
Deepgram → Gemini → Murf Falcon
│
▼
LiveKit
LiveKit handles the real-time communication layer while the AI services handle speech recognition, reasoning, and speech generation.
Once this basic pipeline works, you can progressively add:
- Memory
- Tools
- APIs
- Database
- Telephony
- Human Escalation
- Analytics
- Specialist Agents
The important thing is not to build everything at once.
Build the smallest working voice loop first.
Then add capabilities one at a time.
24. Basic Project Setup
Clone the project:
git clone https://github.com/pratyush06-aec/murf-livekit-starter.git
Create a Python virtual environment:
python -m venv .venv
Activate it on Windows:
.venv\Scripts\Activate.ps1
Install dependencies:
pip install -r requirements.txt
Then configure your environment variables.
For example:
`LIVEKIT_URL=...
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
DEEPGRAM_API_KEY=...
GOOGLE_API_KEY=...
MURF_API_KEY=...
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...`
Never commit these values to GitHub.
Add your environment file to .gitignore:
.env
.env.*
!.env.example
You can safely commit an example configuration:
LIVEKIT_URL=
LIVEKIT_API_KEY=
LIVEKIT_API_SECRET=
DEEPGRAM_API_KEY=
GOOGLE_API_KEY=
MURF_API_KEY=
25. Keeping Secrets Out of the Repository
This became especially important once the project involved:
LiveKit credentials
Gemini credentials
Deepgram credentials
Murf credentials
Twilio credentials
Slack webhooks
Never place these directly inside:
API_KEY = "your-secret-key"
Instead:
import os
API_KEY = os.getenv("MURF_API_KEY")
Keep the real values in your local environment or secure deployment environment.
If a credential is accidentally committed, rotating/revoking it is necessary. Simply deleting it from the latest commit is not sufficient because Git history may still contain it.
26. Running and Testing the Agent
Once the environment variables are configured, start the LiveKit agent according to the project's development command.
Then connect through the frontend or configured LiveKit client.
A useful first test is simply:
"Hello Pooja, can you hear me?"
Then progressively test:
Test 1 — General conversation
"What can you help me with?"
Test 2 — Weather tool
"What's the current weather situation in my district?"
Test 3 — Memory
"My name is Ramesh and I'm in Asansol."
Then reconnect and verify that the information can be retrieved appropriately.
Test 4 — Escalation
"I'm trapped and need urgent help."
Verify that Pooja asks for permission before creating the escalation.
Test 5 — Shelter specialist
"Where is the nearest emergency shelter?"
Verify the handoff.
Test 6 — Outbound call
Trigger the outbound flow and verify the call reaches the configured phone number.
Test 7 — Analytics
Complete a call and verify that the dashboard metrics change based on the actual interaction.
27. What I Learned During the Challenge
The biggest lesson from these 10 days wasn't a particular API or framework.
It was understanding that a useful voice agent is a system, not just a prompt.
A production-oriented architecture needs to think about:
Conversation
+
Latency
+
Memory
+
Tools
+
Data
+
Failures
+
Security
+
Human escalation
+
Observability
+
Specialization
Each layer solves a different problem.
And each layer introduces new failure modes.
28. From "Chatbot" to "Voice System"
At the beginning of the challenge, I was mainly thinking about:
How do I make an AI talk?
By the end, the question had completely changed.
I was thinking about:
How do I build a voice system that can understand, remember, act, fail safely, escalate, communicate with humans, make phone calls, and measure its own outcomes?
That's a much more interesting engineering problem.
*29. What's Next for Pooja?
*
The 10-day challenge gave me the foundation, but there is a lot more I want to explore.
Some potential next steps include:
- Integrating official government disaster datasets
- More accurate geolocation and nearest-shelter calculations
- Multi-region shelter capacity tracking
- SMS/WhatsApp-based emergency follow-ups
- More Indian language support
- Improved emergency classification
- Stronger authentication for operator dashboards
- Better observability and tracing
- More specialist agents
- More robust evaluation of voice-agent quality
- Production-grade deployment and monitoring
The ultimate goal would be to move Pooja from a challenge project toward a reliable voice-first disaster-response platform.
30. Final Thoughts
Building Pooja over these 10 days completely changed the way I think about voice AI.
What started as:
Speech → AI → Speech
eventually became:
Speech
↓
Real-Time Transport
↓
Speech Recognition
↓
LLM Reasoning
↓
Tools + Memory + Live Data
↓
Human Escalation / Specialist Handoff
↓
Analytics
↓
Natural Voice Response
And that's what made this challenge so valuable.
I didn't just learn how to connect an LLM to a TTS API.
I learned how to engineer around an AI model.
Link of Day 1 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-voiceai-ugcPost-7491217998469595138-EDNF/
Link of Day 2 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-livekit-ugcPost-7491575388628967424-Cg3l/
Link of Day 3 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-livekit-ugcPost-7491983933178331136-L4qs/
Link of Day 4 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-gemini-ugcPost-7492308868140703744-t7wA/
Link of Day 5 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-livekit-ugcPost-7492616307574620160-OYKx/
Link of Day 6 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_10daysofvoiceagentschallenge-voiceforbharat-ugcPost-7493022074911879169-c6Df/
Link of Day 7 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-livekit-ugcPost-7493372541093322753-B_Lw/
Link of Day 8 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_voiceforbharat-murfai-livekit-ugcPost-7493765730602065920-_PuB/
Link of Day 9 Demo: https://www.linkedin.com/posts/pratyush-dutta-221b94302_10daysofvoiceagentschallenge-voiceforbharat-ugcPost-7494137786820730881-j4Wj/
Top comments (0)