DEV Community: Subhangi Dutta

The 6.2-Second Silence: Smashing a 'Silent Handoff' Bug in Real-Time Voice AI

Subhangi Dutta — Sat, 22 Aug 2026 07:45:13 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I almost missed this one. It didn't throw an exception. It didn't fail a health check. Every service reported 200 OK. The only evidence was a number my stress-test dashboard quietly rolled over one night: 74% of simulated emergency callers hung up mid-transfer. Nothing crashed. Nothing errored. The system just went silent at the exact moment someone needed it most.

This is the story of how a bug that never once threw an error nearly sank my pilot rollout three days before launch — and what it taught me about the difference between a service being up and a system actually being alive.

The Backstory: Building Sentinel

During natural disasters like flash floods, emergency dispatch lines get overwhelmed within minutes. Traditional interactive voice systems (IVR) fail callers because navigating rigid keypads while trapped in rising water is impossible.

To address this, I built Sentinel—an autonomous, real-time emergency voice AI dispatcher, as a solo hackathon project. Sentinel handles concurrent inbound emergency calls, conducts immediate triage (extracting location, injuries, and flood depth), redacts personally identifiable information (PII), and dynamically delegates specialized routing through WebRTC pipelines.

The underlying stack integrates:

Deepgram Nova-3 for ultra-low latency Speech-to-Text (STT) streaming.
Gemini 1.5 Pro as the core intelligence orchestrator.
LiveKit Agents Framework for WebRTC audio transport and multi-agent coordination.
Murf Falcon 2 for hyper-realistic voice synthesis (TTS).

In an emergency pipeline, latency is not just a user experience metric—it is life-critical. A caller who hangs up assuming the line dropped doesn't call back immediately — in a flood, they go looking for another way out instead. That's the failure mode I was testing against.

The Bug: When Silence Became the Failure Mode

I was three days from a planned pilot rollout with a regional disaster-response team, running concurrency load tests to simulate a flash-flood surge — dozens of overlapping calls hitting Sentinel at once. That's when the abandonment number showed up: over 74% of simulated emergency callers hung up within the first 10 seconds of a transfer.

Fig 1: Initial telemetry capturing 24 failed calls out of 32 during the flood-surge load test—the majority logging: "Caller hung up without responding to Sentinel's opening statement."

When an inbound caller needed specialized shelter triage, the root TriageAgent recognized the intent and executed a delegation handoff to the ShelterSpecialistAgent:

# The delegation trigger inside Sentinel's core triage loop
if detected_intent == "shelter_dispatch":
    await ctx.handoff(
        to_agent=ShelterSpecialistAgent,
        transfer_context=caller_data,
    )

The handoff itself succeeded in the background, but the audio channel went completely dead.

For 6.2 seconds, absolute silence hung on the line. The caller—assuming the call had dropped in the middle of a disaster—hung up. Only after the caller had already disconnected did Sentinel's logs register a late, unprompted audio dispatch.

The Investigation & Root Cause

Debugging asynchronous audio streams across WebRTC connections is notoriously tricky — nothing was throwing, so standard error monitoring was useless. I turned to Sentry's trace spans across the STT → LLM → TTS pipeline, and every service completed cleanly and fast. The delegation handoff() call itself completed in milliseconds. The gap wasn't inside any span at all — it was in the six seconds of nothing between them, a silence no span boundary was built to measure.

That reframed the question I was even asking. I'd been debugging as if the only thing that mattered was did the request succeed? But for a voice system, the question that actually matters is did the caller hear something when they expected to? Every span said yes. The caller would have said no.

Fig 2: Sequence diagram illustrating the unprompted turn-taking standoff across agent lifecycles.

I isolated the root cause to an unprompted turn-taking standoff in the agent delegation lifecycle:

State Transfer Without Audio Activation: The TriageAgent completed its turn, transferred memory context, and relinquished session control to ShelterSpecialistAgent.
The Turn-Taking Deadlock:
- LiveKit's agent loop operates reactively: an agent processes input only after detecting a new VAD (Voice Activity Detection) turn from the user.
- The user, having just spoken their emergency request, was waiting for the assistant to acknowledge the transfer.
- The newly mounted ShelterSpecialistAgent was passively waiting for the user to speak first before generating an LLM response.
The 6-Second Timeout: The system only recovered after a 6-second inactivity timeout triggered an automatic fallback prompt—far too late for a high-stress emergency call.

My first instinct was to shorten the 6-second inactivity timeout — but that just meant the fallback prompt fired faster into an already-lost call. The real problem wasn't timeout tuning; it was that the agent had no reason to speak until the timeout forced it to.

The Fix & Implementation

To break the standoff, the incoming agent needed to claim conversational initiative immediately upon entering the session lifecycle, rather than waiting for an inbound audio frame.

I implemented proactive lifecycle synthesis using the on_enter() hook to immediately kick off speech synthesis with context-aware prompt injection:

from livekit.agents import VoiceAgent, AgentContext

class ShelterSpecialistAgent(VoiceAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    async def on_enter(self, ctx: AgentContext) -> None:
        """
        Fires immediately upon agent delegation handoff.
        Forces proactive speech synthesis to eliminate
        the unprompted turn standoff.
        """
        user_name = ctx.session_data.get("caller_name", "there")
        sector = ctx.session_data.get("sector", "your area")

        # Proactively construct greeting from incoming handoff context
        proactive_greeting = (
            f"I have your details, {user_name}. "
            f"I am actively tracking available rescue "
            f"shelters in {sector}. How can I direct you?"
        )

        # Immediately push synthesis to the WebRTC audio track
        await self.say(
            proactive_greeting,
            allow_interruptions=True,
        )

By binding the initial greeting synthesis directly to on_enter(), the agent pipeline starts streaming audio packets through Murf Falcon 2 the exact millisecond the WebRTC track ownership flips.

The Real-World Impact

Deploying this fix completely resolved the dead-air deadlock and transformed system performance under production load:

Metric	Before Fix (Deadlock State)	After Fix (`on_enter` Lifecycle)
Agent Handoff Latency	~6,240 ms	< 318 ms
Call Abandonment Rate	74.2%	< 2.8%
First-Turn TTS Synthesis	On Inactivity Fallback (~6s)	Instant (< 200ms TTFB)
Caller Triage Completion	25.8% Success Rate	94.6% Resolution Rate

Fig 3: The same dispatcher dashboard, one deploy later — resolved calls climbing steadily across active flood zones instead of stacking up as failures.

Lessons Learned for Voice AI Systems

Asynchronous Multi-Agent Systems Cannot Be Passive: Voice-to-voice agent handoffs require explicit turn management. In Sentinel's case, ShelterSpecialistAgent had no code path that fired on entry — only a path that reacted to input — so there was never a moment where it was actually going to speak first. Never assume an incoming agent should wait for user input; give it an explicit reason to act the instant it takes control.
Telemetry Must Measure Perceived Silence: Standard error-rate monitoring won't catch deadlocks where every service returns 200 OK. What finally exposed the gap wasn't a failed span — it was the absence of one. For voice systems, that means tracking things error monitoring normally ignores: time-to-first-byte on speech synthesis, time between agent handoff and first audio, total duration of dead air, and caller abandonment specifically during transitions — the signals that catch silence no individual service will ever report as a failure.

Building real-time voice agents is as much about managing silence and state transitions as it is about low latency. What I'd actually built was two separate handoffs stacked on top of each other: a state handoff, where session data and memory context move between agents, and a conversational handoff, where ownership of the next spoken turn moves between agents. I had engineered the first one carefully. I had assumed the second one would just happen. By shifting from passive turn-taking to proactive lifecycle hooks, Sentinel achieved sub-second, reliable emergency dispatch handoffs when every second matters most. That 74% abandonment rate is now under 3%, and the handoff that used to take 6.2 seconds now takes under a third of a second.

In mission-critical voice AI, eliminating dead air isn't just an optimization—it ensures the system responds the instant someone needs help.

When the Water Rises: Building Sentinel AI Disaster Response in 10 Days

Subhangi Dutta — Sat, 15 Aug 2026 12:21:54 +0000

It was during the peak monsoon season when I read a distress report from a family stranded on their rooftop. Power was flickering, rain was hammering against the walls, and mobile data was down to a crawling 2G connection. They had a phone with 14% battery, but navigating complex portals or downloading 20-page government advisory PDFs was impossible.

All they could do was make a direct voice call.

When water is rising inside your living room, you cannot type or navigate dropdowns. You need an immediate, natural voice that provides real-time flood advisories, locates relief shelter beds, and dispatches rescue teams without delay.

That became the driving mission behind Sentinel — an autonomous, real-time Voice AI emergency dispatcher built over 10 days during the #VoiceForBharat challenge.

1. The Problem I Wanted to Solve

Emergency management during Indian monsoons faces three critical bottlenecks:

Information Fragmentation: Emergency guidelines, live rainfall alerts, and shelter capacities exist across disconnected departments. Callers need instant answers (e.g., "Where is the nearest shelter in Patna with medical support?").
The Friction of Touch Interfaces: Wet screens, low digital literacy, and high adrenaline make text UIs fail. Voice is the most accessible lifeline.
Context Drop During Escalation: Traditional hotlines often disconnect or force victims to repeat their distress details to multiple operators.

2. System Architecture & Data Flow

To ensure sub-second response times, Sentinel connects audio streaming, AI reasoning, and low-latency speech synthesis into a unified WebRTC pipeline powered by LiveKit, Deepgram Nova-3, Google Gemini, Murf AI, and Open-Meteo:

 Caller Voice (Browser WebRTC / Inbound SIP)
            │
            ▼
 ┌─────────────────────────────────────────────────────────┐
 │               LiveKit WebRTC Transport                  │
 │      (Audio Ingestion & Silero VAD Turn Detection)       │
 └──────────────────────────┬──────────────────────────────┘
                            │
                            ▼
 ┌─────────────────────────────────────────────────────────┐
 │               Deepgram Nova-3 (STT)                     │
 │          (Continuous Multilingual Streaming)           │
 └──────────────────────────┬──────────────────────────────┘
                            │ Real-time Transcript
                            ▼
 ┌─────────────────────────────────────────────────────────┐
 │          Google Gemini 3.5 Flash-Lite (LLM)             │
 │                                                         │
 │  • Emergency Triage & Safety Guardrails                 │
 │  • SQLite Caller Memory & State Tracking                │
 │  • Open-Meteo Weather Telemetry Tools                   │
 │  • Dynamic Specialist Agent Handoff                     │
 └──────────────────────────┬──────────────────────────────┘
                            │ Streaming Text Tokens
                            ▼
 ┌─────────────────────────────────────────────────────────┐
 │               Murf Falcon 2 (TTS)                       │
 │  (Ultra-Fast Indian Voice Stream: Anisha/hi-IN)         │
 └──────────────────────────┬──────────────────────────────┘
                            │
                            ▼
 Direct Audio Stream back to Caller (<500ms Time-to-First-Byte)

3. The Sentinel Voice Interface

Sentinel is designed with a lightweight, accessible interface featuring real-time Indian helpline integrations (112, NDRF 1078, Medical 108) and live ticker alerts for flood and drought monitoring across India.

Sentinel in ready state with live disaster telemetry feeds:

# Low-Latency Voice Pipeline Configuration
session = AgentSession(
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=google.LLM(model="gemini-3.5-flash-lite"),
    tts=murf.TTS(
        voice="Anisha",
        locale="hi-IN",
        style="Conversation",
        tokenizer=tokenize.basic.SentenceTokenizer(min_sentence_len=1),
        text_pacing=True,
    ),
    turn_detection=MultilingualModel(),
    vad=ctx.proc.userdata["vad"],
    preemptive_generation=False,
)

4. Real-Time Telemetry & Dispatch Dashboards

To make Sentinel production-ready for disaster management cells, I built two specialized real-time monitoring consoles:

A. Call Analytics & Success Engine

The Call Metrics Dashboard tracks total inbound WebRTC and SIP telephony calls, analyzing whether each interaction successfully delivered verified telemetry or required human intervention.

Outcome Verification: Automatically parses call logs and session duration.
Success Criteria Standard: Flags calls as successful only when verified relief shelter coordinates are provided or an escalation ticket is generated.

B. Live Emergency Dispatcher & PII Protection

When callers report being trapped or injured, Sentinel flags the situation as EMERGENCY urgency and generates a structured ticket.

Automated PII Redaction: Phone numbers, sensitive identification, and private contact info are masked automatically ([REDACTED_PHONE]) to protect caller privacy while giving rescue units necessary triage notes.
Status Lifecycle: Live sync tracks requests across Open, In Progress, and Resolved states with callback dispatch verification.

5. Specialist Handoffs: Shelter Information Agent

When a caller specifically requests shelter availability or medical resources, Sentinel delegates the session to a Shelter Information Specialist using dynamic agent handoffs:

class ShelterInformationSpecialist(Agent):
    def __init__(self, instructions: str, call_context: dict, location: str):
        super().__init__(instructions=instructions)
        self.location = location

    async def on_enter(self) -> None:
        shelter_data = disaster_data.compute_nearest_shelters(location=self.location)
        greeting = (
            f"Hello, I am the Shelter Information Specialist. "
            f"I have live capacity for {self.location}. {shelter_data} "
            f"Would you like details on pet policies, medical care, or check-in rules?"
        )
        await self.session.say(greeting, allow_interruptions=True)

6. Challenges & Solutions: The "Silent Handoff"

The Problem: When delegating from Sentinel Command to the Shelter Specialist, the previous turn concluded and the audio stream went silent, leaving the user waiting without feedback.
The Solution: Implemented the on_enter() lifecycle method in LiveKit Agents to trigger an immediate audio synthesis turn via session.say() using Murf Falcon the instant the handoff occurs, eliminating dead air.

7. How to Run Sentinel

You need Python 3.10 through 3.13, Node.js 18+, and pnpm 9. Clone the public repository then create local environment files:

cp backend/.env.example backend/.env.local
cp frontend/.env.example frontend/.env.local

Add these required backend values to backend/.env.local:

LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
MURF_API_KEY=your_murf_api_key
GOOGLE_API_KEY=your_gemini_api_key
DEEPGRAM_API_KEY=your_deepgram_api_key

Start the Services

# 1. Clone the repository
git clone https://github.com/subhangidutta23-ship-it/sentinel-voice-ai.git
cd sentinel-voice-ai

# 2. Start the Backend Worker
cd backend
.\.venv\Scripts\python.exe -m src.agent dev

# 3. Start the Frontend Console
cd ../frontend
pnpm install
pnpm dev

Open http://localhost:3000 in your browser to begin an emergency call session.

Thank you to Murf AI and the VoiceForBharat organizers for this 10-day challenge!