Subhangi Dutta

Posted on Aug 22

The 6.2-Second Silence: Smashing a 'Silent Handoff' Bug in Real-Time Voice AI

#devchallenge #bugsmash #ai #python

Summer Bug Smash: Smash Stories 🐛🛹

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

I almost missed this one. It didn't throw an exception. It didn't fail a health check. Every service reported 200 OK. The only evidence was a number my stress-test dashboard quietly rolled over one night: 74% of simulated emergency callers hung up mid-transfer. Nothing crashed. Nothing errored. The system just went silent at the exact moment someone needed it most.

This is the story of how a bug that never once threw an error nearly sank my pilot rollout three days before launch — and what it taught me about the difference between a service being up and a system actually being alive.

The Backstory: Building Sentinel

During natural disasters like flash floods, emergency dispatch lines get overwhelmed within minutes. Traditional interactive voice systems (IVR) fail callers because navigating rigid keypads while trapped in rising water is impossible.

To address this, I built Sentinel—an autonomous, real-time emergency voice AI dispatcher, as a solo hackathon project. Sentinel handles concurrent inbound emergency calls, conducts immediate triage (extracting location, injuries, and flood depth), redacts personally identifiable information (PII), and dynamically delegates specialized routing through WebRTC pipelines.

The underlying stack integrates:

Deepgram Nova-3 for ultra-low latency Speech-to-Text (STT) streaming.
Gemini 1.5 Pro as the core intelligence orchestrator.
LiveKit Agents Framework for WebRTC audio transport and multi-agent coordination.
Murf Falcon 2 for hyper-realistic voice synthesis (TTS).

In an emergency pipeline, latency is not just a user experience metric—it is life-critical. A caller who hangs up assuming the line dropped doesn't call back immediately — in a flood, they go looking for another way out instead. That's the failure mode I was testing against.

The Bug: When Silence Became the Failure Mode

I was three days from a planned pilot rollout with a regional disaster-response team, running concurrency load tests to simulate a flash-flood surge — dozens of overlapping calls hitting Sentinel at once. That's when the abandonment number showed up: over 74% of simulated emergency callers hung up within the first 10 seconds of a transfer.

Fig 1: Initial telemetry capturing 24 failed calls out of 32 during the flood-surge load test—the majority logging: "Caller hung up without responding to Sentinel's opening statement."

When an inbound caller needed specialized shelter triage, the root TriageAgent recognized the intent and executed a delegation handoff to the ShelterSpecialistAgent:

# The delegation trigger inside Sentinel's core triage loop
if detected_intent == "shelter_dispatch":
    await ctx.handoff(
        to_agent=ShelterSpecialistAgent,
        transfer_context=caller_data,
    )

The handoff itself succeeded in the background, but the audio channel went completely dead.

For 6.2 seconds, absolute silence hung on the line. The caller—assuming the call had dropped in the middle of a disaster—hung up. Only after the caller had already disconnected did Sentinel's logs register a late, unprompted audio dispatch.

The Investigation & Root Cause

Debugging asynchronous audio streams across WebRTC connections is notoriously tricky — nothing was throwing, so standard error monitoring was useless. I turned to Sentry's trace spans across the STT → LLM → TTS pipeline, and every service completed cleanly and fast. The delegation handoff() call itself completed in milliseconds. The gap wasn't inside any span at all — it was in the six seconds of nothing between them, a silence no span boundary was built to measure.

That reframed the question I was even asking. I'd been debugging as if the only thing that mattered was did the request succeed? But for a voice system, the question that actually matters is did the caller hear something when they expected to? Every span said yes. The caller would have said no.

Fig 2: Sequence diagram illustrating the unprompted turn-taking standoff across agent lifecycles.

I isolated the root cause to an unprompted turn-taking standoff in the agent delegation lifecycle:

State Transfer Without Audio Activation: The TriageAgent completed its turn, transferred memory context, and relinquished session control to ShelterSpecialistAgent.
The Turn-Taking Deadlock:
- LiveKit's agent loop operates reactively: an agent processes input only after detecting a new VAD (Voice Activity Detection) turn from the user.
- The user, having just spoken their emergency request, was waiting for the assistant to acknowledge the transfer.
- The newly mounted ShelterSpecialistAgent was passively waiting for the user to speak first before generating an LLM response.
The 6-Second Timeout: The system only recovered after a 6-second inactivity timeout triggered an automatic fallback prompt—far too late for a high-stress emergency call.

My first instinct was to shorten the 6-second inactivity timeout — but that just meant the fallback prompt fired faster into an already-lost call. The real problem wasn't timeout tuning; it was that the agent had no reason to speak until the timeout forced it to.

The Fix & Implementation

To break the standoff, the incoming agent needed to claim conversational initiative immediately upon entering the session lifecycle, rather than waiting for an inbound audio frame.

I implemented proactive lifecycle synthesis using the on_enter() hook to immediately kick off speech synthesis with context-aware prompt injection:

from livekit.agents import VoiceAgent, AgentContext

class ShelterSpecialistAgent(VoiceAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    async def on_enter(self, ctx: AgentContext) -> None:
        """
        Fires immediately upon agent delegation handoff.
        Forces proactive speech synthesis to eliminate
        the unprompted turn standoff.
        """
        user_name = ctx.session_data.get("caller_name", "there")
        sector = ctx.session_data.get("sector", "your area")

        # Proactively construct greeting from incoming handoff context
        proactive_greeting = (
            f"I have your details, {user_name}. "
            f"I am actively tracking available rescue "
            f"shelters in {sector}. How can I direct you?"
        )

        # Immediately push synthesis to the WebRTC audio track
        await self.say(
            proactive_greeting,
            allow_interruptions=True,
        )

By binding the initial greeting synthesis directly to on_enter(), the agent pipeline starts streaming audio packets through Murf Falcon 2 the exact millisecond the WebRTC track ownership flips.

The Real-World Impact

Deploying this fix completely resolved the dead-air deadlock and transformed system performance under production load:

Metric	Before Fix (Deadlock State)	After Fix (`on_enter` Lifecycle)
Agent Handoff Latency	~6,240 ms	< 318 ms
Call Abandonment Rate	74.2%	< 2.8%
First-Turn TTS Synthesis	On Inactivity Fallback (~6s)	Instant (< 200ms TTFB)
Caller Triage Completion	25.8% Success Rate	94.6% Resolution Rate

Fig 3: The same dispatcher dashboard, one deploy later — resolved calls climbing steadily across active flood zones instead of stacking up as failures.

Lessons Learned for Voice AI Systems

Asynchronous Multi-Agent Systems Cannot Be Passive: Voice-to-voice agent handoffs require explicit turn management. In Sentinel's case, ShelterSpecialistAgent had no code path that fired on entry — only a path that reacted to input — so there was never a moment where it was actually going to speak first. Never assume an incoming agent should wait for user input; give it an explicit reason to act the instant it takes control.
Telemetry Must Measure Perceived Silence: Standard error-rate monitoring won't catch deadlocks where every service returns 200 OK. What finally exposed the gap wasn't a failed span — it was the absence of one. For voice systems, that means tracking things error monitoring normally ignores: time-to-first-byte on speech synthesis, time between agent handoff and first audio, total duration of dead air, and caller abandonment specifically during transitions — the signals that catch silence no individual service will ever report as a failure.

Building real-time voice agents is as much about managing silence and state transitions as it is about low latency. What I'd actually built was two separate handoffs stacked on top of each other: a state handoff, where session data and memory context move between agents, and a conversational handoff, where ownership of the next spoken turn moves between agents. I had engineered the first one carefully. I had assumed the second one would just happen. By shifting from passive turn-taking to proactive lifecycle hooks, Sentinel achieved sub-second, reliable emergency dispatch handoffs when every second matters most. That 74% abandonment rate is now under 3%, and the handoff that used to take 6.2 seconds now takes under a third of a second.

In mission-critical voice AI, eliminating dead air isn't just an optimization—it ensures the system responds the instant someone needs help.

DEV Community