DEV Community: Lena Hoffmann

Barge-In, VAD, and the Latency Budget: Engineering Realtime Voice

Lena Hoffmann — Sat, 25 Jul 2026 08:09:14 +0000

Realtime voice is the only modality where being right is not good enough — you also have to be right now. A text model can take four seconds and nobody blinks. A voice agent that takes four seconds to start speaking sounds broken. I spent a chunk of last year wiring up bidirectional audio, and most of the hard problems had nothing to do with the model.

The latency budget is a sum, and you don't control most of it

The number users feel is time to first audible byte after they stop talking. That is not one number, it's a chain:

mic capture + encode    15-55 ms
network uplink RTT/2    20-150 ms   (mobile: worse, and variable)
VAD end-of-turn hold    200-700 ms  <-- you own this one
server queue + model    150-600 ms
first TTS chunk         50-300 ms
network downlink        20-150 ms
client jitter buffer    40-120 ms   <-- you own this one too

A "fast" pipeline lands between several hundred milliseconds and over a second. Human conversational turn gaps are famously short — around 200 ms in the conversation-analysis literature — which is why even good voice agents feel slightly like a satellite call.

The two terms you control are the end-of-turn hold and the jitter buffer, and both trade latency against correctness. Shrink the hold and you interrupt people mid-sentence. Shrink the jitter buffer and you get dropouts on flaky Wi-Fi. Instrument every segment before optimizing: almost every time I've been asked to "make the voice agent faster," the culprit was a fixed 500 ms silence timeout nobody had touched, not the model.

VAD is a classifier, so it has a confusion matrix

Voice activity detection sounds like a solved DSP problem until you deploy it. Energy-threshold VAD (frame RMS against an adaptive noise floor) is free and works fine in a quiet room. It falls apart with a TV, a fan, or a café. Neural VAD — Silero and WebRTC's are the two you'll encounter — is far more robust to non-speech noise, but costs a frame or two of lookahead and still can't distinguish your speech from a nearby speaker's.

The part people underestimate: VAD errors are asymmetric in cost.

False negative (you think the user is still talking): the agent sits there. Laggy, but recoverable.
False positive (you cut them off mid-thought): the agent barges in on a pause. Users hate this far more, and it poisons the transcript because you ship a truncated utterance to the model.

So the hold timer should not be a constant. A pause after "so I was thinking, um—" is grammatically incomplete; a pause after "what's the weather tomorrow" is a complete request. If your streaming ASR emits partial hypotheses, use that text as a weak end-of-turn signal and shorten the hold when the transcript looks complete:

def end_of_turn_hold(partial_text: str, pitch_falling: bool) -> int:
    hold = 640
    # Complete-looking utterance -> commit sooner.
    if looks_syntactically_complete(partial_text):
        hold -= 220
    # Falling terminal pitch is a turn-yielding cue in many languages.
    if pitch_falling:
        hold -= 120
    # Filled pauses mean "I'm not done" -> wait longer.
    if partial_text.rstrip().endswith(("um", "uh", "so", "and", "like")):
        hold += 300
    return max(260, hold)

Nothing fancy, and worth more than a better model.

Barge-in is a state machine problem, not an audio problem

The moment your agent can be interrupted, you have concurrency. The user starts talking while TTS audio is still in flight, and three buffers now hold stale audio: the server's generation queue, the network, and the client's playback buffer. Cancelling server-side does nothing if the client already has 800 ms queued — the agent keeps talking for almost a second after being interrupted, which reads as rude.

Cancellation has to be client-first, server-second:

on user_speech_detected(confidence):
  if state == SPEAKING and confidence > BARGE_IN_THRESHOLD:
    playback.flush()               # local, ~0 ms, do this FIRST
    transport.send({type: "cancel", turn_id: current_turn})
    state = LISTENING
    # Truncate history to what the user ACTUALLY heard.
    history.append(turn.truncate_to(playback.played_ms))

That last line is the one everyone forgets. If you log the full generated response but the user only heard the first eight words, your history is now a lie, and the model will confidently reference things it "said" that nobody heard. Track played-milliseconds and truncate to match.

Barge-in also needs a higher confidence threshold than normal turn detection, because the biggest source of false triggers is your own output. Without echo cancellation the agent hears itself through the speaker and interrupts itself in a loop. On the web, getUserMedia with echoCancellation: true covers the common case; on speakerphone in a hard-surfaced room browser AEC struggles, and you want half-duplex gating as a fallback.

Transport: prefer WebRTC once you leave the lab

Sending PCM over a WebSocket is easy and works right up until packet loss. TCP head-of-line blocking means one lost segment stalls everything queued behind it, and your tuned jitter buffer eats a 300 ms spike. WebRTC gives you an SRTP/UDP path with real jitter buffering, loss concealment, and AEC free from the browser stack, at the cost of signaling complexity. My heuristic: WebSocket + Opus is fine for prototypes; anything shipping to mobile networks belongs on WebRTC. Either way send Opus at 16 kHz mono, not raw PCM at 48 kHz.

When I wanted a reference for how these pieces behave alongside other modalities in one session, the GeminiOmni studio was a useful system to poke at, because it surfaces the ugly interaction fast: the moment a voice turn triggers a slow non-voice tool call, your latency budget is blown and you need a filler-speech strategy or users assume it crashed.

One page for my past self: log per-segment timestamps from day one, make the hold adaptive before touching anything else, truncate history to what was played, and test on a phone, on cellular, in a room with a fan running. These systems are all beautiful on a wired desktop with a headset, and that is not where your users are.

Holding Many Modalities in One Session: A Parts-Based API Structure

Lena Hoffmann — Fri, 24 Jul 2026 19:05:07 +0000

Multimodal APIs are usually documented one modality at a time — here is image input, here is audio, here is streaming text. What the docs rarely cover is how to hold them in one coherent session, which is what an actual application needs.

Parts, not endpoints

The mental shift that made this tractable: stop thinking in endpoints, start thinking in typed parts. A request is an ordered list where each element declares its own type:

type Part =
  | { kind: 'text';  text: string }
  | { kind: 'image'; mime: string; data: Uint8Array | { uri: string } }
  | { kind: 'audio'; mime: string; data: Uint8Array }
  | { kind: 'video'; mime: string; uri: string }

Once input is a part list, interleaving stops being a special case. "Here are two images, a question about them, and an audio clip for tone reference" is just a four-element array — not three API calls you have to correlate afterwards.

Inline versus reference, and why it bites

Small assets go inline as base64. Large ones must be uploaded first and passed by URI. The threshold is provider-specific and the failure mode is bad: exceed it and you get an opaque 400, not a helpful "too large, upload it instead."

Worth building a size check into your part constructor so it routes automatically rather than discovering the limit in production.

Also: base64 inflates payloads by roughly a third. A "small enough to inline" image is meaningfully smaller than you assume.

Streaming breaks the tidy model

Text streams token by token. Images arrive whole. Audio may stream as chunks. So your response handler cannot assume a uniform shape — it needs to be a state machine over part-typed events, accumulating text deltas while treating an image part as atomic.

The bug I hit repeatedly: treating every stream event as appendable text, which quietly corrupts binary parts.

Context accumulation is the real cost

Multi-turn multimodal sessions grow fast. Every image stays in context and keeps costing tokens on every subsequent turn.

Two things that helped:

Summarise old turns — after N turns, replace early image parts with a text description of what they were
Explicit pinning — let the user mark which assets stay in context, and drop the rest

Without either, a ten-turn session with images becomes surprisingly expensive, and the cost is invisible until the bill arrives.

I put this architecture into practice at GeminiOmni — image editing, video generation, and live chat sharing one session model.

Caveat

Part-list APIs are converging across providers but are not identical. This structure ports with modest adapter work — it is not free.

What I Learned Building a Multimodal AI Studio Solo on Gemini + Veo

Lena Hoffmann — Wed, 10 Jun 2026 12:44:08 +0000

I spent a weekend wiring Google's Gemini and Veo APIs into a single app just to feel where the edges of multimodal AI actually are. It turned into a small studio I now use daily, and along the way I learned more about these models from plumbing them than from any paper. Here's the honest technical debrief.

Three pipelines, three completely different problems

I wanted one prompt box that could do video, image editing, and document Q&A. Naively I assumed they'd share most of the stack. They don't.

1. Image-to-video: the enemy is time, not pixels

Generating one good frame is solved. Video is about temporal coherence — frame 13 must agree with frame 12 or you get flicker and identity drift. Modern video models treat the clip as one object in space and time (latent diffusion over a width x height x time volume, with spatiotemporal attention) rather than 120 independent images. Conditioning on a reference image as the first frame is what makes image-to-video feel controlled: you've handed the model a strong anchor and asked it to extrapolate motion, not invent a world.

The surprise: native audio sync (Veo 3.1 generating clip + soundtrack jointly) does more for perceived realism than another notch of resolution. A door slam landing on the exact frame the door shuts is uncanny in a good way.

2. Instruction-based image editing: preservation is the hard part

Generating is unconstrained; editing must change one thing and preserve everything else. Condition the diffusion model on both the instruction and the source image's latents, cross-attend the instruction to steer only the referenced region, and bias hard toward preserving unedited latents. Push that preservation too soft and the subject's face quietly morphs across edits — the classic 'character consistency' failure that makes or breaks storytelling use-cases.

3. PDF chat: it's retrieval, not a long context

The naive 'paste the whole PDF' approach dies on long files (models get lost in the middle) and costs you the full document every turn. The version that works is a tiny RAG pipeline: chunk with overlap that respects structure, embed chunks into a vector index, retrieve the few nearest passages per question, and ground the answer in only those passages with a citation. Half the real work is just parsing hostile PDFs (multi-column, scanned, tables) into clean ordered text before any model sees it.

What was genuinely hard solo

Cost control. Every modality has a different price curve. I collapsed everything to one credit balance and route to the cheapest model that clears a quality bar per task. Hard-coding model names at call sites is a trap; put them behind one config.
Latency UX. Video takes seconds-to-minutes. The product is mostly about making waiting feel intentional — optimistic UI, job queues, auto-refunding failed jobs so a timeout never costs a user a credit.
Glue > models. The models are an API call. The studio is chunkers, parsers, queues, a credit ledger, and a lot of error handling. That's the actual product.

The takeaway

If you want to understand these models, stop reading and wire three of them into one app. The cheapest experiment is still the same one I ran: feed a model a single image and watch what it does with time. The result of mine, if you want to poke at it, lives at geminiomni-ai.com — but the real value was the debugging, not the demo.

Happy to compare notes if you're building in this space.