DEV Community

Cover image for Speculative Listening Cuts Full-Duplex Response Start to 74-241 ms — Idle Compute in Chunked Speech Models
devs
devs

Posted on

Speculative Listening Cuts Full-Duplex Response Start to 74-241 ms — Idle Compute in Chunked Speech Models

On 4 November 2026 a paper titled “Decode Before You Hear: Speculative Listening for Chunk-Interleaved Full-Duplex Speech Language Models” landed on arXiv (2611.06982). The authors measured four chunk-interleaved full-duplex speech models and found that listen windows leave 70–89 % of the compute idle while the next audio chunk is still arriving. Speak and tool-call chunks then serialize generation and miss the real-time deadline.

They turned the idle window into useful work. During the listen period the model drafts the next chunk under a pure-silence hypothesis. When the real audio arrives it verifies the draft. Across the models 95.3 % of the drafted chunks were identical to the output under real audio. When they differed, 88 % of the changes started at the very first output token, so a cheap second branch from the runner-up token covers most corrections. The whole scheme is training-free and lossless — it preserves the original distribution.

On their TickTock-Bench (760 sessions, 15 883 chunks) the technique sped speak chunks 2.6–2.8× and tool-call chunks up to 23.4×. Average decoding delay before a response starts fell 73 %, landing in the 74–241 ms band — inside the roughly 200 ms gap humans leave between turns. Maximum lag behind real time dropped 94.4 %.

Sound wave frequency and amplitude

Why the numbers matter for a local desktop pipeline

Cloud full-duplex models can hide a lot of the latency behind a network round-trip and a GPU farm. A Windows process that captures microphone or loopback audio, runs a local STT model, and optionally feeds a short local translation or reply model has a different budget. The capture path itself is not free.

Typical shared-mode WASAPI numbers on a modern laptop:

  • Device + driver buffer: 10–20 ms
  • Windows Audio Engine mix + any communications enhancement or spatial processing: 10–30 ms
  • Your capture client period: another 10–40 ms before you see a stable buffer

That is already 30–90 ms before the first sample reaches your STT runtime. Exclusive mode can shrink the engine contribution, but then you fight device exclusivity and the rest of the desktop. Recent Coval measurements put Qwen3-ASR 1.7B variants around 35 ms time-to-final segment and Soniox STT RT v5 at 47 ms — those numbers assume the audio has already left the capture device.

If your local model finishes inside the remaining budget, end-to-end from speech end to on-screen text can stay under a couple of hundred milliseconds without ever shipping the audio off the machine. Speculative techniques like the one in the paper are interesting for the generation half of a full-duplex loop, but they do not move the capture-side cost. The idle-compute idea only helps after the audio is already in process memory.

Practical controls on the Windows side

  1. Measure the real capture latency, not the model latency. Use a loopback click or a known impulse and timestamp the first non-zero sample that reaches your process. The number is almost always higher than the buffer size you requested.
  2. Prefer a small local STT model (GGUF / ONNX / MLX) that finishes inside the remaining budget rather than a cloud endpoint whose time-to-final looks better on paper but adds 80–150 ms of network.
  3. Keep the credential and config boundary inside the process or a DPAPI-protected store. Nothing that looks like a settings file next to source code should ever contain a live key or a shell command that embeds one.
  4. If you later add a speculative generation step for short replies, draft only under silence or a simple VAD hypothesis and verify against the actual transcript. The paper’s 95 % hit rate is encouraging, but the verification still has to be cheap enough that a miss does not blow the latency budget.

The same discipline applies when the overlay itself is the product. The process that owns the WASAPI client and the model runtime should not write durable config that an agent or a later git commit can accidentally ship.

Sine wave

Architecture sketch

[Mic / Loopback] → WASAPI capture (shared or exclusive)
                 → ring buffer (10–40 ms)
                 → local STT (Qwen3-ASR / similar GGUF)
                 → optional local translate / short reply
                 → overlay draw
Enter fullscreen mode Exit fullscreen mode

No external key is required on the hot path. If a cloud fallback is present it is optional and the key never leaves a protected store. The speculative drafting idea from the paper can be layered on top of the generation step once the transcript is ready; it does not replace the need to keep the capture path tight.

The paper’s result is a useful reminder that a lot of the “AI latency” conversation is still compute that is simply waiting. On a local Windows tool the waiting is often in the audio engine, not the model. Measure both sides, keep the model local when the numbers allow, and the end-to-end number becomes something you can actually ship.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.

Top comments (0)