On 18 November 2026 AssemblyAI shipped three specialized STT-3.7 models. The one that matters for voice agents is AssemblyAI-STT-3.7-Realtime-Agent. It returns the final transcript 86 ms after speech ends, cuts partial-to-final rewrites to 1.0 %, and posts the lowest normalized word error rate (4.53 %) and entity error rate (13.2 %) on their 12,460-call English voice-agent set, ahead of Soniox Realtime v6, ElevenLabs Scribe v3 Live, and Deepgram Nova-3.
Those numbers are real. The 86 ms figure is measured from end-of-speech to final transcript on their harness. Short-reply errors drop, background speech is reduced, and numbers on Voice Code Bench improve dramatically versus the previous Universal-3.6 Pro Realtime. Independent checks on Voice Arena and Speko put it at or near the top of streaming models for US English semantic WER and end-of-turn latency.
For a developer building a local overlay that shows translated suggestions during a technical interview or a sales call in another language, the cloud STT latency is only one line in the budget. The rest lives on the Windows machine.
The dual-track capture path still dominates
A usable live overlay needs two clean PCM streams:
- Local microphone (what you say).
- Loopback of the render device (what the other party says through Teams, Zoom, Meet, or the browser).
Both go through the Windows Audio Engine (audiodg.exe + audioeng.dll) in shared mode so the call client keeps exclusive control of the device. Shared mode is the only practical choice; exclusive mode would break the call.
Typical shared-mode period is 10 ms. You can request a smaller period; the engine frequently ignores it or falls back under load. Communications enhancements, spatial audio, vendor APOs, and Bluetooth HFP/A2DP add another 10–40+ ms and are fragile when the CPU or GPU is busy. Process-wide audio ducking and communication-device policies inject extra silence or force resampling.
A practical stack looks like this:
- Endpoint device → WASAPI shared-mode client (10–20 ms buffer).
- Audio engine mixing and any enabled enhancements.
- Your capture callback (copy PCM, do minimal work, return immediately).
- Framing + optional local VAD or silence detection.
- Network to the STT endpoint (or local model).
- Partial/final handling + translation.
- Overlay paint on a dedicated thread (<16 ms if you stay off the UI thread).
The 86 ms final from AssemblyAI arrives after the capture path has already spent 20–50 ms. Add network RTT, framing, and the time for the first usable partial, and the overlay still appears tens to hundreds of milliseconds after the speaker finishes. That is the floor.
Why dual-track matters more than the model name
Single-stream capture (mic only, or loopback only) is useless for a bilingual call. You need both sides so the translator and the suggestion engine see the full conversation. Mixing them in one buffer loses speaker identity and makes barge-in detection harder. Separate tracks let you:
- Run independent VAD or energy gates.
- Apply different gain or noise profiles.
- Feed the agent-text context (if the model supports it) only for the remote side.
- Keep the overlay responsive even when one track is quiet.
On Windows the cleanest way is two IAudioClient instances: one for the capture device, one for the loopback of the default render device (or the device the call client is using). You must match sample rate and channel count or insert a resampler; the audio engine will not do it for free. The callback must be non-blocking. Any work heavier than a memcpy belongs on another thread.
Latency budget that actually fits
A realistic end-to-end number for a suggestion that appears before the other person finishes their next sentence:
- Capture + framing: 20–40 ms
- Network + first partial: 80–150 ms (region-dependent)
- Final transcript (AssemblyAI agent model): ~86 ms after speech end
- Translation + LLM suggestion: 50–200 ms (local or BYOK)
- Overlay render: <16 ms
Total from end-of-speech to visible text is usually 150–400 ms on a well-tuned machine. The cloud model’s 86 ms is visible in the tail; the capture path sets the floor. Drop the audio engine enhancements, pin the capture thread, and keep the period short, and you reclaim the largest slice.
Local models (faster-whisper, Whisper.cpp, Qwen3-ASR GGUF, or the new open-weight under-1 B parameter options) move the STT step onto the machine and remove the network leg, but they still wait on the same WASAPI buffers. The overlay still lives or dies by how cleanly you open the two clients and how little work you do on the callback.
Practical takeaways
- Treat the AssemblyAI-STT-3.7-Realtime-Agent numbers as the new baseline for cascaded voice agents. The 86 ms final and low rewrite rate are useful.
- On Windows the dual-track WASAPI path remains the dominant latency term for any local overlay that must coexist with a call client.
- Shared mode + short period + separate tracks + non-blocking callback is still the architecture that ships.
- Measure the full path on your hardware; vendor numbers stop at their WebSocket.
The model improved. The audio engine did not. That is the part you still control.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.


Top comments (0)