Soniox STT v6 Agent Text Injection: Residual Echo Filtering and Windows Capture Latency Budgets

Soniox dropped STT v6 on November 3, 2026. The headline features are agent text input and agent voice echo cancellation on the real-time model (stt-rt-v6). You stream the agent’s outgoing text on the same WebSocket as the caller’s audio; the model uses that context for short and ambiguous replies, and it filters residual agent audio that acoustic echo cancellation left behind.

This is the practical problem every real-time audio pipeline on Windows hits. Capture, mixing, AEC, and the network hop eat the latency budget long before the model’s time-to-final becomes the bottleneck.

The two problems v6 targets

Cascaded voice agents normally run STT only on the caller side. The model never sees the question. Short answers collapse: Soniox’s own docs show “River Roast” coming out as “River Road” and “a tow” as “a tour.”

The second failure mode is residual echo. Agent TTS plays out of a speaker or laptop, leaks back into the mic, and gets transcribed as user speech. The agent interrupts itself. Acoustic echo cancellation (AEC) removes most of it; whatever remains still triggers false barge-in.

v6 addresses both by accepting timed agent text frames:

{"type": "agent_text", "text": "I can rebook you through Porto or Faro.", "end_of_segment": true}

Enable it once in the start request with "enable_agent_text_input": true. Messages have no timestamps; the model aligns them by arrival order in the audio stream. Soniox recommends sending each chunk 500 ms–1 s before playback, paced by your TTS timestamps. On barge-in you flush the already-played portion, send end_of_segment: true, and drop the rest. The agent’s text never appears in the returned transcript.

With that context the model separates residual agent audio from caller speech and drops it, while still listening so the user can interrupt.

Public-domain waveform used for illustration. Real capture graphs look messier once Windows audio engine and device quirks enter the path.

Why the Windows capture path still dominates

Even if the cloud model returns a final segment in 100–200 ms (v5 numbers on Pipecat were around 260 ms median TTFS; v6 has no independent numbers yet), the local path on Windows is the larger variable.

Typical Windows real-time capture stack:

1. Microphone / loopback device (WASAPI exclusive or shared mode).
2. Windows Audio Engine mixing, processing, and any communications enhancements.
3. Your capture buffer (10–40 ms is common; lower is fragile).
4. Local AEC / noise suppression (Windows has its own, or you run RNNoise / Speex / custom).
5. Resampling if the device rate ≠ the STT rate (16 kHz or 24 kHz typical).
6. Network send + model + return.

Each stage adds jitter. Shared-mode WASAPI can introduce 10–20 ms extra. Device drivers (especially Bluetooth and some USB headsets) add more. The moment you overlay translated text or run local post-processing, the total round-trip from speech end to on-screen text is dominated by capture + transport, not the model’s internal latency.

Practical command-line checks developers use:

# List WASAPI devices and exclusive-mode support
# (via tools like wasapi-info or your own IAudioClient probe)
powershell -Command "Get-PnpDevice -Class AudioEndpoint | Select-Object FriendlyName, Status"

In code you measure the gap between IAudioCaptureClient::GetBuffer timestamps and the first partial from the STT socket. That delta is your real budget. If it is already 180 ms before the model even starts, a 50 ms model improvement is noise.

Architecture notes for a Windows overlay

For a tool that shows live transcription or translation during a call (job interview, customer call, remote meeting), the pipeline looks like:

• WASAPI loopback or mic capture (prefer exclusive when possible for lower latency).
• Local VAD (Silero or energy + hangover) to gate sends and reduce cost.
• Optional local AEC before the cloud hop.
• WebSocket to stt-rt-v6 with agent_text frames when you control the TTS side.
• Partial results → overlay window (always-on-top, click-through or hotkey).
• Final segment → translation model (local or API) → display.

Tradeoffs:

• Cloud STT with agent context (Soniox, AssemblyAI Universal-3.5 Pro also does agent context) gives better short-reply accuracy and, in Soniox’s case, residual echo filtering. Cost is ~$0.12/hour streaming plus text tokens for the agent side.
• Fully local STT (Whisper.cpp, NVIDIA Canary, etc.) removes the network hop and keeps data on-box, but you lose the free agent-text injection and must implement your own echo handling and short-reply biasing (prompting or hotword lists).
• Hybrid: local VAD + local rough transcript for the overlay, cloud only for high-accuracy finals or when language ID is uncertain.

BYOK and rate-limit realities matter. Streaming WebSockets can be dropped; you need reconnect with sequence numbers and pre-roll audio so you do not lose the first 200 ms of speech. Parsing the JSON frames is trivial; the hard part is keeping the audio clock and the text clock aligned when TTS playback is delayed by the audio engine.

Practical numbers from public sources

• Soniox lists $0.12/hour streaming, $0.10/hour async, diarization and language ID included. Agent text is billed separately as input tokens (~$4/1M).
• v5 (still the only public benchmark numbers) hit 1.34 % semantic WER and 260 ms median time-to-final on the Pipecat set. v6 claims “significantly” better; independent numbers are not out yet.
• AssemblyAI shipped similar agent-context carryover earlier in 2026; the clearer differentiator Soniox is highlighting is the model-level residual echo filter.
• v5 models are scheduled for removal on 15 December 2026; after that requests route to v6 automatically. Test formatting changes (numbers and currencies are written as spoken words).

What still breaks in practice

• Bluetooth headsets and many USB conference speakers have poor or no AEC. Residual echo is large; the model filter helps but is not magic.
• Code-switching mid-sentence still needs testing. v6 claims stronger support, but real interviews mix languages constantly.
• Overlay latency perception is driven by the first partial, not the final. If your first partial is 400 ms late because of capture buffering, users notice even if the final is accurate.
• Local models on NPU/GPU help the translation step, but the capture path is still CPU/driver bound.

The useful pattern is to treat the cloud STT as one stage in a measured pipeline, not the whole system. Instrument the capture clock, the first-byte time on the socket, and the time from final to on-screen text. Then decide where to spend the next 20 ms: exclusive-mode WASAPI, smaller buffers, better local AEC, or a different model.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.