One Model Swallows the Whole Voice Pipeline? NVIDIA Open-Sources VoiceChat 11B: 448ms Turn-Taking, Tool Calls Mid-Conversation

#ai #apigateway #wrouter

Intro: The Three-Stage Voice Stack Just Got Punched Through

On August 9, NVIDIA officially announced NemotronLabs VoiceChat 11B (the model card shows it landed on Hugging Face on August 3) - an 11-billion-parameter open, end-to-end speech-to-speech model. Its most radical move: instead of the classic cascaded ASR -> LLM -> TTS stack, a single unified network performs streaming speech understanding and speech generation at the same time. That means true full-duplex conversation - the model listens while it speaks, users can barge in at any moment, and the agent yields instantly. Measured smooth turn-taking latency is 448 ms, and in user-interruption scenarios the take-over rate hits 1.00 at 480 ms.
Even more notable: it is the first open full-duplex model that can call tools live, mid-conversation. The two pain points voice-agent developers have stared at for two years - latency and tool calling - just got touched by one open model simultaneously.

How Full-Duplex Works Here

Architecture: Three Existing Components Plus One New Channel
VoiceChat 11B is a hybrid Mamba/Transformer. In essence, NVIDIA stitched together three components it already had, then added a new output path:

The model emits three streams at once: agent audio, agent text, and a running transcription of the user's speech. Training consumed roughly 550k hours of real and synthetic audio.
Tool Calls Without Dead Air: The "On-Hold" Line
When a cascaded stack calls a tool, the conversation falls into awkward silence. VoiceChat's answer: tool-call scripts go out on the dedicated side channel, and developers define a per-tool "on-hold" line - the moment the model generates the text that triggers the call, it speaks that line ("One sec, let me check that flight"), the API runs in the background, and the conversation keeps flowing.
The constraints are equally explicit: at most 5 tools per session; no parallel tool calls; users cannot interrupt during tool execution; system prompts and tool responses must be ASCII-only and TTS-friendly.
Benchmarks: #2 Open Full-Duplex, but "Research Only"
On Full-Duplex-Bench 1.0: smooth turn-taking TOR 0.82 at 448 ms, user-interruption TOR 1.00 at 480 ms. NVIDIA reports the model ranks #2 among open full-duplex models on VoiceBench. Weights ship under the permissive OpenMDW 1.1 license and run on vLLM.
But don't rush it into production. NVIDIA explicitly labels the checkpoint "ready for research purposes only," and the repo honestly documents the failure modes: an audio context ceiling of about two minutes; degradation into non-recoverable gibberish after several turns; occasional runaway self-talk after a turn ends; dropped words in user transcription. The hardware bar is a single 80 GB GPU (A100/H100/B200 class), and there is no hosted API today - teams without GPU access cannot even evaluate it.
In Practice: Production Voice Agents Are Still Cascaded Stacks Today
Read the VoiceChat release backwards and the conclusion gets clearer: end-to-end full-duplex is the direction, but a 2-minute context ceiling, a 5-tool cap, and English-ASCII-only constraints mean that for at least the next year, production voice agents will remain cascaded: streaming ASR + an LLM brain + streaming TTS. And in a cascaded stack, both the latency budget and the intelligence ceiling sit on that middle LLM layer.
That layer is exactly where flexibility matters most: fast turn-taking wants a small model; complex requests want a flagship; and when one provider's API wobbles, you need to fail over to a backup model in seconds. Using a model gateway like wrouter.ai as the LLM layer is the natural fix - one OpenAI-compatible endpoint covering the full lineup of mainstream models, a stable interface with no single-provider point of failure, and unified billing across all models, so the high-frequency micro-calls typical of voice workloads don't leave you reconciling invoices across multiple vendor dashboards.
A typical voice-agent brain layer looks like this:

from openai import OpenAI

client = OpenAI(
    base_url="https://wrouter.ai/v1",
    api_key="YOUR_WROUTER_KEY",
)

def voice_brain(transcript: str, complex_task: bool):
    # Small talk goes to a fast model for latency;
    # complex requests switch to a flagship.
    model = "gpt-5.6-sol" if complex_task else "deepseek-v4-flash"
    stream = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "Reply in short, conversational sentences suitable for TTS."},
            {"role": "user", "content": transcript},
        ],
        stream=True,  # stream tokens so TTS can synthesize as they arrive
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

Swapping models is a one-string change, and both the latency budget and the bill live in a single dashboard.

Closing

The significance of VoiceChat 11B is not "usable today" - it is that a complete, open reference implementation of end-to-end full-duplex plus tool calling is now sitting on the table. The cascaded stack's window is still open, but the ceiling has been drawn. Use this window to make your voice agent's LLM layer pluggable and swappable, so you can actually migrate when end-to-end models mature. If you want to spin up a cascaded voice agent quickly, grab a key at wrouter.ai and start testing.

Sources