For a long time the honest objection to running a voice agent on your own hardware was speed. Open models were slower, a single GPU could not keep up, and anyone who wanted a call to feel like a conversation ended up on a hosted API.
That objection has quietly expired, and most of the discussion hasn't caught up. The number people now quote is real. The way they quote it usually isn't.
The 508ms figure, and what's missing from it
Daily built a voice agent on NVIDIA's open models and ran it on one RTX 5090. Server-side voice-to-voice came in at 508ms P50 and 544ms P90. They broke it down by stage: 19ms for speech to text, 171ms for the language model, 108ms for text to speech.
That's a great result. It's also only the part of the call that happens on your server.
On a real phone call the telephony leg typically adds another 300 to 600ms, and it comes out of the same budget. The usual bar for a conversation that feels natural is about 800ms end to end. Do the subtraction and your actual server-side budget lands somewhere between 200 and 500ms, depending on the carrier and the route. So 508ms is a strong signal that local inference is no longer the bottleneck. It isn't proof you're done.
Quote the hardware every single time
Here's the detail that should make everyone more careful with benchmark screenshots. Daily ran the identical stack on a DGX Spark, same software, same measurement. It came back at 1,180ms P50.
Same code, more than double the latency, and well past the point where a caller notices. A latency number with no hardware attached to it tells you very little. If someone sends you a figure for their local voice pipeline, the first question is what box it ran on.
Architecture beats a bigger card
The more interesting finding sits one layer down, in the speech-to-text stage.
The Open ASR Leaderboard paper evaluated 86 systems on identical A100 hardware, in a snapshot dated 27 March 2026. They report throughput as RTFx, which is seconds of audio processed per second of compute, so higher means more concurrent calls per GPU.
Conformer-based encoders averaged 758 RTFx. Whisper-based decoders averaged 201. The fastest single entry hit 6,400.
That's close to four times the throughput between two model families running on exactly the same silicon, and more than thirty times between the fastest entry and the Whisper average. When a self-hosted deployment feels sluggish, the reflex is to buy a bigger GPU. Most of the time the better fix is a different encoder.
Colocation is the part hosted setups can't copy
There's a structural reason local wins on latency once the models are good enough, and it has nothing to do with raw inference speed.
A typical hosted voice pipeline spreads its stages across providers. Speech to text runs on one company's servers, the language model on another's, the voice on a third. Every hand-off is a network round trip carrying audio out and waiting for it to come back.
Put every stage on one machine you own and those hand-offs disappear entirely. The hop doesn't get shorter. There's nothing left to hop between. This only works with open weights, because closed providers don't ship models you can place next to each other. The best a hosted setup can do is pick a region close to the provider's, which shortens the trip without removing it.
A stack that actually fits on one 24GB card
If you want to try this, the sizing is where builds go wrong, so here's a combination that fits.
For the language model, Qwen3.8-27B quantized to 4-bit comes down to 15.93 GiB of weights and peaks around 20,266 MiB with a 64K context. At 16-bit it would need roughly 54GB, which is why quantization is the thing that makes this possible at all. Voice work tolerates the small accuracy cost well, since the job is narrow: follow a workflow and call a tool correctly.
That leaves about 4.2 GiB on a 24GB card. Parakeet TDT 0.6B v2 loads in roughly 2GB and fits, and it's the most accurate of the options at 6.05 percent average WER on the Open ASR Leaderboard, English only. Voxtral Mini 3B is a good model too, but it needs around 9.5GB and simply won't share the card.
For the voice, Magpie TTS multilingual is 357M parameters and covers twelve languages, small enough that synthesis stops being a latency concern. Read its model card before shipping, though. It's open weights under an agreement rather than an open source licence, and it can't clone a voice.
Where this leaves the decision
Once latency is off the table, the reason to go local is control over where the audio goes. Cost is a weak argument on its own, since cloud token prices keep falling and the real expense of self-hosting is the engineer who keeps the stack running.
We build Dograh, an open-source voice agent orchestrator, and the design choice we'd push hardest is keeping that orchestration layer separate from inference. The same workflow, turn-taking, telephony and tool calls should run whether the models underneath are rented from a provider or running as open weights on your own GPU. Plenty of teams start hosted and move inference local once compliance catches up with the pilot. If the orchestrator is locked to a vendor, that move is a rewrite.
The longer guide covers the benchmark caveats and the licence traps we only touched on here. Read it on Dograh.
Top comments (0)