Light does not negotiate. Tokyo to Virginia is about 130ms round-trip and no amount of infrastructure spend changes that. If you want a voice agent to feel like a person and not a call-center IVR, the round trip has to leave the building.
So I ran the whole stack on a CPU-only laptop. No GPU. No cloud LLM. Whisper v3 Turbo for STT, one of five Ollama models for the LLM, a local TTS for the last mile. Wall-clock target: sub-300ms voice-to-first-byte. Here are the seconds.
5 Ollama models, one CPU, one budget
I put each model behind the same Whisper Turbo front end and the same local TTS, and asked it to produce the first spoken token as fast as possible. The audio input was a short customer-support prompt in English and Japanese. The box was a mid-range x86 laptop, no accelerator, running Ollama with Q4_K_M weights where available.
| Model | Size on disk | Tokens/sec | Time to first token | Japanese quality | Verdict for 300ms |
|---|---|---|---|---|---|
| Qwen2.5 1.5B | ~1 GB | 4-8 tok/s | 200-350 ms | Usable | Fits |
| Gemma 3 2B | ~1.5 GB | ~15 tok/s | 220-380 ms | Passable | Fits (best throughput) |
| Qwen3.5 0.8B | ~1 GB | 12-15 tok/s | 150-250 ms | Thinking loop | Fast but unstable |
| Qwen3.5 2B | ~2.7 GB | 8-9 tok/s | 280-450 ms | Thinking loop | Overshoots on retries |
| DeepSeek-R1 1.5B | ~1.1 GB | 3.9-4.0 tok/s | 400-600 ms | JP/CN code-switch | Misses the budget |
Two of five hold the budget with margin. Qwen2.5 1.5B and Gemma 3 2B are the only ones I would put in front of a real user today. The rest have a reason they miss, and the reasons matter more than the seconds.
If you were expecting DeepSeek-R1 to win, you and I both. It is the model most local-LLM tutorials open with. On CPU, at 1.5B, with a Japanese prompt, it thinks in Chinese for a beat before it answers in a mix of both. The stopwatch says the latency is fine. The user hangs up anyway.
Why Whisper v3 Turbo made this fight winnable
Whisper Large v3 Turbo is 809M parameters with 4 decoder layers instead of the Large v3's 32. That single architectural change cuts per-chunk decode time in half at a 0.3-0.7pp WER cost. Simplismart's production benchmark put the served RTF at 1300x, and Groq's hosted variant runs at 216x real-time. On my CPU box I get nothing close to 1300x, but I get 50-150ms of STT on a 3-second utterance and that is what the budget needs.
The distilled English-only cousin, distil-whisper large-v3, runs at ~90x RTF on a GPU, which is faster on paper. It also drops multilingual, so for the Japanese half of my traffic Turbo is the only one that survives. If your product is English-only and you have a GPU, distil-whisper is worth a look. Turbo is the model I ship.
Where the 300ms actually goes
Adding up the budget line by line is the boring part, and it is also the part everyone gets wrong.
VAD (voice activity detect): 0-5 ms
Whisper Turbo STT (3s audio): 50-150 ms
Qwen2.5 1.5B LLM TTFT: 200-350 ms ← the fat one
Local TTS TTFB: 40-75 ms
Network: 0 ms ← the win
--------------------------------------------
Total to first spoken byte: 290-580 ms
The LLM's time-to-first-token is the fat variable and everything else is thin. If Qwen2.5 lands at 200ms you clear 300ms. If it lands at 350ms you clear 400ms. There is nothing else to tune. Speeding up Whisper by 40ms buys you 40ms. Speeding up the LLM by 40ms buys you 40ms on every subsequent token too, because that is the model that keeps talking.
Which is why my "winning" configuration is Qwen2.5 1.5B, and Qwen2.5 is not the fastest of the five. Gemma 3 2B has better tokens-per-second by roughly 15 vs 4-8, and I still ship Qwen2.5 when the traffic is Japanese-heavy, because Qwen's TTFT variance is tighter and the output quality does not embarrass me. Gemma is my English-first pick.
The two failure modes nobody warns you about
Thinking-loop overshoot. Qwen3.5 in the 0.8B and 2B sizes will start a chain-of-thought passage before it answers. On the desktop that is a feature. In a voice loop it burns 200ms and outputs <thinking> tokens the TTS then tries to speak. If you want a reasoning model for your voice agent, wrap it in a stop-on-first-answer regex. Do not skip that step.
Cross-language code-switching. DeepSeek-R1 1.5B on a Japanese prompt produces answers with Chinese phrases mixed in. This is a distillation artifact from the base model, and prompt tuning does not remove it. If your users speak the language the model bleeds from, you will hear it in the TTS output. I noticed on the second test call. Users will notice on the first.
Both of these fail in production and pass in benchmark scripts, which is exactly the opposite of what you want.
The Pipecat vs LiveKit context, briefly
If you are doing this in a framework rather than raw Python, LiveKit's baseline is 750-900ms end-to-end and Pipecat's is 800-950ms with a standard cloud stack. Both frameworks can reach sub-500ms p95 in 2026 with streaming STT, partial TTS, and prefix caching, assuming a GPU somewhere in the loop.
The CPU-only path is aiming at a different product entirely. 300ms from a device that is not on the internet and does not care about a regional outage. That is what the physics buys you when you take the trip out of the loop.
The realistic edge + cloud hybrid
The pure-edge configuration is a technical achievement, and it is also not what I would ship to most customers. This is the version I actually recommend:
- Edge: VAD, Whisper Turbo STT, a small local LLM as a fallback / cache.
- Cloud: primary LLM (whatever your quality tier demands) and TTS.
- Only text crosses the wire. Voice data stays local.
That configuration lands at 300-350ms voice-to-first-byte with the same Whisper Turbo front end and a cloud LLM, gets you the quality of a large model, and keeps the privacy story that the pure cloud path cannot buy at any price.
Constraints worth naming
Local inference on CPU costs you memory, battery, and thermal headroom. 1.5B at Q4_K_M is about a gigabyte of RAM permanently pinned. A long call drains a laptop battery faster than a video call. Sustained inference will thermal-throttle a fanless device inside ten minutes.
None of that is a dealbreaker. All of it is a reason to have a fallback path. My rule: local first, cloud on thermal or memory pressure, log which path served each request so I can see the shape of the tradeoff a month later.
What I would do differently
If I were starting fresh I would benchmark Phi-4 Mini alongside Qwen2.5. Phi-4 Mini is 12 tok/s on CPU at 3.8B, which is a good spot in the CPU-bandwidth-bound curve, and it did not exist when I ran this test. I would also test the newer distil-whisper INT8 variants for the English-only case, since the 6x speedup is real and the WER cost is smaller than the reputation suggests.
I would not switch off Whisper Turbo. The 6.3x speedup over Large v3 with a tiny WER penalty is the single most important line item in the whole stack, and every downstream design decision falls out of that one architectural change.
The 300ms wall was physics. Whisper Turbo and a 1.5B LLM on a laptop CPU turned it into an engineering problem, and engineering problems ship.
The long form of this stack (Pipecat design patterns, filler-word strategies for hiding the last 50ms, the full latency breakdown for hybrid cloud, and the parts I cut from this piece for length) is the book that grew out of the work: Voice AI 300ms UX (English edition).
Top comments (0)