My local voice assistant stuttered until I made the LLM wait for the TTS

korexil — Fri, 16 Oct 2026 10:11:14 +0000

I wanted small ESP32 voice-assistant boards (running the open-source xiaozhi firmware) to hold real conversations offline, served from my laptop: an RTX 4060 Laptop GPU (8 GB) and 16 GB of RAM. Speech detection and recognition run on the CPU; an LLM (Gemma 4 E4B, QAT q4_0, on llama.cpp) writes the answer while a voice-cloning TTS (Qwen3-TTS 1.7B) speaks it sentence by sentence.

Fitting both models into 8 GB was the first problem. Making them share the card was the harder one: long answers kept stalling mid-speech.

One live run of each, played at 2x speed. Red bars are where the audio ran out.

Why it stalled

Measured on its own, the TTS engine needs about 0.6 seconds to generate 1 second of audio (a real-time factor, RTF, of about 0.6). Inside the pipeline the first sentence is what matters. Counting from the request to its last audio block, the first sentence took 1.3 to 1.5 times its own length to generate while the LLM was decoding on the same GPU, against about 0.6 times when the LLM wasn't. Generating it took longer than playing it, so the playback buffer ran dry. Later sentences ran at about 0.6 either way; the slowdown is in the first sentence, while the LLM is busy. (Corrected on 2026-10-18: the first version of this paragraph said 1.0 and 0.84 to 0.89, from trace events stamped when the server's event loop dequeued each block rather than when it arrived.)

My guess, which I haven't confirmed with a profiler, is that the TTS decodes frame by frame with many small kernels, and each step ends up waiting behind the LLM's work. Throttling the LLM (to 20, then 14 tokens/s) and changing the GPU scheduling priority in Windows didn't help in my tests.

The fix: take turns

So I stopped running them in parallel. The LLM writes one sentence and pauses. The TTS synthesizes that sentence alone, at full speed. Then the LLM continues. At 61 to 66 tokens/s, writing a sentence takes the LLM far less time than speaking it, so the audio stays ahead for the whole answer.

It isn't free: the LLM now spends most of its time waiting. In the runs below it finished writing at about 54 s instead of about 12 s, inside answers that play for 65 to 84 s. For speech that's fine, because the listener is still hearing the answer anyway.

What changed, step by step

Same story-telling question at every step, sent as text (so speech detection and recognition aren't included), all four steps measured in one sitting, 3 runs each. "First audio" is when the server sends its first audio frame to the board. Stalls are estimated by the server against the board's 0.3 s buffer, not measured at the speaker.

Setup	First audio, avg (range)	Stalls per answer: avg total (count range)
PyTorch TTS on GPU, LLM on CPU	4.4 s (3.5 to 6.0)	4.59 s (48 to 60)
Native C++ TTS on GPU, LLM on CPU	5.3 s (5.2 to 5.4)	0.05 s (0 to 2)
Both on GPU, running at once	3.9 s (3.8 to 4.1)	2.84 s (9 to 17)
Both on GPU, taking turns	2.3 s (1.9 to 2.7)	0.09 s (0 to 1)

Switching to a native engine removed almost all of the stutter on its own, but the first word came no sooner (5.3 s against 4.4 s, and the baseline varied a lot between runs). The PyTorch TTS process also held about 8.9 GB of private memory, mostly paged out to disk on my 16 GB laptop, so don't read the first row as a general PyTorch-versus-native comparison. Moving the LLM onto the GPU made the first word come sooner but brought the stalls back. Taking turns made the first word come even sooner and removed that regression.

A separate controlled test changes only the scheduling: Japanese prompts, a simulated player with a 0.5 s buffer, identical generated text in both modes, 6 runs each. Running at once, 6 of 6 runs stalled. Taking turns, 0 of 6. The buffer and prompts differ, so its numbers aren't directly comparable with the table.

What a PCM trace showed

After I posted this on the Hugging Face forum, a reader suggested logging when each audio block actually arrives instead of looking only at averages. So the server now records every sentence handed to the TTS, every audio block with its duration, and every batch of frames released to the sender. I recorded three fresh runs each of the last two setups (separate from the table). Replaying the sender on those timestamps reproduces the stalls the server reported, and showed two things.

Running at once, the TTS falls behind while the LLM is generating. The first audio arrives 2.3 to 2.5 s after the request instead of 1.0 to 1.2 s, and in all three runs every stall falls in the first 13 seconds, while the LLM is still writing or has just finished; none come later. On the server's timeline, a player would have to wait 1.9 to 3.4 s before starting to avoid any stall (a lower bound: it assumes exact 1x playback and ignores decoding on the board). With turn-taking it is 0 to 0.12 s. That "how long would the player have to wait" number is the one I'd compare across GPUs.

One remaining stall came from my own server, not the GPU. To trim trailing silence, the server holds back the last 1.2 s of each segment's audio. The TTS delivers its first 1.0 to 1.2 s as one burst, so nothing goes out until the next block arrives about 0.19 s later. In one of the three turn-taking runs only a single 60 ms frame was released at that point and the board ran dry for 0.13 s. Replaying the first segment as if audio could be sent on arrival removes that stall and sends the first frame 0.19 s sooner. Fixed since: the server now asks the TTS for the pause it wants per sentence instead of holding audio back, the first frame leaves with the first block (0.19 s sooner) and none of the three runs has the early gap; in the one matched pair I compared, the pauses are unchanged. Details in the repo under bench/results/traces-2026-10-18.

Fitting into 8 GB

The TTS runs on a patched C++ engine based on qwen3-tts.cpp. Two patches matter most: deduplicating the Q/K/V weights saves 488 MiB of VRAM, and keeping the text embedding table on the CPU saves another 594 MiB. With both models loaded the card shows 7,448 of 8,188 MiB in use; about 6.8 GB of that is the two models, the rest is the desktop.

Limits

Self-hosted, nothing is hosted for you. Tested on one machine (RTX 4060 Laptop, Windows 11); it has never run on Linux or macOS.
The LLM side doesn't support tool calling yet.
You don't need a board: it also runs as a local LLM + TTS HTTP service.

The code, the measurements with raw data, the PCM traces and the pitfalls I hit along the way are here (Apache-2.0): https://github.com/korexil/gpu-duet

If you run an LLM and a TTS on the same consumer GPU, I'd like to know whether taking turns still beats running in parallel on your card.

Update 2026-10-18. Interrupting an answer with the board's button and asking again at once, with a TTS error or a hang injected on the new answer: in one run per case, nothing of the old answer leaked into the new one, a TTS error ended the answer in 25 ms, and a hung TTS was cut off after 10 s (it used to be 60). Write-up: bench/results/interrupt-2026-10-18 in the repo.

Written with help from an AI assistant. The measurements were run on my laptop, and the raw data is in the repo.