I wanted small ESP32 voice-assistant boards (running the open-source xiaozhi firmware) to hold real conversations offline, served from my laptop: an RTX 4060 Laptop GPU (8 GB) and 16 GB of RAM. Speech detection and recognition run on the CPU; an LLM (Gemma 4 E4B, QAT q4_0, on llama.cpp) writes the answer while a voice-cloning TTS (Qwen3-TTS 1.7B) speaks it sentence by sentence.
Fitting both models into 8 GB was the first problem. Making them share the card was the harder one: long answers kept stalling mid-speech.
One live run of each, played at 2x speed. Red bars are where the audio ran out.
Why it stalled
Measured on its own, the TTS engine needs about 0.6 seconds to generate 1 second of audio (a real-time factor, RTF, of about 0.6). Inside the pipeline the first sentence is what matters. Counting from the request to its last audio block, the first sentence took about 1.5 times its own length to generate while the LLM was decoding on the same GPU, against about 1.0 times when the LLM wasn't. Generating it took longer than playing it, so the playback buffer ran dry. Over a whole answer, the median per-sentence RTF was the same either way (0.84 to 0.89); the slowdown shows up in the first sentence.
My guess, which I haven't confirmed with a profiler, is that the TTS decodes frame by frame with many small kernels, and each step ends up waiting behind the LLM's work. Throttling the LLM (to 20, then 14 tokens/s) and changing the GPU scheduling priority in Windows didn't help in my tests.
The fix: take turns
So I stopped running them in parallel. The LLM writes one sentence and pauses. The TTS synthesizes that sentence alone, at full speed. Then the LLM continues. At 61 to 66 tokens/s, writing a sentence takes the LLM far less time than speaking it, so the audio stays ahead for the whole answer.
It isn't free: the LLM now spends most of its time waiting. In the runs below it finished writing at about 54 s instead of about 12 s, inside answers that play for 65 to 84 s. For speech that's fine, because the listener is still hearing the answer anyway.
What changed, step by step
Same story-telling question at every step, sent as text (so speech detection and recognition aren't included), all four steps measured in one sitting, 3 runs each. "First audio" is when the server sends its first audio frame to the board. Stalls are estimated by the server against the board's 0.3 s buffer, not measured at the speaker.
| Setup | First audio, avg (range) | Stalls per answer: avg total (count range) |
|---|---|---|
| PyTorch TTS on GPU, LLM on CPU | 4.4 s (3.5 to 6.0) | 4.59 s (48 to 60) |
| Native C++ TTS on GPU, LLM on CPU | 5.3 s (5.2 to 5.4) | 0.05 s (0 to 2) |
| Both on GPU, running at once | 3.9 s (3.8 to 4.1) | 2.84 s (9 to 17) |
| Both on GPU, taking turns | 2.3 s (1.9 to 2.7) | 0.09 s (0 to 1) |
Switching to a native engine removed almost all of the stutter on its own, but the first word came no sooner (5.3 s against 4.4 s, and the baseline varied a lot between runs). The PyTorch TTS process also held about 8.9 GB of private memory, mostly paged out to disk on my 16 GB laptop, so don't read the first row as a general PyTorch-versus-native comparison. Moving the LLM onto the GPU made the first word come sooner but brought the stalls back. Taking turns made the first word come even sooner and removed that regression.
A separate controlled test changes only the scheduling: Japanese prompts, a simulated player with a 0.5 s buffer, identical generated text in both modes, 6 runs each. Running at once, 6 of 6 runs stalled. Taking turns, 0 of 6. The buffer and prompts differ, so its numbers aren't directly comparable with the table.
What a PCM trace showed
After I posted this on the Hugging Face forum, a reader suggested logging when each audio block actually arrives instead of looking only at averages. So the server now records every sentence handed to the TTS, every audio block with its duration, and every batch of frames released to the sender. I recorded three fresh runs each of the last two setups (separate from the table). Replaying the sender on those timestamps reproduces the stalls the server reported, and showed two things.
Running at once, the TTS falls behind while the LLM is generating. The first audio arrives 2.3 to 2.5 s after the request instead of 1.0 to 1.2 s, and in all three runs every stall falls in the first 13 seconds, while the LLM is still writing or has just finished; none come later. On the server's timeline, a player would have to wait 1.9 to 3.4 s before starting to avoid any stall (a lower bound: it assumes exact 1x playback and ignores decoding on the board). With turn-taking it is 0 to 0.12 s. That "how long would the player have to wait" number is the one I'd compare across GPUs.
One remaining stall came from my own server, not the GPU. To trim trailing silence, the server holds back the last 1.2 s of each segment's audio. The TTS delivers its first 1.0 to 1.2 s as one burst, so nothing goes out until the next block arrives about 0.19 s later. In one of the three turn-taking runs only a single 60 ms frame was released at that point and the board ran dry for 0.13 s. Replaying the first segment as if audio could be sent on arrival removes that stall and sends the first frame 0.19 s sooner. That's the next thing to fix.
Fitting into 8 GB
The TTS runs on a patched C++ engine based on qwen3-tts.cpp. Two patches matter most: deduplicating the Q/K/V weights saves 488 MiB of VRAM, and keeping the text embedding table on the CPU saves another 594 MiB. With both models loaded the card shows 7,448 of 8,188 MiB in use; about 6.8 GB of that is the two models, the rest is the desktop.
Limits
- Self-hosted, nothing is hosted for you. Tested on one machine (RTX 4060 Laptop, Windows 11); it has never run on Linux or macOS.
- The LLM side doesn't support tool calling yet.
- You don't need a board: it also runs as a local LLM + TTS HTTP service.
The code, the measurements with raw data, the PCM traces and the pitfalls I hit along the way are here (Apache-2.0): https://github.com/korexil/gpu-duet
If you run an LLM and a TTS on the same consumer GPU, I'd like to know whether taking turns still beats running in parallel on your card.
Written with help from an AI assistant. The measurements were run on my laptop, and the raw data is in the repo.
Top comments (2)
The separation between sender-timeline estimates and speaker measurements makes these results easier to read. For the next fixture, I'd interrupt mid-sentence while the LLM is paused for TTS, then immediately ask a new question. Does cancellation release the turn-taking gate and discard old queued audio before the new answer starts? A generation ID on sentence, PCM and sender events would let the same trace show stale audio crossing into the next turn. A TTS error during that handoff is worth including too, so a failed sentence cannot leave the LLM waiting indefinitely.
Good questions. I checked the code but haven't traced this case yet, so this is what the code says, not what a run showed.
One caveat first: by default the server is half-duplex and drops the microphone audio while it speaks, so talking over it does nothing by itself. On my board, pressing the button sends an abort, and the server stops the answer when it gets one. (A config option,
voice.barge_in, turns voice barge-in on, but it only makes sense on a board that cancels its own echo.)By default there's no separate gate to release: the LLM and the TTS run one after the other in the same worker thread (a demo switch,
llm.overlap, lets the LLM run ahead instead). Each answer gets its own audio queue, pacer and cancel flag. The abort cancels the running answer's task and waits for that task to finish; the worker thread isn't joined, it stops at its next check, between PCM blocks (100 ms each) or between streamed LLM pieces. Anything it still produces lands in the old queue, which nothing reads anymore, so stale audio shouldn't reach the next answer.The device side is the other half. The pacer runs about 300 ms ahead, so those frames are already sent; the withheld tail is dropped. After the abort the server only sends
tts stop, and whether the device drops its buffer is up to the firmware.A TTS error (connection refused, 500, 503) ends the answer: the worker pushes an error and an end marker, the remaining sentences are dropped, and the LLM isn't left waiting. A hang is the weak spot:
Your fixture is the right way to check this: press the button while the TTS is synthesizing, ask again right away, inject a TTS error and a hang at that point, and put a generation ID on the sentence, PCM and frame events. Apart from the once-a-second metrics ticks, every turn event is published from the event-loop side; the one exception is
llm_done, which comes from the worker side, so an ID would show exactly whether anything from the old answer crosses over. I plan to add it next to the existing traces.