DEV Community

Todd Sullivan
Todd Sullivan

Posted on

I Made My Voice Agent Feel Faster by Streaming Sentences, Not Audio

#ai #python #devops #voiceai

The annoying thing about voice agents is that “the model is fast” does not mean the experience is fast.

I had a small voice assistant running on a local device, talking to a hosted chat backend. The actual LLM call was only one part of the wait. The full path looked more like this:

  1. wake word detection
  2. speech recognition
  3. authenticated /chat call
  4. model response
  5. local TTS synthesis
  6. audio playback

If you wait for step 4 to finish before starting step 5, the user hears nothing until the entire reply is done. That feels dead, even when the backend is technically fine.

So I changed the contract. The hardware client now calls the chat endpoint with stream_tts: true:

response = self.session.post(
    f"{CHAT_API_BASE}/chat",
    json={"message": message, "stream_tts": True},
    timeout=30,
    stream=True,
)
Enter fullscreen mode Exit fullscreen mode

The backend yields text chunks as they arrive from the model. The device keeps a small buffer, splits complete sentences, and starts synthesizing each sentence immediately:

_SENTENCE_BOUNDARY_RE = re.compile(r'(?<=[.!?])\s+')

def split_complete_sentences(buffer: str) -> tuple[list[str], str]:
    *sentences, remainder = _SENTENCE_BOUNDARY_RE.split(buffer)
    return [s.strip() for s in sentences if s.strip()], remainder
Enter fullscreen mode Exit fullscreen mode

That is deliberately boring. Not phoneme streaming. Not a custom audio protocol. Just sentence-level pipelining.

The next useful bit was overlapping synthesis and playback. A single background worker waits for synthesized WAV files and plays them in order, while a one-worker ThreadPoolExecutor starts rendering the next sentence as soon as it is complete.

for sentence in sentences:
    future = synth_executor.submit(self.tts_engine.synthesize, sentence)
    playback_queue.put((sentence, future))
Enter fullscreen mode Exit fullscreen mode

That removed the worst gap: “sentence one finished playing, now start thinking about sentence two’s audio.” The hardware now does the obvious thing a human expects — keep talking.

I also cut backend time-to-first-token by doing less. For this conversational path, I turned off extended model thinking:

_NO_THINKING = types.ThinkingConfig(thinking_budget=0)
Enter fullscreen mode Exit fullscreen mode

And I stopped advertising Google Search on every request. The search tool is only added when the prompt smells like it needs current/external information. Most turns do not.

if self._needs_search(built_contents):
    all_functions.append(self.google_search)
Enter fullscreen mode Exit fullscreen mode

The result was about a 5x cut in chat time-to-first-byte for the common path, plus a much better perceived response because speech starts before the full answer exists.

The lesson was not “stream everything.” It was smaller than that:

  • stream at the boundary the product can actually use
  • overlap the slow local work with the slow network work
  • do not give the model tools or reasoning budget unless the turn needs them
  • log chunks, sentence counts, gaps, and total time so you can see where the pause moved

Voice agents do not need heroic architecture to feel better. Sometimes the fix is a regex, a queue, and deleting the expensive defaults.

Top comments (0)