How to Build a Low-Latency Voice Bot with Streaming STT and TTS

A voice bot can be described in three steps:

1. Listen to the user.
2. Decide what to say.
3. Speak the response.

That sounds straightforward until you try to make the loop happen in real time.

Speech recognition has latency. Your reasoning layer has latency. Speech synthesis has latency. Network transit adds more. Audio arrives continuously rather than as neat request-response messages, and users expect to interrupt, hesitate, change direction, and speak over the system.

That is why building a usable voice bot is less about connecting three APIs and more about designing the entire audio pipeline around streaming, turn-taking, cancellation, and latency.

This article walks through the architecture behind that pipeline, using the same STT → reasoning → TTS pattern that powers many production voice applications. Smallest AI provides these layers through Pulse, Electron, Lightning, and its voice-agent platform, but the architectural principles apply regardless of which components you choose.

What a voice bot actually does

At the highest level, the loop looks like this:

User audio
    ↓
Speech-to-Text
    ↓
Transcript
    ↓
LLM or deterministic logic
    ↓
Response text
    ↓
Text-to-Speech
    ↓
Audio response

The architecture is easy to understand.

The difficulty is that every boundary adds delay.

If you treat the system as a conventional sequence of synchronous API calls, you can easily end up with this behavior:

record full utterance
→ wait for transcription
→ wait for complete model response
→ wait for complete speech generation
→ begin playback

Every stage blocks the next.

For offline processing, that may be acceptable. For a live conversation, it feels slow.

A real-time voice bot should instead try to overlap work wherever possible:

audio ───────────────►
       STT ──────────►
            LLM ─────►
                 TTS ─────────► audio

Streaming is what turns a serial pipeline into an overlapping one.

Choose the stack before writing the orchestration layer

Three components determine most of the behavior of a traditional voice bot:

• Speech-to-text for incoming audio
• A reasoning or decision layer
• Text-to-speech for outgoing audio

Choosing each component independently gives you flexibility, but it also creates more interfaces to manage.

Authentication, connection lifecycle, audio formats, retries, versioning, latency measurement, observability, and billing can all become integration concerns.

For some applications, that control is worth it. For others, a unified platform reduces the amount of orchestration code your team has to own.

Pick STT for conversations, not offline transcription

A good transcription model for prerecorded files is not automatically a good STT engine for a voice bot.

For real-time use, you should care about:

• Streaming transcription
• Stable partial transcripts
• Endpoint detection
• Word timestamps when required
• Language coverage
• Performance on your actual acoustic environment
• Whether processing can keep up with live audio

One useful metric is real-time factor, or RTF.

If a recognizer takes one second to process one second of speech, its RTF is 1.0. For a live system, you generally want processing comfortably below that level so work does not accumulate behind the incoming audio stream.

Smallest AI’s Pulse is designed for real-time and prerecorded speech recognition. For a voice bot, the important architectural capability is that transcription can happen while speech is still arriving rather than after an entire recording has been uploaded.

Pick TTS for time to first audio

For a live voice application, speech quality is only one part of TTS performance.

You also need to know how quickly playback can begin.

A system that eventually produces excellent audio but leaves a long silence before the first syllable still feels broken in conversation.

Streaming TTS addresses this by returning audio incrementally instead of waiting for synthesis of the entire response.

Smallest AI’s Lightning is designed around this streaming model and can be used as the speech-generation layer of a real-time pipeline.

Other practical considerations include:

• Voice consistency
• Language support
• Audio encoding
• Streaming behavior
• Prosody and pacing
• Voice cloning when your application requires a consistent custom voice

Decide whether you actually need an LLM

Not every voice bot needs a general-purpose language model.

Consider a phone router that only needs to recognize requests such as:

billing
technical support
cancel subscription
check order
speak to an agent

A deterministic intent layer may be faster, cheaper, and easier to audit than a large model.

An LLM becomes more useful when users can phrase requests unpredictably or when the system needs multi-turn reasoning, tool calls, or context-sensitive responses.

Smallest AI’s Electron can act as the conversational reasoning layer in this architecture. You can also use another compatible reasoning system if your application requires it.

If you do not want to assemble the orchestration and agent infrastructure yourself, the Smallest AI voice-agent platform provides a managed path for building and deploying voice agents.

Architecture blueprint: make every stage stream

A practical voice bot architecture looks roughly like this:

                      ┌─────────────────┐
Microphone / Call ───►│ Streaming STT   │
                      └────────┬────────┘
                               │
                        partial + final
                          transcripts
                               │
                               ▼
                      ┌─────────────────┐
                      │ Orchestrator    │
                      │ VAD / state /   │
                      │ cancellation    │
                      └────────┬────────┘
                               │
                               ▼
                      ┌─────────────────┐
                      │ LLM or rules    │
                      └────────┬────────┘
                               │
                         streamed text
                               │
                               ▼
                      ┌─────────────────┐
                      │ Streaming TTS   │
                      └────────┬────────┘
                               │
                               ▼
                         Audio playback

WebSockets are a natural transport for this kind of architecture because the connection stays open while audio and events move continuously.

The important idea is not WebSockets specifically. It is avoiding a workflow where every stage waits for a complete payload from the previous one.

For a deeper breakdown of how the STT, LLM, TTS, tools, and latency budget interact, Smallest AI’s guide to designing voice assistants around STT, LLMs, TTS, and latency is a useful companion to this implementation view.

Create and store the API key

If you are prototyping the pipeline with the Smallest AI API, keep the API key in an environment variable rather than hard-coding it into the application.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

export SMALLEST_API_KEY="your-api-key-here"

Every authenticated request sends the value through the Authorization header:

Authorization: Bearer <SMALLEST_API_KEY value>

Keep the key on your server.

Do not expose it in browser JavaScript, mobile application code, public repositories, screenshots, query parameters, or client-side logs.

For production deployments, put the value in the secret-management system used by your infrastructure rather than a checked-in configuration file.

Building the core voice loop

The easiest way to reason about the application is as several concurrent tasks rather than one long function.

You typically have independent flows for:

microphone → STT
STT events → conversation state
conversation state → reasoning
reasoning output → TTS
TTS audio → playback
user interruption → cancellation

An asynchronous runtime such as Python’s asyncio, Node.js, or another event-driven environment maps naturally to this model.

1. Capture and stream incoming audio

Start with the audio source.

Depending on the application, that may be:

• A browser microphone
• A native mobile microphone
• A WebRTC session
• A telephony stream
• A SIP connection
• Another realtime media transport

Do not accumulate several seconds of speech before beginning transcription.

Send audio frames to STT continuously.

Frame sizes in the tens-of-milliseconds range are common because they balance responsiveness against packet and processing overhead. A 16 kHz stream is also a common baseline for speech-recognition workloads, although your actual settings should match the STT service and source audio you are using.

The recognizer can then begin returning partial transcription results while the person is still speaking.

2. Handle VAD and endpointing separately

Recognizing words is only part of the problem.

The bot also has to determine when the user has finished a turn.

This is where voice activity detection and endpointing become critical.

Consider someone saying:

“Can you… uh… move my appointment to Friday?”

A simplistic silence timer may decide the utterance ended after “Can you.”

Set the threshold too aggressively and the bot interrupts natural pauses.

Set it too conservatively and every turn gains an uncomfortable delay.

The correct value depends heavily on the environment and task.

A tightly scripted support call may tolerate aggressive turn-taking. A conversational assistant whose users frequently pause to think may need more patience.

Do not tune endpointing exclusively with clean test recordings. Use audio that represents actual microphones, background noise, accents, network conditions, and speaking patterns from your deployment.

Do not wait for the entire model response

After STT produces a finalized user turn, the transcript enters the reasoning layer.

A common first implementation looks like this:

complete transcript
        ↓
generate entire LLM response
        ↓
send complete response to TTS
        ↓
play audio

It works, but it wastes time.

If your reasoning system supports streaming output, begin preparing speech before the entire answer exists.

You do not necessarily want to synthesize individual tokens. TTS needs enough context to produce natural phrasing and prosody.

A better strategy is to buffer the model output into speakable chunks such as complete clauses or sentences:

LLM tokens
   ↓
small text buffer
   ↓
natural speech boundary
   ↓
TTS

While TTS is synthesizing the first chunk, the model can continue generating the next.

That overlap is one of the most important ways to reduce perceived response time.

Stream synthesized audio back immediately

The same principle applies after text enters the TTS layer.

Do not wait for the complete audio file if your provider supports streaming.

Start playback as soon as enough audio has arrived.

The end-to-end flow becomes:

User still speaking
       │
       ├── STT processing audio
       │
User finishes
       │
       ├── final transcript
       │
       ├── reasoning begins
       │
       ├── first usable text chunk
       │
       ├── TTS begins
       │
       └── first audio begins playing
              while later text
              and audio are still
              being generated

For network delivery, codecs designed for interactive audio are usually better suited to real-time conversations than formats optimized primarily for downloadable media.

Opus is commonly used in real-time communication because it performs well at relatively low bitrates and is designed for interactive audio. Raw PCM is convenient when debugging or when downstream systems need uncompressed samples.

Whatever format you choose, avoid unnecessary encode/decode conversions between services. Every conversion creates another place for buffering, CPU work, or format mismatches.

Barge-in changes the architecture

A voice bot that cannot be interrupted will quickly feel artificial.

Suppose the bot starts saying:

“Your current subscription includes—”

and the user interrupts:

“Actually, I want to cancel it.”

The system should stop the outgoing response and process the new utterance.

That means several things have to happen almost simultaneously:

detect user speech
→ stop or cancel TTS generation
→ flush queued playback audio
→ preserve the new incoming speech
→ return control to the listening state

The important consequence is that your STT path cannot simply disappear while the bot is speaking.

You need to continue monitoring incoming audio so the system can detect the interruption.

This is also why echo cancellation matters. Without it, the microphone may feed the bot’s own synthesized speech back into STT, and the system can mistake its response for a new user utterance.

Barge-in is not a small feature you bolt onto the end of development. It affects the audio pipeline, state machine, playback buffer, VAD behavior, and cancellation strategy.

Design for it early.

Treat latency as a pipeline budget

Developers often optimize whichever component has the most obvious latency number.

That is not enough.

A conversational delay is the sum of several pieces:

endpoint detection
+ STT
+ network
+ reasoning
+ tool calls
+ TTS startup
+ playback buffering

Improving one component may not noticeably change the experience if another stage dominates the total.

Instrument the boundaries individually.

For each turn, record timestamps such as:

speech_started
speech_ended
final_transcript_received
llm_started
first_llm_chunk_received
tts_started
first_audio_received
playback_started

Then measure distributions rather than only averages.

P50 tells you what a typical user experiences.

P90 and P99 reveal the slow turns that users are more likely to remember as failures.

A related metric is task completion. A low-latency system that misunderstands users or fails to complete the requested workflow is not successful simply because it responds quickly.

Latency and task success have to be evaluated together.

Four mistakes that break voice bots outside the demo

1. Building a chatbot with a microphone attached

Voice is not just a different input widget for a text application.

You now have to handle:

• Silence
• Natural pauses
• Background noise
• Echo
• Interruptions
• Audio encoding
• Turn detection
• Playback state

Ignoring these problems is why many prototypes work perfectly at a developer’s desk and fall apart during real calls.

2. Batching every stage

If the system waits for:

full audio
→ full transcript
→ full model response
→ full TTS file

all of the delays are serialized.

Stream wherever the component supports it.

The goal is not merely faster APIs. It is overlapping stages.

3. Load testing HTTP requests instead of conversations

Fifty text API requests are not equivalent to fifty simultaneous calls.

Each live conversation can involve:

• A persistent connection
• Continuous audio
• STT state
• VAD state
• Conversation history
• TTS generation
• Audio playback
• Cancellation events

Load test the architecture using realistic concurrent audio sessions.

Watch memory, open connections, queue depth, model latency, reconnection behavior, and tail latency.

4. Treating one voice as correct for every workflow

The voice itself is part of the interface.

A scheduling assistant, support agent, sales workflow, and collections system may have very different tone requirements.

If your application needs a consistent custom voice, voice cloning is one option, but it should be evaluated alongside latency, language coverage, intelligibility, and the context in which the bot will speak.

Production features that the MVP usually skips

Getting audio through STT, a model, and TTS proves the basic pipeline.

It does not make the bot production-ready.

Multi-turn conversation state

A useful voice bot has to remember what happened earlier in the conversation.

For shorter sessions, you can carry recent turns into the model context.

Longer interactions require a strategy to keep context from growing indefinitely.

Common approaches include:

sliding window
summarization
retrieval of relevant past context
structured application state

Do not rely entirely on a raw transcript when the application has business state such as an appointment date, order ID, account selection, or workflow stage.

Store important state explicitly.

Telephony integration

If the bot will handle phone calls, speech generation is only part of the system.

You eventually encounter:

• SIP/PSTN connectivity
• Phone-number management
• DTMF
• Transfers
• Hold behavior
• Recording
• Regional routing
• Compliance requirements

At this point, teams must decide whether they want to own the telephony and orchestration infrastructure themselves or use an agent platform that abstracts some of it.

Smallest AI’s managed voice-agent platform is one option when you want the speech stack and agent infrastructure integrated rather than assembled independently.

Observability

Log enough information to reconstruct why a bad conversation happened.

Useful measurements include:

• STT latency
• Endpointing delay
• LLM time to first output
• Tool-call latency
• TTS startup latency
• End-to-end time to first audio
• Interruption frequency
• Connection failures
• Task-completion rate

Be thoughtful about what you store.

Voice interactions can contain sensitive information, and observability should not become an excuse to dump raw transcripts, credentials, or unnecessary user data into logs.

Should you build the stack or use a managed platform?

There is no universal answer.

Building the pipeline yourself gives you direct control over:

audio transport
STT
reasoning
tool execution
TTS
state
telephony
observability

That control can be valuable if voice is a core part of your product infrastructure.

But you also inherit every integration boundary.

A managed platform moves more of that responsibility into the provider.

Smallest AI’s stack combines Pulse for STT, Electron for conversational reasoning, Lightning for TTS, and its voice-agent tooling for orchestration and deployment.

The important engineering question is not whether a unified or modular architecture is always better.

It is how much of the real-time voice infrastructure your team wants to build, operate, monitor, and debug itself.

A practical build checklist

Before calling the voice bot production-ready, verify that you have made explicit decisions about:

• Streaming versus batching at every stage
• STT endpointing and VAD thresholds
• Audio sample rate and encoding
• LLM versus deterministic routing
• TTS chunk boundaries
• Barge-in and cancellation
• Echo handling
• Conversation-state management
• Tool-call latency
• Reconnection behavior
• Concurrent session limits
• P50, P90, and P99 latency measurement
• Secret management
• Sensitive transcript logging
• Real-world audio testing

Most production failures happen because one of these areas was treated as an implementation detail instead of an architectural decision.

The real optimization is overlap

The basic voice-bot loop has not changed:

listen → think → speak

What separates a responsive voice application from a sluggish one is how much work happens concurrently.

Stream audio into STT instead of waiting for a recording.

Start reasoning as soon as the user’s turn is complete.

Stream model output toward TTS instead of waiting for the full response.

Start playback as soon as audio is available.

Keep listening while the bot speaks so users can interrupt.

Measure every boundary instead of treating latency as one opaque number.

Once those pieces are in place, the technology choices become easier to evaluate because you can judge them inside the architecture that will actually run in production.

If you want to prototype the pipeline using Pulse, Electron, and Lightning, start building with the Smallest AI API and test it against your own audio, latency targets, and conversation patterns.

FAQ

Which programming language should I use for a voice bot?

Python is a practical choice when you are already working with AI APIs and asynchronous processing. Node.js is also a strong option for browser, WebSocket, and WebRTC-heavy backends.

For high-concurrency deployments, other languages may make sense depending on your infrastructure.

The best choice is usually the runtime your team can operate reliably while handling persistent connections and concurrent audio streams.

How do I reduce voice-bot latency?

Start with architecture rather than micro-optimizations.

Stream STT, reasoning output, and TTS. Tune endpointing carefully. Avoid unnecessary audio transformations. Reduce network hops where possible. Measure time to first output for each component, and inspect tail latency rather than relying only on averages.

Do I need an LLM for every voice bot?

No.

A deterministic intent or rules layer can be the better choice for narrow, predictable workflows.

Use an LLM when the application needs flexible language understanding, multi-turn reasoning, dynamic responses, or tool selection.

Many production systems use a combination of deterministic logic and models.

What is the difference between a voice bot and a voice assistant?

The terms overlap.

A voice bot usually describes a more task-specific system: customer support, scheduling, routing, qualification, or another bounded workflow.

A voice assistant often implies broader, more open-ended interaction.

Both can use the same underlying STT → reasoning → TTS architecture. The main difference is usually the scope of the jobs they are expected to handle.