A voice assistant can be technically correct and still feel broken.
The failure often begins after the user finishes speaking. Nothing dramatic happens. There is simply a pause: long enough for the user to wonder whether the system heard them, but short enough for each service dashboard to report an acceptable result.
That pause is the product.
Users do not experience speech recognition, a language model, a tool call, and speech synthesis as separate services. They experience one conversational turn. If any handoff is late, uncertain, or difficult to cancel, the whole interaction begins to feel like a phone tree rather than a conversation.
The useful question is therefore not, "Which model is fastest?" It is:
How quickly can the system begin a trustworthy, speakable response?
This article develops that idea from the original voice-assistant architecture guide, but focuses on what developers can measure, instrument, and change in a production pipeline.
The guide was published by Smallest.ai, whose work focuses on real-time speech models and voice-agent systems.
Measure one turn, not four services
A typical cascaded assistant looks simple:
microphone
-> streaming speech-to-text (STT)
-> turn decision
-> language model and optional tools
-> stable text buffer
-> streaming text-to-speech (TTS)
-> interruptible playback
The diagram becomes misleading when every arrow is treated as a clean, serial boundary. In a responsive system, useful work overlaps. Audio is transcribed while the user is speaking. The model may begin after the turn is committed but before every transcript artifact is finalized. TTS can start when a stable, speakable clause exists rather than waiting for the complete answer.
Do not compress all of this into a single ambiguous "time to first audio" metric. Record the boundaries separately:
Turn-decision delay: from the user's last speech frame to the point at which the system commits the turn.
STT finalization delay: the time needed to produce a transcript that is safe to send downstream.
LLM time to first token (TTFT): from the model request to its first token.
Time to first speakable chunk: from the model request to the first stable clause that TTS can render safely.
TTS time to first audio: from the synthesis request to the first playable audio chunk.
End-of-turn-to-playback: from the user's last speech frame to audio actually beginning at the client.
The last measurement is the user-facing result. The others explain why it happened.
Turn detection is not the same as VAD
Voice activity detection answers a narrow question: does this audio frame contain speech?
Turn detection answers a harder question: has the speaker finished the thought?
VAD can contribute evidence, but silence alone is not enough. A production turn detector may also use:
finalized and interim transcript timing;
confidence or transcript stability;
punctuation or semantic completeness;
domain-specific patterns, such as a phone number that may continue;
the user's speaking rate and recent pause behavior;
the cost of interrupting versus waiting in the current workflow.
The distinction matters because a fixed silence threshold creates opposite failure modes. Commit too quickly and the assistant cuts off a user who paused mid-sentence. Wait too long and every turn feels hesitant.
Endpointing remains heuristic: background noise can prevent reliable silence detection, while transcript-based gap detection behaves differently and may be more useful for some utterances. The correct threshold is not an industry constant; tune it against real audio from the deployment environment.
STT accuracy is about consequential errors
Word error rate is useful, but it should not be treated as a universal pass/fail score for an assistant.
An incorrect filler word may have no downstream effect. One wrong digit in an account number, a misspelled surname, or a reversed negation may change the entire action. Conventional WER gives equal weight to errors with very different effects on meaning and readability, so it should be paired with entity-level and task-level evaluation.
For a voice agent, evaluate at least three layers:
Transcription quality: WER or another suitable ASR metric.
Entity accuracy: names, dates, amounts, identifiers, addresses, and domain terminology.
Task success: whether the downstream system understood the intent and performed the correct action.
Test with the audio users will actually produce: noisy rooms, phone codecs, weak connections, accents, hesitations, and overlapping speech. A clean studio recording is not a substitute for a deployment test set.
Streaming ASR can reduce emission delay, but it introduces unstable partial hypotheses. Research such as FastEmit demonstrates that latency and recognition quality must be evaluated together; its measured gains belong to the tested models and datasets, not to every streaming implementation.
The first token is not yet a spoken answer
LLM TTFT is important, but it is not the end of the model stage.
Suppose the model emits:
Sure — let me...
The first token arrived, but the assistant still has nothing useful to say. TTS may also need to wait for a stable clause so it does not synthesize an opening that later becomes awkward or incorrect.
A better voice prompt front-loads the answer:
Your appointment is confirmed for Thursday at 3 PM.
I can also send a reminder if you want one.
The first sentence is complete, useful, and independently speakable.
That creates another metric worth tracking: time to first speakable chunk. It includes model TTFT plus the time needed to accumulate a safe synthesis boundary.
A simple clause buffer might look like this:
const boundary = /[.!?]\s$|[,;:]\s$/;
let pending = "";
function onModelToken(token: string) {
pending += token;
const enoughText = pending.trim().length >= 24;
const hasBoundary = boundary.test(pending);
if (enoughText && hasBoundary) {
tts.enqueue(pending);
pending = "";
}
}
function onModelComplete() {
if (pending.trim()) tts.enqueue(pending);
}
This is intentionally simple. A production buffer should also consider abbreviations, numbers, markup, pronunciation hints, language-specific punctuation, maximum wait time, and whether already-enqueued speech can be cancelled.
Tool latency only hurts when it blocks the critical path
Tool calls are often described as an additive delay: a 300 ms lookup supposedly adds exactly 300 ms to the turn.
That is true only when the lookup sits on the serial critical path.
Some work can be hidden or reduced through:
prefetching likely context after the session starts;
caching results with a clear freshness policy;
running independent tools in parallel;
starting a safe verbal acknowledgement while a slow operation continues;
using deterministic routing when a model decision is unnecessary;
cancelling stale work when the user changes direction.
Do not use filler speech to disguise arbitrary latency. An acknowledgement such as "I'll check the live inventory" is useful only when it communicates real progress and does not make an unsupported promise.
Trace every tool call with its queue time, network time, server time, result size, cache status, retry count, and whether it blocked the first speakable chunk. That final field is more informative than duration alone.
Streaming is controlled overlap
"Stream every stage" sounds attractive but is too absolute.
Partial output is valuable only when it is useful and safely reversible. Some tools return atomic results. Partial transcripts can change. Premature TTS can speak text the model would have revised. Speculation can also increase cost and make cancellation harder.
A better rule is:
Stream where partial output is useful, and overlap work only when errors can be contained or reversed.
Incremental TTS research demonstrates how synthesis can begin from partial text while later segments are still being generated. The prefix-to-prefix TTS study is useful evidence for the architecture, but it does not justify a universal millisecond saving for every model, sentence, device, or network.
The engineering work behind low-latency voice systems extends beyond inference. Media transport, jitter, packet loss, session ownership, routing, WebRTC behavior, and infrastructure placement all affect the pause and belong in the same design discussion.
Build an explicit latency budget
A latency budget is a design constraint, not a chart created after launch. Start with an end-of-turn-to-playback target, then assign provisional limits to the stages on the critical path.
Here is an illustrative, deliberately aggressive 800 ms budget. It is an example for planning, not an industry benchmark:
Stage
Example budget
Turn decision
150 ms
Transcript stabilization
100 ms
First speakable model chunk
300 ms
First playable TTS audio
150 ms
Transport and client buffering
100 ms
Total
800 ms
Your allocation will change with the product. A hands-free command may prioritize speed. A medical intake flow may tolerate a longer wait to avoid interrupting the speaker. A tool-heavy transaction may need an acknowledgement before the final answer.
Measure distributions rather than averages. At minimum, report p50, p95, and p99 by:
interaction type;
geography and network;
language;
device or telephony provider;
tool-free versus tool-dependent turns;
successful, interrupted, cancelled, and retried turns.
An acceptable p50 can hide a painful p95.
Instrument the complete turn
The following TypeScript example records the event boundaries without tying the implementation to a specific STT, model, or TTS provider:
const marks = new Map();
function mark(name: string, at = performance.now()) {
marks.set(name, at);
}
function duration(start: string, end: string) {
const a = marks.get(start);
const b = marks.get(end);
return a === undefined || b === undefined ? undefined : b - a;
}
function voiceTurnMetrics() {
return {
turnDecisionMs: duration("speech_last_frame", "turn_committed"),
transcriptReadyMs: duration("turn_committed", "stt_final"),
llmTtftMs: duration("llm_started", "llm_first_token"),
firstSpeakableMs: duration("llm_started", "first_speakable_chunk"),
ttsFirstAudioMs: duration("tts_started", "tts_first_audio"),
clientBufferMs: duration("tts_first_audio", "playback_started"),
endToEndMs: duration("speech_last_frame", "playback_started"),
};
}
Call mark() from the actual callbacks in the pipeline:
mark("speech_last_frame", vad.lastSpeechTimestamp());
mark("turn_committed");
mark("stt_final");
mark("llm_started");
mark("llm_first_token");
mark("first_speakable_chunk");
mark("tts_started");
mark("tts_first_audio");
mark("playback_started");
Use one trace ID across the browser or phone gateway, STT, orchestration layer, tools, model, TTS, and playback client. Without cross-service correlation, teams often optimize the service with the most visible dashboard rather than the stage responsible for the user's wait.
If you are comparing a managed stack with a custom pipeline, Smallest.ai's voice-agent platform is one integrated implementation to evaluate. Apply the same event boundaries and p50, p95, and p99 measurements to it—or any other platform—rather than assuming that an integrated stack is automatically low latency.
Barge-in must cancel the old turn
In a full-duplex assistant, inbound audio capture and speech detection normally remain active while the assistant speaks. When new user speech is confirmed, the system should stop treating the old response as current.
The control path may resemble:
async function onUserSpeechStarted() {
generation.abort();
await tts.cancel();
playback.stopAndFlush();
tools.cancelNonReusableWork();
conversation.truncateAssistantMessage({
toPlayedAudioTimestamp: playback.lastPlayedTimestamp(),
});
}
The exact APIs will differ, but the responsibilities are stable:
cancel model generation;
cancel synthesis;
stop and flush queued playback;
discard tool results that are no longer relevant;
preserve only the portion of the assistant response the user actually heard;
continue processing the incoming speech without clipping its beginning.
Barge-in cannot be added cleanly as a final UI feature. It changes audio capture, state management, model context, tool cancellation, and playback architecture.
Production starts where the demo ends
A convincing demo proves that the happy path can answer. Production requires tests for human behavior and imperfect infrastructure.
Before launch, include:
quiet, noisy, reverberant, and low-bitrate audio;
short commands and long, hesitant utterances;
mid-sentence pauses and self-corrections;
names, phone numbers, dates, amounts, and domain terminology;
tool-free, cached, slow, failed, and retried tool calls;
barge-in during early and late playback;
packet loss, jitter, reconnects, and duplicated events;
multiple languages and code-switching;
long conversations with context compaction;
p95 latency under realistic concurrency.
Also test the failure semantics. What happens if the tool succeeds after the user interrupts? If TTS emits audio after cancellation? If two final transcripts arrive? If the client reconnects while audio is queued? A low-latency system that produces stale actions is not a good system.
When reviewing an implementation, study how it represents processors, transports, interruption, events, cancellation, and session state. Do not copy defaults blindly; validate each design choice against your own latency traces and failure modes.
The pause is the architecture
The fastest model will not rescue a pipeline with slow turn detection, serial tool calls, unstable partial transcripts, or excessive audio buffering.
Measure the complete turn from the user's last speech frame to audible playback. Separate that duration into named stages. Optimize the stage that dominates p95, then test the change against accuracy, cancellation behavior, and task success.
A natural voice assistant is not a collection of fast components. It is one coordinated participant whose timing, state, and failure modes have been designed as a whole.
Ready to test this architecture in a working stack? Open Smallest.ai's self-serve application, build a voice agent, instrument the event boundaries above, and compare its p50 and p95 end-to-end latency with your current pipeline.
Top comments (0)