Voice Assistants Are Designed in the Silence

A voice assistant can be technically correct and still feel broken.

The failure often begins after the user finishes speaking.

Nothing dramatic happens. There is simply a pause—long enough for the user to wonder whether the system heard them, but short enough for every individual service dashboard to report an acceptable result.

That pause is the product.

Users do not experience speech recognition, a language model, a tool call, and speech synthesis as separate services. They experience one conversational turn.

When any handoff is late, uncertain, or difficult to cancel, the interaction starts to feel more like a phone tree than a conversation.

The useful question is not:

Which model is fastest?

It is:

How quickly can the system begin a trustworthy, speakable response?

This guide focuses on what developers can measure, instrument, and improve in a production voice pipeline.

Measure One Conversational Turn, Not Four Services

A typical cascaded voice assistant looks simple:

Microphone
  ↓
Streaming speech-to-text
  ↓
Turn decision
  ↓
Language model and optional tools
  ↓
Stable text buffer
  ↓
Streaming text-to-speech
  ↓
Interruptible playback

This diagram becomes misleading when every arrow is treated as a clean, serial boundary.

In a responsive system, useful work overlaps:

• Audio is transcribed while the user is speaking.
• The model may begin after the turn is committed but before every transcript artifact is finalized.
• TTS can start when a stable, speakable clause exists instead of waiting for the complete response.

Do not compress the entire pipeline into one ambiguous time to first audio metric.

Record the important boundaries separately.

Metric	What it measures
Turn-decision delay	Time between the user’s last speech frame and the system committing the turn
STT finalization delay	Time required to produce a transcript that is safe to send downstream
LLM time to first token	Time between the model request and its first generated token
Time to first speakable chunk	Time between the model request and the first stable clause that TTS can safely render
TTS time to first audio	Time between the synthesis request and the first playable audio chunk
End-of-turn to playback	Time between the user’s last speech frame and audio beginning on the client

The last metric represents the user-facing experience.

The other metrics explain why that experience occurred.

Turn Detection Is Not the Same as VAD

Voice activity detection answers a narrow question:

Does this audio frame contain speech?

Turn detection answers a harder question:

Has the speaker finished their thought?

VAD can contribute evidence, but silence alone is not enough.

A production turn detector may also consider:

• Finalized and interim transcript timing
• Transcript confidence or stability
• Punctuation and semantic completeness
• Domain-specific patterns
• The user’s speaking rate
• Recent pause behaviour
• The cost of interrupting versus waiting

For example, a phone number may continue after a brief pause. A support caller may hesitate before stating an account identifier. A user may pause naturally in the middle of a longer question.

A fixed silence threshold creates two opposite failure modes:

1. Commit too quickly: the assistant cuts off a user who paused mid-sentence.
2. Wait too long: every response feels hesitant.

Endpointing remains heuristic.

Background noise can prevent reliable silence detection, while transcript-based gap detection behaves differently and may perform better for some utterances.

There is no universal endpointing threshold. Tune it using real audio from the intended deployment environment.

STT Accuracy Should Focus on Consequential Errors

Word error rate is useful, but it should not be treated as a universal pass-or-fail score for a voice assistant.

An incorrect filler word may have no downstream effect.

One incorrect digit in an account number, a misspelled surname, or a reversed negation may change the entire action.

Conventional WER assigns similar importance to errors that have very different effects on meaning and task completion.

For a voice agent, evaluate at least three layers.

1. Transcription quality

Measure WER or another suitable ASR metric.

2. Entity accuracy

Evaluate high-impact entities such as:

• Names
• Dates
• Amounts
• Phone numbers
• Account identifiers
• Addresses
• Domain-specific terminology

3. Task success

Determine whether the downstream system:

• Understood the user’s intent
• Extracted the correct information
• Performed the correct action

Test using the audio users will actually produce:

• Noisy rooms
• Phone codecs
• Weak connections
• Accents
• Hesitations
• Overlapping speech
• Low-quality microphones

A clean studio recording is not a substitute for a deployment test set.

Streaming ASR can reduce emission delay, but it also introduces unstable partial hypotheses. Latency and recognition quality must therefore be evaluated together.

The First Token Is Not Yet a Spoken Answer

LLM time to first token is important, but it does not represent the end of the model stage.

Suppose the model begins with:

Sure — let me...

The first token arrived quickly, but the assistant still has nothing useful to say.

TTS may also need to wait for a stable clause so it does not synthesize an opening that later becomes awkward, incomplete, or incorrect.

A stronger voice response front-loads the answer:

Your appointment is confirmed for Thursday at 3 PM.

I can also send a reminder.

The first sentence is:

• Complete
• Useful
• Independently speakable
• Safe to send to TTS

This creates another useful metric:

Time to first speakable chunk

It includes the model’s TTFT plus the time required to accumulate a safe synthesis boundary.

A simple clause buffer may look like this:

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

const boundary = /[.!?]\s$|[,;:]\s$/;

let pending = "";

function onModelToken(token: string) {
  pending += token;

  const enoughText = pending.trim().length >= 24;
  const hasBoundary = boundary.test(pending);

  if (enoughText && hasBoundary) {
    tts.enqueue(pending);
    pending = "";
  }
}

function onModelComplete() {
  if (pending.trim()) {
    tts.enqueue(pending);
  }
}

This example is intentionally simple.

A production buffer should also consider:

• Abbreviations
• Numbers
• Markdown or other markup
• Pronunciation hints
• Language-specific punctuation
• Maximum waiting time
• Whether enqueued speech can still be cancelled

Tool Latency Only Hurts When It Blocks the Critical Path

Tool calls are often described as additive latency.

For example:

A 300 ms lookup adds 300 ms to the response.

That is true only when the lookup sits directly on the serial critical path.

Some tool latency can be hidden or reduced by:

• Prefetching likely context after a session starts
• Caching results with a clear freshness policy
• Running independent tools in parallel
• Starting a truthful acknowledgement while a slow operation continues
• Using deterministic routing when a model decision is unnecessary
• Cancelling stale work when the user changes direction

A useful acknowledgement might be:

I’ll check the live inventory now.

However, filler speech should not be used merely to disguise arbitrary latency.

An acknowledgement is useful only when it communicates real progress and does not make an unsupported promise.

Trace every tool call with:

• Queue time
• Network time
• Server processing time
• Result size
• Cache status
• Retry count
• Whether it blocked the first speakable chunk

The final field is often more useful than the tool’s total duration.

Streaming Is Controlled Overlap

“Stream every stage” sounds attractive, but it is too absolute.

Partial output is valuable only when it is useful and safely reversible.

Potential problems include:

• Partial transcripts changing
• Tools returning only atomic results
• TTS speaking text the model would have revised
• Speculative work increasing cost
• Cancellation becoming harder
• Stale results entering the conversation state

A better rule is:

Stream when partial output is useful, and overlap work only when errors can be contained or reversed.

Incremental TTS can begin synthesis from partial text while later segments are still being generated.

However, no research result or benchmark justifies assuming the same millisecond improvement across every:

• Model
• Sentence
• Device
• Language
• Network
• Deployment architecture

Low-latency engineering also extends beyond model inference.

Media transport, jitter, packet loss, WebRTC behaviour, session ownership, routing, and infrastructure placement all influence the pause.

Build an Explicit Latency Budget

A latency budget should be a design constraint, not a chart created after launch.

Start with an end-of-turn-to-playback target, then assign provisional limits to each stage on the critical path.

Here is an illustrative and deliberately aggressive 800 ms budget.

This is a planning example, not an industry benchmark.

Stage	Example budget
Turn decision	150 ms
Transcript stabilization	100 ms
First speakable model chunk	300 ms
First playable TTS audio	150 ms
Transport and client buffering	100 ms
Total	800 ms

The allocation should change based on the product.

A hands-free command may prioritize speed.

A medical intake flow may tolerate a longer delay to reduce the chance of interrupting the user.

A tool-heavy transaction may need a short acknowledgement before the final answer.

Measure distributions, not averages

At minimum, report:

• p50
• p95
• p99

Break these metrics down by:

• Interaction type
• Geography
• Network type
• Language
• Device
• Telephony provider
• Tool-free versus tool-dependent turns
• Successful turns
• Interrupted turns
• Cancelled turns
• Retried turns

An acceptable p50 can hide a painful p95.

Instrument the Complete Turn

The following TypeScript example records useful event boundaries without coupling the implementation to a specific STT, LLM, or TTS provider.

const marks = new Map<string, number>();

function mark(name: string, at = performance.now()) {
  marks.set(name, at);
}

function duration(start: string, end: string) {
  const startTime = marks.get(start);
  const endTime = marks.get(end);

  if (startTime === undefined || endTime === undefined) {
    return undefined;
  }

  return endTime - startTime;
}

function voiceTurnMetrics() {
  return {
    turnDecisionMs: duration(
      "speech_last_frame",
      "turn_committed"
    ),

    transcriptReadyMs: duration(
      "turn_committed",
      "stt_final"
    ),

    llmTtftMs: duration(
      "llm_started",
      "llm_first_token"
    ),

    firstSpeakableMs: duration(
      "llm_started",
      "first_speakable_chunk"
    ),

    ttsFirstAudioMs: duration(
      "tts_started",
      "tts_first_audio"
    ),

    clientBufferMs: duration(
      "tts_first_audio",
      "playback_started"
    ),

    endToEndMs: duration(
      "speech_last_frame",
      "playback_started"
    ),
  };
}

Call mark() from the actual callbacks in your pipeline:

mark("speech_last_frame", vad.lastSpeechTimestamp());
mark("turn_committed");
mark("stt_final");
mark("llm_started");
mark("llm_first_token");
mark("first_speakable_chunk");
mark("tts_started");
mark("tts_first_audio");
mark("playback_started");

Use one trace ID across:

• Browser or phone gateway
• STT service
• Orchestration layer
• External tools
• Language model
• TTS service
• Playback client

Without cross-service correlation, teams often optimize the service with the most visible dashboard instead of the stage responsible for the user’s wait.

When comparing a managed voice stack with a custom pipeline, apply the same:

• Event boundaries
• p50 measurements
• p95 measurements
• p99 measurements
• Accuracy tests
• Cancellation tests

An integrated stack should not automatically be assumed to have low end-to-end latency.

Barge-In Must Cancel the Old Turn

In a full-duplex assistant, inbound audio capture and speech detection should normally remain active while the assistant is speaking.

When new user speech is confirmed, the system must stop treating the old response as current.

The control path may resemble this:

async function onUserSpeechStarted() {
  generation.abort();

  await tts.cancel();

  playback.stopAndFlush();

  tools.cancelNonReusableWork();

  conversation.truncateAssistantMessage({
    toPlayedAudioTimestamp: playback.lastPlayedTimestamp(),
  });
}

The exact APIs will differ, but the responsibilities remain similar.

The system should:

1. Cancel model generation.
2. Cancel speech synthesis.
3. Stop and flush queued playback.
4. Discard tool results that are no longer relevant.
5. Preserve only the response portion the user actually heard.
6. Continue processing incoming speech without clipping its beginning.

Barge-in cannot be added cleanly as a final UI feature.

It affects:

• Audio capture
• Session state
• Model context
• Tool cancellation
• Playback architecture
• Conversation history

Production Starts Where the Demo Ends

A convincing demo proves that the happy path can answer.

Production requires testing for real human behaviour and imperfect infrastructure.

Before launch, test:

• Quiet audio
• Noisy audio
• Reverberant rooms
• Low-bitrate audio
• Short commands
• Long and hesitant utterances
• Mid-sentence pauses
• Self-corrections
• Names and phone numbers
• Dates and monetary amounts
• Domain-specific terminology
• Tool-free calls
• Cached tool calls
• Slow tool calls
• Failed and retried tool calls
• Barge-in during early playback
• Barge-in during late playback
• Packet loss
• Jitter
• Reconnects
• Duplicated events
• Multiple languages
• Code-switching
• Long conversations
• Context compaction
• Realistic concurrency

Also test failure semantics.

Ask questions such as:

• What happens if a tool succeeds after the user interrupts?
• What happens if TTS emits audio after cancellation?
• What happens if two final transcripts arrive?
• What happens if the client reconnects while audio remains queued?
• What happens if a stale tool result returns after the conversation has changed?

A low-latency system that performs stale actions is not a good system.

When reviewing an implementation, study how it represents:

• Processors
• Transports
• Interruptions
• Events
• Cancellation
• Session state

Do not copy default settings blindly.

Validate each design choice against your own traces, traffic, users, and failure modes.

The Pause Is the Architecture

The fastest model will not rescue a pipeline with:

• Slow turn detection
• Serial tool calls
• Unstable partial transcripts
• Excessive client buffering
• Weak cancellation
• Poor session state management

Measure the complete turn from the user’s last speech frame to audible playback.

Separate that duration into named stages.

Optimize the stage dominating p95, then test the change against:

• Recognition accuracy
• Entity accuracy
• Task success
• Cancellation behaviour
• User interruption patterns

A natural voice assistant is not simply a collection of fast components.

It is one coordinated participant whose timing, state, and failure modes have been designed as a whole.

Build and Measure a Voice Agent

Ready to test this architecture in a working voice stack?

Build a voice agent with the Smallest.ai API and instrument the event boundaries described above.

Compare your:

• p50 end-to-end latency
• p95 end-to-end latency
• Time to first speakable chunk
• Barge-in cancellation time
• Tool-dependent latency

against your current production pipeline.