DEV Community: Smallest AI

Voice Assistants Are Designed in the Silence

Smallest AI — Mon, 27 Jul 2026 16:17:29 +0000

A voice assistant can be technically correct and still feel broken.
The failure often begins after the user finishes speaking. Nothing dramatic happens. There is simply a pause: long enough for the user to wonder whether the system heard them, but short enough for each service dashboard to report an acceptable result.
That pause is the product.
Users do not experience speech recognition, a language model, a tool call, and speech synthesis as separate services. They experience one conversational turn. If any handoff is late, uncertain, or difficult to cancel, the whole interaction begins to feel like a phone tree rather than a conversation.
The useful question is therefore not, "Which model is fastest?" It is:
How quickly can the system begin a trustworthy, speakable response?
This article develops that idea from the original voice-assistant architecture guide, but focuses on what developers can measure, instrument, and change in a production pipeline.
The guide was published by Smallest.ai, whose work focuses on real-time speech models and voice-agent systems.
Measure one turn, not four services
A typical cascaded assistant looks simple:
microphone
-> streaming speech-to-text (STT)
-> turn decision
-> language model and optional tools
-> stable text buffer
-> streaming text-to-speech (TTS)
-> interruptible playback
The diagram becomes misleading when every arrow is treated as a clean, serial boundary. In a responsive system, useful work overlaps. Audio is transcribed while the user is speaking. The model may begin after the turn is committed but before every transcript artifact is finalized. TTS can start when a stable, speakable clause exists rather than waiting for the complete answer.
Do not compress all of this into a single ambiguous "time to first audio" metric. Record the boundaries separately:
Turn-decision delay: from the user's last speech frame to the point at which the system commits the turn.
STT finalization delay: the time needed to produce a transcript that is safe to send downstream.
LLM time to first token (TTFT): from the model request to its first token.
Time to first speakable chunk: from the model request to the first stable clause that TTS can render safely.
TTS time to first audio: from the synthesis request to the first playable audio chunk.
End-of-turn-to-playback: from the user's last speech frame to audio actually beginning at the client.
The last measurement is the user-facing result. The others explain why it happened.
Turn detection is not the same as VAD
Voice activity detection answers a narrow question: does this audio frame contain speech?
Turn detection answers a harder question: has the speaker finished the thought?
VAD can contribute evidence, but silence alone is not enough. A production turn detector may also use:
finalized and interim transcript timing;
confidence or transcript stability;
punctuation or semantic completeness;
domain-specific patterns, such as a phone number that may continue;
the user's speaking rate and recent pause behavior;
the cost of interrupting versus waiting in the current workflow.
The distinction matters because a fixed silence threshold creates opposite failure modes. Commit too quickly and the assistant cuts off a user who paused mid-sentence. Wait too long and every turn feels hesitant.
Endpointing remains heuristic: background noise can prevent reliable silence detection, while transcript-based gap detection behaves differently and may be more useful for some utterances. The correct threshold is not an industry constant; tune it against real audio from the deployment environment.
STT accuracy is about consequential errors
Word error rate is useful, but it should not be treated as a universal pass/fail score for an assistant.
An incorrect filler word may have no downstream effect. One wrong digit in an account number, a misspelled surname, or a reversed negation may change the entire action. Conventional WER gives equal weight to errors with very different effects on meaning and readability, so it should be paired with entity-level and task-level evaluation.
For a voice agent, evaluate at least three layers:
Transcription quality: WER or another suitable ASR metric.
Entity accuracy: names, dates, amounts, identifiers, addresses, and domain terminology.
Task success: whether the downstream system understood the intent and performed the correct action.
Test with the audio users will actually produce: noisy rooms, phone codecs, weak connections, accents, hesitations, and overlapping speech. A clean studio recording is not a substitute for a deployment test set.
Streaming ASR can reduce emission delay, but it introduces unstable partial hypotheses. Research such as FastEmit demonstrates that latency and recognition quality must be evaluated together; its measured gains belong to the tested models and datasets, not to every streaming implementation.
The first token is not yet a spoken answer
LLM TTFT is important, but it is not the end of the model stage.
Suppose the model emits:
Sure — let me...
The first token arrived, but the assistant still has nothing useful to say. TTS may also need to wait for a stable clause so it does not synthesize an opening that later becomes awkward or incorrect.
A better voice prompt front-loads the answer:
Your appointment is confirmed for Thursday at 3 PM.
I can also send a reminder if you want one.
The first sentence is complete, useful, and independently speakable.
That creates another metric worth tracking: time to first speakable chunk. It includes model TTFT plus the time needed to accumulate a safe synthesis boundary.
A simple clause buffer might look like this:
const boundary = /[.!?]\s$|[,;:]\s$/;
let pending = "";

function onModelToken(token: string) {
pending += token;

const enoughText = pending.trim().length >= 24;
const hasBoundary = boundary.test(pending);

if (enoughText && hasBoundary) {
tts.enqueue(pending);
pending = "";
}
}

function onModelComplete() {
if (pending.trim()) tts.enqueue(pending);
}
This is intentionally simple. A production buffer should also consider abbreviations, numbers, markup, pronunciation hints, language-specific punctuation, maximum wait time, and whether already-enqueued speech can be cancelled.
Tool latency only hurts when it blocks the critical path
Tool calls are often described as an additive delay: a 300 ms lookup supposedly adds exactly 300 ms to the turn.
That is true only when the lookup sits on the serial critical path.
Some work can be hidden or reduced through:
prefetching likely context after the session starts;
caching results with a clear freshness policy;
running independent tools in parallel;
starting a safe verbal acknowledgement while a slow operation continues;
using deterministic routing when a model decision is unnecessary;
cancelling stale work when the user changes direction.
Do not use filler speech to disguise arbitrary latency. An acknowledgement such as "I'll check the live inventory" is useful only when it communicates real progress and does not make an unsupported promise.
Trace every tool call with its queue time, network time, server time, result size, cache status, retry count, and whether it blocked the first speakable chunk. That final field is more informative than duration alone.
Streaming is controlled overlap
"Stream every stage" sounds attractive but is too absolute.
Partial output is valuable only when it is useful and safely reversible. Some tools return atomic results. Partial transcripts can change. Premature TTS can speak text the model would have revised. Speculation can also increase cost and make cancellation harder.
A better rule is:
Stream where partial output is useful, and overlap work only when errors can be contained or reversed.
Incremental TTS research demonstrates how synthesis can begin from partial text while later segments are still being generated. The prefix-to-prefix TTS study is useful evidence for the architecture, but it does not justify a universal millisecond saving for every model, sentence, device, or network.
The engineering work behind low-latency voice systems extends beyond inference. Media transport, jitter, packet loss, session ownership, routing, WebRTC behavior, and infrastructure placement all affect the pause and belong in the same design discussion.
Build an explicit latency budget
A latency budget is a design constraint, not a chart created after launch. Start with an end-of-turn-to-playback target, then assign provisional limits to the stages on the critical path.
Here is an illustrative, deliberately aggressive 800 ms budget. It is an example for planning, not an industry benchmark:
Stage
Example budget
Turn decision
150 ms
Transcript stabilization
100 ms
First speakable model chunk
300 ms
First playable TTS audio
150 ms
Transport and client buffering
100 ms
Total
800 ms

Your allocation will change with the product. A hands-free command may prioritize speed. A medical intake flow may tolerate a longer wait to avoid interrupting the speaker. A tool-heavy transaction may need an acknowledgement before the final answer.
Measure distributions rather than averages. At minimum, report p50, p95, and p99 by:
interaction type;
geography and network;
language;
device or telephony provider;
tool-free versus tool-dependent turns;
successful, interrupted, cancelled, and retried turns.
An acceptable p50 can hide a painful p95.
Instrument the complete turn
The following TypeScript example records the event boundaries without tying the implementation to a specific STT, model, or TTS provider:
const marks = new Map();

function mark(name: string, at = performance.now()) {
marks.set(name, at);
}

function duration(start: string, end: string) {
const a = marks.get(start);
const b = marks.get(end);
return a === undefined || b === undefined ? undefined : b - a;
}

function voiceTurnMetrics() {
return {
turnDecisionMs: duration("speech_last_frame", "turn_committed"),
transcriptReadyMs: duration("turn_committed", "stt_final"),
llmTtftMs: duration("llm_started", "llm_first_token"),
firstSpeakableMs: duration("llm_started", "first_speakable_chunk"),
ttsFirstAudioMs: duration("tts_started", "tts_first_audio"),
clientBufferMs: duration("tts_first_audio", "playback_started"),
endToEndMs: duration("speech_last_frame", "playback_started"),
};
}
Call mark() from the actual callbacks in the pipeline:
mark("speech_last_frame", vad.lastSpeechTimestamp());
mark("turn_committed");
mark("stt_final");
mark("llm_started");
mark("llm_first_token");
mark("first_speakable_chunk");
mark("tts_started");
mark("tts_first_audio");
mark("playback_started");
Use one trace ID across the browser or phone gateway, STT, orchestration layer, tools, model, TTS, and playback client. Without cross-service correlation, teams often optimize the service with the most visible dashboard rather than the stage responsible for the user's wait.
If you are comparing a managed stack with a custom pipeline, Smallest.ai's voice-agent platform is one integrated implementation to evaluate. Apply the same event boundaries and p50, p95, and p99 measurements to it—or any other platform—rather than assuming that an integrated stack is automatically low latency.
Barge-in must cancel the old turn
In a full-duplex assistant, inbound audio capture and speech detection normally remain active while the assistant speaks. When new user speech is confirmed, the system should stop treating the old response as current.
The control path may resemble:
async function onUserSpeechStarted() {
generation.abort();
await tts.cancel();
playback.stopAndFlush();
tools.cancelNonReusableWork();

conversation.truncateAssistantMessage({
toPlayedAudioTimestamp: playback.lastPlayedTimestamp(),
});
}
The exact APIs will differ, but the responsibilities are stable:
cancel model generation;
cancel synthesis;
stop and flush queued playback;
discard tool results that are no longer relevant;
preserve only the portion of the assistant response the user actually heard;
continue processing the incoming speech without clipping its beginning.
Barge-in cannot be added cleanly as a final UI feature. It changes audio capture, state management, model context, tool cancellation, and playback architecture.
Production starts where the demo ends
A convincing demo proves that the happy path can answer. Production requires tests for human behavior and imperfect infrastructure.
Before launch, include:
quiet, noisy, reverberant, and low-bitrate audio;
short commands and long, hesitant utterances;
mid-sentence pauses and self-corrections;
names, phone numbers, dates, amounts, and domain terminology;
tool-free, cached, slow, failed, and retried tool calls;
barge-in during early and late playback;
packet loss, jitter, reconnects, and duplicated events;
multiple languages and code-switching;
long conversations with context compaction;
p95 latency under realistic concurrency.
Also test the failure semantics. What happens if the tool succeeds after the user interrupts? If TTS emits audio after cancellation? If two final transcripts arrive? If the client reconnects while audio is queued? A low-latency system that produces stale actions is not a good system.
When reviewing an implementation, study how it represents processors, transports, interruption, events, cancellation, and session state. Do not copy defaults blindly; validate each design choice against your own latency traces and failure modes.
The pause is the architecture
The fastest model will not rescue a pipeline with slow turn detection, serial tool calls, unstable partial transcripts, or excessive audio buffering.
Measure the complete turn from the user's last speech frame to audible playback. Separate that duration into named stages. Optimize the stage that dominates p95, then test the change against accuracy, cancellation behavior, and task success.
A natural voice assistant is not a collection of fast components. It is one coordinated participant whose timing, state, and failure modes have been designed as a whole.
Ready to test this architecture in a working stack? Open Smallest.ai's self-serve application, build a voice agent, instrument the event boundaries above, and compare its p50 and p95 end-to-end latency with your current pipeline.

Your First Voice AI App Shouldn’t Start With a WebSocket

Smallest AI — Mon, 27 Jul 2026 16:16:46 +0000

The first successful voice feature usually looks unimpressive: one audio file goes in, and one useful transcript comes out.

That is not a toy result. It is the shortest route to the questions that decide whether a voice product will work. Does the transcript preserve names and numbers? Are speaker turns separated? Are timestamps useful downstream? Can the system handle the audio your users will actually produce?

It is tempting to begin with live microphones, persistent connections, partial transcripts, browser permissions, and animated interfaces. Those pieces feel like voice AI. They also make failures harder to isolate. A safer rule is simple: prove the transcript’s value first, then earn the complexity of streaming.

Start with the product’s clock

The important architectural choice is not Python versus another language. It is batch versus real time.

If you are processing a voicemail, podcast, meeting recording, or archive, the entire file already exists. An HTTP request matches the problem: send the audio, wait for the result, and store the transcript. There is no benefit in pretending a completed file is a live conversation.

A voice assistant, live-captioning tool, or call workflow runs on a different clock. It needs partial results while the speaker is talking. The system must receive small audio frames, interpret interim hypotheses, decide when an utterance is final, and recover when the connection drops. That is where a WebSocket belongs.

Smallest.ai publishes a roughly 64 ms time-to-first-transcript figure for Pulse streaming. Treat that as a provider specification, not a guarantee for your application: network path, audio framing, region, load, and transcript-stability requirements all affect what users experience. The Pulse speech-to-text product page is the current product reference, while your own p50 and p95 traces should make the architecture decision.

The smallest useful Python proof

A batch prototype should stay intentionally boring. Put the API key in an environment variable, read a representative audio file, request only the metadata you need, and inspect the returned JSON.

import os
import requests

endpoint = "https://api.smallest.ai/waves/v1/stt/"
params = {
    "model": "pulse",
    "language": "en",
    "word_timestamps": "true",
    "diarize": "true",
}
headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}",
    "Content-Type": "application/octet-stream",
}

with open("meeting.wav", "rb") as audio:
    response = requests.post(
        endpoint,
        params=params,
        headers=headers,
        data=audio,
        timeout=120,
    )
    response.raise_for_status()
    result = response.json()

print(result["transcription"])
print(result.get("words", [])[:5])

This uses the current unified endpoint documented in the pre-recorded STT API reference. In production, also handle timeouts, retries, rate limits, response-schema changes, and request identifiers. Do not log the authorization header or raw audio by default.

That short loop gives you a useful evaluation surface. Test several accents and speaking styles. Use clean and noisy recordings. Include telephony audio if production will receive it. Check the JSON your next component must consume, not only the sentence printed to the terminal.

Measure the transcript you need, not the demo you want

A single successful file proves connectivity. A small evaluation set begins to prove usefulness.

Use audio captured from the microphones, codecs, channels, and environments your users will have.

Record failures involving names, numbers, abbreviations, product terms, and code-switching—not only average text quality.

Inspect speaker labels and timestamp drift if downstream features depend on them.

Track request latency at p50 and p95; one fast example says little about production behavior.

Define what happens when the API times out, rejects a file, or returns an empty transcript.

This stage also tells you whether streaming is necessary. If users upload recordings and return later, a persistent connection may add operational cost without improving the product. If the transcript drives a live response, batch processing will eventually reveal its limit.

Streaming is not a faster POST request

Moving to streaming changes the application, not merely the transport. A live client sends raw audio frames and receives partial and final transcript messages while the connection remains open. Sending and receiving must happen concurrently.

Batch waits for the complete recording; streaming turns a moving signal into usable partial results.

import asyncio
import json
import os
from urllib.parse import urlencode

import websockets

CHUNK_SIZE = 4096

params = {
    "language": "en",
    "encoding": "linear16",
    "sample_rate": "16000",
    "word_timestamps": "true",
}
url = (
    "wss://api.smallest.ai/waves/v1/stt/live?model=pulse&"
    + urlencode(params)
)
headers = {"Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}"}


async def send_audio(ws, path: str):
    with open(path, "rb") as audio:
        while chunk := audio.read(CHUNK_SIZE):
            await ws.send(chunk)
            await asyncio.sleep(0.05)

    await ws.send(json.dumps({"type": "close_stream"}))


async def receive_transcripts(ws):
    async for message in ws:
        event = json.loads(message)
        label = "final" if event.get("is_final") else "partial"
        print(label, event.get("transcript", ""))

        if event.get("is_last"):
            return


async def transcribe_file(path: str):
    async with websockets.connect(
        url,
        additional_headers=headers,
    ) as ws:
        sender = asyncio.create_task(send_audio(ws, path))
        receiver = asyncio.create_task(receive_transcripts(ws))
        await asyncio.gather(sender, receiver)


asyncio.run(transcribe_file("audio.pcm"))

The current real-time WebSocket documentation recommends 4096-byte chunks and a close_stream control message when a one-off stream is complete. The sample above expects headerless, 16-bit linear PCM at 16 kHz mono. A WAV file normally includes a container header, so do not rename a WAV file to .pcm and assume the bytes are equivalent.

The concurrency model matters more than the socket

A production microphone flow should separate audio capture, network sending, transcript receiving, and UI updates. If one task blocks the others, buffers grow and the interface feels stale even when recognition is fast.

Audio producer: reads fixed-size frames and applies bounded buffering.

Network sender: writes frames, handles backpressure, and stops cleanly.

Transcript receiver: distinguishes revisable partial text from final text.

Application state: stores only stable transcript segments and exposes connection status.

Recovery path: reconnects deliberately and prevents duplicate or out-of-order segments.

Partial transcripts are a preview, not an append-only log. Replace the current partial segment when a new hypothesis arrives; persist it only when the server marks it final. Otherwise, readers see duplicated words and downstream systems process text that the recognizer later corrected.

Keep the browser simple—and the key off it

A browser can capture microphone audio and render transcript updates, but it should not receive a long-lived service API key. Put the authenticated upstream connection behind your server. The browser connects to your application, and your server authorizes the user, opens the provider connection, applies limits, and forwards only the events the interface needs.

That boundary gives you a place to enforce session duration, input format, concurrency, logging policy, and abuse controls. It also creates a clean progression: file uploads can use a normal server route, while microphone audio uses a streaming proxy. Batch and live transcription remain two deliberate modes instead of one over-engineered pipeline.

A transcript is rarely the final product

Raw text is enough for a connectivity demo. Production workflows usually need structure. Word timestamps support captions and review tools. Speaker diarization separates participants. Redaction can reduce sensitive information passed downstream. Each feature should exist because a downstream workflow needs it, not because a parameter is available.

If your product depends on speaker separation, read the speaker diarization implementation guide before choosing a transcript schema. For conversational systems, the STT–LLM–TTS latency-budget guide explains why a fast recognition stage does not guarantee a fast audible response.

Know when the prototype has earned real time

Move from batch to streaming when at least one requirement cannot be met by waiting for a complete file:

The user needs words on screen while speaking.

A conversational system must begin reasoning before the entire recording exists.

Turn-taking, interruption, or live routing depends on partial results.

The recording is long enough that upload-then-process creates unacceptable delay.

When you cross that boundary, measure the complete path: audio capture, frame queueing, network transit, first usable partial, finalization, downstream processing, and—if the system speaks—first audible response. Optimizing only the model’s headline latency can hide the stage users are actually waiting on.

Build outward from one trustworthy transcript

A voice product does not become serious when it opens a WebSocket. It becomes serious when every added layer solves a problem the simpler version exposed.

Start with representative audio. Produce a transcript. Verify the structure and failure modes. Add streaming only when the user’s clock demands it. Then keep the API key behind your server and measure the whole path under realistic load.

If you want to test the same batch-to-streaming path with Smallest.ai, open the self-serve application and create an API key—no demo booking required. The complete Python implementation guide provides the broader walkthrough. What requirement in your product genuinely forces the architecture to become real time?