DEV Community: Smallest AI

Your Speech-to-Text API Is Only as Good as the Audio You Send It

Smallest AI — Mon, 03 Aug 2026 18:27:51 +0000

A speech-to-text API can perform well on clean benchmark recordings and still struggle with your users’ actual audio.

That gap is not always caused by the model.

It can come from:

Background noise
Incorrect sample rates
Unsupported audio formats
Accents and code-switching
Domain-specific vocabulary
Choosing batch transcription when the product needs streaming
Reading the wrong field from the API response

The fastest way to evaluate speech recognition is not to upload one perfect recording. It is to test the API against the audio your application will encounter in production.

This guide shows how to integrate the Pulse speech-to-text API from Python and Node.js, process pre-recorded audio, build a real-time WebSocket client, and avoid the response-handling mistakes that often break otherwise successful integrations.

Pulse supports two transcription modes

Pulse is the speech-recognition model from Smallest AI.

It supports two distinct integration patterns.

Pre-recorded transcription

Use pre-recorded transcription when the entire audio file already exists.

Common examples include:

Call recordings
Voicemails
Podcast episodes
Uploaded videos
Meeting archives
Offline transcription jobs

The application sends the complete file through an HTTPS request and receives one structured response.

Audio file
    ↓
HTTPS request
    ↓
Complete transcript
    ↓
Storage or downstream processing

Real-time transcription

Use real-time transcription when audio is still being captured.

Common examples include:

Voice agents
Live captions
Phone calls
Browser microphones
Meeting assistants
Real-time conversation analytics

The application keeps a WebSocket connection open, sends audio in small binary chunks, and receives partial and final transcript events while the speaker is still talking.

Live audio
    ↓
WebSocket audio chunks
    ↓
Partial transcript events
    ↓
Final transcript events
    ↓
Application action

The choice is not primarily about which transport looks more advanced.

Use batch transcription when the product can wait. Use streaming when the product must react while the audio is still arriving.

Create and store the API key

Keep the API key in an environment variable rather than hard-coding it into the application.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

export SMALLEST_API_KEY="your-api-key-here"

Every authenticated request sends the value through the Authorization header:

Authorization: Bearer SMALLEST_API_KEY

Keep the key on your server. Do not expose it in browser JavaScript, mobile application code, public repositories, screenshots, or client-side logs.

Transcribe a pre-recorded file with Python

The current pre-recorded endpoint is:

POST https://api.smallest.ai/waves/v1/stt/

The model, language, and optional features are supplied as query parameters. The audio file is sent as raw bytes in the request body.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import os
from pathlib import Path
from typing import Any

import requests


STT_ENDPOINT = "https://api.smallest.ai/waves/v1/stt/"


def transcribe_file(
    file_path: str,
    language: str = "en",
) -> dict[str, Any]:
    audio_path = Path(file_path)

    if not audio_path.is_file():
        raise FileNotFoundError(f"Audio file not found: {audio_path}")

    with audio_path.open("rb") as audio_file:
        response = requests.post(
            STT_ENDPOINT,
            params={
                "model": "pulse",
                "language": language,
                "word_timestamps": "true",
            },
            headers={
                "Authorization": (
                    f"Bearer {os.environ['SMALLEST_API_KEY']}"
                ),
                "Content-Type": "application/octet-stream",
            },
            data=audio_file,
            timeout=120,
        )

    response.raise_for_status()
    return response.json()


result = transcribe_file("recording.wav")

print(result["transcription"])
print(result.get("words", [])[:5])

One response-handling detail matters immediately:

The pre-recorded response uses transcription, not transcript.

A simplified response can look like this:

{
  "status": "success",
  "transcription": "Hello, this is a test transcription.",
  "words": [
    {
      "start": 0.48,
      "end": 1.12,
      "word": "Hello,"
    },
    {
      "start": 1.12,
      "end": 1.28,
      "word": "this"
    }
  ],
  "utterances": [
    {
      "start": 0.48,
      "end": 3.76,
      "text": "Hello, this is a test transcription."
    }
  ]
}

The returned structures support different downstream workflows:

Field	Useful for
`transcription`	Complete readable transcript
`words`	Captions, subtitle alignment and audio navigation
`utterances`	Sentence-level segments and readable conversation records
`language`	Language-aware routing and transcript metadata
Speaker fields	Multi-speaker calls and meetings

Do not assume every optional field will always be present. Use .get() or validate the response against a schema before sending it downstream.

Transcribe audio from a URL

Pulse can also process an audio file hosted at a publicly accessible URL.

This is useful when recordings already live in object storage and downloading them to the application server would add unnecessary work.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import os
from typing import Any

import requests


STT_ENDPOINT = "https://api.smallest.ai/waves/v1/stt/"


def transcribe_from_url(
    audio_url: str,
    language: str = "en",
) -> dict[str, Any]:
    response = requests.post(
        STT_ENDPOINT,
        params={
            "model": "pulse",
            "language": language,
            "word_timestamps": "true",
        },
        headers={
            "Authorization": (
                f"Bearer {os.environ['SMALLEST_API_KEY']}"
            ),
            "Content-Type": "application/json",
        },
        json={"url": audio_url},
        timeout=120,
    )

    response.raise_for_status()
    return response.json()


result = transcribe_from_url(
    "https://example.com/recordings/customer-call.wav"
)

print(result["transcription"])

The remote file must be accessible to the API. A URL that requires browser cookies, local authentication, or access to a private network will not work without a suitable signed URL or public access policy.

Make the same request from Node.js

Modern Node.js versions provide fetch without requiring node-fetch.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import { readFile } from "node:fs/promises";

const STT_ENDPOINT = "https://api.smallest.ai/waves/v1/stt/";

async function transcribeFile(filePath, language = "en") {
  const audio = await readFile(filePath);

  const params = new URLSearchParams({
    model: "pulse",
    language,
    word_timestamps: "true",
  });

  const response = await fetch(`${STT_ENDPOINT}?${params}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SMALLEST_API_KEY}`,
      "Content-Type": "application/octet-stream",
    },
    body: audio,
  });

  if (!response.ok) {
    const errorBody = await response.text();

    throw new Error(
      `Transcription failed with ${response.status}: ${errorBody}`,
    );
  }

  return response.json();
}

const result = await transcribeFile("recording.wav");

console.log(result.transcription);
console.log(result.words?.slice(0, 5) ?? []);

For a remotely hosted file, change the request body and content type.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

const STT_ENDPOINT = "https://api.smallest.ai/waves/v1/stt/";

async function transcribeFromUrl(audioUrl, language = "en") {
  const params = new URLSearchParams({
    model: "pulse",
    language,
    word_timestamps: "true",
  });

  const response = await fetch(`${STT_ENDPOINT}?${params}`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SMALLEST_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url: audioUrl,
    }),
  });

  if (!response.ok) {
    const errorBody = await response.text();

    throw new Error(
      `Transcription failed with ${response.status}: ${errorBody}`,
    );
  }

  return response.json();
}

const result = await transcribeFromUrl(
  "https://example.com/recordings/customer-call.wav",
);

console.log(result.transcription);

Enable only the enrichment features you need

A production transcript often needs more than plain text.

Pulse supports optional features such as:

Word timestamps
Sentence-level utterances
Speaker diarization
Emotion detection
Gender detection
Language detection
Keyword boosting
PII and PCI redaction
Inverse text normalization

These capabilities are enabled through query parameters.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import os

import requests


with open("call_recording.wav", "rb") as audio_file:
    response = requests.post(
        "https://api.smallest.ai/waves/v1/stt/",
        params={
            "model": "pulse",
            "language": "en",
            "word_timestamps": "true",
            "diarize": "true",
            "emotion_detection": "true",
            "gender_detection": "true",
            "itn_normalize": "true",
        },
        headers={
            "Authorization": (
                f"Bearer {os.environ['SMALLEST_API_KEY']}"
            ),
            "Content-Type": "application/octet-stream",
        },
        data=audio_file,
        timeout=120,
    )

response.raise_for_status()

result = response.json()

print(result["transcription"])

Enable features because the workflow requires them, not simply because the parameters exist.

Word timestamps

Useful for:

Captions
Subtitle tracks
Transcript review
Search
Audio playback navigation

Timing information is difficult to reconstruct accurately after transcription. Enable it early when the product may eventually need synchronized text.

Speaker diarization

Useful for:

Customer calls
Interviews
Meetings
Podcasts
Sales conversations

Diarization converts a wall of text into speaker-attributed turns.

Emotion detection

Useful for enriching offline call analysis and quality-assurance workflows.

Treat emotional scores as model-generated signals rather than definitive statements about a person.

Inverse text normalization

Useful when spoken expressions should appear in conventional written form.

For example:

"twenty five dollars"

can become:

"$25"

This matters for dates, currencies, phone numbers, amounts, and other entities that downstream software must parse.

Accuracy begins with representative audio

A published benchmark is not a substitute for testing your own workload.

Build an evaluation set that includes the conditions your users will create:

Clean studio recordings
Browser microphones
Mobile devices
Telephony audio
Background conversations
Fast speech
Hesitations
Multiple speakers
Regional accents
Domain-specific vocabulary
Mixed-language conversations

Track more than overall Word Error Rate.

A transcript can be mostly correct and still fail the application by misrecognizing the exact information that matters.

Evaluate:

Area	What to inspect
General transcription	Missing, substituted or invented words
Entity accuracy	Names, dates, amounts, addresses and identifiers
Domain vocabulary	Product names, abbreviations and technical terms
Speaker separation	Whether speaker labels remain consistent
Timestamp quality	Alignment and drift
Language handling	Accents, language switching and detection
Failure behavior	Empty responses, timeouts and invalid files
Downstream usefulness	Whether the next system can reliably use the result

For domain terms, product names, and uncommon vocabulary, keyword boosting may be more valuable than repeatedly switching providers.

Audio format is part of the model input

Speech recognition cannot recover information that the audio pipeline has already damaged.

Before blaming the API, inspect:

Sample rate
Channel count
Codec
Container format
Clipping
Background noise
Compression artifacts
Volume
Packet loss

For speech workloads, mono audio at an appropriate speech-oriented sample rate is usually more useful than sending oversized stereo files with no additional spoken information.

Real-time streaming also requires the client to describe the incoming encoding correctly. Sending μ-law audio while declaring linear16, for example, will produce unusable results even though the WebSocket connection itself succeeds.

Language selection should be deliberate

When you know the language, send its explicit language code.

For example:

en
hi
de
fr
es

When the language is unknown or the conversation may switch between related languages, use the appropriate regional auto-detection option supported by the chosen transcription mode.

Do not assume that batch and streaming expose identical language sets or regional routes. Check the current model documentation before shipping a multilingual workflow.

Explicit language selection is usually the safer choice when the source language is known.

Real-time transcription changes the application

Batch transcription produces one result after the file is complete.

Streaming produces a sequence of changing hypotheses.

The current WebSocket endpoint is:

wss://api.smallest.ai/waves/v1/stt/live?model=pulse

A client must:

Open and authenticate the connection
Send correctly encoded binary audio
Receive events concurrently
Replace partial hypotheses
Commit final segments
Handle connection failures
Close the stream deliberately

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import WebSocket from "ws";

const params = new URLSearchParams({
  model: "pulse",
  language: "en",
  encoding: "linear16",
  sample_rate: "16000",
  word_timestamps: "true",
});

const websocketUrl =
  `wss://api.smallest.ai/waves/v1/stt/live?${params}`;

const ws = new WebSocket(websocketUrl, {
  headers: {
    Authorization: `Bearer ${process.env.SMALLEST_API_KEY}`,
  },
});

ws.on("open", () => {
  console.log("Connected to Pulse STT");
});

ws.on("message", (rawMessage) => {
  const event = JSON.parse(rawMessage.toString());

  if (event.is_final) {
    console.log("\n[FINAL]", event.transcript);
  } else {
    process.stdout.write(`\r[PARTIAL] ${event.transcript ?? ""}`);
  }

  if (event.is_last) {
    console.log("\nStream complete");
  }
});

ws.on("error", (error) => {
  console.error("WebSocket error:", error);
});

function sendAudioChunk(audioBuffer) {
  if (ws.readyState !== WebSocket.OPEN) {
    throw new Error("WebSocket is not open");
  }

  ws.send(audioBuffer);
}

function closeStream() {
  if (ws.readyState === WebSocket.OPEN) {
    ws.send(
      JSON.stringify({
        type: "close_stream",
      }),
    );
  }
}

Send audio as binary frames. A commonly recommended chunk size is 4096 bytes, although the correct capture and buffering strategy still depends on the source encoding and application.

When the audio is complete, send:

{
  "type": "close_stream"
}

This asks the service to flush the remaining buffered audio and return the last event with is_last: true.

Stream microphone audio from Python

A real-time microphone client needs separate send and receive loops. Otherwise, waiting for one operation can prevent the other from progressing.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import asyncio
import json
import os
from urllib.parse import urlencode

import pyaudio
import websockets


SAMPLE_RATE = 16_000
CHUNK_SIZE = 4_096

params = {
    "model": "pulse",
    "language": "en",
    "encoding": "linear16",
    "sample_rate": str(SAMPLE_RATE),
    "word_timestamps": "true",
}

websocket_url = (
    "wss://api.smallest.ai/waves/v1/stt/live?"
    + urlencode(params)
)

headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}",
}


async def stream_microphone() -> None:
    audio = pyaudio.PyAudio()

    microphone = audio.open(
        format=pyaudio.paInt16,
        channels=1,
        rate=SAMPLE_RATE,
        input=True,
        frames_per_buffer=CHUNK_SIZE,
    )

    async with websockets.connect(
        websocket_url,
        additional_headers=headers,
    ) as websocket:
        print("Listening. Press Ctrl+C to stop.")

        async def send_audio() -> None:
            try:
                while True:
                    chunk = microphone.read(
                        CHUNK_SIZE,
                        exception_on_overflow=False,
                    )
                    await websocket.send(chunk)
                    await asyncio.sleep(0.01)
            except asyncio.CancelledError:
                await websocket.send(
                    json.dumps(
                        {
                            "type": "close_stream",
                        }
                    )
                )
                raise

        async def receive_events() -> None:
            async for message in websocket:
                event = json.loads(message)
                transcript = event.get("transcript", "")

                if event.get("is_final"):
                    print(f"\n[FINAL] {transcript}")
                else:
                    print(
                        f"\r[PARTIAL] {transcript}",
                        end="",
                        flush=True,
                    )

                if event.get("is_last"):
                    return

        sender = asyncio.create_task(send_audio())

        try:
            await receive_events()
        finally:
            sender.cancel()
            await asyncio.gather(
                sender,
                return_exceptions=True,
            )

            microphone.stop_stream()
            microphone.close()
            audio.terminate()


asyncio.run(stream_microphone())

Production code should also handle:

Operating-system audio permissions
Device disconnection
Queue limits
Slow network writes
Reconnection
Session timeouts
Duplicate segments
Graceful shutdown

Partial transcripts are not permanent text

A streaming recognizer may emit:

I need to update

Then revise it to:

I need to update my address

Then finalize:

I need to update my billing address.

Do not append every partial result to the transcript.

Maintain two states:

Stable final segments
Current replaceable partial segment

Only persist or act on text after the API marks the segment final, unless the product explicitly accepts the risks of acting on provisional text.

The source article’s related guide on handling reconnects, dropouts and duplicated streaming transcripts covers the next layer of production complexity.

Batch and streaming return different structures

This distinction frequently causes silent bugs.

Pre-recorded response

Expect fields such as:

transcription
words
utterances
language

Real-time response

Expect a sequence of events containing fields such as:

transcript
is_final
is_last
session_id

Notice the difference:

Batch:     transcription
Streaming: transcript

Do not route both modes through one untested parser and assume their payloads are interchangeable.

A useful application-level interface can normalize both into your own internal format:

text
is_final
start_time
end_time
speaker
language
source_mode

That keeps provider-specific response details at the integration boundary.

Choose the mode based on the product’s clock

Use pre-recorded transcription when:

The full file already exists.
The workflow can wait for one complete response.
Simpler HTTP request-response code is preferable.
You are processing archived audio.
You are running offline batch jobs.

Use real-time transcription when:

Partial words must appear while the user speaks.
A voice agent must begin reasoning before the turn is complete.
Live captions must update continuously.
Turn detection drives application logic.
You are processing a live phone call or microphone.
User experience depends on low first-transcript latency.

A WebSocket is not automatically a better architecture.

It is justified only when waiting for the complete recording would prevent the product from meeting its user-facing requirement.

Production readiness is more than a successful transcript

Before launch, test the complete pipeline.

Input

Are audio formats validated?
Is the declared encoding accurate?
Are sample rates and channels appropriate?
Are unsupported or corrupted files rejected clearly?

Authentication

Is the API key server-side?
Are secrets excluded from logs?
Can keys be rotated safely?

Response handling

Does the application distinguish transcription from transcript?
Are optional fields handled safely?
Are partial results replaced instead of appended?
Are final events persisted exactly once?

Reliability

Are timeouts configured?
Are transient failures retried?
Can retries create duplicate processing?
Can the WebSocket reconnect safely?
Are queues bounded?

Evaluation

Does the test set represent real users?
Are names, numbers and domain terms measured separately?
Are latency percentiles tracked?
Are failures visible through logs and metrics?

Start with one representative recording

The first useful integration does not need a microphone, animated waveform, or persistent WebSocket.

Start with:

One representative audio file
        ↓
One authenticated HTTP request
        ↓
One structured transcript
        ↓
Evaluation

Then inspect:

Accuracy
Entity preservation
Speaker labels
Timestamps
Language handling
Response structure
Failure behavior

Move to real-time streaming only when the application genuinely needs results before the audio ends.

That progression keeps the integration understandable and gives every layer of complexity a clear reason to exist.

Build with Pulse STT

To test the workflow:

Create a Smallest AI API key.
Open the Pulse speech-to-text product page.
Start with one recording captured under realistic conditions.
Validate the transcript and response fields.
Add timestamps or diarization only when the workflow needs them.
Move to WebSocket streaming when the product requires live results.

You can also explore the original Smallest AI integration guide and the Smallest AI Cookbook for additional examples.

What caused the hardest speech-to-text failure in your application: noisy audio, response handling, language coverage, or streaming state?

Your TTS Isn’t Robotic. Your Delivery Pipeline Is.

Smallest AI — Mon, 03 Aug 2026 18:08:32 +0000

Natural speech emerges when code, timing, and expression are designed as one system.

A text-to-speech demo can be impressive for exactly thirty seconds. Then someone types a date, an abbreviation, a long sentence, or a line that needs actual feeling. The voice suddenly rushes through a pause, stresses the wrong word, or waits so long to begin that the illusion breaks.

That moment reveals an uncomfortable truth: realistic speech is not mainly a model-selection problem. It is a delivery problem.

Python makes it easy to send text to a speech service and save a WAV file. The hard part is deciding how the text should be normalized, where phrases should break, how quickly audio should begin, and what happens when many requests arrive together. Those choices determine whether a voice feels present or merely audible.

Naturalness lives between the words

Developers often judge TTS by voice identity and sample quality. Those matter, but listeners notice rhythm first. A believable voice varies pitch, emphasis, pauses, and pace in ways that support meaning. This is prosody: the structure that turns a sequence of words into an utterance.

A higher sample rate can preserve more audio detail, but it cannot repair flat phrasing. Likewise, a convincing cloned voice can still sound mechanical if every sentence has the same contour. The model creates the acoustic material; the pipeline decides how that material reaches the listener.

Modern neural systems typically normalize text, predict timing and acoustic features, and use a neural vocoder to produce the waveform. If you want a deeper technical explanation of that process, Smallest.ai’s guide to how neural TTS works provides the useful model-level context.

Realism is the difference between producing audio and shaping an expressive utterance.

Start with one honest Python request

The best first test is deliberately small: one sentence, one voice, one output file. That isolates synthesis from playback, networking, and conversational orchestration. It also gives you a repeatable artifact to listen to and compare.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

import os

import requests

api_key = os.environ["SMALLEST_API_KEY"]
url = "https://api.smallest.ai/waves/v1/tts"

response = requests.post(
    url,
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
        "Accept": "audio/wav",
    },
    json={
        "text": (
            "A natural voice is shaped by timing, emphasis, "
            "and the space between phrases."
        ),
        "voice_id": "meher",
        "model": "lightning_v3.1_pro",
        "sample_rate": 24000,
        "output_format": "wav",
    },
    timeout=30,
)

response.raise_for_status()

with open("output.wav", "wb") as audio_file:
    audio_file.write(response.content)

print("Saved generated speech to output.wav")

This is enough to validate authentication, voice selection, file output, and the basic sound of the model. The current Lightning text-to-speech product is the relevant Smallest.ai product surface for this workflow.

Do not confuse a successful file with a production system

A synchronous request is perfectly reasonable for narration, prototypes, and offline generation. It becomes a problem when your application needs to synthesize many responses or begin playback before the entire utterance exists.

For batch work, concurrency matters because blocking calls serialize the queue. For live applications, streaming matters because the listener experiences the delay before the first playable audio—not the total time required to finish the sentence.

The practical design question is therefore not “Does this API support streaming?” It is “Where can my application safely begin?” Sending tiny text fragments may reduce delay but can damage phrasing. Waiting for a complete paragraph preserves context but creates an obvious pause. A stable clause or short sentence is often the useful unit: enough context for expression, but not so much that playback feels late.

This is also why real-time voice systems are asynchronous by nature. Text arrives in chunks, speech is synthesized in chunks, audio is buffered, and stale output may need to be cancelled. Smallest.ai’s article on streaming architecture for real-time voice agents expands on that coordination problem.

Production quality is a set of trade-offs

Once the basic request works, realism becomes an operational discipline. Test numbers, dates, acronyms, currency, names, and mixed-language phrases. Listen for awkward joins between streamed chunks. Measure time to first audio as well as total synthesis time. Retry transient failures without generating duplicate playback. Keep API keys on the server, never in browser or mobile code.

Cost should be evaluated against the real workload, not a single headline rate. Character or byte volume, concurrency, caching, output format, and repeated prompts all change the bill. Long-form narration and conversational agents may use the same TTS model but need very different buffering and delivery strategies.

The broader voice stack matters too. In an assistant, speech recognition, reasoning, tool calls, and TTS share one latency budget. Optimizing synthesis in isolation may produce a faster component without producing a faster conversation. The guide to designing voice assistants around a full latency budget is a useful next step when TTS becomes part of a larger agent.

The most realistic voice is the one that arrives correctly

The Python call is the easy part. A believable speech experience comes from everything around it: clean text normalization, sensible phrase boundaries, expressive prosody, early but stable streaming, secure credentials, and measurement under real load.

That is the memorable test for any TTS system: not whether one sample sounds human, but whether the voice still feels human when the text is messy, the network is busy, and the application has to respond now.

If you are working through the same trade-offs, share what has been hardest in the comments, explore the original Python TTS guide, or follow more voice AI work from Smallest.ai.

Voice Assistants Are Designed in the Silence

Smallest AI — Mon, 27 Jul 2026 16:17:29 +0000

A voice assistant can be technically correct and still feel broken.

The failure often begins after the user finishes speaking.

Nothing dramatic happens. There is simply a pause—long enough for the user to wonder whether the system heard them, but short enough for every individual service dashboard to report an acceptable result.

That pause is the product.

Users do not experience speech recognition, a language model, a tool call, and speech synthesis as separate services. They experience one conversational turn.

When any handoff is late, uncertain, or difficult to cancel, the interaction starts to feel more like a phone tree than a conversation.

The useful question is not:

Which model is fastest?

It is:

How quickly can the system begin a trustworthy, speakable response?

This guide focuses on what developers can measure, instrument, and improve in a production voice pipeline.

Measure One Conversational Turn, Not Four Services

A typical cascaded voice assistant looks simple:

Microphone
  ↓
Streaming speech-to-text
  ↓
Turn decision
  ↓
Language model and optional tools
  ↓
Stable text buffer
  ↓
Streaming text-to-speech
  ↓
Interruptible playback

This diagram becomes misleading when every arrow is treated as a clean, serial boundary.

In a responsive system, useful work overlaps:

Audio is transcribed while the user is speaking.
The model may begin after the turn is committed but before every transcript artifact is finalized.
TTS can start when a stable, speakable clause exists instead of waiting for the complete response.

Do not compress the entire pipeline into one ambiguous time to first audio metric.

Record the important boundaries separately.

Metric	What it measures
Turn-decision delay	Time between the user’s last speech frame and the system committing the turn
STT finalization delay	Time required to produce a transcript that is safe to send downstream
LLM time to first token	Time between the model request and its first generated token
Time to first speakable chunk	Time between the model request and the first stable clause that TTS can safely render
TTS time to first audio	Time between the synthesis request and the first playable audio chunk
End-of-turn to playback	Time between the user’s last speech frame and audio beginning on the client

The last metric represents the user-facing experience.

The other metrics explain why that experience occurred.

Turn Detection Is Not the Same as VAD

Voice activity detection answers a narrow question:

Does this audio frame contain speech?

Turn detection answers a harder question:

Has the speaker finished their thought?

VAD can contribute evidence, but silence alone is not enough.

A production turn detector may also consider:

Finalized and interim transcript timing
Transcript confidence or stability
Punctuation and semantic completeness
Domain-specific patterns
The user’s speaking rate
Recent pause behaviour
The cost of interrupting versus waiting

For example, a phone number may continue after a brief pause. A support caller may hesitate before stating an account identifier. A user may pause naturally in the middle of a longer question.

A fixed silence threshold creates two opposite failure modes:

Commit too quickly: the assistant cuts off a user who paused mid-sentence.
Wait too long: every response feels hesitant.

Endpointing remains heuristic.

Background noise can prevent reliable silence detection, while transcript-based gap detection behaves differently and may perform better for some utterances.

There is no universal endpointing threshold. Tune it using real audio from the intended deployment environment.

STT Accuracy Should Focus on Consequential Errors

Word error rate is useful, but it should not be treated as a universal pass-or-fail score for a voice assistant.

An incorrect filler word may have no downstream effect.

One incorrect digit in an account number, a misspelled surname, or a reversed negation may change the entire action.

Conventional WER assigns similar importance to errors that have very different effects on meaning and task completion.

For a voice agent, evaluate at least three layers.

1. Transcription quality

Measure WER or another suitable ASR metric.

2. Entity accuracy

Evaluate high-impact entities such as:

Names
Dates
Amounts
Phone numbers
Account identifiers
Addresses
Domain-specific terminology

3. Task success

Determine whether the downstream system:

Understood the user’s intent
Extracted the correct information
Performed the correct action

Test using the audio users will actually produce:

Noisy rooms
Phone codecs
Weak connections
Accents
Hesitations
Overlapping speech
Low-quality microphones

A clean studio recording is not a substitute for a deployment test set.

Streaming ASR can reduce emission delay, but it also introduces unstable partial hypotheses. Latency and recognition quality must therefore be evaluated together.

The First Token Is Not Yet a Spoken Answer

LLM time to first token is important, but it does not represent the end of the model stage.

Suppose the model begins with:

Sure — let me...

The first token arrived quickly, but the assistant still has nothing useful to say.

TTS may also need to wait for a stable clause so it does not synthesize an opening that later becomes awkward, incomplete, or incorrect.

A stronger voice response front-loads the answer:

Your appointment is confirmed for Thursday at 3 PM.

I can also send a reminder.

The first sentence is:

Complete
Useful
Independently speakable
Safe to send to TTS

This creates another useful metric:

Time to first speakable chunk

It includes the model’s TTFT plus the time required to accumulate a safe synthesis boundary.

A simple clause buffer may look like this:

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

const boundary = /[.!?]\s$|[,;:]\s$/;

let pending = "";

function onModelToken(token: string) {
  pending += token;

  const enoughText = pending.trim().length >= 24;
  const hasBoundary = boundary.test(pending);

  if (enoughText && hasBoundary) {
    tts.enqueue(pending);
    pending = "";
  }
}

function onModelComplete() {
  if (pending.trim()) {
    tts.enqueue(pending);
  }
}

This example is intentionally simple.

A production buffer should also consider:

Abbreviations
Numbers
Markdown or other markup
Pronunciation hints
Language-specific punctuation
Maximum waiting time
Whether enqueued speech can still be cancelled

Tool Latency Only Hurts When It Blocks the Critical Path

Tool calls are often described as additive latency.

For example:

A 300 ms lookup adds 300 ms to the response.

That is true only when the lookup sits directly on the serial critical path.

Some tool latency can be hidden or reduced by:

Prefetching likely context after a session starts
Caching results with a clear freshness policy
Running independent tools in parallel
Starting a truthful acknowledgement while a slow operation continues
Using deterministic routing when a model decision is unnecessary
Cancelling stale work when the user changes direction

A useful acknowledgement might be:

I’ll check the live inventory now.

However, filler speech should not be used merely to disguise arbitrary latency.

An acknowledgement is useful only when it communicates real progress and does not make an unsupported promise.

Trace every tool call with:

Queue time
Network time
Server processing time
Result size
Cache status
Retry count
Whether it blocked the first speakable chunk

The final field is often more useful than the tool’s total duration.

Streaming Is Controlled Overlap

“Stream every stage” sounds attractive, but it is too absolute.

Partial output is valuable only when it is useful and safely reversible.

Potential problems include:

Partial transcripts changing
Tools returning only atomic results
TTS speaking text the model would have revised
Speculative work increasing cost
Cancellation becoming harder
Stale results entering the conversation state

A better rule is:

Stream when partial output is useful, and overlap work only when errors can be contained or reversed.

Incremental TTS can begin synthesis from partial text while later segments are still being generated.

However, no research result or benchmark justifies assuming the same millisecond improvement across every:

Model
Sentence
Device
Language
Network
Deployment architecture

Low-latency engineering also extends beyond model inference.

Media transport, jitter, packet loss, WebRTC behaviour, session ownership, routing, and infrastructure placement all influence the pause.

Build an Explicit Latency Budget

A latency budget should be a design constraint, not a chart created after launch.

Start with an end-of-turn-to-playback target, then assign provisional limits to each stage on the critical path.

Here is an illustrative and deliberately aggressive 800 ms budget.

This is a planning example, not an industry benchmark.

Stage	Example budget
Turn decision	150 ms
Transcript stabilization	100 ms
First speakable model chunk	300 ms
First playable TTS audio	150 ms
Transport and client buffering	100 ms
Total	800 ms

The allocation should change based on the product.

A hands-free command may prioritize speed.

A medical intake flow may tolerate a longer delay to reduce the chance of interrupting the user.

A tool-heavy transaction may need a short acknowledgement before the final answer.

Measure distributions, not averages

At minimum, report:

Break these metrics down by:

Interaction type
Geography
Network type
Language
Device
Telephony provider
Tool-free versus tool-dependent turns
Successful turns
Interrupted turns
Cancelled turns
Retried turns

An acceptable p50 can hide a painful p95.

Instrument the Complete Turn

The following TypeScript example records useful event boundaries without coupling the implementation to a specific STT, LLM, or TTS provider.

const marks = new Map<string, number>();

function mark(name: string, at = performance.now()) {
  marks.set(name, at);
}

function duration(start: string, end: string) {
  const startTime = marks.get(start);
  const endTime = marks.get(end);

  if (startTime === undefined || endTime === undefined) {
    return undefined;
  }

  return endTime - startTime;
}

function voiceTurnMetrics() {
  return {
    turnDecisionMs: duration(
      "speech_last_frame",
      "turn_committed"
    ),

    transcriptReadyMs: duration(
      "turn_committed",
      "stt_final"
    ),

    llmTtftMs: duration(
      "llm_started",
      "llm_first_token"
    ),

    firstSpeakableMs: duration(
      "llm_started",
      "first_speakable_chunk"
    ),

    ttsFirstAudioMs: duration(
      "tts_started",
      "tts_first_audio"
    ),

    clientBufferMs: duration(
      "tts_first_audio",
      "playback_started"
    ),

    endToEndMs: duration(
      "speech_last_frame",
      "playback_started"
    ),
  };
}

Call mark() from the actual callbacks in your pipeline:

mark("speech_last_frame", vad.lastSpeechTimestamp());
mark("turn_committed");
mark("stt_final");
mark("llm_started");
mark("llm_first_token");
mark("first_speakable_chunk");
mark("tts_started");
mark("tts_first_audio");
mark("playback_started");

Use one trace ID across:

Browser or phone gateway
STT service
Orchestration layer
External tools
Language model
TTS service
Playback client

Without cross-service correlation, teams often optimize the service with the most visible dashboard instead of the stage responsible for the user’s wait.

When comparing a managed voice stack with a custom pipeline, apply the same:

Event boundaries
p50 measurements
p95 measurements
p99 measurements
Accuracy tests
Cancellation tests

An integrated stack should not automatically be assumed to have low end-to-end latency.

Barge-In Must Cancel the Old Turn

In a full-duplex assistant, inbound audio capture and speech detection should normally remain active while the assistant is speaking.

When new user speech is confirmed, the system must stop treating the old response as current.

The control path may resemble this:

async function onUserSpeechStarted() {
  generation.abort();

  await tts.cancel();

  playback.stopAndFlush();

  tools.cancelNonReusableWork();

  conversation.truncateAssistantMessage({
    toPlayedAudioTimestamp: playback.lastPlayedTimestamp(),
  });
}

The exact APIs will differ, but the responsibilities remain similar.

The system should:

Cancel model generation.
Cancel speech synthesis.
Stop and flush queued playback.
Discard tool results that are no longer relevant.
Preserve only the response portion the user actually heard.
Continue processing incoming speech without clipping its beginning.

Barge-in cannot be added cleanly as a final UI feature.

It affects:

Audio capture
Session state
Model context
Tool cancellation
Playback architecture
Conversation history

Production Starts Where the Demo Ends

A convincing demo proves that the happy path can answer.

Production requires testing for real human behaviour and imperfect infrastructure.

Before launch, test:

Quiet audio
Noisy audio
Reverberant rooms
Low-bitrate audio
Short commands
Long and hesitant utterances
Mid-sentence pauses
Self-corrections
Names and phone numbers
Dates and monetary amounts
Domain-specific terminology
Tool-free calls
Cached tool calls
Slow tool calls
Failed and retried tool calls
Barge-in during early playback
Barge-in during late playback
Packet loss
Jitter
Reconnects
Duplicated events
Multiple languages
Code-switching
Long conversations
Context compaction
Realistic concurrency

Also test failure semantics.

Ask questions such as:

What happens if a tool succeeds after the user interrupts?
What happens if TTS emits audio after cancellation?
What happens if two final transcripts arrive?
What happens if the client reconnects while audio remains queued?
What happens if a stale tool result returns after the conversation has changed?

A low-latency system that performs stale actions is not a good system.

When reviewing an implementation, study how it represents:

Processors
Transports
Interruptions
Events
Cancellation
Session state

Do not copy default settings blindly.

Validate each design choice against your own traces, traffic, users, and failure modes.

The Pause Is the Architecture

The fastest model will not rescue a pipeline with:

Slow turn detection
Serial tool calls
Unstable partial transcripts
Excessive client buffering
Weak cancellation
Poor session state management

Measure the complete turn from the user’s last speech frame to audible playback.

Separate that duration into named stages.

Optimize the stage dominating p95, then test the change against:

Recognition accuracy
Entity accuracy
Task success
Cancellation behaviour
User interruption patterns

A natural voice assistant is not simply a collection of fast components.

It is one coordinated participant whose timing, state, and failure modes have been designed as a whole.

Build and Measure a Voice Agent

Ready to test this architecture in a working voice stack?

Build a voice agent with the Smallest.ai API and instrument the event boundaries described above.

Compare your:

p50 end-to-end latency
p95 end-to-end latency
Time to first speakable chunk
Barge-in cancellation time
Tool-dependent latency

against your current production pipeline.

Your First Voice AI App Shouldn’t Start With a WebSocket

Smallest AI — Mon, 27 Jul 2026 16:16:46 +0000

The first successful voice feature usually looks unimpressive:

One audio file goes in, and one useful transcript comes out.

That is not a toy result.

It is the shortest route to the questions that determine whether a voice product will actually work:

Does the transcript preserve names and numbers?
Are speaker turns separated correctly?
Are timestamps useful downstream?
Can the system handle the audio your users will really produce?

It is tempting to begin with:

Live microphones
Persistent connections
Partial transcripts
Browser permissions
Animated interfaces

Those pieces feel like voice AI.

They also make failures harder to isolate.

A safer rule is:

Prove the transcript’s value first. Then earn the complexity of streaming.

Start With the Product’s Clock

The important architectural choice is not Python versus another language.

It is:

Batch or real time?

If you are processing a:

Voicemail
Podcast
Meeting recording
Customer call archive
Uploaded audio file

then the entire recording already exists.

An HTTP request matches the problem:

Upload audio
   ↓
Wait for processing
   ↓
Receive transcript
   ↓
Store or analyze result

There is no benefit in pretending a completed file is a live conversation.

A voice assistant, live-captioning tool, or real-time call workflow runs on a different clock.

It needs to:

Receive small audio frames
Return partial transcripts while the user is speaking
Decide when an utterance is final
Recover when the connection drops
Handle interruptions and timing-sensitive actions

That is where a WebSocket belongs.

Smallest AI publishes a roughly 64 ms time-to-first-transcript figure for Pulse streaming.

Treat that as a provider specification, not a guarantee for your application.

Your actual latency will depend on:

Network path
Audio framing
Region
Application load
Buffering
Transcript-stability requirements

The provider specification is useful.

Your own p50 and p95 traces should make the architecture decision.

The Smallest Useful Python Proof

A batch prototype should stay intentionally boring. Put the API key in an environment variable, read a representative audio file, request only the metadata you need, and inspect the returned JSON.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

It should:

Read a representative audio file.
Send it to the speech-to-text API.
Request only the metadata you need.
Inspect the returned JSON.
Record failures.

Here is a minimal Python example:

import os

import requests

endpoint = "https://api.smallest.ai/waves/v1/stt/"

params = {
    "model": "pulse",
    "language": "en",
    "word_timestamps": "true",
    "diarize": "true",
}

headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}",
    "Content-Type": "application/octet-stream",
}

with open("meeting.wav", "rb") as audio:
    response = requests.post(
        endpoint,
        params=params,
        headers=headers,
        data=audio,
        timeout=120,
    )

response.raise_for_status()
result = response.json()

print(result["transcription"])
print(result.get("words", [])[:5])

This uses the unified endpoint documented in the pre-recorded STT API reference.

In production, also handle:

Timeouts
Retries
Rate limits
Invalid audio
Empty transcripts
Response-schema changes
Request identifiers
Unexpected status codes

Do not log the authorization header.

Do not log raw audio by default.

That short loop already gives you a useful evaluation surface.

Test:

Multiple accents
Different speaking styles
Clean recordings
Noisy recordings
Telephony audio
Fast speech
Hesitant speech
Multiple speakers

Do not evaluate only the sentence printed to the terminal.

Inspect the JSON structure your next component must consume.

Measure the Transcript You Need

A single successful file proves connectivity.

A small evaluation set begins to prove usefulness.

Use audio captured from the:

Microphones
Codecs
Channels
Devices
Rooms
Networks

your users will actually have.

Track failures involving:

Names
Phone numbers
Dates
Amounts
Abbreviations
Product names
Domain terminology
Code-switching
Speaker changes

Do not measure only average text quality.

A transcript can look mostly correct while still failing on the information your application needs most.

Useful evaluation areas

Area	What to inspect
Transcription quality	Missing, substituted, or hallucinated words
Entity accuracy	Names, numbers, dates, addresses, identifiers
Speaker separation	Whether participants are assigned consistently
Timestamp quality	Drift, missing words, alignment usefulness
Latency	p50, p95, and timeout rate
Failure handling	Rejected files, empty output, malformed responses
Downstream usefulness	Whether the next system can use the transcript reliably

This stage also tells you whether streaming is necessary.

If users upload recordings and return later, a persistent connection may add operational cost without improving the product.

If the transcript drives a live response, batch processing will eventually reveal its limit.

Streaming Is Not a Faster POST Request

Moving from batch to streaming changes the application.

It does not merely change the transport.

A streaming client must:

Send audio frames continuously
Receive transcript events concurrently
Handle partial and final results
Manage connection state
Apply backpressure
Recover from failures
Avoid duplicated segments

Batch waits for the complete recording.

Streaming turns a moving signal into usable partial results.

Here is a simplified Python example:

import asyncio
import json
import os
from urllib.parse import urlencode

import websockets

CHUNK_SIZE = 4096

params = {
    "language": "en",
    "encoding": "linear16",
    "sample_rate": "16000",
    "word_timestamps": "true",
}

url = (
    "wss://api.smallest.ai/waves/v1/stt/live?model=pulse&"
    + urlencode(params)
)

headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}"
}


async def send_audio(ws, path: str) -> None:
    with open(path, "rb") as audio:
        while chunk := audio.read(CHUNK_SIZE):
            await ws.send(chunk)
            await asyncio.sleep(0.05)

    await ws.send(
        json.dumps(
            {
                "type": "close_stream",
            }
        )
    )


async def receive_transcripts(ws) -> None:
    async for message in ws:
        event = json.loads(message)

        label = "final" if event.get("is_final") else "partial"
        transcript = event.get("transcript", "")

        print(label, transcript)

        if event.get("is_last"):
            return


async def transcribe_file(path: str) -> None:
    async with websockets.connect(
        url,
        additional_headers=headers,
    ) as ws:
        sender = asyncio.create_task(send_audio(ws, path))
        receiver = asyncio.create_task(receive_transcripts(ws))

        await asyncio.gather(sender, receiver)


asyncio.run(transcribe_file("audio.pcm"))

The current real-time WebSocket documentation recommends:

4096-byte chunks
A close_stream control message when the stream is complete

The example expects:

Headerless audio
16-bit linear PCM
16 kHz
Mono

A WAV file normally includes a container header.

Do not rename a .wav file to .pcm and assume the bytes are equivalent.

The Concurrency Model Matters More Than the Socket

A production microphone flow should separate responsibilities.

Audio producer

Reads fixed-size frames and places them in a bounded queue.

Network sender

Writes frames to the provider, handles backpressure, and stops cleanly.

Transcript receiver

Receives partial and final transcript events.

Application state

Stores stable transcript segments and exposes connection status.

Recovery path

Reconnects deliberately and prevents duplicate or out-of-order segments.

A useful architecture might look like this:

Microphone capture
      ↓
Bounded audio queue
      ↓
Network sender ───────────────┐
                              │
Provider WebSocket            │
                              │
Transcript receiver ◀─────────┘
      ↓
Partial transcript state
      ↓
Final transcript state
      ↓
UI or downstream workflow

If one task blocks the others, buffers grow.

The interface then feels stale even when speech recognition itself is fast.

Partial Transcripts Are a Preview

Partial transcripts are not an append-only log.

Suppose the server emits:

I need to update

Then:

I need to update my address

Then:

I need to update my billing address

Appending every partial result would produce duplicated text.

Instead:

Replace the current partial segment when a new hypothesis arrives.
Display it as provisional.
Persist it only when the server marks it final.

A simple state model could look like this:

stable_segments: list[str] = []
current_partial = ""


def handle_event(event: dict) -> None:
    global current_partial

    transcript = event.get("transcript", "")

    if event.get("is_final"):
        stable_segments.append(transcript)
        current_partial = ""
    else:
        current_partial = transcript

This distinction matters beyond the interface.

Downstream systems should normally process stable text, not every revisable hypothesis.

Otherwise, the application may act on text the recognizer later corrects.

Keep the Browser Simple

A browser can:

Capture microphone audio
Display transcript updates
Show connection state
Let the user stop or restart a session

It should not receive a long-lived service API key.

Put the authenticated provider connection behind your server.

Browser
   ↓
Your authenticated application server
   ↓
Speech-to-text provider

The browser connects to your application.

Your server:

Authorizes the user
Opens the provider connection
Applies rate limits
Enforces session duration
Validates input format
Controls concurrency
Applies logging policy
Forwards only the events the interface needs

This creates a clean progression:

Uploaded audio
→ normal server route
→ batch transcription

Microphone audio
→ streaming proxy
→ live transcription

Batch and live transcription remain two deliberate modes instead of one over-engineered pipeline.

A Transcript Is Rarely the Final Product

Raw text is enough for a connectivity demo.

Production workflows usually need structure.

Word timestamps

Useful for:

Captions
Transcript review
Search
Audio navigation
Highlight reels

Speaker diarization

Useful for:

Meetings
Interviews
Support calls
Sales conversations
Multi-participant recordings

Redaction

Useful for reducing sensitive information before sending transcripts downstream.

Confidence or stability signals

Useful when the application must decide whether to:

Accept a result
Ask for confirmation
Flag a segment for review
Delay a downstream action

Each feature should exist because a workflow requires it.

Do not enable every parameter merely because it is available.

Know When the Prototype Has Earned Real Time

Move from batch to streaming when at least one requirement cannot be met by waiting for the complete file.

Streaming is justified when:

The user needs words on screen while speaking.
A conversational system must begin reasoning before the full recording exists.
Turn-taking depends on partial results.
Interruption handling depends on live audio.
Live routing depends on the speaker’s current words.
The recording is too long for upload-then-process.
Waiting for completion creates unacceptable product delay.

If none of those requirements apply, batch may still be the better architecture.

Measure the Complete Streaming Path

Once you move to real time, measure more than model latency.

Record:

Stage	What it captures
Audio capture delay	Time between speech and frame availability
Queue delay	Time waiting before frames are sent
Network transit	Time spent reaching the provider
First usable partial	Time until text becomes useful to the interface
Finalization delay	Time until the utterance becomes stable
Downstream processing	Time used by search, tools, or an LLM
First audible response	Time until the user hears the system respond

Optimizing only the model’s headline latency can hide the stage users are actually waiting on.

A useful first-partial metric should begin at the audio-capture boundary:

First usable partial latency
=
first useful transcript timestamp
-
first relevant audio-frame timestamp

For conversational systems, the full user-facing measurement should extend to audible playback.

Build Outward From One Trustworthy Transcript

A voice product does not become serious when it opens a WebSocket.

It becomes serious when every added layer solves a problem the simpler version exposed.

Start with:

Representative audio
      ↓
One API request
      ↓
One structured transcript
      ↓
Evaluation

Then verify:

Transcript accuracy
Entity preservation
Speaker labels
Timestamps
Failure modes
Response structure
Batch latency

Add streaming only when the user’s clock demands it.

Then:

Keep the API key behind your server.
Separate sending and receiving.
Treat partial text as revisable.
Add bounded buffering.
Design recovery deliberately.
Measure the whole path under realistic load.

The goal is not to use a WebSocket. The goal is to build a voice feature whose complexity matches the problem.

Try the Batch-to-Streaming Path

To test the same progression with Smallest AI:

Open the Smallest AI self-serve application.
Create an API key.
Start with one representative recording.
Validate the returned transcript and metadata.
Move to streaming only when the product requires live results.

Create an API key with Smallest AI

What requirement in your product genuinely forces the architecture to become real time?