Your First Voice AI App Shouldn’t Start With a WebSocket

The first successful voice feature usually looks unimpressive:

One audio file goes in, and one useful transcript comes out.

That is not a toy result.

It is the shortest route to the questions that determine whether a voice product will actually work:

• Does the transcript preserve names and numbers?
• Are speaker turns separated correctly?
• Are timestamps useful downstream?
• Can the system handle the audio your users will really produce?

It is tempting to begin with:

• Live microphones
• Persistent connections
• Partial transcripts
• Browser permissions
• Animated interfaces

Those pieces feel like voice AI.

They also make failures harder to isolate.

A safer rule is:

Prove the transcript’s value first. Then earn the complexity of streaming.

Start With the Product’s Clock

The important architectural choice is not Python versus another language.

It is:

Batch or real time?

If you are processing a:

• Voicemail
• Podcast
• Meeting recording
• Customer call archive
• Uploaded audio file

then the entire recording already exists.

An HTTP request matches the problem:

Upload audio
   ↓
Wait for processing
   ↓
Receive transcript
   ↓
Store or analyze result

There is no benefit in pretending a completed file is a live conversation.

A voice assistant, live-captioning tool, or real-time call workflow runs on a different clock.

It needs to:

• Receive small audio frames
• Return partial transcripts while the user is speaking
• Decide when an utterance is final
• Recover when the connection drops
• Handle interruptions and timing-sensitive actions

That is where a WebSocket belongs.

Smallest AI publishes a roughly 64 ms time-to-first-transcript figure for Pulse streaming.

Treat that as a provider specification, not a guarantee for your application.

Your actual latency will depend on:

• Network path
• Audio framing
• Region
• Application load
• Buffering
• Transcript-stability requirements

The provider specification is useful.

Your own p50 and p95 traces should make the architecture decision.

The Smallest Useful Python Proof

A batch prototype should stay intentionally boring. Put the API key in an environment variable, read a representative audio file, request only the metadata you need, and inspect the returned JSON.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

It should:

1. Read a representative audio file.
2. Send it to the speech-to-text API.
3. Request only the metadata you need.
4. Inspect the returned JSON.
5. Record failures.

Here is a minimal Python example:

import os

import requests

endpoint = "https://api.smallest.ai/waves/v1/stt/"

params = {
    "model": "pulse",
    "language": "en",
    "word_timestamps": "true",
    "diarize": "true",
}

headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}",
    "Content-Type": "application/octet-stream",
}

with open("meeting.wav", "rb") as audio:
    response = requests.post(
        endpoint,
        params=params,
        headers=headers,
        data=audio,
        timeout=120,
    )

response.raise_for_status()
result = response.json()

print(result["transcription"])
print(result.get("words", [])[:5])

This uses the unified endpoint documented in the pre-recorded STT API reference.

In production, also handle:

• Timeouts
• Retries
• Rate limits
• Invalid audio
• Empty transcripts
• Response-schema changes
• Request identifiers
• Unexpected status codes

Do not log the authorization header.

Do not log raw audio by default.

That short loop already gives you a useful evaluation surface.

Test:

• Multiple accents
• Different speaking styles
• Clean recordings
• Noisy recordings
• Telephony audio
• Fast speech
• Hesitant speech
• Multiple speakers

Do not evaluate only the sentence printed to the terminal.

Inspect the JSON structure your next component must consume.

Measure the Transcript You Need

A single successful file proves connectivity.

A small evaluation set begins to prove usefulness.

Use audio captured from the:

• Microphones
• Codecs
• Channels
• Devices
• Rooms
• Networks

your users will actually have.

Track failures involving:

• Names
• Phone numbers
• Dates
• Amounts
• Abbreviations
• Product names
• Domain terminology
• Code-switching
• Speaker changes

Do not measure only average text quality.

A transcript can look mostly correct while still failing on the information your application needs most.

Useful evaluation areas

Area	What to inspect
Transcription quality	Missing, substituted, or hallucinated words
Entity accuracy	Names, numbers, dates, addresses, identifiers
Speaker separation	Whether participants are assigned consistently
Timestamp quality	Drift, missing words, alignment usefulness
Latency	p50, p95, and timeout rate
Failure handling	Rejected files, empty output, malformed responses
Downstream usefulness	Whether the next system can use the transcript reliably

This stage also tells you whether streaming is necessary.

If users upload recordings and return later, a persistent connection may add operational cost without improving the product.

If the transcript drives a live response, batch processing will eventually reveal its limit.

Streaming Is Not a Faster POST Request

Moving from batch to streaming changes the application.

It does not merely change the transport.

A streaming client must:

• Send audio frames continuously
• Receive transcript events concurrently
• Handle partial and final results
• Manage connection state
• Apply backpressure
• Recover from failures
• Avoid duplicated segments

Batch waits for the complete recording.

Streaming turns a moving signal into usable partial results.

Here is a simplified Python example:

import asyncio
import json
import os
from urllib.parse import urlencode

import websockets

CHUNK_SIZE = 4096

params = {
    "language": "en",
    "encoding": "linear16",
    "sample_rate": "16000",
    "word_timestamps": "true",
}

url = (
    "wss://api.smallest.ai/waves/v1/stt/live?model=pulse&"
    + urlencode(params)
)

headers = {
    "Authorization": f"Bearer {os.environ['SMALLEST_API_KEY']}"
}


async def send_audio(ws, path: str) -> None:
    with open(path, "rb") as audio:
        while chunk := audio.read(CHUNK_SIZE):
            await ws.send(chunk)
            await asyncio.sleep(0.05)

    await ws.send(
        json.dumps(
            {
                "type": "close_stream",
            }
        )
    )


async def receive_transcripts(ws) -> None:
    async for message in ws:
        event = json.loads(message)

        label = "final" if event.get("is_final") else "partial"
        transcript = event.get("transcript", "")

        print(label, transcript)

        if event.get("is_last"):
            return


async def transcribe_file(path: str) -> None:
    async with websockets.connect(
        url,
        additional_headers=headers,
    ) as ws:
        sender = asyncio.create_task(send_audio(ws, path))
        receiver = asyncio.create_task(receive_transcripts(ws))

        await asyncio.gather(sender, receiver)


asyncio.run(transcribe_file("audio.pcm"))

The current real-time WebSocket documentation recommends:

• 4096-byte chunks
• A close_stream control message when the stream is complete

The example expects:

Headerless audio
16-bit linear PCM
16 kHz
Mono

A WAV file normally includes a container header.

Do not rename a .wav file to .pcm and assume the bytes are equivalent.

The Concurrency Model Matters More Than the Socket

A production microphone flow should separate responsibilities.

Audio producer

Reads fixed-size frames and places them in a bounded queue.

Network sender

Writes frames to the provider, handles backpressure, and stops cleanly.

Transcript receiver

Receives partial and final transcript events.

Application state

Stores stable transcript segments and exposes connection status.

Recovery path

Reconnects deliberately and prevents duplicate or out-of-order segments.

A useful architecture might look like this:

Microphone capture
      ↓
Bounded audio queue
      ↓
Network sender ───────────────┐
                              │
Provider WebSocket            │
                              │
Transcript receiver ◀─────────┘
      ↓
Partial transcript state
      ↓
Final transcript state
      ↓
UI or downstream workflow

If one task blocks the others, buffers grow.

The interface then feels stale even when speech recognition itself is fast.

Partial Transcripts Are a Preview

Partial transcripts are not an append-only log.

Suppose the server emits:

I need to update

Then:

I need to update my address

Then:

I need to update my billing address

Appending every partial result would produce duplicated text.

Instead:

1. Replace the current partial segment when a new hypothesis arrives.
2. Display it as provisional.
3. Persist it only when the server marks it final.

A simple state model could look like this:

stable_segments: list[str] = []
current_partial = ""


def handle_event(event: dict) -> None:
    global current_partial

    transcript = event.get("transcript", "")

    if event.get("is_final"):
        stable_segments.append(transcript)
        current_partial = ""
    else:
        current_partial = transcript

This distinction matters beyond the interface.

Downstream systems should normally process stable text, not every revisable hypothesis.

Otherwise, the application may act on text the recognizer later corrects.

Keep the Browser Simple

A browser can:

• Capture microphone audio
• Display transcript updates
• Show connection state
• Let the user stop or restart a session

It should not receive a long-lived service API key.

Put the authenticated provider connection behind your server.

Browser
   ↓
Your authenticated application server
   ↓
Speech-to-text provider

The browser connects to your application.

Your server:

• Authorizes the user
• Opens the provider connection
• Applies rate limits
• Enforces session duration
• Validates input format
• Controls concurrency
• Applies logging policy
• Forwards only the events the interface needs

This creates a clean progression:

Uploaded audio
→ normal server route
→ batch transcription

Microphone audio
→ streaming proxy
→ live transcription

Batch and live transcription remain two deliberate modes instead of one over-engineered pipeline.

A Transcript Is Rarely the Final Product

Raw text is enough for a connectivity demo.

Production workflows usually need structure.

Word timestamps

Useful for:

• Captions
• Transcript review
• Search
• Audio navigation
• Highlight reels

Speaker diarization

Useful for:

• Meetings
• Interviews
• Support calls
• Sales conversations
• Multi-participant recordings

Redaction

Useful for reducing sensitive information before sending transcripts downstream.

Confidence or stability signals

Useful when the application must decide whether to:

• Accept a result
• Ask for confirmation
• Flag a segment for review
• Delay a downstream action

Each feature should exist because a workflow requires it.

Do not enable every parameter merely because it is available.

Know When the Prototype Has Earned Real Time

Move from batch to streaming when at least one requirement cannot be met by waiting for the complete file.

Streaming is justified when:

• The user needs words on screen while speaking.
• A conversational system must begin reasoning before the full recording exists.
• Turn-taking depends on partial results.
• Interruption handling depends on live audio.
• Live routing depends on the speaker’s current words.
• The recording is too long for upload-then-process.
• Waiting for completion creates unacceptable product delay.

If none of those requirements apply, batch may still be the better architecture.

Measure the Complete Streaming Path

Once you move to real time, measure more than model latency.

Record:

Stage	What it captures
Audio capture delay	Time between speech and frame availability
Queue delay	Time waiting before frames are sent
Network transit	Time spent reaching the provider
First usable partial	Time until text becomes useful to the interface
Finalization delay	Time until the utterance becomes stable
Downstream processing	Time used by search, tools, or an LLM
First audible response	Time until the user hears the system respond

Optimizing only the model’s headline latency can hide the stage users are actually waiting on.

A useful first-partial metric should begin at the audio-capture boundary:

First usable partial latency
=
first useful transcript timestamp
-
first relevant audio-frame timestamp

For conversational systems, the full user-facing measurement should extend to audible playback.

Build Outward From One Trustworthy Transcript

A voice product does not become serious when it opens a WebSocket.

It becomes serious when every added layer solves a problem the simpler version exposed.

Start with:

Representative audio
      ↓
One API request
      ↓
One structured transcript
      ↓
Evaluation

Then verify:

• Transcript accuracy
• Entity preservation
• Speaker labels
• Timestamps
• Failure modes
• Response structure
• Batch latency

Add streaming only when the user’s clock demands it.

Then:

• Keep the API key behind your server.
• Separate sending and receiving.
• Treat partial text as revisable.
• Add bounded buffering.
• Design recovery deliberately.
• Measure the whole path under realistic load.

The goal is not to use a WebSocket. The goal is to build a voice feature whose complexity matches the problem.

Try the Batch-to-Streaming Path

To test the same progression with Smallest AI:

1. Open the Smallest AI self-serve application.
2. Create an API key.
3. Start with one representative recording.
4. Validate the returned transcript and metadata.
5. Move to streaming only when the product requires live results.

Create an API key with Smallest AI

What requirement in your product genuinely forces the architecture to become real time?