Your Speech Recognition Demo Works. So Why Does It Fail in Production?

At first, everything looks fine. The microphone works, the browser returns text, and the transcript appears almost immediately. But when the user says something longer, the text changes halfway through the sentence. An identifier gets transcribed incorrectly, or the recording stops when the network connection drops.

The problem is not necessarily the speech recognition model. Your application also has to manage audio capture, interim results, browser compatibility, and the path between transcription and the action the user wants to perform.

A working transcript is only the beginning.

To build this feature properly, we'll start with the browser's built-in recognition API, examine where it becomes limiting, and then look at a server-side streaming architecture. Along the way, we'll address authentication, audio formats, transcript state, and the failures that tend to appear outside controlled development environments.

First, decide what the browser needs to recognize

Suppose our application lets users search for orders by speaking rather than typing.

The browser captures audio, speech recognition turns it into text, and the application uses that text to search its backend.

The distinction between speech recognition and voice recognition matters here. Speech recognition determines what someone said. Voice recognition concerns who is speaking. Our search feature needs the words, not the speaker's identity.

The underlying technology is automatic speech recognition (ASR). It processes incoming audio and estimates the spoken text. Depending on the implementation, recognition can run locally, through a browser-managed service, or through an API connected to your application.

That gives us two practical starting points:

• Browser-native recognition: Let the browser manage speech recognition and return the transcript through JavaScript events.

• Server-side ASR: Capture audio in the browser, send it to your backend, and use a dedicated recognition service to produce the transcript.

Neither approach eliminates the need to handle application state. The difference is how much control you have over the recognition pipeline.

Start with the Web Speech API

For our first implementation, assume the application is an internal tool and the development team controls the supported browser.

The Web Speech API is a reasonable place to start. It provides SpeechRecognition for converting speech into text and SpeechSynthesis for generating spoken output. We only need recognition.

Create a simple HTML interface:

HTML

<button id="startBtn">Start speaking</button>
<button id="stopBtn" disabled>Stop</button>
<p id="status" role="status">Ready</p>
<p id="output" aria-live="polite"></p>

Next, connect the buttons to the recognition API. The important detail is that recognition can produce multiple results. Some are provisional, while others are marked final. We should not treat every update as a new permanent line of text.

JAVASCRIPT

const startBtn = document.getElementById("startBtn");
const stopBtn = document.getElementById("stopBtn");
const output = document.getElementById("output");
const status = document.getElementById("status");

const Recognition =
  window.SpeechRecognition || window.webkitSpeechRecognition;

if (!Recognition) {
  status.textContent =
    "Speech recognition is unavailable. Please use text input.";

  startBtn.disabled = true;
} else {
  const recognition = new Recognition();

  recognition.lang = "en-US";
  recognition.interimResults = true;
  recognition.maxAlternatives = 1;

  recognition.onstart = () => {
    status.textContent = "Listening...";
    startBtn.disabled = true;
    stopBtn.disabled = false;
  };

  recognition.onresult = (event) => {
    let confirmed = "";
    let interim = "";

    for (const result of event.results) {
      const text = result[0].transcript;

      if (result.isFinal) {
        confirmed += text + " ";
      } else {
        interim += text;
      }
    }

    output.textContent = confirmed + interim;
  };

  recognition.onerror = (event) => {
    status.textContent =
      `Recognition error: ${event.error}`;
  };

  recognition.onend = () => {
    if (status.textContent === "Listening...") {
      status.textContent = "Stopped";
    }

    startBtn.disabled = false;
    stopBtn.disabled = true;
  };

  startBtn.addEventListener("click", () => {
    output.textContent = "";
    recognition.start();
  });

  stopBtn.addEventListener("click", () => {
    recognition.stop();
  });
}

This corrects a subtle problem in many minimal examples: reading only one transcript without accounting for its finality.

When the user speaks, an interim result may change as the recognizer receives more audio. The UI should show that progress without repeatedly appending the same words.

Before treating the example as complete, test it in your target browsers. MDN's SpeechRecognition reference documents the API and its browser support limitations.

Chrome's conventional speech recognition implementation sends audio to a remote recognition service. Browser-native therefore does not necessarily mean offline or private to the user's device. Some implementations also provide experimental on-device recognition features, but availability and language-pack requirements need separate verification.

For a controlled prototype, those limitations may be acceptable. For a customer-facing feature that needs more predictable control over the audio pipeline, we need another architecture.

Move audio capture out of the recognition API

Now imagine extending the order-search feature into a customer-facing application.

Users arrive with different browsers, microphones, accents, and network conditions. You may also need to evaluate different recognition providers against your application's vocabulary.

A useful architectural separation is to let the browser handle capture and the backend handle recognition.

The flow becomes:

Browser microphone → application WebSocket → backend audio processing → ASR service → transcript → browser UI

The frontend no longer needs to know which ASR model your backend uses. This also creates an important security boundary. Your application can authenticate the user, manage the recognition session, and keep the speech provider's API credentials on the server.

For example, Smallest AI offers the Pulse speech-to-text service, which supports streaming transcription through WebSocket. It can serve as the recognition layer behind this architecture.

The browser still needs to send compatible audio. It cannot assume that whatever MediaRecorder produces will be accepted directly by the ASR service.

That is the next implementation decision.

Capture microphone audio without assuming every browser uses the same format

The browser's getUserMedia() API provides access to the microphone after the user grants permission. MediaRecorder can then divide that audio into chunks for transmission.

The following example implements the browser-capture portion of our application. It assumes that you have configured an application-owned WebSocket endpoint at wss://YOUR_APP_HOST/asr. That endpoint must be implemented on your backend. It is not a Smallest AI API URL.

JAVASCRIPT

async function startCapture() {
  const mimeType = "audio/webm;codecs=opus";

  if (!MediaRecorder.isTypeSupported(mimeType)) {
    throw new Error(
      "This browser does not support the selected audio format."
    );
  }

  let stream;

  try {
    stream = await navigator.mediaDevices.getUserMedia({
      audio: true
    });

    const recorder = new MediaRecorder(stream, {
      mimeType
    });

    const socket = new WebSocket(
      "wss://YOUR_APP_HOST/asr"
    );

    socket.onopen = () => {
      recorder.start(250);
    };

    recorder.ondataavailable = (event) => {
      if (
        event.data.size > 0 &&
        socket.readyState === WebSocket.OPEN
      ) {
        socket.send(event.data);
      }
    };

    socket.onmessage = (event) => {
      try {
        const result = JSON.parse(event.data);

        if (typeof result.transcript === "string") {
          document.getElementById("output").textContent =
            result.transcript;
        }
      } catch {
        console.error("Invalid transcript message");
      }
    };

    socket.onerror = () => {
      document.getElementById("status").textContent =
        "The transcription connection failed.";
    };

    socket.onclose = () => {
      if (recorder.state !== "inactive") {
        recorder.stop();
      }

      stream.getTracks().forEach((track) => track.stop());
    };

    return {
      stop() {
        if (recorder.state !== "inactive") {
          recorder.stop();
        }

        stream.getTracks().forEach((track) => track.stop());

        if (socket.readyState === WebSocket.OPEN) {
          socket.close();
        }
      }
    };
  } catch (error) {
    stream?.getTracks().forEach((track) => track.stop());
    throw error;
  }
}

The 250 ms recording interval is an illustrative configuration inherited from the capture pattern. It is not a guaranteed chunk duration or an optimal value for every ASR provider.

There is also an audio-format distinction worth preserving. audio/webm;codecs=opus describes Opus audio in a WebM container. An ASR service that supports Opus or Ogg Opus does not automatically accept WebM-wrapped audio.

For Pulse, the official streaming audio specifications document supported encodings, sample rates, and channel requirements.

Your backend must deliver audio that matches those requirements. One documented option is 16 kHz, mono, 16-bit PCM using the linear16 encoding.

If you choose that format, you need a compatible browser capture path or a backend conversion step. Changing the encoding parameter in the API URL does not convert the audio itself.

The code above is the browser-facing transport layer, not a complete provider integration. It also needs an application-specific user-authentication mechanism and production reconnection policy.

Before wiring these pieces together, it is worth testing the ASR connection independently.

Create and store the API key

Keep the API key in an environment variable rather than hard-coding it into the application.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

BASH

export SMALLEST_API_KEY="your-api-key-here"

Every authenticated Smallest AI request sends the value through the Authorization header:

TEXT

Authorization: Bearer <SMALLEST_API_KEY value>

Keep the key on your server. Do not expose it in browser JavaScript, mobile application code, public repositories, screenshots, query parameters, or client-side logs.

For production deployments, use a server-side secrets manager and restrict access to the application components that need the credential.

Test the streaming ASR connection independently

Before connecting live browser audio to the backend, verify that your server can authenticate with the recognition service and receive transcript messages.

The Smallest AI API provides access to Pulse. Its documented streaming endpoint is:

TEXT

wss://api.smallest.ai/waves/v1/stt/live?model=pulse

The following Node.js test uses a local file containing raw 16 kHz, mono, signed 16-bit PCM audio. It is an isolated provider test, not the complete browser relay.

Install the WebSocket dependency:

BASH

npm install ws

If your source is a WAV file, you can prepare the raw PCM test input locally with FFmpeg:

BASH

ffmpeg -i sample.wav -ar 16000 -ac 1 -f s16le sample-16k-mono-s16le.raw

This creates headerless PCM. Sending the original WAV bytes while declaring linear16 would include container data that the raw streaming example does not expect.

Before running the snippet, create a Smallest.ai API key in the dashboard and store it in the SMALLEST_API_KEY environment variable.

JAVASCRIPT

// save as test-pulse.cjs
const fs = require("node:fs");
const { once } = require("node:events");
const { setTimeout: sleep } = require("node:timers/promises");
const WebSocket = require("ws");

const apiKey = process.env.SMALLEST_API_KEY;

if (!apiKey) {
  throw new Error("SMALLEST_API_KEY is not set");
}

const audioFile = "sample-16k-mono-s16le.raw";
const sampleRate = 16000;

const url = new URL(
  "wss://api.smallest.ai/waves/v1/stt/live?model=pulse"
);

url.searchParams.set("language", "en");
url.searchParams.set("encoding", "linear16");
url.searchParams.set("sample_rate", String(sampleRate));

const socket = new WebSocket(url.toString(), {
  headers: {
    Authorization: `Bearer ${apiKey}`
  }
});

socket.on("message", (data) => {
  try {
    const result = JSON.parse(data.toString());

    if (typeof result.transcript === "string") {
      console.log(
        result.is_final ? "Final:" : "Interim:",
        result.transcript
      );
    }

    if (result.is_last) {
      socket.close();
    }
  } catch {
    console.error("Unexpected response format");
  }
});

socket.on("error", () => {
  console.error("Speech recognition connection failed");
});

async function run() {
  await once(socket, "open");

  const stream = fs.createReadStream(audioFile, {
    highWaterMark: 4096
  });

  for await (const chunk of stream) {
    if (socket.readyState !== WebSocket.OPEN) {
      throw new Error("WebSocket connection closed");
    }

    socket.send(chunk);

    // Approximate real-time pacing for 16-bit mono PCM.
    const chunkDurationMs =
      (chunk.length / (sampleRate * 2)) * 1000;

    await sleep(chunkDurationMs);
  }

  socket.send(JSON.stringify({
    type: "close_stream"
  }));
}

run().catch((error) => {
  console.error(error.message);
  socket.terminate();
  process.exitCode = 1;
});

Run it with:

BASH

node test-pulse.cjs

Check whether the service returns transcript messages and marks completed segments with is_final.

The close_stream message signals that audio transmission has finished. The documented is_last field identifies the final session response.

This test isolates authentication, audio formatting, and provider response handling. It does not prove that your browser capture and backend relay are correct.

Once it passes, you can connect the application-owned WebSocket to Pulse, forwarding appropriately converted audio upstream and transcript messages downstream. That relay needs its own session lifecycle, user authorization, resource limits, and failure handling.

Keep interim text separate from confirmed text

Return to the user searching for an order number.

If the interface replaces the entire field with every incoming partial transcript, the displayed text can appear to flicker. If it permanently appends each interim update, words can appear more than once.

The application needs two distinct states: confirmed text and text that may still change.

For a streaming service that emits separate transcript segments, the application can maintain:

JAVASCRIPT

let confirmed = "";
let interim = "";

function handleTranscript(result) {
  if (result.is_final) {
    confirmed += result.transcript + " ";
    interim = "";
  } else {
    interim = result.transcript;
  }

  document.getElementById("output").textContent =
    confirmed + interim;
}

This assumes the service emits finalized segments rather than repeatedly sending the entire cumulative transcript. Confirm the exact response semantics before using it with another provider.

The important engineering rule is independent of the provider: do not let provisional text trigger irreversible application behavior.

For our order-search example, showing a partial transcript is fine. Submitting the search or changing an order based on an incomplete identifier is a different decision.

A command-oriented application may use a confirmed segment or an explicit user confirmation before performing an action.

If your provider offers word-level confidence or timestamps, those can help with review and alignment. They should not be assumed to exist in every response.

For more background on ASR failure modes, Smallest AI's guide to accents, noise, and speech recognition challenges provides additional evaluation context.

Treat microphone permission as part of the feature

Our example begins with a button click for a reason.

Microphone access through getUserMedia() requires a secure context. In a deployed web application, that normally means HTTPS. Localhost is treated as a trustworthy origin for development.

Users should understand why the application is asking for microphone access. Request permission when they choose to record, rather than attempting to open the microphone during page initialization.

Once recording begins, the interface should make that state visible and provide an accessible stop control.

The application also needs to distinguish failures that have different remedies:

• Permission denied: Explain that microphone access is blocked and provide a text-input alternative.

• Unsupported browser: Detect unavailable APIs and offer another input method.

• Network interruption: Stop or suspend capture safely and make the incomplete transcript visible.

• Audio-format mismatch: Check the actual audio encoding, sample rate, and channel count before changing recognition settings.

• No-speech timeout: Let the user restart without losing previously confirmed text.

WebSocket reconnection deserves particular care. Reopening a socket does not guarantee that the previous recognition session can resume.

If your backend starts a fresh session, preserve confirmed application state and explicitly decide how to handle audio that was not processed. Do not silently replay a command that might already have triggered an action.

Privacy also extends beyond microphone permission. If you retain transcripts or audio, document what is stored, how long it remains available, and how users can request deletion.

A transcript is not yet an application action

Once recognition works, our order-search feature still needs to interpret the text.

The full application path is:

Audio capture → ASR → transcript processing → intent or entity extraction → application action

Not every application needs a language model after transcription. For a simple voice-search field, ordinary search logic may be sufficient. A command interface might use keyword or rule-based intent matching. A more conversational application may need additional language processing.

The important boundary is between recognizing words and deciding what those words mean for your application.

Post-processing can also include punctuation restoration, normalization, speaker labeling, or domain-specific vocabulary handling, depending on the recognition provider and the requirements of the task.

An incorrect identifier should not be silently corrected into a plausible but different identifier. If the application needs an exact order number, confirmation may be safer than guessing.

That is why end-to-end evaluation should include the action produced from the transcript, not only whether individual words were recognized correctly.

Test with the audio your users will actually produce

The final test should reproduce the conditions that made our initial implementation unreliable.

Record a small, permissioned evaluation set containing representative order IDs, names, commands, and phrases. Include the accents, background noise, microphones, and network conditions your product is expected to handle.

Then measure the stages separately:

• Time from speech capture to the first transcript update.

• Time until the relevant transcript segment is finalized.

• Recognition accuracy on application-specific vocabulary.

• Frequency of incorrect or duplicated confirmed text.

• Behavior during microphone denial, connection loss, and unsupported formats.

• Whether downstream actions use the intended final transcript.

Word error rate can help compare recognition outputs against reference transcripts, but generic benchmarks cannot tell you whether a particular order-search workflow is reliable.

You also need to distinguish recognition delay from UI update delay and backend processing time. Improving one component does not automatically fix the complete user experience.

The browser-native implementation may be enough if your supported environment is narrow and the feature is low-stakes. A dedicated streaming ASR service offers another integration path when you need more control over capture, processing, and provider selection.

Either way, the architecture should make failures observable rather than hiding them behind a single error message.

Return to the user speaking an order number. The goal is not merely to display text while the microphone is active. It is to preserve the correct identifier, present the result clearly, and let the user complete the search.

If you're evaluating a server-side approach, you can start building with the Smallest AI API and test Pulse with your own audio before integrating it into your web application's capture and transcript-handling workflow.