Why Speech-to-Text Is More Than Calling an AI API

A speech-to-text demo can be wonderfully small:

const result = await provider.transcribe(audioUrl);
return result.text;

That code is enough to prove that a model can recognize speech.

It is nowhere near enough to prove that a user can trust the result.

Give the demo a two-hour interview instead of a 20-second clip. Let the upload lose its connection at 87%. Let the provider send the same webhook twice. Let two words overlap by 80 milliseconds, then let the next word arrive ten seconds out of order. Let the user correct a name while an AI summary is waiting in a queue.

At that point, speech-to-text stops being an API integration. It becomes a document system, a media pipeline, and a distributed-systems problem wearing headphones.

While building Echoryte, a workspace that turns recordings into editable, time-linked transcripts, we found a useful way to frame the problem:

A speech API returns a hypothesis. A transcription product must turn that hypothesis into a durable, navigable, and explainable document.

This article is not a comparison of speech models. Models and pricing change too quickly for that to age well. Instead, it is a map of the engineering boundaries that remain regardless of which provider you call.

The real pipeline

The naive architecture has two boxes:

recording -> speech API -> text

A useful system looks closer to this:

upload
  -> inspect the actual media
  -> prepare a stable audio derivative
  -> reserve work and enqueue a job
  -> select a compatible provider/model
  -> submit an attempt
  -> wait through webhook and polling paths
  -> validate and normalize the result
  -> publish a transcript revision
  -> edit, search, translate, summarize, and export

Every arrow is a place where state can be lost, repeated, corrupted, or made ambiguous.

The important shift is to treat each boundary as a contract rather than a convenient function call.

1. A file name is a claim, not evidence

A user uploads interview.mp3. The browser reports audio/mpeg. It is tempting to trust both.

Neither is authoritative.

Extensions can be wrong. MIME types are supplied by clients. Containers may hold several audio tracks, no audio track, damaged timestamps, or codecs your downstream provider cannot decode. Long recordings may also be variable-bitrate media whose reported duration is not what you expect.

Before transcription, inspect the bytes with a media probe such as ffprobe and answer concrete questions:

• What container is this really?
• Is there a usable audio stream?
• What is the measured duration?
• Which track should be transcribed?
• Can the media be decoded within your CPU, memory, disk, and time budgets?

In Echoryte's pipeline, ingestion produces a provider-friendly audio derivative and a browser-friendly playback derivative. Doing this once creates a stable input for both recognition and later review.

This also improves error messages. “The API failed” is not useful. “This file contains no audio stream” or “the measured duration exceeds your plan” tells the user what to do next.

A good rule is:

Validate business limits against inspected media, not against metadata declared by the client.

2. A transcription job is not a provider request

This distinction sounds academic until the first retry.

We model three separate things:

type TranscriptionJob = {
  id: string;
  fileId: string;
  requestedTier: "fast" | "standard" | "precision";
  languageHint?: string;
  diarization: boolean;
};

type TranscriptionAttempt = {
  id: string;
  jobId: string;
  provider: string;
  model: string;
  providerJobId?: string;
  status: "submitted" | "waiting" | "succeeded" | "failed" | "ignored";
};

type TranscriptPublication = {
  jobId: string;
  version: number;
  revision: number;
  objectKey: string;
};

The job represents user intent: “transcribe this recording with these capabilities.”

An attempt represents one execution of that intent against one provider and model.

A publication represents the result that users are allowed to see and edit.

Separating them gives you several useful properties:

• A transient provider failure can create another attempt without inventing another user job.
• A late callback can be attached to the correct historical attempt.
• Cancellation can ignore an old result instead of publishing it accidentally.
• Retranscription can keep the existing transcript readable until the replacement succeeds.
• Cost and latency can be measured per attempt rather than guessed per job.

The last point matters more than it appears. If a provider succeeds but your worker crashes before publication, the provider request happened, the cost happened, and the user-visible result did not. One status field cannot describe all three facts.

3. “Use provider B if provider A fails” is not a routing strategy

Speech providers do not have one-dimensional capability.

Support varies by model, language, automatic language detection, word timestamps, speaker diarization, latency, and sometimes region. A fallback that returns plain text when the product promises word-level navigation is not a fallback. It is a silent contract change.

A better router first removes incompatible candidates and only then ranks the survivors:

function isCompatible(model: ModelCapability, request: RequestFeatures) {
  const language = resolveLanguage(model, request.languageHint);

  return (
    model.tiers.includes(request.tier) &&
    language.supportsWordTimestamps &&
    (!request.diarize || language.supportsDiarization) &&
    (request.languageHint || language.supportsAutomaticDetection)
  );
}

const candidates = capabilitySnapshot.models
  .filter(model => isCompatible(model, request))
  .sort(rankByQualityLatencyAndCost);

Notice the capability snapshot. Provider documentation, model IDs, and language support change. Versioning the routing data lets you answer a difficult operational question later:

“Why did this job choose that model on that day?”

Failover should use the same compatibility filter and exclude capabilities already attempted. Otherwise, a retry loop can bounce between equivalent failures or quietly drop a requested feature.

4. The happy path is asynchronous; the failure paths are more asynchronous

Long recordings do not belong in an HTTP request-response cycle. They need durable background work.

That introduces at-least-once behavior almost everywhere:

• A queue can deliver the same job again.
• A worker can stop after an external side effect but before saving local state.
• A webhook can be delayed, duplicated, or arrive out of order.
• A polling request can race with a webhook.
• The user can cancel while the provider is still processing.

Our useful mental model is a state machine backed by the database:

queued -> claimed -> submitted -> waiting -> normalizing -> published
             |            |          |            |
             +------------+----------+------------+-> failed / ignored

The queue schedules work; it is not the source of truth.

When a worker claims a job, it receives a lease token and expiry. It renews the lease while doing slow work. Every later write checks the same token. If the lease is lost, that execution can no longer publish.

This prevents an old worker from waking up after a pause and overwriting the result produced by a newer worker.

External operations also need stable identities. Creating an attempt, attaching a provider job ID, storing a raw result, reserving a transcript version, and publishing should all be safe to replay.

For callback-based providers, the webhook handler should do very little:

1. Authenticate the callback.
2. Record or mark the event idempotently.
3. Wake the durable job.
4. Return quickly.

Polling remains useful as a recovery path when a callback never arrives. Webhooks reduce latency; polling closes the reliability gap.

One more subtle point: progress is a user-interface estimate, not provider truth. A bar that moves smoothly to 73% does not mean 73% of the words exist. Show stages and historical time ranges, and label estimates as estimates.

5. Provider JSON is untrusted input

Even a successful provider response is external data. Parse it at runtime.

A TypeScript interface cannot reject NaN, negative timestamps, missing fields, invalid confidence values, or a word whose start is after its end. A runtime schema can.

After validation, normalize the result into a provider-independent structure:

type Word = {
  text: string;
  startMs: number;
  endMs: number;
  confidence?: number;
};

type Segment = {
  id: string;
  speakerId: string | null;
  startMs: number;
  endMs: number;
  words: Word[];
};

type Transcript = {
  version: number;
  durationMs: number;
  language: string;
  speakers: Speaker[];
  segments: Segment[];
};

Normalization is not just renaming fields. It is where you define what “valid time” means.

For example, Echoryte's current normalizer tolerates a small, bounded word overlap by clipping the next start time. It rejects a large overlap or a word that reverses the timeline. It normalizes Unicode, removes control characters, and refuses words that become empty.

Segments are then built around editing and reading behavior, not around arbitrary provider paragraphs. A speaker change forces a boundary. So does a meaningful silence. Very long segments are split at safe word boundaries.

The principle is more important than the exact thresholds:

Repair only what you can repair without changing meaning. Reject ambiguity before it becomes durable data.

Silently sorting wildly disordered words may produce JSON that passes a schema, but it can make clicking a quote jump to the wrong moment. That is worse than an explicit failure.

Keeping the original provider result separately is also valuable. You can re-run normalization after improving your rules, investigate disputes, or compare provider behavior without calling the API again.

6. The product is a time-linked document, not a string

Plain text throws away the most valuable part of a transcript: its relationship to the recording.

Once every word has time, several product behaviors become possible:

• Click a word to seek the audio.
• Highlight words during playback.
• Find a quote and verify it in context.
• Export subtitles with real cue boundaries.
• Associate speaker corrections with stable segments.
• Generate summaries whose references jump back to the source.

But editing creates another problem. If you replace the transcript as one giant text blob, you lose stable identities and make concurrent changes difficult to reason about.

A patch model is often a better fit:

type TranscriptPatch =
  | { type: "replaceWordText"; segmentId: string; wordIndex: number; text: string }
  | { type: "setSegmentSpeaker"; segmentId: string; speakerId: string | null }
  | { type: "updateSpeakerLabel"; speakerId: string; label: string }
  | { type: "splitSegment"; segmentId: string; wordIndex: number };

Each accepted batch advances a revision. Periodically, patches can be compacted into a new snapshot.

Revision numbers solve a quiet but serious race:

1. The user requests a summary from revision 12.
2. The summary waits in a queue.
3. The user fixes several names, producing revision 15.
4. The summary finishes.

The result is not necessarily wrong, but it is based on an older source. Store sourceRevision: 12 with the output and mark it as stale relative to revision 15. Do not pretend it reflects edits it never saw.

The same rule applies to translations and exports: pin the input revision when the request is created.

7. AI post-processing should have less authority than it wants

After transcription, it is natural to ask an LLM to add punctuation, produce chapters, extract action items, or answer questions.

The dangerous shortcut is to let the model rewrite the canonical transcript freely.

For punctuation enhancement, we use a smaller contract: the model may propose structured operations such as “add punctuation after word 42” or “insert a paragraph break after word 108.” It cannot replace the recognized words. Every index and punctuation mark is validated; invalid output falls back to the original transcript.

For summaries and chat, the transcript is data, not trusted instructions. Delimit it clearly in the prompt. Validate time references against the recording duration. Sanitize rendered Markdown. Never allow text spoken inside an uploaded recording to redefine the system prompt.

Translation needs similar honesty. Segment timing can remain attached to the source segment, but translated words do not magically receive forced-alignment-quality timestamps. If word timing is interpolated, label it as estimated.

A useful hierarchy is:

recording evidence
  -> normalized transcript
      -> user edits
          -> derived AI outputs

Derived data should point back to its source. It should not quietly become the source.

8. Observability should explain failures without recording the recording

Transcription systems need rich diagnostics, but recordings often contain interviews, research, classes, customer calls, or other sensitive material.

You can observe the pipeline without logging content.

Useful fields include:

• Internal file, job, and attempt IDs
• Provider and model
• Capability snapshot version
• Media duration and coarse size bucket
• Queue wait and stage latency
• Retry, failover, and callback-recovery counts
• Stable error codes
• Billed units and estimated cost

Avoid logging filenames, transcript text, signed media URLs, webhook secrets, or user-supplied source URLs with query strings.

This forces better operational design. “Provider returned 429 for attempt A” is searchable and actionable. A dump of the user's entire response payload is neither necessary nor safe.

A practical build order

You do not need the full architecture for a weekend prototype. You do need to know which shortcuts you are taking.

A reasonable progression is:

1. Call one provider with a small known-good audio file.
2. Add runtime validation and a canonical transcript schema.
3. Move long work into a durable queue.
4. Separate user jobs from provider attempts.
5. Make every state transition replay-safe.
6. Inspect and normalize media before submission.
7. Add revisioned editing before adding AI features.
8. Pin exports, translations, and summaries to a source revision.
9. Add capability-based routing before multi-provider failover.
10. Add content-safe observability, cost accounting, deletion, and retention rules.

Before calling the system production-ready, ask:

• Can an interrupted upload resume safely?
• Can the same job run twice without publishing twice?
• Can a late callback overwrite newer work?
• Can every published word be mapped to a valid time range?
• Does failover preserve the features the user requested?
• Does retranscription preserve the last good transcript on failure?
• Can derived outputs reveal which revision they used?
• Can an operator debug a failure without reading user content?

If several answers are “no,” the speech model may still be excellent. The product is not finished.

The takeaway

The API call is the impressive part of the demo. The surrounding contracts are the valuable part of the product.

A trustworthy transcription system must preserve three kinds of truth at once:

1. Media truth: what was actually uploaded and when speech occurred.
2. Execution truth: which provider attempt ran, failed, retried, or arrived late.
3. Document truth: which transcript revision the user saw, edited, translated, summarized, or exported.

Keep those truths separate, connect them with stable identities, and make every transition safe to repeat.

The speech API recognizes words. The system around it earns the user's trust.

What failure mode surprised you the first time you built an AI pipeline?