How to Benchmark Speech-to-Text APIs Without Fooling Yourself

The best speech-to-text API is the route that meets your workload's accuracy, latency, reliability, data, and cost requirements. A useful benchmark must use the same audio and scoring rules across providers and calculate cost per accepted transcript—not merely compare advertised price per minute.

This tutorial builds the local scoring layer for that benchmark. It deliberately avoids fake provider wrappers: each speech API has its own current authentication, upload, streaming, and asynchronous-job contract. Export every provider result into one CSV, then score them consistently.

What the benchmark measures

The evaluation tracks six production metrics:

1. Word error rate (WER) under one normalization policy.
2. p95 latency, because averages hide slow requests.
3. Failure rate, including empty output and transport errors.
4. Acceptance rate based on the product's actual criteria.
5. Human review time per audio hour.
6. Cost per accepted transcript, including review labor.

Public list prices are useful for shortlisting. They do not replace this benchmark. In the July 27, 2026 snapshot, recorded-audio routes range from AssemblyAI Universal-2 at $0.15/hour to Google standard recognition at $0.96/hour for the first 500,000 monthly minutes. Those routes differ in urgency, channel billing, features, and operational fit.

Environment

• Python 3.11 or later
• jiwer for word error rate
• CSV exports from your provider integrations

Install the dependency:

python -m pip install jiwer

Define one result schema

Create stt_results.csv with this header:

provider,clip_id,audio_seconds,channels,api_cost_usd,latency_ms,review_minutes,accepted,status,reference,hypothesis

Each row represents one provider attempt on one audio clip.

• api_cost_usd must include channel and feature charges for that request.
• latency_ms means upload-to-final for batch jobs or utterance-end-to-final for live jobs.
• accepted is true only when the transcript passes the product contract.
• status should distinguish ok, timeout, empty, rate_limited, and other failures.
• reference and hypothesis contain the approved transcript and provider output.

Do not put batch and live routes in the same report. Their latency and product behavior are not comparable.

Use synthetic rows only to validate the script

The following fixture is not a provider benchmark. Replace it with real results before drawing any conclusion.

provider,clip_id,audio_seconds,channels,api_cost_usd,latency_ms,review_minutes,accepted,status,reference,hypothesis
route_a,fixture_01,30,1,0.010,900,1.0,true,ok,"the quick brown fox","the quick brown fox"
route_b,fixture_01,30,1,0.020,600,0.5,true,ok,"the quick brown fox","the quick brown box"
route_a,fixture_02,45,2,0.025,1500,3.0,false,ok,"call account number five","call a count number five"
route_b,fixture_02,45,2,0.040,800,1.0,true,ok,"call account number five","call account number five"

Add the scoring script

Save this as score_stt.py:

from __future__ import annotations

import argparse
import csv
import math
from collections import defaultdict
from dataclasses import dataclass

from jiwer import wer


@dataclass
class Result:
    provider: str
    clip_id: str
    audio_seconds: float
    channels: int
    api_cost_usd: float
    latency_ms: float
    review_minutes: float
    accepted: bool
    status: str
    reference: str
    hypothesis: str


def parse_bool(value: str) -> bool:
    return value.strip().lower() in {"1", "true", "yes", "y"}


def load_results(path: str) -> list[Result]:
    rows: list[Result] = []
    with open(path, newline="", encoding="utf-8") as handle:
        for row in csv.DictReader(handle):
            rows.append(
                Result(
                    provider=row["provider"],
                    clip_id=row["clip_id"],
                    audio_seconds=float(row["audio_seconds"]),
                    channels=int(row["channels"]),
                    api_cost_usd=float(row["api_cost_usd"]),
                    latency_ms=float(row["latency_ms"]),
                    review_minutes=float(row["review_minutes"]),
                    accepted=parse_bool(row["accepted"]),
                    status=row["status"],
                    reference=row["reference"],
                    hypothesis=row["hypothesis"],
                )
            )
    return rows


def percentile(values: list[float], p: float) -> float:
    if not values:
        return math.nan
    ordered = sorted(values)
    index = (len(ordered) - 1) * p
    lower = math.floor(index)
    upper = math.ceil(index)
    if lower == upper:
        return ordered[lower]
    fraction = index - lower
    return ordered[lower] + (ordered[upper] - ordered[lower]) * fraction


def summarize(rows: list[Result], reviewer_hourly_usd: float) -> None:
    grouped: dict[str, list[Result]] = defaultdict(list)
    for row in rows:
        grouped[row.provider].append(row)

    header = (
        "provider",
        "runs",
        "wer",
        "p95_ms",
        "fail_%",
        "accept_%",
        "review_min/audio_hr",
        "cost/accepted_usd",
    )
    print("\t".join(header))

    for provider, provider_rows in sorted(grouped.items()):
        completed = [row for row in provider_rows if row.status == "ok"]
        accepted = [row for row in provider_rows if row.accepted]
        failures = [row for row in provider_rows if row.status != "ok"]

        references = [row.reference for row in completed]
        hypotheses = [row.hypothesis for row in completed]
        provider_wer = wer(references, hypotheses) if completed else math.nan

        total_audio_hours = sum(row.audio_seconds for row in provider_rows) / 3600
        total_review_minutes = sum(row.review_minutes for row in provider_rows)
        review_per_audio_hour = (
            total_review_minutes / total_audio_hours if total_audio_hours else math.nan
        )

        api_cost = sum(row.api_cost_usd for row in provider_rows)
        review_cost = total_review_minutes / 60 * reviewer_hourly_usd
        cost_per_accepted = (
            (api_cost + review_cost) / len(accepted) if accepted else math.inf
        )

        values = (
            provider,
            str(len(provider_rows)),
            f"{provider_wer:.4f}",
            f"{percentile([row.latency_ms for row in completed], 0.95):.0f}",
            f"{100 * len(failures) / len(provider_rows):.1f}",
            f"{100 * len(accepted) / len(provider_rows):.1f}",
            f"{review_per_audio_hour:.2f}",
            f"{cost_per_accepted:.4f}",
        )
        print("\t".join(values))


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("csv_path")
    parser.add_argument("--reviewer-hourly-usd", type=float, default=30.0)
    args = parser.parse_args()

    summarize(load_results(args.csv_path), args.reviewer_hourly_usd)

Run it:

python score_stt.py stt_results.csv --reviewer-hourly-usd 30

The script prints tab-separated results that can be pasted into a spreadsheet. It does not hide failed requests or divide cost only by successful API responses; it uses the stricter accepted field.

Build a production-like audio set

A benchmark should resemble the traffic the product will receive. Start with 60 to 80 licensed clips:

• 20 clean meetings with names and domain terms
• 20 noisy calls with interruptions, crosstalk, and hold music
• 20 accented or multilingual clips with real code switching
• 10 long-form files
• 10 short live utterances with silence, false starts, and barge-in

Use one reference-transcript and normalization policy for every route. Otherwise WER becomes a comparison of annotation styles rather than providers.

The 2026 GigaSpeechBench paper is a useful reminder: its 680-hour real-world evaluation reports substantial degradation for leading models and commercial APIs in difficult languages, dialects, accents, domains, and age groups. Your own traffic can be just as different from a clean public dataset.

Track more than WER

WER is necessary but incomplete.

Metric	Why it matters
Named-term accuracy	Product names and numbers can create disproportionate review work
Speaker attribution	Required for interviews, compliance, and contact-center analytics
Formatting quality	Punctuation and number formatting affect downstream usability
p95 finalization	Captures the delays users actually feel
Failure and retry rate	Converts reliability problems into cost
Review minutes/audio hour	Connects transcript quality with labor
Cost per accepted transcript	Combines processing, features, retries, and correction

For live speech, measure first partial, stable partial, and finalization separately. Time to first text can look good while the stable result remains slow.

Normalize the integration conditions

Before running the comparison, align:

• audio codec and sample rate
• selected region
• mono, stereo, or multichannel handling
• vocabulary hints and keyterms
• diarization and redaction flags
• timeouts and retries
• model versions
• timestamp and formatting options

Log every feature flag with each result. Deepgram, AssemblyAI, and ElevenLabs publish separate prices for some add-ons, while Google and AssemblyAI channel billing can multiply the base duration.

Use list prices only for the shortlist

The source research normalized public prices to one audio hour as of July 27, 2026:

Provider route	Public price snapshot	Important condition
AssemblyAI Universal-2	$0.15/hr	Multichannel billed per channel
OpenAI mini transcribe	$0.18/hr	25 MB uploaded-file limit
Google Dynamic Batch	$0.18/hr	Lower processing urgency
ElevenLabs Scribe v2	$0.22/hr	Entity and keyterm features cost extra
Deepgram Nova-3 pre-recorded	$0.462/hr	Diarization/redaction priced separately
Google standard recognition	$0.96/hr	First 500,000 monthly minutes; per-channel billing

These rows are not an accuracy ranking and do not represent equal service levels. Verify the official pricing pages immediately before using them in a procurement decision.

Turn the benchmark into a routing policy

Do not force one winner across unrelated workloads.

A practical production policy might use:

• one low-cost asynchronous route for archives
• one streaming specialist for live English calls
• one multilingual route for meetings and media
• one cloud-native route when data residency and procurement dominate

Teams evaluating supported OpenAI-compatible speech models can keep part of the client setup consistent with the CometAPI Quickstart. Verify the current model ID, input limits, feature support, and data terms before adding any route to the benchmark.