Build a caption QA harness in Python: WER, missed entities, timing and reading rate

TL;DR

We're building a caption evaluation harness that scores a WebVTT file on four axes instead of one:
word error rate under a fixed normalizer, missed entity rate on domain terms, median cue timing
offset, and reading rate in characters per second. Python 3.12, jiwer, whisper_normalizer,
webvtt-py. Run it on every model or vendor change.

A caption file can score 96% accurate and still be unusable. WER counts substitutions, insertions and
deletions and weighs each one the same, so "fifteen milligrams" becoming "fifty milligrams" costs
exactly as much as "the" becoming "a". It also throws away every timestamp before it starts, which
means synchronization and readability are invisible to it. Let's measure the other three things.

0. Setup 🛠️

python3 -m venv .venv && source .venv/bin/activate
pip install jiwer whisper_normalizer webvtt-py

$ pip list | grep -Ei 'jiwer|whisper|webvtt'
jiwer               <your version>
webvtt-py           <your version>
whisper-normalizer  <your version>

Pin whatever you install, and pin it in CI. The APIs below move between majors, which is exactly why
the next tip exists.

💡 Tip: jiwer.compute_measures() is gone in recent versions. It is jiwer.process_words() now,
and it returns a WordOutput dataclass. Most blog posts you will find still use the old name.

1. Parse the VTT into text plus timings

# captions.py
from dataclasses import dataclass
import webvtt

@dataclass
class Cue:
    start: float
    end: float
    text: str

    @property
    def duration(self) -> float:
        return self.end - self.start

    @property
    def lines(self) -> list[str]:
        return self.text.split("\n")

    @property
    def flat(self) -> str:
        return " ".join(l.strip() for l in self.lines)

    @property
    def chars_per_second(self) -> float:
        return len(self.flat) / self.duration if self.duration > 0 else float("inf")

def _to_seconds(ts: str) -> float:
    h, m, s = ts.split(":")
    return int(h) * 3600 + int(m) * 60 + float(s)

def load_vtt(path: str) -> list[Cue]:
    # keep the line breaks: they are the thing the readability check measures
    return [
        Cue(_to_seconds(c.start), _to_seconds(c.end), c.text.strip())
        for c in webvtt.read(path)
    ]

def full_text(cues: list[Cue]) -> str:
    return " ".join(c.flat for c in cues)

2. Normalize both sides with the same normalizer

This is the step people skip, and skipping it is why vendor-published WER numbers are not comparable.
Without normalization, "3" against "three" and "Dr." against "doctor" count as errors even though the
recognizer heard perfectly. The normalizer from the Whisper paper (Appendix C) is the closest thing to
a default: it strips bracketed phrases, drops fillers like "um" and "hmm", and expands contractions.

# score_wer.py
import jiwer
from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()

def wer(reference: str, hypothesis: str) -> dict:
    ref, hyp = normalizer(reference), normalizer(hypothesis)
    out = jiwer.process_words(ref, hyp)
    return {
        "wer": out.wer,
        "substitutions": out.substitutions,
        "deletions": out.deletions,
        "insertions": out.insertions,
        "ref_words": sum(len(r) for r in out.references),
    }

$ python -c "from score_wer import wer; print(wer('the dose is fifteen milligrams','the dose is fifty milligrams'))"
{'wer': 0.2, 'substitutions': 1, 'deletions': 0, 'insertions': 0, 'ref_words': 5}

One substitution. Same score you would get for turning "the" into "a". That is the whole problem in
one line of output.

3. Missed entity rate: the metric that actually correlates with complaints

Score the words that carry meaning separately. Your entity list is domain specific and you have to
build it: product names, people, numbers, units, jargon. Two hundred terms is plenty to start.

# score_entities.py
import re
from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()
NUMBER_RE = re.compile(r"\b\d+(?:\.\d+)?\b")

def _terms(text: str, lexicon: list[str]) -> list[str]:
    norm = normalizer(text)
    found = []
    # findall, not search: a term said three times should count three times, the same
    # way numbers do. Iterate a sorted list so `examples` is stable across runs.
    for t in sorted(lexicon):
        found += re.findall(rf"\b{re.escape(t)}\b", norm)
    found += NUMBER_RE.findall(norm)
    return found

def missed_entity_rate(reference: str, hypothesis: str, lexicon: list[str]) -> dict:
    ref_terms = _terms(reference, lexicon)
    hyp_terms = _terms(hypothesis, lexicon)

    remaining = list(hyp_terms)
    missed = []
    for t in ref_terms:
        if t in remaining:
            remaining.remove(t)
        else:
            missed.append(t)

    total = len(ref_terms)
    return {
        "entity_total": total,
        "entity_missed": len(missed),
        "missed_entity_rate": len(missed) / total if total else 0.0,
        "examples": missed[:10],
    }

Numbers are pulled in automatically because they are almost always high-value and almost always the
thing that gets misheard.

4. Timing: median cue offset

WER cannot see this at all. A word-perfect transcript that appears 1.5 seconds late is a bad caption
file. Compare cue start times against a reference VTT by matching on normalized text.

# score_timing.py
import statistics
from whisper_normalizer.english import EnglishTextNormalizer
from captions import Cue

normalizer = EnglishTextNormalizer()

def _tokens(text: str) -> set[str]:
    return set(normalizer(text).split())

def cue_offsets(ref: list[Cue], hyp: list[Cue], window: float = 4.0,
                min_overlap: float = 0.6) -> dict:
    """Match cues by token overlap, not string equality.

    ASR almost never segments cues the same way a human transcriber does, so requiring
    identical normalized text would match nothing on real data. We take the best
    Jaccard overlap inside a time window instead.
    """
    offsets = []
    for r in ref:
        r_tok = _tokens(r.flat)
        if not r_tok:
            continue

        best, best_score = None, 0.0
        for h in hyp:
            if abs(h.start - r.start) > window:
                continue
            h_tok = _tokens(h.flat)
            if not h_tok:
                continue
            score = len(r_tok & h_tok) / len(r_tok | h_tok)
            if score > best_score:
                best, best_score = h, score

        if best is not None and best_score >= min_overlap:
            offsets.append(best.start - r.start)

    if not offsets:
        return {"matched_cues": 0, "median_offset_s": None, "p90_abs_offset_s": None}

    absolute = sorted(abs(o) for o in offsets)
    return {
        "matched_cues": len(offsets),
        "median_offset_s": round(statistics.median(offsets), 3),
        "p90_abs_offset_s": round(absolute[int(0.9 * (len(absolute) - 1))], 3),
    }

A consistently positive median means the whole file is late, which is usually a pipeline bug (an
offset applied at the wrong stage) rather than a model problem. Random scatter with a high p90 is a
model or alignment problem. Worth distinguishing before you go blame a vendor.

5. Readability: reading rate and cue duration

No reference needed for this one, which makes it the cheapest check you can run and the one you can
apply to your entire production library today.

# score_readability.py
from captions import Cue

# Thresholds are editorial choices, not standards. These are in the range major
# streaming style guides use for adult English subtitles; set yours deliberately.
MAX_CPS = 20.0          # characters per second
MIN_DURATION = 5 / 6    # ~0.833s, below this a cue flashes
MAX_LINE_CHARS = 42

def readability(cues: list[Cue]) -> dict:
    too_fast   = [c for c in cues if c.chars_per_second > MAX_CPS]
    too_short  = [c for c in cues if c.duration < MIN_DURATION]
    too_wide   = [c for c in cues if any(len(l) > MAX_LINE_CHARS for l in c.lines)]
    rates = sorted(c.chars_per_second for c in cues if c.duration > 0)

    return {
        "cues": len(cues),
        "median_cps": round(rates[len(rates) // 2], 1) if rates else None,
        "pct_over_cps_limit": round(100 * len(too_fast) / len(cues), 1) if cues else 0.0,
        "cues_under_min_duration": len(too_short),
        "cues_over_line_length": len(too_wide),
    }

A file that dumps forty words into one four second cue has a perfect WER and cannot be read. This
function catches it in milliseconds.

6. Tie it together 📊

# run_eval.py
import json, pathlib, sys
from captions import load_vtt, full_text
from score_wer import wer
from score_entities import missed_entity_rate
from score_timing import cue_offsets
from score_readability import readability

LEXICON = set(json.loads(pathlib.Path("lexicon.json").read_text()))

def evaluate(ref_vtt: str, hyp_vtt: str) -> dict:
    ref, hyp = load_vtt(ref_vtt), load_vtt(hyp_vtt)
    ref_text, hyp_text = full_text(ref), full_text(hyp)
    return {
        "file": pathlib.Path(hyp_vtt).name,
        **wer(ref_text, hyp_text),
        **missed_entity_rate(ref_text, hyp_text, LEXICON),
        **cue_offsets(ref, hyp),
        **readability(hyp),
    }

if __name__ == "__main__":
    results = [evaluate(r, h) for r, h in zip(sys.argv[1::2], sys.argv[2::2])]
    print(json.dumps(results, indent=2))

$ python run_eval.py refs/clip01.vtt out/clip01.vtt
[
  {
    "file": "clip01.vtt",
    "wer": 0.043,
    "substitutions": 18,
    "deletions": 5,
    "insertions": 3,
    "ref_words": 604,
    "entity_total": 41,
    "entity_missed": 6,
    "missed_entity_rate": 0.146,
    "examples": ["nifedipine", "2026", "okonkwo"],
    "matched_cues": 71,
    "median_offset_s": 0.42,
    "p90_abs_offset_s": 0.91,
    "cues": 112,
    "median_cps": 17.4,
    "pct_over_cps_limit": 9.8,
    "cues_under_min_duration": 4,
    "cues_over_line_length": 7
  }
]

⚠️ Note: matched_cues will always be lower than cues, because the matcher only accepts pairs
above the overlap threshold. If it drops near zero, your hypothesis segmentation has diverged
completely from the reference, which is itself a finding worth acting on.

Read that output the way it is meant to be read: 4.3% WER looks excellent, and 14.6% of the words
that actually matter are wrong. Those two numbers describe the same file.

7. The reference set is the real work

Thirty to fifty clips, chosen for spread rather than convenience: your noisiest recording, your
heaviest accent, your densest jargon, two people talking over each other, one that is mostly silence.
Human-transcribe them once. That is the entire capital cost of the program, and it amortizes over
every model change you will ever make.

Then run the harness in CI on every vendor or model change. The reason to automate is not that the
evaluation is hard, it is that nobody rechecks captions after the initial selection, and the model
behind a hosted API changes without an announcement.

What's next

• Add cpWER if you ship speaker labels. A transcript that gets every word right and swaps who said what scores zero errors on standard WER and is seriously wrong.
• Track the four metrics over time on the same reference set. The interesting signal is divergence: WER improving while entity rate worsens is exactly what you see when a model gets better at common words and starts producing plausible replacements for rare ones.
• Worth knowing before someone quotes it at you: there is no 99% accuracy rule in FCC Part 79. The Commission declined to set a numeric threshold and uses a de minimis test on accuracy, synchronicity, completeness and placement. The 99% figure comes from vendor contracts, not regulation. WCAG 2.2 SC 1.2.2 is likewise qualitative.

Four numbers, one reference set, and a CI job.