A regex matched my LLM pipeline on clean text. Then synthetic speech broke both.

An operator on a plant radio says "Close valve two-oh-three, line Bravo." The field technician answers "Copy, closing two-oh-eight, line Bravo." Valve 208 is on a different line. The read-back exists so somebody catches that, and sometimes nobody on the channel does.

I spent the AMD Developer Hackathon (ACT III, lablab.ai) building SayAgain, a checker for exactly that moment. This post is not the launch tour. It is the part I didn't expect: on clean text a plain regex did as well as the language model, and when I fed the pipeline synthetic speech, both of them got worse in a way that a headline number hides.

Live replay: https://sayagain.edycu.dev/replay/ · Repo (MIT): https://github.com/edycutjong/sayagain · Demo video: https://youtu.be/6a4IF-iJmxg

The design: the model points, code decides

The obvious build is "ask an LLM whether the read-back matches the order." I didn't want a model making a safety verdict, so the model's only job is to point at words: which span of the transcript is the action, which is the equipment tag, which is the line. It runs on the AMD Token Factory (an OpenAI-compatible endpoint, JSON mode), and every call writes a receipt.

Then plain Python takes over. The spans must be words really present in the transcript (grounding). Spoken digits and NATO letters become plant tags through alias tables. The plant registry decides whether a tag exists and which line it is on. And a pure function returns the verdict:

def verdict(
    instr: list[CanonCommand],
    readback: list[CanonCommand] | None,
    *,
    span_conf: float | None = None,
    floor: float | None = None,
    disagreement: bool = False,
) -> Verdict:
    """VERIFIED · MISMATCH · NO_READBACK · ABSTAIN for one paired exchange (canonical inputs only)."""
    bad = validate_instruction(instr)
    if bad:
        return Verdict(ABSTAIN, bad, trace=[f"instruction: {bad}"])
    trace = ["instruction: " + ", ".join(f"{c.tag} line {c.line or c.registry_line}" for c in instr) + " (registry ok)"]
    if readback is None:
        return Verdict(NO_READBACK, trace=[*trace, "pairing: no read-back inside the window"])
    if any(r.tag_status == "ambiguous" for r in readback):
        return Verdict(ABSTAIN, R_AMBIGUOUS, trace=[*trace, "read-back: bare number shared by several tags"])
    if disagreement:
        return Verdict(ABSTAIN, R_AUDIO, trace=[*trace, "audio: prompted and unprompted transcripts disagree"])
    if floor is not None and span_conf is not None and span_conf < floor:
        return Verdict(ABSTAIN, R_AUDIO, trace=[*trace, f"audio: span confidence {span_conf:.3f} < floor {floor:.3f}"])
    diffs = compare(instr, readback)
    if diffs:
        return Verdict(MISMATCH, diffs=diffs, trace=[*trace, "compare: " + "; ".join(str(d) for d in diffs)])
    return Verdict(VERIFIED, trace=[*trace, "compare: every field equal"])

(referee/verdict.py — the module imports no model code.)

Notice how many exits lead to ABSTAIN. If the system can't read something, it says so and hands the card to a human. A failed model call is NOT_AUDITED, never a guess. Every card a supervisor sees ends with Confirm or Dismiss.

The rule I froze before the first model call

Before calling any model, I committed an evaluation protocol and tagged it (prereg-v1). The ladder rule: walk up from the cheapest extractor (a regex, no model) through larger models, and pick the smallest rung that lets zero wrong read-backs through as VERIFIED while keeping false alarms within a budget a shift can absorb.

That rule was written for the audio row: real recordings, real radio. I didn't have recordings during the hackathon, so I applied it to the recording cards' written scripts and logged that as a deviation. Everything below says which condition it is.

Result 1: on clean words, the regex was enough

On 89 held-out scripted exchanges (41 with a seeded error, 48 correct):

• No rung passed a wrong read-back as VERIFIED. 0 for the regex and 0 for all four models on the AMD Token Factory.
• The plain regex met the rule outright: 41/41 seeded errors flagged, 0/48 false alarms.
• Gemma-4-31B-IT was the only model that met it: 0 passed, 2/48 false alarms — after one re-audit of 118 transcripts whose first call failed at the gateway (108 HTTP 500s, 10 time-outs).
• An LLM-as-judge baseline (given the whole exchange, no registry) passed 2 wrong read-backs as VERIFIED, both missing the action word, and also cleared an order for a valve the plant doesn't have.

The honest reading: by my own pre-registered rule, on scripted words no language model is needed. I put that in the README in those words. It's the kind of sentence a project write-up usually leaves out, and it is the most useful thing the benchmark told me: clean text is not where the model earns its place.

One result did favour the design. When the same model returned normalised values instead of copying words from the transcript, false alarms went from 2/48 to 6/48; in 4 of those it turned a spoken pump or motor-valve tag into a valve tag the registry doesn't have. Making the model copy words, and making code check them, matters.

Result 2: synthetic speech broke both

To exercise the speech half, I ran an exploratory pass: ElevenLabs text-to-speech of the 145 card scripts (two voices), through a simulated radio path, transcribed by local Whisper-large-v3, then the same frozen referee. This was added after the text rows, it is not pre-registered, and it is not human speech. Same 89 held-out cards.

• Still 0 wrong read-backs passed as VERIFIED in every row that ran.
• Radio, no added noise: Whisper's word error rate was 0.87%. The regex flagged 39/41 errors, but false alarms went from 0/48 to 16/48; Gemma went from 2/48 to 17/48. Neither meets the false-alarm budget anymore.
• Errors that vanish: at no added noise, 2 of 41 seeded errors raised no card at all. Whisper wrote "for three Bravo" and "tutu, hotel" for the tag in the instruction, so no order was heard and the wrong read-back had nothing to be compared against. At −6 dB machine noise, 26 of 41 seeded errors raised no card.

That third bullet is the finding. "0 wrong read-backs passed" stayed true the whole time, and it would make a clean headline. But an error that produces no card is not caught either; it just never reaches a human. The stricter form of my own rule counts those as failures, and the README now says so.

Where the false alarms came from is also worth knowing, because none of it is the model being clever or dumb:

• Whisper writes spoken digit runs as "two, four, three", and the referee abstains on an unknown tag.
• I run Whisper twice (with and without a vocabulary prompt) and abstain when they disagree; the unprompted pass mishears plant words, so the gate fires on correct read-backs.
• On short or noisy turns Whisper invents text.

All of these fail toward a human, which is the right direction. They still cost the supervisor's attention, which is the budget the rule protects.

What I'd tell myself at the start

1. Write the selection rule down before the first model call, and keep the boring baseline in it. Without the regex rung I would have reported "Gemma meets the rule" and stopped.
2. "Zero false accepts" is necessary, not sufficient. Track the errors that never form a decision at all. A pipeline can be perfectly safe on what it judges and still miss what it never saw.
3. Clean-text evals flatter speech systems. The 0/48 → 16/48 jump came entirely from transcription, before any model judged anything.
4. Receipts make retries honest. The gateway failed most first-pass calls in one run (265 of 312), so re-audits were routine. Because every call is receipted and every result re-derives offline from stored outputs (scripts/eval.py --rescore), a re-audit is visible, not quietly merged.

Limits, plainly

• No human recording has been evaluated. Every number here is written scripts or synthetic voices on a fictional plant registry.
• The misheard-instruction failure needs a referee change, which means a new pre-registration, not a quiet patch.
• The live link is a static replay of stored runs; the live queue (FastAPI + server-sent events) runs locally.

The next step is the row the rule was written for: real people, real radios, real plant noise.

Try the refused one: https://sayagain.edycu.dev/replay/ · Code: https://github.com/edycutjong/sayagain · Video: https://youtu.be/6a4IF-iJmxg

If you work with plant or field radio, I'd value one sentence on whether operators actually read back tag numbers and NATO letters like this.