ASR-induced failure rate: did the speech path break it?

Part 1 ended in the failure-review meeting: the ASR team says the transcripts were fine and the NLU misread them, the NLU team says the transcripts were garbage, QA says the test oracle is stale. Everyone is arguing from the same green dashboard, and nobody has an experiment.

This post is the experiment.

Run everything twice

For every semantic test case in your voice suite, execute it twice through the same frozen downstream:

• Text control: reference text straight into the NLU/agent. No audio, no ASR.
• Audio path: audio through the ASR, then into that same NLU/agent.

Then read the four outcomes — the attribution table from part 1:

Text control	Audio path	Attribution
Pass	Pass	HEALTHY
Pass	Fail	SPEECH_ATTRIBUTABLE
Fail	Fail	DOWNSTREAM_OR_TEST
Fail	Pass	CONTEXT_OR_ORACLE

This is a controlled experiment, not a metric. The text control holds everything downstream constant; the only thing that changes between the two runs is the speech path. When the two runs disagree, the speech path is the difference, and the argument about whose fault it is ends — the table already routed the ticket.

The per-trial verdict rolls up to a suite-level number: ASR-IFR, the ASR-induced failure rate — the fraction of the full paired suite where the text control passes but the audio path fails. In the implementation, each trial contributes 1 if its attribution is SPEECH_ATTRIBUTABLE and 0 otherwise; ASR-IFR is the mean over all trials. The numerator is the speech-attributable count. The denominator is the whole suite.

The denominator choice is the part worth thinking about, because it's doing more work than the numerator.

Why the denominator matters more than the numerator

The numerator is just a count — 142 trials, in the pilot below. Anyone can count failures. The denominator decides what the number means.

ASR-IFR divides by all trials, not by control-passing trials. The alternative — "of the tests that worked, what fraction did speech break" — is also a meaningful number (0.394 / 0.800 ≈ 0.49 in the pilot). But the unconditional rate answers the question operations actually asks: what fraction of my suite is broken by the speech path right now? It stays comparable across runs even when downstream health changes — if the NLU regresses and text-control accuracy drops, a conditional rate would quietly rebase itself and hide the change; the unconditional rate doesn't move unless the speech path moved. That's what makes it gateable: --max-asr-ifr-delta 0.02 against your last approved baseline means something, because the denominator can't shift under you.

And there is a ceiling built into the definition that keeps the metric honest. ASR-IFR can never exceed text-control accuracy — a trial only counts if the control passed. In the pilot, text control is 0.800, so the worst ASR-IFR can ever be is 0.800. When ASR-IFR approaches that ceiling, the reading is unambiguous: nearly every test the downstream could pass is being broken by speech. The metric degrades gracefully instead of exploding — compare that with WER, which cheerfully reports 1.229 when insertions pile up.

The control is the keystone

None of this works without the text control, and the control has to be boring to be useful. In the pilot's reproduced results, text-control accuracy is 0.800 in every condition — clean, 15 dB noise, 5 dB noise, fast speech, telephony, telephony-plus-noise. The downstream never changed, so every bit of movement in ASR-IFR is the speech path. That flat 0.800 is the entire epistemic foundation: without it, you cannot attribute anything, and you're back to arguing in the meeting.

The control also does triage the metric can't. If text-control accuracy drops between releases, that's a downstream or test-oracle problem — the NLU, the agent, or a stale test — and it goes to those owners, not the ASR team. ASR-IFR moving while text control holds flat is a recognizer problem. WER moving while both hold flat is lexical churn: investigate, then ship. Three numbers, three owners, no meeting.

What the numbers look like across conditions

The paired design is implemented in CAFA-IVR (the worked example for this series), and the 360-trial Conformer-CTC pilot ships per-condition reproductions. Sixty trials each:

Condition	WER	ASR-IFR	Text control
Clean	0.115	0.10	0.800
Telephony (8 kHz)	0.272	0.20	0.800
Fast speech (1.15×)	0.412	0.317	0.800
15 dB noise	0.741	0.367	0.800
Telephony + 15 dB noise	1.121	0.667	0.800
5 dB noise	1.229	0.717	0.800
Overall	0.648	0.394	0.800

Read the 5 dB row the way an operator would. WER 1.229 is nearly uninterpretable — worse than 100%, because insertions pile up past the reference length. ASR-IFR 0.717 says something plain: of the 80% of tests the downstream could pass, speech broke almost 90% of them (0.717 / 0.800). Two out of every five tests in the suite, broken by the speech path. One of those two numbers helps you decide whether to ship. It isn't the first one.

And notice what the table separates that WER alone cannot. In the telephony-plus-noise condition, WER 1.121 and ASR-IFR 0.667 move together; in fast speech, WER 0.412 looks moderate while ASR-IFR 0.317 says nearly a third of the suite broke. The relationship between transcription damage and task damage isn't fixed — it depends on which words the recognizer corrupts, which is exactly what part 2's CEER is for. ASR-IFR tells you the speech path broke it; CEER tells you what it broke. You want both numbers, in that order.

Overall: 360 trials, 142 speech-attributable failures, ASR-IFR 0.394. Nearly two in five working tests, broken by speech. That's the release-gate number:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --max-asr-ifr-delta 0.02

A candidate that moves ASR-IFR more than two points against baseline doesn't ship until someone explains the movement. WER can move however it likes in the meantime — it's diagnostic, not decisive.

The honest boundary

Stated plainly, as in parts 1 and 2: this is a 360-trial pilot on controlled synthetic speech with a constrained in-domain Conformer-CTC recognizer. It demonstrates the method — the paired design, the attribution table, the gate. It is not a production-ASR benchmark and should not be read as one. Your numbers will differ; the experimental design won't.

The operational takeaway

The question to take back to your team is the one from part 1: for every failing voice test you have right now, can you say whether the speech path caused it? If you run the paired design, the answer comes with a number attached — and the number routes itself. ASR-IFR up, text control flat: fix the recognizer. Text control down: fix the NLU or the test. WER up, both flat: ship.

Three numbers, three owners, no meeting. That's the whole pitch.

Next in the series: CIER — weighting failures by realized consequence, or why "the task failed" is still not the end of the story.