DEV Community: Vijay Kumar Sridharan

The fail-closed gate: a release-gate design story

Vijay Kumar Sridharan — Mon, 19 Oct 2026 15:04:06 +0000

The fail-closed gate: a release-gate design story

Draft — for review. Part 5 of the CAFA-IVR series.
Author: Vijay Kumar Sridharan, IEEE Senior Member

Parts 1 through 4 built the measurement pipeline: WER lies, CEER sees entities, ASR-IFR attributes blame, CIER weighs consequence. This post is about the last mile — the thing that turns those numbers into a shipping decision. And about a one-line design principle that changed between v1.0 and v1.1.0, because the first version got it wrong.

The principle: a quality gate that cannot evaluate its metric must fail, not pass.

The most expensive gate outcome

Every release gate has four possible outcomes, and only three of them are honest:

Red, correctly — the candidate regressed; someone investigates.
Green, correctly — nothing regressed; the release ships.
Red, incorrectly — a false alarm; someone investigates, finds nothing, ships late and annoyed.
Green, incorrectly — the gate couldn't check and passed anyway.

Outcome 4 is the most expensive by an order of magnitude, and it's the one nobody dashboards. A false red costs an investigation. A false green costs the entire premise of having a gate: every release that passed through it is now of unknown quality, and you don't know which ones. The gate didn't just fail to catch a regression — it certified ignorance as approval. When you find out, months later, the question isn't "why did this regress" but "how many releases shipped under a green light that meant nothing."

In v1.0 of this framework's compare command, outcome 4 was reachable. If a summary CSV arrived with a missing, non-numeric, or non-finite metric — a corrupted export, a column rename upstream, a nan from a divide-by-zero in someone's pipeline — the gate couldn't evaluate and the pipeline moved on. The v1.1.0 changelog states the change plainly: "Baseline/candidate comparison now fails closed for missing, nonnumeric, nonfinite, or empty enabled metrics. Pipelines that previously passed malformed input now exit with status 2."

"Pipelines that previously passed malformed input." Read that twice. That's the confession, and it's the reason this post exists instead of a footnote.

What fail-closed means, concretely

In the v1.1.0 implementation, every enabled gate demands a present, numeric, finite metric in both summaries — baseline and candidate. The check is mechanical, in _metric_value:

metric missing or empty → fail
metric not parseable as a number → fail
metric parses to infinity or NaN → fail
summary file unreadable or empty → fail

Any of these prints CAFA-IVR release gate: FAIL with the specific offending input named, and exits with status 2 — deliberately distinct from status 1, so CI systems and wrapper scripts can distinguish "the gate rejected this" from "the test harness itself broke." A gate that fails ambiguously is a gate people learn to ignore; the exit code is part of the design.

Here's the part worth sitting with: the framework's own shipped pilot dataset cannot pass its own CEER gate. The 360-trial reproduction carries no critical-entity annotations — its summary reports ceer: null, honestly, as parts 2 and 4 documented. Run the comparison without thinking:

cafa-ivr compare \
  --baseline empirical/cafa_reproduced_summary_360.csv \
  --candidate empirical/cafa_reproduced_summary_360.csv

CAFA-IVR release gate: FAIL
 - invalid summary input: empirical/cafa_reproduced_summary_360.csv: required metric 'ceer' is missing

Exit 2. The gate looked at the project's own reference data and refused it. That is not a bug — that is the entire point. A gate that waves through its own unevaluable input would wave through yours too.

Explicit opt-out over silent skip

The alternative to failing would have been to skip the CEER check when the metric is absent — "no entity annotations, so nothing to compare." That's the fail-open design, and it's seductive because it never annoys anyone. It's also how outcome 4 happens at scale: six months later, someone removes the annotation step from the pipeline to save time, the gate silently stops checking entities, and every release after that is green on a check that no longer exists. Nobody notices, because the gate never complained.

So v1.1.0 requires the opt-out to be explicit:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --no-ceer-gate

--no-ceer-gate says, in the pipeline definition where everyone can see it: this dataset has no entity annotations, and we accept that the entity check does not run. The difference between a skipped check and an opted-out check is a line in a config file — which is exactly the difference between "we decided" and "it happened." Decisions are auditable. Happenings are not.

This is the generalizable half of the design: every silent skip in a quality gate should require an explicit, visible, version-controlled opt-out. If the skip is legitimate, writing it down costs nothing. If the skip is covering a regression in your own process, the act of writing it down is the mechanism that catches it — usually at code-review time, when someone asks "why are we disabling the entity gate on this pipeline?"

The WER gate that defaults to off

One more detail from the implementation, because it ties the whole series together. The compare command accepts --max-wer-delta — a WER regression gate exists. Its default is None: disabled. The framework that spent part 1 arguing WER is the wrong unit of analysis does not gate releases on WER unless you explicitly ask it to. The gated quantities are ASR-IFR and CEER deltas — did the speech path break more tests, did entity corruption increase — with WER available as an opt-in diagnostic for teams that want it.

That's not an oversight; it's the thesis compiled into a default. Defaults are the strongest statement a tool makes — most users never change them — so they should encode what the author actually believes. The author believes WER is diagnostic, not decisive. The default says so.

The honest boundary, again

The README states it, and it bears repeating because defaults invite cargo-culting: "The default numeric gates are examples for tooling demonstration, not universal acceptance criteria. Production teams should govern their own thresholds by risk tier and historical baseline."

--max-asr-ifr-delta 0.02 and --max-ceer-delta 0.01 are starting points. A payments IVR and a music IVR should not share a gate any more than they should share a consequence scale (part 4). The framework's job is the mechanism — present, numeric, finite, fail-closed, explicit opt-outs. Your job is the policy: which deltas, at which risk tier, against which baseline, reviewed how often. A gate whose thresholds nobody owns is outcome 4 wearing a nicer suit.

The takeaway for anyone building gates

You don't need this framework to use the principle. Wherever you have a quality gate — test coverage, latency budgets, error-rate alerts, model evals — ask the outcome-4 question: if the metric this gate checks became unavailable tomorrow, would the gate go red or green? If the answer is green, you don't have a gate; you have a green light wired to nothing. Fix it the way v1.1.0 did: require the metric's presence as part of the check, fail loudly and specifically when it's absent, and make every skip an explicit, visible, reviewable decision.

A red gate you investigate is an annoyance. A green gate you trusted blindly is an incident report waiting for its date.

That closes the five-post arc: why WER lies (#1), the entity metric (#2), the attribution experiment (#3), consequence weighting (#4), and the gate that holds it all accountable (#5). The question to take back to your team is the one from this post: if your release gate's metric went missing tomorrow, would it go red — or green?

Consequence-weighted impact: not all failures cost the same

Vijay Kumar Sridharan — Thu, 15 Oct 2026 14:56:56 +0000

Consequence-weighted impact: not all failures cost the same

Draft — for review. Part 4 of the CAFA-IVR series.
Author: Vijay Kumar Sridharan, IEEE Senior Member

Parts 1 through 3 built a pipeline. ASR-IFR answers whose fault: did the speech path break an otherwise working test? CEER answers what broke: which governed entities got corrupted along the way? Both are necessary. Neither tells you what to fix first.

Your ASR error backlog has two hundred entries this quarter. Entry #47 is a misheard filler word on a balance inquiry. Entry #112 is a misheard amount on a funds transfer. Sorted by WER, they're neighbors. Sorted by what they cost the business, they live on different planets — and your backlog is sorted by WER.

The industry is starting to price this in, too: SquadStack's Arth V2, launched October 12, headlines a "semantic word error rate" — 8.85% on its own benchmark — that counts only meaning-changing errors, not every misheard word.

CIER — the conversational impact error rate — is the framework's answer to "how much does it matter." It weights each realized task failure by its consequence tier, so the transfer counts forty times more than the filler word. Not approximately. Arithmetically.

What it measures

Every trial in your suite gets a consequence tier, L0 through L4, assigned by your team:

| Tier | Meaning | Weight |
|---|---|
| L0 | Healthy | 0.00 |
| L1 | Benign lexical error | 0.00 |
| L2 | Semantic / entity error | 0.25 |
| L3 | Task / dialogue failure | 0.65 |
| L4 | Critical consequence | 1.00 |

CIER is then the severity-weighted failure rate over the impact-labelled trials: each trial contributes its tier weight if the audio-path task failed, zero if it succeeded, and trials with no recognized tier are excluded entirely — null, not zero, the same honesty rule CEER uses.

Three properties of that definition are doing the real work:

1. It weights the task outcome, not the words. CEER scores entity corruption; CIER scores what happened. A trial can have mangled entities and still contribute zero if the task succeeded, and a trial with a perfect transcript contributes its full tier weight if the task failed. The unit of analysis is the realized consequence.

2. It doesn't care whose fault it was. ASR-IFR counts only speech-attributable failures — the text control passed, the audio path failed. CIER counts realized audio-path task failures regardless of attribution, including trials where the text control failed too. The spec is explicit: if you want the ASR-attributable consequence picture, filter to SPEECH_ATTRIBUTABLE first, then summarize CIER. Unfiltered, CIER is the operations number: what did our callers actually experience, weighted by what it cost.

3. The weights are policy, not physics. The spec labels the L0–L4 weights "illustrative policy parameters" and says they must not be represented as universal industry thresholds. Your L4 might be 1.0 and your competitor's might be 10.0 — the framework doesn't care. What it insists on is that the tiers exist, that they're assigned by governance rather than guessed by the metric, and that unlabelled trials stay null instead of silently counting as zero-consequence.

That last point is the same split part 2 drew for CEER: the framework specifies the mechanism (the impact_level column, the weighting formula, the null rule); your team owns the judgment (which trials are L4, what the weights should be for your risk posture). "Critical" depends on what your system is allowed to do. A music IVR and a payments IVR should not share a consequence scale, and nothing in the framework asks them to.

Four vignettes, one scorer

Same method as part 2: hand-built illustrative inputs — not measurements — run through the actual reference scorer instead of hand arithmetic. Four trials, four consequence tiers:

Trial	What happened	WER	CEER	Tier	CIER contribution
V01	"Pay my bill on the fifth" → "on the ninth." Payment lands four days late.	0.167	2.0	L2	0.25
V02	"Transfer fifty dollars" → "fifteen dollars." Real money moves wrong.	0.20	2.0	L4	1.00
V03	"…balance please" → "…balance peas." Task succeeds anyway.	0.143	—	L1	0.00
V04	"Lock my card" transcribed perfectly; the NLU failed it anyway.	0.00	—	L3	0.65

Aggregate CIER: 0.475.

Read V01 against V02 first, because that's the whole post. Near-identical WER (0.167 vs 0.20). Identical CEER (2.0 — one date destroyed and replaced, one amount destroyed and replaced; the entity machinery can't tell them apart either). But CIER scores them 0.25 and 1.00 — a factor of four — because a late payment and a wrong transfer are not the same failure, and every practitioner knows it except the dashboard.

Then read V03: nonzero WER, zero CIER. The filler word that WER dutifully counted contributes nothing, because the task succeeded. And V04, the interesting one: WER is literally zero — a perfect transcript — yet CIER records 0.65, because the task failed at L3 consequence. The attribution column says DOWNSTREAM_OR_TEST, not SPEECH_ATTRIBUTABLE. CIER doesn't care. The caller still didn't get their card locked. This is what "realized consequence" means: it's the operations number, not the blame number.

How the three metrics compose

You now have the full sentence. For any trial in the suite:

ASR-IFR says whether the speech path broke it (attribution).
CEER says what governed information got corrupted (precision).
CIER says what it cost (consequence).

"Speech broke it, this entity, this cost" — three numbers, each answering a question the other two can't. Sort the quarter's ASR backlog by WER and you fix V03 first. Sort by CEER and you fix V01 and V02 first. Sort by CIER and you fix V02 first, then V04 (which isn't even an ASR problem — it's the highest-consequence item on the list and WER will never surface it). Your compliance team, your billing-disputes team, and your on-call rotation all agree with the third ordering.

One honest limitation of the current implementation: the v1.1.0 compare gate ships --max-asr-ifr-delta and --max-ceer-delta, but no CIER delta gate. CIER is computed and reported; it just isn't a gate yet. That's consistent with its status in the spec — "optional in v1.0" — and with how teams actually use it: ASR-IFR and CEER gate releases, CIER prioritizes the backlog between them. If your team wants a CIER gate, the weights and the formula are there; the flag is a small change.

The honest boundary

Stated plainly, as in parts 1 through 3: the 360-trial pilot shipped with the framework carries no impact labels, so its reproduced summary reports cier: null. CIER is specified in the spec (§6.4), implemented in the scorer, and covered by the unit tests — but it is not empirically exercised by that dataset. The vignettes above are illustrative inputs run through the real reference scorer, not measurements. The first real CIER numbers will come from consequence-labelled suites, which is a governance exercise for adopting teams, not something a framework can ship for you.

The operational takeaway

The question to take back to your team: of the ASR failures in your last release, which ones actually cost you something — and is your backlog sorted that way, or is it sorted by word error rate? If it's the latter, you're spending engineering quarters fixing V03s while V02s compound.

Label a sample of your suite L0–L4. It doesn't have to be the whole suite — CIER aggregates only labelled trials, and the null rule keeps the unlabelled ones honest. Then sort the backlog by consequence and watch the ordering change. That's the entire adoption path for this metric: one column in a CSV, and suddenly the dashboard speaks operations.

Next in the series: the v1.1.0 fail-closed gate design story — why every comparison gate in the framework fails closed on invalid input, and the engineering judgment behind it.

ASR-induced failure rate: did the speech path break it?

Vijay Kumar Sridharan — Tue, 13 Oct 2026 14:57:31 +0000

Part 1 ended in the failure-review meeting: the ASR team says the transcripts were fine and the NLU misread them, the NLU team says the transcripts were garbage, QA says the test oracle is stale. Everyone is arguing from the same green dashboard, and nobody has an experiment.

This post is the experiment.

Run everything twice

For every semantic test case in your voice suite, execute it twice through the same frozen downstream:

Text control: reference text straight into the NLU/agent. No audio, no ASR.
Audio path: audio through the ASR, then into that same NLU/agent.

Then read the four outcomes — the attribution table from part 1:

Text control	Audio path	Attribution
Pass	Pass	HEALTHY
Pass	Fail	SPEECH_ATTRIBUTABLE
Fail	Fail	DOWNSTREAM_OR_TEST
Fail	Pass	CONTEXT_OR_ORACLE

This is a controlled experiment, not a metric. The text control holds everything downstream constant; the only thing that changes between the two runs is the speech path. When the two runs disagree, the speech path is the difference, and the argument about whose fault it is ends — the table already routed the ticket.

The per-trial verdict rolls up to a suite-level number: ASR-IFR, the ASR-induced failure rate — the fraction of the full paired suite where the text control passes but the audio path fails. In the implementation, each trial contributes 1 if its attribution is SPEECH_ATTRIBUTABLE and 0 otherwise; ASR-IFR is the mean over all trials. The numerator is the speech-attributable count. The denominator is the whole suite.

The denominator choice is the part worth thinking about, because it's doing more work than the numerator.

Why the denominator matters more than the numerator

The numerator is just a count — 142 trials, in the pilot below. Anyone can count failures. The denominator decides what the number means.

ASR-IFR divides by all trials, not by control-passing trials. The alternative — "of the tests that worked, what fraction did speech break" — is also a meaningful number (0.394 / 0.800 ≈ 0.49 in the pilot). But the unconditional rate answers the question operations actually asks: what fraction of my suite is broken by the speech path right now? It stays comparable across runs even when downstream health changes — if the NLU regresses and text-control accuracy drops, a conditional rate would quietly rebase itself and hide the change; the unconditional rate doesn't move unless the speech path moved. That's what makes it gateable: --max-asr-ifr-delta 0.02 against your last approved baseline means something, because the denominator can't shift under you.

And there is a ceiling built into the definition that keeps the metric honest. ASR-IFR can never exceed text-control accuracy — a trial only counts if the control passed. In the pilot, text control is 0.800, so the worst ASR-IFR can ever be is 0.800. When ASR-IFR approaches that ceiling, the reading is unambiguous: nearly every test the downstream could pass is being broken by speech. The metric degrades gracefully instead of exploding — compare that with WER, which cheerfully reports 1.229 when insertions pile up.

The control is the keystone

None of this works without the text control, and the control has to be boring to be useful. In the pilot's reproduced results, text-control accuracy is 0.800 in every condition — clean, 15 dB noise, 5 dB noise, fast speech, telephony, telephony-plus-noise. The downstream never changed, so every bit of movement in ASR-IFR is the speech path. That flat 0.800 is the entire epistemic foundation: without it, you cannot attribute anything, and you're back to arguing in the meeting.

The control also does triage the metric can't. If text-control accuracy drops between releases, that's a downstream or test-oracle problem — the NLU, the agent, or a stale test — and it goes to those owners, not the ASR team. ASR-IFR moving while text control holds flat is a recognizer problem. WER moving while both hold flat is lexical churn: investigate, then ship. Three numbers, three owners, no meeting.

What the numbers look like across conditions

The paired design is implemented in CAFA-IVR (the worked example for this series), and the 360-trial Conformer-CTC pilot ships per-condition reproductions. Sixty trials each:

Condition	WER	ASR-IFR	Text control
Clean	0.115	0.10	0.800
Telephony (8 kHz)	0.272	0.20	0.800
Fast speech (1.15×)	0.412	0.317	0.800
15 dB noise	0.741	0.367	0.800
Telephony + 15 dB noise	1.121	0.667	0.800
5 dB noise	1.229	0.717	0.800
Overall	0.648	0.394	0.800

Read the 5 dB row the way an operator would. WER 1.229 is nearly uninterpretable — worse than 100%, because insertions pile up past the reference length. ASR-IFR 0.717 says something plain: of the 80% of tests the downstream could pass, speech broke almost 90% of them (0.717 / 0.800). Two out of every five tests in the suite, broken by the speech path. One of those two numbers helps you decide whether to ship. It isn't the first one.

And notice what the table separates that WER alone cannot. In the telephony-plus-noise condition, WER 1.121 and ASR-IFR 0.667 move together; in fast speech, WER 0.412 looks moderate while ASR-IFR 0.317 says nearly a third of the suite broke. The relationship between transcription damage and task damage isn't fixed — it depends on which words the recognizer corrupts, which is exactly what part 2's CEER is for. ASR-IFR tells you the speech path broke it; CEER tells you what it broke. You want both numbers, in that order.

Overall: 360 trials, 142 speech-attributable failures, ASR-IFR 0.394. Nearly two in five working tests, broken by speech. That's the release-gate number:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --max-asr-ifr-delta 0.02

A candidate that moves ASR-IFR more than two points against baseline doesn't ship until someone explains the movement. WER can move however it likes in the meantime — it's diagnostic, not decisive.

The honest boundary

Stated plainly, as in parts 1 and 2: this is a 360-trial pilot on controlled synthetic speech with a constrained in-domain Conformer-CTC recognizer. It demonstrates the method — the paired design, the attribution table, the gate. It is not a production-ASR benchmark and should not be read as one. Your numbers will differ; the experimental design won't.

The operational takeaway

The question to take back to your team is the one from part 1: for every failing voice test you have right now, can you say whether the speech path caused it? If you run the paired design, the answer comes with a number attached — and the number routes itself. ASR-IFR up, text control flat: fix the recognizer. Text control down: fix the NLU or the test. WER up, both flat: ship.

Three numbers, three owners, no meeting. That's the whole pitch.

Next in the series: CIER — weighting failures by realized consequence, or why "the task failed" is still not the end of the story.

Critical-entity error rate: the metric WER can't see

Vijay Kumar Sridharan — Tue, 06 Oct 2026 14:59:27 +0000

Part 2 of the CAFA-IVR series.

In part 1, WER scored "peas" and a ten-thousand-dollar mis-transfer as the same error. This post is about the machinery for telling them apart.

The errors that break IVR tasks don't distribute evenly across the vocabulary. They cluster in a tiny governed set: amounts, dates, account numbers, yes/no decisions, negation, cancel/confirm actions. A recognizer can mangle every filler word in a call and the task still completes; corrupt one governed token and the task completes wrong — which is worse than not completing at all.

CEER — critical-entity error rate — counts corruption in that governed set, and nothing else.

What it measures

For each trial, the framework compares two multisets: the critical entities the reference should have produced against the ones the speech path actually produced. Entities are (type, normalized value) pairs — ("amount", "fifty dollars") — and normalization runs before scoring, so 500 and five hundred count as the same entity rather than two different failures.

The count itself is simple: every expected entity that never arrived (missing) plus every observed entity that was never expected (spurious), divided by the number of expected entities.

Two properties worth noticing before we go further:

It's unbounded. A substituted amount counts twice — once for the amount you lost, once for the amount you invented. CEER can exceed 1.0, and does: the framework's own demo output reports CEER 2.0, which the docs call expected behavior for an unbounded error count. That's not a bug. It's the framework refusing to pretend that acting on the wrong amount is the same as acting on no amount.
It's null when there's nothing to govern. Trials with no annotated entities don't get a CEER of zero — they get no CEER at all. Zero would mean "no corruption," which is a claim. Null means "not measured," which is the truth.

Three vignettes

Hand-built illustrative inputs — not measurements. But I ran them through the actual reference scorer instead of doing the arithmetic by hand, because the arithmetic is the easy part to get wrong. (Vignette 3 is the proof.)

Vignette 1: the amount. Reference: "Transfer fifty dollars to savings." ASR: "Transfer fifteen dollars to savings." One substitution in five words — WER 0.20, the kind of number that sails through a regression gate without a second look. The entity inventory tells the other story: one expected amount, one observed amount, and they disagree. Missing one, spurious one: CEER 2.0. WER says one word in five was wrong. CEER says the entire governed payload of the utterance was destroyed and replaced with a fiction. Your billing-disputes team would recognize the second description.

Vignette 2: the date. "Pay my bill on the fifth" → "on the ninth." WER 0.17 — smaller still, safer-looking still. CEER 2.0 again. The payment lands four days late, the late fee posts, a human agent spends six minutes unwinding it. WER 0.17 will never explain that ticket. CEER 2.0 at least points at the right word.

Vignette 3: the negation — and the inversion. "Please do not cancel my card" → "Please cancel my card." Two words dropped: WER 0.33, the highest WER of the three. But the entity inventory here has two entries — negation and action — and only one was corrupted. CEER 0.5, the lowest of the three.

Read that again, because it's the whole argument: the trial with the worst WER has the best CEER, and the trials with the best WER have the worst CEER. Across all three, mean WER is 0.23 and CEER is 1.25 — the two numbers aren't even pointing in the same direction. CEER is not WER with the volume turned up. It's a different unit measuring a different thing: corruption of governed information, not corruption of words. Sort your ASR error backlog by WER and you fix vignette 3 first. Sort by CEER and you fix vignettes 1 and 2 first — and your compliance team agrees with the second ordering.

What's specified, what's yours

The framework specifies the mechanism: the entity columns (expected_entities_json, predicted_entities_json), the normalization, the missing-plus-spurious count, the null-not-zero rule. It deliberately does not specify the inventory. What counts as critical — which entity types, which values — is a governance decision your team makes, because "critical" depends on what your system is allowed to do with the words. A music IVR and a payments IVR should not share an entity inventory, and the framework doesn't ask them to.

That also means CEER is only as honest as the annotations. Unannotated trials don't silently pass; they report null, and the summary aggregates only annotated trials. If your CEER coverage is 12% of the suite, the number means "of the 12% we bothered to govern." The framework won't let you forget the other 88%.

How it composes with attribution

Part 1's four-state table answers whether the speech path broke the task. CEER answers what governed information it corrupted along the way. All three vignettes above came back SPEECH_ATTRIBUTABLE — text control passed, audio path failed. But the CEER column splits that single verdict into two different tickets: vignettes 1 and 2 are entity failures (the recognizer corrupted governed tokens — an acoustic/model problem for the ASR backlog), while a SPEECH_ATTRIBUTABLE trial with CEER 0 would be a lexical failure the downstream survived everywhere except the task oracle (a robustness problem for the NLU backlog). Same attribution, different owner, different fix. That's why you want both numbers.

The honest boundary

Stated plainly: the 360-trial pilot shipped with the framework carries no critical-entity annotations, so its reproduced summary reports ceer: null. CEER is specified, implemented, and covered by the test suite — but it is not empirically exercised by that dataset. The vignettes above are illustrative inputs run through the reference scorer, not measurements. The first real CEER numbers will come from annotated suites, which don't exist in the repo yet. (A worked CEER walkthrough is on the project's issue list.)

The operational takeaway

Gate releases on entity corruption, not word corruption. The compare gate already supports it:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --max-ceer-delta 0.01

A candidate that moves WER by a point but moves CEER not at all is a lexical reshuffle — ship it. A candidate that holds WER flat while CEER ticks up has learned to corrupt the words that matter while looking better on the dashboard. That one doesn't ship, and WER would never have told you.

The question to take back to your team: of the ASR errors in your last release, how many touched a governed entity — and would you even know? If the answer is no, you're gating on WER while your entities go unguarded.

Next in the series: the ASR-IFR deep-dive — from attribution verdicts to the release-gate metric, and why the denominator matters more than the numerator.

Why WER lies to you in production IVR

Vijay Kumar Sridharan — Sun, 04 Oct 2026 17:19:46 +0000

Part 1 of the CAFA-IVR series.

Your ASR dashboard is green and your callers are furious.

Word error rate dropped two points last quarter. The vendor deck says accuracy is up. Meanwhile containment is flat, agent handle time is up, and the failure-review meeting has the same argument it always has: the ASR team says the transcripts were fine and the NLU misread them, the NLU team says the transcripts were garbage, and QA says the test oracle is stale. Everyone is arguing from the same green dashboard.

The dashboard is measuring the wrong thing. WER is the wrong unit of analysis for production IVR — not because it's a bad metric, but because it answers a question nobody in operations is asking.

What WER actually measures

WER counts substitutions, deletions, and insertions, divided by the number of reference words. Every word weighs the same. It was designed for one job: transcription fidelity — how close the hypothesis is to what was said.

That is a reasonable thing to measure. It is not the thing that fails in production. In production, what fails is the task: the transfer went to the wrong account, the card got cancelled when the caller said don't, the payment landed on the wrong date. WER treats every one of those the same way it treats a misheard "um." Same numerator, same denominator, same shrug.

Three vignettes. All illustrative — simplified cases, not measurements — but everyone who has run a voice system has lived some version of each.

Vignette 1: the ten-cent error and the ten-thousand-dollar error. Two calls, eight words each, one substitution each.

Call A — reference: "I want to check my balance please." ASR: "I want to check my balance peas." WER: 12.5%. Intent (check_balance) resolves fine. Nothing happens. Nobody notices.

Call B — reference: "Transfer five hundred dollars to savings." ASR: "Transfer fifteen hundred dollars to savings." WER: 12.5%. Intent resolves fine too — the system confidently does the wrong thing, and real money moves to the wrong place.

Identical WER. One is a rounding error in a log file; the other is a billing dispute.

Vignette 2: the vanishing "don't." Reference: "Please don't cancel my card." ASR: "Please cancel my card." One deletion. WER: 20%. The system executes the exact opposite of the caller's request — in a regulated environment, against an explicit instruction the caller gave on a recorded line. WER scored this as a single missing word, barely worse than "peas." Your compliance team would score it very differently.

Vignette 3: the ninth instead of the fifth. Reference: "Pay my bill on the fifth." ASR: "Pay my bill on the ninth." One substitution. WER: ~17%. Late fee, angry caller, a human agent spending six minutes unwinding it. The word that was wrong happened to be the only word that mattered.

WER scored all three as "one word wrong." Operations scored them as nothing, a compliance incident, and a billing complaint. The metric and the consequences live in different universes, and the metric is the one on the dashboard.

The attribution question

This is where the failure-review meeting goes wrong. When an end-to-end voice test fails, the interesting question is almost never "how many words were wrong." It's this:

Did adding the speech path cause an otherwise valid test to fail?

That question has a clean experimental design, and most teams don't run it. For every semantic test case, execute it twice:

Text control: reference text straight into the NLU/agent. No audio, no ASR.
Audio path: audio through ASR, then into the same frozen NLU/agent.

Then read the four outcomes:

Text control	Audio path	Attribution
Pass	Pass	HEALTHY
Pass	Fail	SPEECH_ATTRIBUTABLE
Fail	Fail	DOWNSTREAM_OR_TEST
Fail	Pass	CONTEXT_OR_ORACLE

This ends the meeting argument. If the text control fails, the ASR team is excused — the failure is downstream or the test itself is stale, and it gets routed to the NLU/agent/test owners. If the text control passes and the audio path fails, the failure appeared when the speech path was introduced, and the ASR side owns the investigation. A SPEECH_ATTRIBUTABLE verdict narrows the failure surface; it doesn't prove a single acoustic root cause, but it tells you whose backlog the ticket belongs in.

The primary metric falls out of the table: ASR-IFR, the ASR-attributable intent failure rate — the fraction of the full paired suite where the text control passes but the audio path fails. It is a task metric, not a transcription metric. It counts the failures your callers actually experienced, attributed to the layer that introduced them.

What the numbers look like when you run it

The paired design is implemented in CAFA-IVR, an open reference framework (the worked example for this series), and it ships with a 360-trial scoring reproduction: a controlled Conformer-CTC pilot across clean, telephony, noise, and speaking-rate conditions. Two properties of the results are worth your attention.

First, the text-control accuracy is 0.800 in every condition — clean, 15 dB noise, 5 dB noise, fast speech, telephony, all of them. The downstream system never changed, so every bit of movement in the results is the speech path. That flat 0.800 is the whole point of the control: it holds the rest of the stack constant while the audio degrades.

Second, WER and task outcome stop moving together once conditions get hard:

Clean: WER 0.115, ASR-IFR 0.10. Fine. The metric and the task agree.
Telephony (8 kHz): WER 0.272, ASR-IFR 0.20. WER more than doubled; speech-attributable failures doubled with it.
Severe noise (5 dB SNR): WER 1.229 — worse than 100%, because insertions pile up — while ASR-IFR is 0.72. The WER number is nearly uninterpretable at that point. The ASR-IFR number says something plain: 72% of previously working tests now fail, and it's the speech path.
Overall: 360 trials, 142 speech-attributable failures, mean WER 0.648, ASR-IFR 0.394.

Read that last line again. Mean WER of 0.65 tells you the transcripts were rough. ASR-IFR of 0.394 tells you two out of five working tests broke because of the speech path. One of those numbers helps you decide whether to ship. It isn't the first one.

The honest boundaries, stated plainly: this is a 360-trial pilot on controlled synthetic speech with a constrained in-domain recognizer. It demonstrates the method; it is not a production-ASR benchmark and should not be read as one. The framework also defines two consequence-aware companions — CEER (critical-entity error rate: amounts, dates, negation, actions — the words from the vignettes above) and CIER (a severity-weighted rate over realized consequences) — which the pilot doesn't exercise and which get their own posts in this series.

The release-gate principle

None of this means WER is useless. It means WER has a job description, and release approval isn't in it. The policy pattern that falls out of the paired design:

WER regressions get investigated — always. It's still your best lexical diagnostic.
But WER alone never approves or blocks a release. ASR-IFR deltas against your last approved baseline do.
Text-control regressions go to the NLU/agent/test owners, not the ASR team. The table already did the triage.

The question to take back to your team is the attribution question: for every failing voice test you have right now, can you say whether the speech path caused it? If the answer is no, you're managing WER while your callers manage the consequences — and the dashboard will stay green all the way down.

Next in the series: CEER — counting the words that matter (amounts, dates, negation), and why entity-aware scoring changes which ASR errors you fix first.