Consequence-weighted impact: not all failures cost the same

Draft — for review. Part 4 of the CAFA-IVR series.
Author: Vijay Kumar Sridharan, IEEE Senior Member

Parts 1 through 3 built a pipeline. ASR-IFR answers whose fault: did the speech path break an otherwise working test? CEER answers what broke: which governed entities got corrupted along the way? Both are necessary. Neither tells you what to fix first.

Your ASR error backlog has two hundred entries this quarter. Entry #47 is a misheard filler word on a balance inquiry. Entry #112 is a misheard amount on a funds transfer. Sorted by WER, they're neighbors. Sorted by what they cost the business, they live on different planets — and your backlog is sorted by WER.

The industry is starting to price this in, too: SquadStack's Arth V2, launched October 12, headlines a "semantic word error rate" — 8.85% on its own benchmark — that counts only meaning-changing errors, not every misheard word.

CIER — the conversational impact error rate — is the framework's answer to "how much does it matter." It weights each realized task failure by its consequence tier, so the transfer counts forty times more than the filler word. Not approximately. Arithmetically.

What it measures

Every trial in your suite gets a consequence tier, L0 through L4, assigned by your team:

| Tier | Meaning | Weight |
|---|---|
| L0 | Healthy | 0.00 |
| L1 | Benign lexical error | 0.00 |
| L2 | Semantic / entity error | 0.25 |
| L3 | Task / dialogue failure | 0.65 |
| L4 | Critical consequence | 1.00 |

CIER is then the severity-weighted failure rate over the impact-labelled trials: each trial contributes its tier weight if the audio-path task failed, zero if it succeeded, and trials with no recognized tier are excluded entirely — null, not zero, the same honesty rule CEER uses.

Three properties of that definition are doing the real work:

1. It weights the task outcome, not the words. CEER scores entity corruption; CIER scores what happened. A trial can have mangled entities and still contribute zero if the task succeeded, and a trial with a perfect transcript contributes its full tier weight if the task failed. The unit of analysis is the realized consequence.

2. It doesn't care whose fault it was. ASR-IFR counts only speech-attributable failures — the text control passed, the audio path failed. CIER counts realized audio-path task failures regardless of attribution, including trials where the text control failed too. The spec is explicit: if you want the ASR-attributable consequence picture, filter to SPEECH_ATTRIBUTABLE first, then summarize CIER. Unfiltered, CIER is the operations number: what did our callers actually experience, weighted by what it cost.

3. The weights are policy, not physics. The spec labels the L0–L4 weights "illustrative policy parameters" and says they must not be represented as universal industry thresholds. Your L4 might be 1.0 and your competitor's might be 10.0 — the framework doesn't care. What it insists on is that the tiers exist, that they're assigned by governance rather than guessed by the metric, and that unlabelled trials stay null instead of silently counting as zero-consequence.

That last point is the same split part 2 drew for CEER: the framework specifies the mechanism (the impact_level column, the weighting formula, the null rule); your team owns the judgment (which trials are L4, what the weights should be for your risk posture). "Critical" depends on what your system is allowed to do. A music IVR and a payments IVR should not share a consequence scale, and nothing in the framework asks them to.

Four vignettes, one scorer

Same method as part 2: hand-built illustrative inputs — not measurements — run through the actual reference scorer instead of hand arithmetic. Four trials, four consequence tiers:

Trial	What happened	WER	CEER	Tier	CIER contribution
V01	"Pay my bill on the fifth" → "on the ninth." Payment lands four days late.	0.167	2.0	L2	0.25
V02	"Transfer fifty dollars" → "fifteen dollars." Real money moves wrong.	0.20	2.0	L4	1.00
V03	"…balance please" → "…balance peas." Task succeeds anyway.	0.143	—	L1	0.00
V04	"Lock my card" transcribed perfectly; the NLU failed it anyway.	0.00	—	L3	0.65

Aggregate CIER: 0.475.

Read V01 against V02 first, because that's the whole post. Near-identical WER (0.167 vs 0.20). Identical CEER (2.0 — one date destroyed and replaced, one amount destroyed and replaced; the entity machinery can't tell them apart either). But CIER scores them 0.25 and 1.00 — a factor of four — because a late payment and a wrong transfer are not the same failure, and every practitioner knows it except the dashboard.

Then read V03: nonzero WER, zero CIER. The filler word that WER dutifully counted contributes nothing, because the task succeeded. And V04, the interesting one: WER is literally zero — a perfect transcript — yet CIER records 0.65, because the task failed at L3 consequence. The attribution column says DOWNSTREAM_OR_TEST, not SPEECH_ATTRIBUTABLE. CIER doesn't care. The caller still didn't get their card locked. This is what "realized consequence" means: it's the operations number, not the blame number.

How the three metrics compose

You now have the full sentence. For any trial in the suite:

• ASR-IFR says whether the speech path broke it (attribution).
• CEER says what governed information got corrupted (precision).
• CIER says what it cost (consequence).

"Speech broke it, this entity, this cost" — three numbers, each answering a question the other two can't. Sort the quarter's ASR backlog by WER and you fix V03 first. Sort by CEER and you fix V01 and V02 first. Sort by CIER and you fix V02 first, then V04 (which isn't even an ASR problem — it's the highest-consequence item on the list and WER will never surface it). Your compliance team, your billing-disputes team, and your on-call rotation all agree with the third ordering.

One honest limitation of the current implementation: the v1.1.0 compare gate ships --max-asr-ifr-delta and --max-ceer-delta, but no CIER delta gate. CIER is computed and reported; it just isn't a gate yet. That's consistent with its status in the spec — "optional in v1.0" — and with how teams actually use it: ASR-IFR and CEER gate releases, CIER prioritizes the backlog between them. If your team wants a CIER gate, the weights and the formula are there; the flag is a small change.

The honest boundary

Stated plainly, as in parts 1 through 3: the 360-trial pilot shipped with the framework carries no impact labels, so its reproduced summary reports cier: null. CIER is specified in the spec (§6.4), implemented in the scorer, and covered by the unit tests — but it is not empirically exercised by that dataset. The vignettes above are illustrative inputs run through the real reference scorer, not measurements. The first real CIER numbers will come from consequence-labelled suites, which is a governance exercise for adopting teams, not something a framework can ship for you.

The operational takeaway

The question to take back to your team: of the ASR failures in your last release, which ones actually cost you something — and is your backlog sorted that way, or is it sorted by word error rate? If it's the latter, you're spending engineering quarters fixing V03s while V02s compound.

Label a sample of your suite L0–L4. It doesn't have to be the whole suite — CIER aggregates only labelled trials, and the null rule keeps the unlabelled ones honest. Then sort the backlog by consequence and watch the ordering change. That's the entire adoption path for this metric: one column in a CSV, and suddenly the dashboard speaks operations.

Next in the series: the v1.1.0 fail-closed gate design story — why every comparison gate in the framework fails closed on invalid input, and the engineering judgment behind it.