The fail-closed gate: a release-gate design story

Draft — for review. Part 5 of the CAFA-IVR series.
Author: Vijay Kumar Sridharan, IEEE Senior Member

Parts 1 through 4 built the measurement pipeline: WER lies, CEER sees entities, ASR-IFR attributes blame, CIER weighs consequence. This post is about the last mile — the thing that turns those numbers into a shipping decision. And about a one-line design principle that changed between v1.0 and v1.1.0, because the first version got it wrong.

The principle: a quality gate that cannot evaluate its metric must fail, not pass.

The most expensive gate outcome

Every release gate has four possible outcomes, and only three of them are honest:

1. Red, correctly — the candidate regressed; someone investigates.
2. Green, correctly — nothing regressed; the release ships.
3. Red, incorrectly — a false alarm; someone investigates, finds nothing, ships late and annoyed.
4. Green, incorrectly — the gate couldn't check and passed anyway.

Outcome 4 is the most expensive by an order of magnitude, and it's the one nobody dashboards. A false red costs an investigation. A false green costs the entire premise of having a gate: every release that passed through it is now of unknown quality, and you don't know which ones. The gate didn't just fail to catch a regression — it certified ignorance as approval. When you find out, months later, the question isn't "why did this regress" but "how many releases shipped under a green light that meant nothing."

In v1.0 of this framework's compare command, outcome 4 was reachable. If a summary CSV arrived with a missing, non-numeric, or non-finite metric — a corrupted export, a column rename upstream, a nan from a divide-by-zero in someone's pipeline — the gate couldn't evaluate and the pipeline moved on. The v1.1.0 changelog states the change plainly: "Baseline/candidate comparison now fails closed for missing, nonnumeric, nonfinite, or empty enabled metrics. Pipelines that previously passed malformed input now exit with status 2."

"Pipelines that previously passed malformed input." Read that twice. That's the confession, and it's the reason this post exists instead of a footnote.

What fail-closed means, concretely

In the v1.1.0 implementation, every enabled gate demands a present, numeric, finite metric in both summaries — baseline and candidate. The check is mechanical, in _metric_value:

• metric missing or empty → fail
• metric not parseable as a number → fail
• metric parses to infinity or NaN → fail
• summary file unreadable or empty → fail

Any of these prints CAFA-IVR release gate: FAIL with the specific offending input named, and exits with status 2 — deliberately distinct from status 1, so CI systems and wrapper scripts can distinguish "the gate rejected this" from "the test harness itself broke." A gate that fails ambiguously is a gate people learn to ignore; the exit code is part of the design.

Here's the part worth sitting with: the framework's own shipped pilot dataset cannot pass its own CEER gate. The 360-trial reproduction carries no critical-entity annotations — its summary reports ceer: null, honestly, as parts 2 and 4 documented. Run the comparison without thinking:

cafa-ivr compare \
  --baseline empirical/cafa_reproduced_summary_360.csv \
  --candidate empirical/cafa_reproduced_summary_360.csv

CAFA-IVR release gate: FAIL
 - invalid summary input: empirical/cafa_reproduced_summary_360.csv: required metric 'ceer' is missing

Exit 2. The gate looked at the project's own reference data and refused it. That is not a bug — that is the entire point. A gate that waves through its own unevaluable input would wave through yours too.

Explicit opt-out over silent skip

The alternative to failing would have been to skip the CEER check when the metric is absent — "no entity annotations, so nothing to compare." That's the fail-open design, and it's seductive because it never annoys anyone. It's also how outcome 4 happens at scale: six months later, someone removes the annotation step from the pipeline to save time, the gate silently stops checking entities, and every release after that is green on a check that no longer exists. Nobody notices, because the gate never complained.

So v1.1.0 requires the opt-out to be explicit:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --no-ceer-gate

--no-ceer-gate says, in the pipeline definition where everyone can see it: this dataset has no entity annotations, and we accept that the entity check does not run. The difference between a skipped check and an opted-out check is a line in a config file — which is exactly the difference between "we decided" and "it happened." Decisions are auditable. Happenings are not.

This is the generalizable half of the design: every silent skip in a quality gate should require an explicit, visible, version-controlled opt-out. If the skip is legitimate, writing it down costs nothing. If the skip is covering a regression in your own process, the act of writing it down is the mechanism that catches it — usually at code-review time, when someone asks "why are we disabling the entity gate on this pipeline?"

The WER gate that defaults to off

One more detail from the implementation, because it ties the whole series together. The compare command accepts --max-wer-delta — a WER regression gate exists. Its default is None: disabled. The framework that spent part 1 arguing WER is the wrong unit of analysis does not gate releases on WER unless you explicitly ask it to. The gated quantities are ASR-IFR and CEER deltas — did the speech path break more tests, did entity corruption increase — with WER available as an opt-in diagnostic for teams that want it.

That's not an oversight; it's the thesis compiled into a default. Defaults are the strongest statement a tool makes — most users never change them — so they should encode what the author actually believes. The author believes WER is diagnostic, not decisive. The default says so.

The honest boundary, again

The README states it, and it bears repeating because defaults invite cargo-culting: "The default numeric gates are examples for tooling demonstration, not universal acceptance criteria. Production teams should govern their own thresholds by risk tier and historical baseline."

--max-asr-ifr-delta 0.02 and --max-ceer-delta 0.01 are starting points. A payments IVR and a music IVR should not share a gate any more than they should share a consequence scale (part 4). The framework's job is the mechanism — present, numeric, finite, fail-closed, explicit opt-outs. Your job is the policy: which deltas, at which risk tier, against which baseline, reviewed how often. A gate whose thresholds nobody owns is outcome 4 wearing a nicer suit.

The takeaway for anyone building gates

You don't need this framework to use the principle. Wherever you have a quality gate — test coverage, latency budgets, error-rate alerts, model evals — ask the outcome-4 question: if the metric this gate checks became unavailable tomorrow, would the gate go red or green? If the answer is green, you don't have a gate; you have a green light wired to nothing. Fix it the way v1.1.0 did: require the metric's presence as part of the check, fail loudly and specifically when it's absent, and make every skip an explicit, visible, reviewable decision.

A red gate you investigate is an annoyance. A green gate you trusted blindly is an incident report waiting for its date.

---

That closes the five-post arc: why WER lies (#1), the entity metric (#2), the attribution experiment (#3), consequence weighting (#4), and the gate that holds it all accountable (#5). The question to take back to your team is the one from this post: if your release gate's metric went missing tomorrow, would it go red — or green?

Comments

[Max Quimby]

Outcome 4 — "green, incorrectly, because the gate couldn't check" — is the one that should be framed and hung on a wall. The line "it certified ignorance as approval" is exactly it: a false red costs you one investigation, a false green costs you the entire premise of the gate, retroactively, for every release that passed under it. And you can't even dashboard it, because by definition the gate didn't know it was failing.

We learned the same lesson on a pipeline that screens generated media with mechanical checks before publish. Our early version treated "checker threw / returned null / empty input" as a pass-through, and the quiet failures were always the expensive ones — a renamed column, a nan from a divide-by-zero, an OCR step that silently returned empty on a malformed file. Flipping every one of those to exit-nonzero was the single highest-leverage change we made. The subtle trap on the other side: once you fail-closed, a flaky upstream can red-gate every release and train people to rubber-stamp the override, which quietly rebuilds outcome 4 with a human in the loop. How do you keep a legitimately-unevaluable input (transient, not a real regression) from eroding trust in the gate until the fail-closed exit becomes noise people click past?

[Dean Lee]

The framing of Outcome 4 as "certifying ignorance as approval" captures the core mechanism of model risk in automated pipelines.

When a gate fails open on missing or non-finite inputs, the engineering team is essentially shorting volatility. You collect a steady stream of small, visible wins in the form of green CI runs and uninterrupted release cadence, while accumulating unpriced tail risk in production drift. Because the metric was null, the true loss function was unobserved rather than bounded. When the regression eventually surfaces weeks or months later, diagnosing the root cause costs far more than the developer time saved by skipping the evaluation.

The hardest part of maintaining a fail-closed gate is usually organizational incentive alignment. Release velocity is a direct performance metric for feature teams, whereas verification failure creates immediate friction. If an exit code 2 halts a critical deployment because an upstream pipeline emitted an unexpected NaN or renamed a summary column, the immediate institutional impulse is to demand an override flag or loosen the schema. Unless the cost of shipping unverified artifacts is formally priced into the deployment contract, fail-closed gates face constant pressure to degrade back into advisory warnings.