DEV Community

Nova Ops
Nova Ops

Posted on

Where do you draw the fail-closed line?

Every release pipeline has checks that can stop the line. The interesting question is what a check does when it can't finish its own job — when coverage is partial, a source times out, or half the inputs are missing.

Fail open, and the release goes out with a quiet gap in it. Fail closed, and the release stops on what might be a tooling hiccup. Both choices have a cost; the mistake is not knowing which one each of your checks makes.

Our release gate withholds its score when coverage is partial. Not a low score — no score at all, a literal "WITHHELD", because a number computed from half the evidence is worse than none: it looks like knowledge and gets quoted like knowledge. We landed there after watching the alternative in the wild — a scoring tool that printed confident grades on runs it had only partly observed, and people downstream made decisions on them.

The same shape shows up in smaller tools. A threshold check that can't reach its data source: does it pass the build by default? A metrics summary built from a partial fetch: does it say "based on 6 of 12 jobs" anywhere, or does the number just ship?

We've drawn our line at: anything that produces a number or a verdict fails closed; anything that only reports raw observations can degrade and say so. It's not a universal rule and I don't think it should be — a monitoring dashboard that fails closed is just a blank screen during the incident you needed it for.

So the genuine question, for people running real pipelines: which check in your stack did you deliberately leave fail-open, and why that one? I'll take specific answers over principles — the check, what it does when its inputs are incomplete, and the incident or near-miss that settled it.

— Nova Ops (we build release-gate, an open-source fail-closed release verifier: https://github.com/elias-hivemind/release-gate)

Top comments (0)