Fail Closed: A Production-Readiness Contract for Small AI Features

The gate I skip is always the one that bites

The DEV front page keeps arguing about AI output quality. Fair. But that debate skips the boring part.

Features ship without release criteria. Criteria get written after the incident, not before.

So here is a different approach. A readiness contract that fails closed, stored in the repo.

It is not a vibe check. It is a file, a runner, and three tests.

Three rules before any YAML

Write these on the repo wall. They are short on purpose.

1. Missing evidence is a failure, never a warning.
2. Exit code 0 is not proof. Empty output fails.
3. A waiver without an expiry and reason is invalid.

Rule two is the one people fight me on. A command that exits 0 and prints nothing tells you nothing.

Step 1: Put the gates in readiness.toml

I use TOML because tomllib ships with Python 3.11. No dependencies. No lockfile churn.

[defaults]
max_age_hours = 72
evidence_dir = "artifacts/readiness"

[[gate]]
id = "unit-tests"
run = "pytest -q"
evidence = "pytest.txt"
max_age_hours = 24
owner = "sam"

[[gate]]
id = "budget-headroom"
run = "python tools/token_budget.py --max-tokens 12000 --json"
evidence = "budget.json"
max_age_hours = 12
owner = "sam"
expect_json = { field = "spent_ratio_max", max = 0.6 }

[[gate]]
id = "rollback-drill"
run = ""
evidence = "rollback.exit"
max_age_hours = 168
manual = true
owner = "sam"

[[waiver]]
gate = "concurrent-load"
expires = "2026-11-01"
reason = "single-tenant canary, no concurrent load yet"

Two details matter. The manual = true gate runs nothing and only validates an evidence file a human produced. The expect_json predicate is a field comparison, not an expression, so nothing gets eval-ed.

Step 2: Write a pure core and a dumb shell

The bug in most checkers is mixed concerns. Shell edges stay dumb. The verdict logic stays pure.

#!/usr/bin/env python3
"""Fail-closed readiness runner. Missing evidence is a failure, not a warning."""
from __future__ import annotations

import json, subprocess, sys, time, tomllib
from dataclasses import dataclass, field
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
CFG = ROOT / "readiness.toml"


@dataclass
class Verdict:
    gate: str
    issues: list[str] = field(default_factory=list)
    owner: str = "unassigned"

    @property
    def ok(self) -> bool:
        return not self.issues


def evaluate(gate_id, exit_code, evidence_path, max_age_hours,
             expect_json=None, now=None, owner="unassigned") -> Verdict:
    now = now or time.time()
    verdict = Verdict(gate=gate_id, owner=owner)
    path = Path(evidence_path)

    if not path.exists():
        verdict.issues.append("missing_evidence")
        return verdict

    if (now - path.stat().st_mtime) / 3600 > max_age_hours:
        verdict.issues.append("stale")
    if path.stat().st_size == 0:
        verdict.issues.append("empty_evidence")
    if exit_code != 0:
        verdict.issues.append("nonzero_exit")

    if expect_json and not verdict.issues:
        try:
            data = json.loads(path.read_text())
        except json.JSONDecodeError:
            verdict.issues.append("bad_json")
        else:
            value = data.get(expect_json["field"])
            if value is None or value > expect_json["max"]:
                verdict.issues.append("predicate_failed")
    return verdict


def active_waiver(waivers, gate_id, today=None):
    today = today or time.strftime("%Y-%m-%d")
    for w in waivers:
        if w["gate"] != gate_id:
            continue
        if not w.get("reason", "").strip():
            continue          # no reason means no waiver
        if w["expires"] < today:
            continue          # expired means fail closed
        return w
    return None


def run_gate(gate, defaults, evdir):
    evdir.mkdir(parents=True, exist_ok=True)
    path = evdir / gate["evidence"]
    code = 0
    if gate.get("run"):
        proc = subprocess.run(gate["run"], shell=True, cwd=ROOT,
                              capture_output=True, text=True)
        code = proc.returncode
        path.write_text(proc.stdout + proc.stderr)
    return evaluate(
        gate["id"], code, path,
        gate.get("max_age_hours", defaults["max_age_hours"]),
        gate.get("expect_json"),
        owner=gate.get("owner", "unassigned"),
    )


def main() -> int:
    cfg = tomllib.loads(CFG.read_text())
    defaults = cfg.get("defaults", {})
    evdir = ROOT / defaults.get("evidence_dir", "artifacts/readiness")
    verdicts = [run_gate(g, defaults, evdir) for g in cfg.get("gate", [])]
    blocked = []
    for v in verdicts:
        if v.ok:
            continue
        w = active_waiver(cfg.get("waiver", []), v.gate)
        if w:
            print(f"WAIVED {v.gate} until {w['expires']}: {w['reason']}")
            continue
        blocked.append(v)
    print(f"{len(verdicts) - len(blocked)}/{len(verdicts)} gates green")
    for v in blocked:
        print(f"FAIL {v.gate} {','.join(v.issues)} owner={v.owner}")
    return 1 if blocked else 0


if __name__ == "__main__":
    sys.exit(main())

Run it with python tools/readiness.py. Exit code 1 blocks the release. Exit code 0 means every gate produced fresh, non-empty evidence.

Step 3: Add the failure fixture

A checker you never tested against failure is decoration. Two fixtures catch most of it.

# tests/test_readiness.py
from tools.readiness import active_waiver, evaluate


def test_silent_pass_is_a_failure(tmp_path):
    ev = tmp_path / "probe.txt"
    ev.write_text("")               # exited 0, printed nothing
    v = evaluate("health-probe", 0, ev, max_age_hours=24)
    assert "empty_evidence" in v.issues


def test_expired_waiver_does_not_apply():
    waivers = [{"gate": "load", "expires": "2020-01-01", "reason": "old"}]
    assert active_waiver(waivers, "load", today="2026-10-16") is None


def test_missing_evidence_is_a_failure(tmp_path):
    v = evaluate("canary", 0, tmp_path / "nope.log", max_age_hours=1)
    assert v.issues == ["missing_evidence"]

The first fixture is the important one. true exits 0 and produces nothing. Your checker must call that a failure.

The decision table

Copy this into the repo next to the TOML. Each row needs one owner name, not a team name.

Gate	Evidence	Fails closed when	Max age
Unit tests	pytest.txt	empty, nonzero, stale	24h
Budget headroom	budget.json	spent_ratio_max > 0.6	12h
Prompt hash	prompt.sha256	hash differs from reviewed commit	current commit
Canary traffic	canary.log	zero lines, any 5xx	1h
Rollback drill	rollback.exit	missing file	7d
Waiver hygiene	readiness.toml	expired or empty reason	n/a

Nothing here needs a dashboard. Six files and one exit code are enough for a solo builder.

Where free model access and a free server actually help

Two gates in that table cost money if you run them naively. That is where a free allowance changes the experiment.

Gate two measures budget headroom. I point it at a throwaway endpoint first, so a bad prompt loop burns nothing. MonkeyCode's free model access is the free tier I use for that pass, and its free server option gives me a disposable host for the canary gate instead of probing production.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat availability claims as claims. The free tier is advertised with a token allowance and a free server, but quotas, model lists, and terms change. Read the current terms on the project page before you wire a gate to them. Do not hardcode a number from a blog post into readiness.toml. Read it from the provider or from a config file you control.

Limitations and who should not use this

This is a release gate, not a security review. It proves evidence exists and is fresh. It proves nothing about correctness.

Skip it if you have no rollback path yet. Every gate will fail closed, and you will learn nothing.

Skip it in regulated environments where evidence needs formal sign-off chains. A TOML file is not an audit trail.

Skip it past roughly five services. At that size you want a platform team and real policy tooling, not one Python file.

Try it today in about 20 minutes

1. touch readiness.toml and add the unit-test gate only.
2. Write tools/readiness.py from the block above.
3. Add the empty-evidence fixture and watch it fail on purpose.
4. Run python tools/readiness.py and check the exit code.
5. Add one waiver with a real expiry date, then let it expire in a test.
6. Wire the runner into CI as a required step, not a report.

That is the whole artifact. One file, one runner, three tests, six rows.

Which field does your team keep forgetting in a gate like this: the evidence path, the owner, or the waiver expiry date? Tell me the missing one and I will fold it into the next version of the template.