Green Tests Are Not a Runtime: A Fail-Closed Production-Readiness Checklist

Green unit tests do not mean the change can take traffic. If an AI-assisted PR exposes a route, a job, or a migration, you block merge until rollback, capacity, and probe evidence exist as files your pipeline can fail closed on.

That is the whole rule. The rest of this article is a copy-paste checklist, a schema, and a small checker that treats TBD as a red gate.

Why "tests passed" is the wrong merge signal

AI-assisted diffs arrive faster than review habits. The suite goes green because the same session that wrote the handler also wrote the mocks. You get a PR that looks finished. Production does not care.

You need evidence that is boring and named. Not a screenshot of a local 200. A file in the repo that CI can parse, and a command that actually rolls the change back.

If a field is missing, you do not negotiate. You fail closed.

What this checklist covers

This is a runtime production-readiness gate. It answers one question: if this lands on a server that receives real requests, can you stop it, bound it, and see it?

It does not replace load tests in a prod-like environment. It does not certify a security review. It does not prove the model that drafted the code was correct. It only proves that someone named the operational contract before merge.

Do not use this as a substitute for an SRE handbook you already trust. Use it when AI-authored changes start skipping the messy parts: rollback, flags, probes, and numbers with units.

The fail-closed gates

Copy these nine gates. Each gate has required evidence and a fail-closed rule. Empty strings, TODO, TBD, and n/a without a reason code all fail.

1. Rollback is a command

• Evidence: ops/rollback.md contains one fenced command block tagged rollback that a pager-holder can paste.
• Fail closed: no tagged block, or the block is kubectl apply -f . with no version pin, or it says "revert the PR" with no SHA strategy.

A paragraph is not a rollback. If the on-call cannot paste one command, you do not have a rollback.

2. Probes have failure semantics

• Evidence: ops/probes.yml names liveness and readiness paths, success codes, and the exact condition that must flip readiness to false.
• Fail closed: readiness equals liveness, or either probe hits a dependency that can deadlock the process.

Liveness answers "should we kill this process?" Readiness answers "should we send it traffic?" If those are the same URL, you will either flap or serve errors with a smile.

3. Capacity is a number with a unit

• Evidence: ops/capacity.yml sets max_rps, max_payload_bytes, and max_concurrency as integers.
• Fail closed: any value is 0, negative, or a string like plenty.

Models love adjectives. Production wants integers. You can be conservative. You cannot be vague.

4. Timeouts are hard deadlines

• Evidence: inbound request timeout and outbound client timeout are integers in milliseconds, and inbound must exceed outbound.
• Fail closed: missing timeouts, or inbound shorter than outbound.

This is not a retry policy. Retries without a deadline just move the fire. You pin the wall-clock budget for one attempt so a hung dependency cannot hold a worker forever.

5. Flags default closed

• Evidence: every new path is behind a flag whose default in config/flags.yml is false.
• Fail closed: default true, or no flag name on a newly exported route.

Default-on is how a "small demo" becomes Friday traffic. Keep the flag name in the PR title so reviewers can grep it.

6. Migrations reverse

• Evidence: db/forward.sql and db/reverse.sql both exist, and reverse is not a comment that says "restore from backup".
• Fail closed: forward-only DDL, or reverse drops data with no expand/contract note.

Backups are a disaster plan. A reverse migration is a merge requirement. If you cannot unwind the schema, you cannot unwind the release.

7. Correlation IDs on every new path

• Evidence: the new handler reads or mints x-request-id and logs it as a structured field.
• Fail closed: print debugging, or logs without a request key.

When the page fires, you will search one ID across gateway, app, and store. If that ID was never written, you are reading poetry.

8. On-call owner is a person-shaped string

• Evidence: ops/owner.yml has team and pager fields that match your roster file.
• Fail closed: owner: ai or owner: whoever-merged.

Models do not carry pagers. A team name that cannot be resolved in the roster is the same as no owner.

9. Secrets stay out of the diff

• Evidence: CI runs a scan; ops/secret_scan.txt is the last clean report path.
• Fail closed: any private key PEM, AKIA prefixes, or .env files in the change set.

AI assistants paste from local env files with alarming confidence. The gate does not "trust the model." It trusts a scan artifact.

How to pick the numbers without lying

You do not need a week of benchmarking to fill max_rps. You need a ceiling you will defend at 2 a.m.

Start from last week's production for the parent service, not from a model suggestion. If you have no parent, start from the smallest integer that still lets a canary through: max_concurrency: 2, max_rps: 5, max_payload_bytes equal to the documented API limit.

Write the source of the number in a comment above the field. Comments are not parsed. They are for the reviewer who will ask "why 40?"

If you cannot name a source, the number is fiction. Leave the field out and let the checker fail. That failure is cheaper than a silent 500 storm.

Evidence schema you can commit

Put this at ops/prod_gates.yml. The checker below reads it. Do not keep a second copy in a wiki.

version: 1
service: payments-api
change_id: pr-1842
rollback:
  command: |
    kubectl set image deploy/payments-api payments-api=ghcr.io/example/payments-api:sha-9f3c1a
  verified_in: staging
probes:
  liveness: /healthz
  readiness: /readyz
  readiness_fails_when: "dependency_ping != ok"
capacity:
  max_rps: 40
  max_payload_bytes: 65536
  max_concurrency: 16
deadlines_ms:
  inbound: 800
  outbound: 300
flags:
  - name: payments_api_v2_charge
    default: false
migration:
  forward: db/forward.sql
  reverse: db/reverse.sql
observability:
  request_id_header: x-request-id
owner:
  team: payments-runtime
  pager: payments-oncall
secret_scan: ops/secret_scan.txt

Those numbers are placeholders. Replace them with your service's real ceilings. The gate does not care that 40 RPS is right. It cares that you typed an integer you are willing to own.

Keep reverse SQL next to forward SQL:

-- db/forward.sql
ALTER TABLE charges ADD COLUMN IF NOT EXISTS idempotency_key TEXT;

-- db/reverse.sql
ALTER TABLE charges DROP COLUMN IF EXISTS idempotency_key;

If reverse would destroy rows, stop. Split the change into expand, migrate, contract. The checker cannot see data loss. You can.

Checker that fails closed

Save as scripts/check_prod_gates.py. Run it in CI. Treat exit code 1 as a merge block, not a warning.

#!/usr/bin/env python3
"""Fail-closed production-readiness checker. Missing or TBD fields block merge."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this gate\n")
    raise SystemExit(2)

BAD = {"", "tbd", "todo", "n/a", "none", "unknown", "ai"}


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def load(path: Path) -> dict:
    if not path.is_file():
        fail(f"missing {path}")
    data = yaml.safe_load(path.read_text()) or {}
    if not isinstance(data, dict):
        fail("prod_gates.yml must be a mapping")
    return data


def need(v, label: str) -> None:
    if v is None:
        fail(f"{label} is missing")
    if isinstance(v, str) and v.strip().lower() in BAD:
        fail(f"{label} is a placeholder")


def main() -> None:
    root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
    cfg = load(root / "ops" / "prod_gates.yml")

    rb = cfg.get("rollback") or {}
    need(rb.get("command"), "rollback.command")
    cmd = str(rb.get("command"))
    if "revert the pr" in cmd.lower() and "sha" not in cmd.lower():
        fail("rollback.command must pin an image tag or git SHA")

    probes = cfg.get("probes") or {}
    need(probes.get("liveness"), "probes.liveness")
    need(probes.get("readiness"), "probes.readiness")
    if probes.get("liveness") == probes.get("readiness"):
        fail("readiness must not equal liveness")
    need(probes.get("readiness_fails_when"), "probes.readiness_fails_when")

    cap = cfg.get("capacity") or {}
    for key in ("max_rps", "max_payload_bytes", "max_concurrency"):
        val = cap.get(key)
        if not isinstance(val, int) or val <= 0:
            fail(f"capacity.{key} must be a positive int")

    dl = cfg.get("deadlines_ms") or {}
    inbound, outbound = dl.get("inbound"), dl.get("outbound")
    if not isinstance(inbound, int) or not isinstance(outbound, int):
        fail("deadlines_ms.inbound and outbound must be ints")
    if inbound <= 0 or outbound <= 0:
        fail("deadlines must be positive")
    if inbound <= outbound:
        fail("inbound deadline must exceed outbound deadline")

    flags = cfg.get("flags") or []
    if not flags:
        fail("flags must list at least one default-closed flag")
    for flag in flags:
        need(flag.get("name"), "flag.name")
        if flag.get("default") is not False:
            fail(f"flag {flag.get('name')} must default to false")

    mig = cfg.get("migration") or {}
    for key in ("forward", "reverse"):
        rel = mig.get(key)
        need(rel, f"migration.{key}")
        if not (root / str(rel)).is_file():
            fail(f"migration.{key} file missing: {rel}")

    obs = cfg.get("observability") or {}
    need(obs.get("request_id_header"), "observability.request_id_header")

    owner = cfg.get("owner") or {}
    need(owner.get("team"), "owner.team")
    need(owner.get("pager"), "owner.pager")

    scan = cfg.get("secret_scan")
    need(scan, "secret_scan")
    if not (root / str(scan)).is_file():
        fail(f"secret_scan report missing: {scan}")

    print("PASS: production-readiness gates are named and parseable")


if __name__ == "__main__":
    main()

Run it locally until it fails for the right reason:

python3 -m pip install pyyaml
python3 scripts/check_prod_gates.py .

Catch leftover fiction before CI does:

grep -nEi 'tbd|todo|n/a|plenty|revert the pr' ops/prod_gates.yml && echo 'placeholders found' && exit 1

Makefile target you can teach the team in one line:

.PHONY: prod-gates
prod-gates:
    python3 scripts/check_prod_gates.py .

CI sketch:

# .github/workflows/prod-gates.yml
name: prod-gates
on: [pull_request]
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python3 scripts/check_prod_gates.py .

If the workflow is continue-on-error: true, you do not have a gate. You have a diary.

Decision table for reviewers

Signal in the PR	Merge?	Why
Tests green, ops/prod_gates.yml missing	No	No runtime contract
Gates file present, rollback.command is prose	No	Cannot execute a paragraph
Capacity numbers copied from another service	No until edited	Copied ceilings are fiction
Flag default true "because demo"	No	Demo defaults leak
Reverse migration is "restore backup"	No	Disaster plan, not a reverse
Checker PASS, load test still pending	Yes, behind the flag only	Gate is necessary, not sufficient

Paste this table into the PR template. Reviewers should not re-litigate the rules in comments. The file either parses or it does not.

Where a scratch environment helps

You want the checker to run somewhere that is not your laptop and not production. A short dry run catches YAML that looks valid and still fails the integer rules.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a throwaway box plus model access to draft ops/prod_gates.yml from a diff and then execute scripts/check_prod_gates.py, MonkeyCode's free model access and free server option can host that loop. Do not treat that box as staging. Do not paste production secrets into it. Use it to get a red/green signal on the evidence files before you open the PR.

Limitations

The checker believes files. A determined author can write max_rps: 40 with no benchmark behind it. Fail-closed schema is not a load test. It is an honesty prompt with teeth.

Placeholder detection is string-based. max_rps: 999999 passes. You still need a human who knows the service.

Probe paths are not probed. The script does not HTTP-get /readyz. Hook that in staging separately.

Deadline math is local. It does not see the mesh timeout in front of you. If the proxy is 200ms and you wrote 800ms inbound, production will disagree. Add the proxy number to the YAML when you know it.

Who should not use this

Skip this approach if you already enforce production contracts in a service catalog with real owners and automated rollbacks. You would be duplicating a stronger system.

Skip it if your compliance process requires signed evidence in a GRC tool. A YAML file in git is not that artifact.

Skip it if the change cannot receive traffic — docs-only, comment-only, or lockfile bumps with no runtime surface. Forcing nine gates on a README change trains people to ignore gates.

Do not use a scratch-server path for regulated data, customer payloads, or production credentials. The checklist assumes public-ish metadata: probe paths, integer ceilings, flag names.

What you do tomorrow morning

Pick one service that recently merged an AI-assisted route. Add ops/prod_gates.yml with honest numbers, even if they are conservative. Run the checker locally until it fails for the right reasons, then until it passes.

Then attach the workflow. The first red CI is the point. After that, "tests passed" stops being the whole story, and traffic gets a named budget before it arrives.