DEV Community: Taylor Zhu

Expire the Eval or Don't Enable: A Fail-Closed Measurement Checklist

Taylor Zhu — Mon, 21 Sep 2026 13:34:35 +0000

A model-backed endpoint is not ready because the unit suite is green. It is ready when the measurement that can stop enablement is named, hashed, dated, owned, and fail-closed. If that measurement is missing, you are not shipping a feature. You are hoping.

This is not another reminder that tests exist. You already have those. This is the gate that decides whether a prompt change, a retrieval tweak, or a tool-schema edit may take production traffic.

What you are actually gating

Unit tests check deterministic code. Model-backed paths are not deterministic in the way your CI likes. You still need a number. You still need a cutoff. You still need a clock on the set that produced that number.

Without a clock, last quarter's goldens keep blessing this week's behavior. That is how a "measured" system drifts into an unmeasured one. Quietly.

You will copy four things:

A fail-closed rule.
A gate matrix you can paste into a PR template.
A YAML manifest the job must refuse to run without.
A small runner that exits non-zero when the evidence is incomplete.

None of this requires a vendor. A free model path and a spare server help when you want the job to run without burning a paid key. They are optional. The gate is not.

The fail-closed rule

If any of the following is true, the enablement does not happen. No silent skip. No "we'll backfill the eval."

The eval set has no eval_id.
The eval set has no content hash recorded in the repo.
The eval set is older than its own max_age_days.
The eval set has no human owner.
The job cannot load the set from the path the manifest names.
The score is below threshold and there is no waiver file with an expiry.
The waiver is past expiry, or it does not name a ticket.

Read that again. A missing eval is a failed eval. An expired eval is a failed eval. An owned-by-nobody eval is a failed eval.

You do not "degrade to best effort" on the measurement that authorizes traffic. Best effort is for retries. Measurement is for permission.

Missing is red, not yellow

Teams love a skip. Skip looks kind in a dashboard. Skip is how a prompt PR lands on Friday and the goldens catch up "next sprint." Next sprint is where stale answers become product behavior.

If the job cannot run, the flag stays off. That is the whole method. Write it on the PR template so nobody has to remember it under pressure.

Gate matrix you can copy

Paste this into the PR that touches prompts, tools, retrieval, or the model-backed route.

Gate	Evidence you must attach	Fail-closed if missing
Identity	`eval_id` in `eval_gate.yaml`	Enablement blocked
Integrity	SHA-256 of the dataset file	Hash mismatch or empty hash
Freshness	`generated_at` + `max_age_days`	Age exceeded
Ownership	GitHub handle or team alias	Owner blank
Execution	CI log of the runner, same commit SHA	Job skipped or ran on another SHA
Threshold	`min_pass_rate` compared to scored output	Score absent or below cutoff
Waiver	`waivers/<eval_id>.md` with ticket + expiry	Waiver missing, expired, or unnamed
Rollback	Kill switch / flag name in the PR	No named flag, no enablement

You do not need all eight to be fancy. You need all eight to be present. Fancy comes later.

Treat the matrix as a receipt, not a vibe. If a cell is empty, the row failed. If the row failed, the flag stays down.

Manifest the job must refuse to invent

Create eval_gate.yaml next to the route you are protecting. Keep it boring.

# Proposal: commit this beside the model-backed route.
# Unexecuted example — wire paths to your repo.
eval_id: checkout_assist_v3
dataset_path: evals/checkout_assist_v3.jsonl
dataset_sha256: REPLACE_WITH_REAL_SHA256
generated_at: "2026-09-14"
max_age_days: 14
owner: "checkout-ml"
min_pass_rate: 0.86
blocking: true
feature_flag: "checkout_assist_v3"
model_path_env: MODEL_ENDPOINT

Two notes. First: blocking: true means CI red is the product. If you set it false, you are collecting telemetry, not a gate. Second: model_path_env names an environment variable, not a model. You pin the path in deploy config. You do not hide it in a notebook.

Hash the dataset in CI so a silent edit cannot ride along:

# Label: local helper, run from repo root.
python - <<'PY'
from pathlib import Path
import hashlib, sys
p = Path("evals/checkout_assist_v3.jsonl")
if not p.is_file():
    sys.exit("eval dataset missing")
digest = hashlib.sha256(p.read_bytes()).hexdigest()
print(digest)
PY

If that digest does not match dataset_sha256, the job fails. You do not "update the hash later." You update it in the same PR that changed the set, or you do not change the set.

Refresh without laundering the gate

Goldens have to move. Domains shift. That is not a license to move the cutoff in the same diff as the prompt.

Use a split:

Freeze the old hash in git history. Do not rewrite it.
Add rows in a dataset-only PR. New generated_at. New hash. Same min_pass_rate.
Land prompt or tool changes in a second PR that must beat the already-merged set.
Raise min_pass_rate only after the new set has survived a week of blocking CI.

If you change the prompt, the set, and the cutoff together, you did not measure a regression. You authored a new story and called it a score.

Keep the files dull and findable:

evals/checkout_assist_v3.jsonl
evals/checkout_assist_v3.README.md
waivers/
eval_gate.yaml
eval_gate.py

The README names who labels a row, what "correct" means, and which ticket last rebuilt the set. If that file is empty, the owner field in YAML is theater.

Runner that fails closed

The runner below is a proposal. It does not call a vendor SDK. It scores a JSONL file of {id, input, expect} rows against a callable you already own. Wire your own client. Keep the control flow.

# eval_gate.py — proposal / unexecuted example
from __future__ import annotations

import hashlib
import json
import os
import sys
from datetime import date, datetime, timezone
from pathlib import Path

import yaml


def fail(msg: str) -> None:
    print(f"FAIL-CLOSED: {msg}", file=sys.stderr)
    raise SystemExit(1)


def load_manifest(path: Path) -> dict:
    if not path.is_file():
        fail(f"manifest missing: {path}")
    data = yaml.safe_load(path.read_text()) or {}
    for key in (
        "eval_id",
        "dataset_path",
        "dataset_sha256",
        "generated_at",
        "max_age_days",
        "owner",
        "min_pass_rate",
        "blocking",
        "feature_flag",
    ):
        if data.get(key) in (None, ""):
            fail(f"manifest field empty: {key}")
    return data


def assert_fresh(generated_at: str, max_age_days: int) -> None:
    born = date.fromisoformat(generated_at)
    age = (date.today() - born).days
    if age > int(max_age_days):
        fail(f"eval expired: age={age}d max={max_age_days}d")


def assert_hash(dataset: Path, expected: str) -> None:
    digest = hashlib.sha256(dataset.read_bytes()).hexdigest()
    if digest != expected:
        fail(f"dataset hash mismatch: got={digest}")


def assert_waiver_if_needed(eval_id: str, pass_rate: float, min_pass: float) -> None:
    if pass_rate >= min_pass:
        return
    waiver = Path("waivers") / f"{eval_id}.md"
    if not waiver.is_file():
        fail(f"score {pass_rate:.3f} < {min_pass} and no waiver")
    text = waiver.read_text()
    if "ticket:" not in text.lower() or "expires:" not in text.lower():
        fail("waiver missing ticket: or expires:")
    # Keep parsing strict and local. Do not fetch the ticket tracker here.


def score_rows(dataset: Path, call) -> float:
    rows = [json.loads(line) for line in dataset.read_text().splitlines() if line.strip()]
    if not rows:
        fail("dataset empty")
    hits = 0
    for row in rows:
        got = call(row["input"])
        # Exact match on purpose. Fuzzy scoring belongs in a named scorer file.
        if str(got).strip() == str(row["expect"]).strip():
            hits += 1
    return hits / len(rows)


def main() -> None:
    manifest = load_manifest(Path("eval_gate.yaml"))
    if not manifest["blocking"]:
        fail("blocking is false; this job is not a gate")
    dataset = Path(manifest["dataset_path"])
    if not dataset.is_file():
        fail(f"dataset missing: {dataset}")
    assert_hash(dataset, manifest["dataset_sha256"])
    assert_fresh(manifest["generated_at"], manifest["max_age_days"])
    if not os.environ.get(manifest.get("model_path_env", "MODEL_ENDPOINT")):
        fail("model path env is unset; refuse to score against an implicit default")

    def call(prompt: str) -> str:
        # Wire your client here. Do not silently fall back to a different model.
        raise RuntimeError("plug in your model client")

    pass_rate = score_rows(dataset, call)
    assert_waiver_if_needed(manifest["eval_id"], pass_rate, float(manifest["min_pass_rate"]))
    print(json.dumps({
        "eval_id": manifest["eval_id"],
        "pass_rate": pass_rate,
        "feature_flag": manifest["feature_flag"],
        "commit": os.environ.get("GIT_SHA", ""),
        "scored_at": datetime.now(timezone.utc).isoformat(),
    }, indent=2))


if __name__ == "__main__":
    main()

Notice what the runner does not do. It does not default the model path. It does not treat blocking: false as success. It does not accept an empty dataset as 100%. Those are the holes teams leave "for later." Later is how stale goldens become production truth.

Exact match is a feature here. If your task is generative, do not sprinkle if "close enough" into this file. Add scorer.py, hash that file too, and name it in the manifest. Hidden fuzzy logic is a second model you forgot to version.

CI shape, not CI poetry

Keep the workflow stupid. One job. Same commit SHA as the merge.

# .github/workflows/eval-gate.yml — proposal
name: eval-gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "eval_gate.yaml"
      - "src/checkout_assist/**"
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python eval_gate.py
        env:
          MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
          GIT_SHA: ${{ github.sha }}

If the path filter never matches your prompt files, the gate never runs. That is a hole. Put the prompt files where the filter looks, or drop the filter.

Print the JSON on success and archive it. A gate without a receipt is a story you tell in standup. You already know how those rot.

Store the receipt next to the flag name. When someone asks "why is v3 on?" you point at a commit SHA, a hash, a pass rate, and a date. You do not point at a Slack memory.

Where a free model path actually helps

You need a real MODEL_ENDPOINT for the runner. A laptop that is asleep is not a runtime. A paid key that someone rotated is not a runtime either.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you are already using MonkeyCode, the useful part here is mundane: free model access and a free server option give you a place to keep MODEL_ENDPOINT pointed at something that answers when CI asks. That is the whole integration. The checklist does not care which product serves the bytes. It cares that the path is explicit, the job is blocking, and the goldens can expire.

Do not point production traffic at the same endpoint you use to score goldens unless that is a decision you have written down. Scoring and serving are different jobs. Mixing them is how a load spike looks like an eval outage, or the reverse.

One practical split:

CI eval uses the free server and free model access.
Production serving uses the pinned path your deploy config already named.
Both paths show up as env values. Neither is a default in code.

If you do not use MonkeyCode, run the same split against whatever endpoint you already trust. The artifact above still applies. Fork the checklist first. The product is optional.

Limitations

Exact string match will punish valid paraphrases. If your task is generative, replace score_rows with a named scorer file and version that file like the dataset. Do not hide the scorer in a prompt.

This checklist does not measure fairness, latency, or cost. If those can block a release, they need their own manifests. Do not overload min_pass_rate with three meanings.

max_age_days: 14 is an example, not a standard. Set the clock from how often your domain shifts. A tax table and a chitchat bot do not share a freshness budget.

The waiver path can become a culture hole. If every PR carries a waiver, you no longer have a gate. You have a diary. Cap concurrent waivers at one per eval_id, and make expiry shorter than max_age_days.

A hashed JSONL file is not a sampling strategy. If production traffic is 90% language A and your goldens are 90% language B, the gate is precise and wrong. Fix the set. Do not celebrate the score.

Who should not use this

Skip this if you do not have a model-backed route in production, and no plan to enable one. A checklist without a flag to protect is ceremony.

Skip this if your "eval" is a handful of vibe checks in a chat window. Write the JSONL first. Then gate.

Skip this if a human must approve every response at runtime. Your measurement problem is different. You need an audit log, not a pass rate.

Skip this if you cannot fail closed — regulated change windows, vendor-owned models you cannot pin, or an org that treats red CI as optional. The runner will only annoy you.

What you do on Monday

Pick one endpoint. One. Write eval_gate.yaml with a real hash or refuse to merge. Put blocking: true. Name the feature flag. Run the job on the PR that would have enabled it.

If the job cannot run, you do not enable. Stale goldens are not a vibe problem. They are an authorization problem. Treat them that way.

Cut Over the Model Path or Don't Ship: A Fail-Closed Inference Checklist

Taylor Zhu — Sat, 19 Sep 2026 09:39:15 +0000

If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started.

Treat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak.

This is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch.

The failure you are actually shipping

You added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies.

The bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks complete().

You need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering.

What “cut over” means here

Cut over is not “we created a prod API key.” Cut over means all four of these are true at once:

The production base URL is an allowlisted origin, not an environment default inherited from .env.local.
The model identifier is an explicit production contract, not default, latest, or an empty string.
Timeouts, token ceilings, and retry counts are set in prod config and are smaller than “keep hammering.”
Fallback is deny-by-default: no lab host, no alternate vendor you have not reviewed, no “best effort” second try against the drafting server.

If you cannot show those four, you are still in the sandbox. Ship the feature flag off.

Six gates, each with a receipt

Every gate needs an artifact. A Slack “looks good” is not a receipt.

Gate 1 — Named production origin

Pass: INFERENCE_BASE_URL in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel.

Fail closed: any prod manifest that still contains localhost, 127.0.0.1, ngrok, trycloudflare, or a hostname tagged lab, dev, sandbox, or draft.

Receipt: the commit SHA of inference-allowlist.json plus the secret-store version id.

Gate 2 — Model identity is not a moving alias

Pass: production sets INFERENCE_MODEL_ID to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook.

Fail closed: latest, auto, empty, or a name that only exists on the drafting server.

Receipt: a one-line contract in the repo: model id, max tokens, and who owns rotation.

Gate 3 — Lab host denylist in CI

Pass: CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts.

Fail closed: denylist bypassed with “temporary” comments, or the check only scans src/ and ignores deploy/.

Receipt: the CI job log URL for the merge commit.

Gate 4 — Budgets are numbers, not folklore

Pass: production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination.

Fail closed: unlimited retries, exponential backoff with no cap, or a retry that swaps base_url.

Receipt: the config snippet and a test that asserts the client is constructed once with prod settings.

Gate 5 — Fallback cannot re-enter the lab

Pass: on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client.

Fail closed: “if prod fails, call the free server so the demo still works.”

Receipt: a failing integration test that stubs prod errors and asserts the lab host is never dialed.

Gate 6 — Traffic identity

Pass: production requests send a stable X-Service / User-Agent and an environment tag prod. Logs can answer “which app, which model id, which base URL” without reading source.

Fail closed: the drafting client and the prod client are the same binary with the same defaults.

Receipt: one redacted log line from a staging call that already uses the prod origin.

Copy-paste checklist

Use this in the PR. Check a box only when the artifact exists.

[ ] inference-allowlist.json lists the single prod origin (or the exact set of regional origins).
[ ] inference-denylist.json lists lab, draft, and tunnel hosts.
[ ] Prod overlay sets INFERENCE_BASE_URL and INFERENCE_MODEL_ID; neither is blank.
[ ] Client construction test fails if base_url is missing.
[ ] Error-path test fails if any HTTP call target is on the denylist.
[ ] Timeout, retry cap, and token ceiling are set in prod, not only in a README.
[ ] Feature flag default is off until Gate 1–5 receipts are linked.
[ ] Runbook names the owner of model-id rotation.
[ ] Rollback is “flag off,” not “point DNS back at the sandbox.”

If a box depends on “we will add it after launch,” the gate failed.

Artifact: a policy file CI can reject

Commit this as inference-policy.json. Keep it boring. Boring is reviewable.

{
  "allow_base_urls": [
    "https://inference.prod.example.internal"
  ],
  "deny_host_substrings": [
    "localhost",
    "127.0.0.1",
    "ngrok",
    "trycloudflare",
    "lab.",
    "sandbox.",
    "draft.",
    "dev-inference"
  ],
  "require_env": [
    "INFERENCE_BASE_URL",
    "INFERENCE_MODEL_ID",
    "INFERENCE_TIMEOUT_MS",
    "INFERENCE_MAX_RETRIES",
    "INFERENCE_MAX_OUTPUT_TOKENS"
  ],
  "max_retries": 1,
  "forbid_fallback_base_url": true
}

Replace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay.

Artifact: a fail-closed checker

The script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log.

#!/usr/bin/env python3
"""Fail closed if prod config can still reach a lab inference host."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path
from urllib.parse import urlparse

POLICY = Path("inference-policy.json")
SCAN_ROOTS = [Path("deploy"), Path("k8s"), Path("infra"), Path(".")]
SCAN_SUFFIXES = {".yml", ".yaml", ".json", ".tf", ".env", ".toml"}


def load_policy() -> dict:
    if not POLICY.exists():
        print("FAIL: inference-policy.json missing")
        sys.exit(2)
    return json.loads(POLICY.read_text())


def host_of(url: str) -> str:
    parsed = urlparse(url if "://" in url else f"https://{url}")
    return (parsed.hostname or "").lower()


def denied(host: str, needles: list[str]) -> str | None:
    for needle in needles:
        if needle.lower() in host:
            return needle
    return None


def check_env(policy: dict) -> list[str]:
    errors = []
    for key in policy["require_env"]:
        if not os.environ.get(key):
            errors.append(f"missing env {key}")
    base = os.environ.get("INFERENCE_BASE_URL", "")
    if base:
        host = host_of(base)
        hit = denied(host, policy["deny_host_substrings"])
        if hit:
            errors.append(f"INFERENCE_BASE_URL host matches denylist '{hit}': {host}")
        allowed_hosts = {host_of(u) for u in policy["allow_base_urls"]}
        if host not in allowed_hosts:
            errors.append(f"INFERENCE_BASE_URL host not allowlisted: {host}")
    model = os.environ.get("INFERENCE_MODEL_ID", "")
    if model.lower() in {"", "latest", "auto", "default"}:
        errors.append(f"INFERENCE_MODEL_ID is not pinned: {model!r}")
    try:
        retries = int(os.environ.get("INFERENCE_MAX_RETRIES", "99"))
    except ValueError:
        retries = 99
        errors.append("INFERENCE_MAX_RETRIES is not an int")
    if retries > int(policy["max_retries"]):
        errors.append(f"retries {retries} exceed policy max {policy['max_retries']}")
    return errors


def check_files(policy: dict) -> list[str]:
    errors = []
    needles = policy["deny_host_substrings"]
    for root in SCAN_ROOTS:
        if not root.exists():
            continue
        for path in root.rglob("*"):
            if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES:
                continue
            if path.name == POLICY.name:
                continue
            # Skip local-only samples; prod overlays must still pass.
            if ".local." in path.name or path.name.endswith(".example"):
                continue
            text = path.read_text(errors="ignore")
            lower = text.lower()
            for needle in needles:
                if needle.lower() in lower and "prod" in path.parts:
                    errors.append(f"{path}: denylist hit {needle!r}")
    return errors


def main() -> int:
    policy = load_policy()
    errors = check_env(policy) + check_files(policy)
    if errors:
        print("FAIL-CLOSED: inference cutover incomplete")
        for item in errors:
            print(f" - {item}")
        return 1
    print("PASS: inference path looks cut over (config scan only)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Wire it so merge is impossible when it exits 1:

# .github/workflows/inference-cutover.yml
name: inference-cutover
on:
  pull_request:
    paths:
      - "deploy/**"
      - "k8s/**"
      - "infra/**"
      - "inference-policy.json"
      - ".github/workflows/inference-cutover.yml"
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Require prod env in the CI job (from secrets, not the PR)
        env:
          INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }}
          INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }}
          INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }}
          INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }}
          INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }}
        run: python3 scripts/check_inference_cutover.py

Secrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate.

Decision table

Symptom	What you might tell yourself	Fail-closed action
Drafting host still in prod overlay	“Staging needs it”	Split overlays. Prod overlay cannot parse the lab hostname.
Model id is `latest`	“We want improvements automatically”	Pin an id. Rotate with a ticket and a replay test.
Prod 429, client calls lab	“Users should still get an answer”	Return an error. Page the owner. Never change `base_url` in retry.
Unit tests mock the SDK	“Coverage is high”	Add one test that inspects the constructed URL.
Tunnel URL in a hotfix	“Only for this incident”	Incident flag off. Do not hot-patch origin.
Free drafting server is fast today	“We can launch on it”	Launch is a contract plus capacity you control. Speed is not a contract.

Where a free drafting path belongs

Drafting against a shared or free inference endpoint is fine. Shipping that endpoint is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a place to sketch clients, prompts, and the checker itself, MonkeyCode’s free model access and free server option can stay on the drafting side of the line. Keep that origin on the denylist for production overlays. Generate the policy file there if you want. Then run the checker against the repo you actually deploy. The product mention ends here because the gate does not care which editor you used. It cares whether prod can still dial the lab.

Who should not use this approach

Do not use this checklist as a substitute for load tests, eval suites, or a real vendor contract. It will not tell you the model is accurate. It will not size GPUs. It will not prove GDPR. It only proves you did not leave a drafting path wired into the live service.

Skip it if you have no production inference origin yet. In that case the honest gate is “do not ship the feature,” not “scan a policy file that allowlists a wish.”

Also skip it if your app does not call a model at runtime. Generated code that never performs inference is a different review: dependency pins, tests, and side effects. This article is only for the path that answers complete() after deploy.

Limitations

The scanner is string-based. Obfuscated URLs, runtime service discovery, and hosts injected by a sidecar can evade it. Pair it with egress policy in the mesh or network layer. If the cluster can resolve the lab hostname, add a network deny.

The allowlist does not prove capacity. A pinned origin can still 429. Fail closed on that too: user-visible error, metric, page. Do not reopen the sandbox to absorb overflow.

Aliases will drift. Re-run the job when you rotate model ids. If nobody owns rotation, you do not have a contract. You have a default.

Ship rule

Name the production origin. Pin the model id. Deny the lab host in CI. Cap retries without changing destination. Link the job log on the PR. If you cannot, leave the flag off.

That is the whole method. The drafting environment can be free, shared, or noisy. Production cannot inherit it.

The Lab Host Is Not Prod: A Fail-Closed Promotion Checklist

Taylor Zhu — Fri, 18 Sep 2026 08:00:54 +0000

You can prototype on a free model and a free server. You cannot promote that host. Treat the lab as disposable, pin a promotion contract, and fail closed when the evidence files are missing.

That is the rule. Everything below is a copy-paste checklist for the cutover, not another merge ritual.

The leak you will actually ship

AI-assisted services rarely blow up because a generated loop is ugly. They blow up because the lab leaked.

You scaffold a handler. You run it on the same box that hosts the assistant. You paste the lab URL into a client. You leave a debug token in .env. A teammate then deploys "the same thing."

Unit tests will not catch that. A PR checklist about tools or retries will not catch it either. Promotion is a different gate. If the process that built the service still owns the runtime, you do not have production. You have a shared notebook with a port.

What counts as a lab

Call it a lab when all of these are true:

The coding assistant, the app process, and scratch data can share a host.
Model access is convenience-first, not contract-first.
Secrets are whatever unblocked the demo.
Nobody can name the production identity of the service.

Free model access and a free server option are useful in that phase. They become a problem only when you skip the cutover and ship the box you happened to be using.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a scratch environment with free model access and a free server option, MonkeyCode is an open-source project some teams park that work on. The checklist does not depend on it. Strip the product name out and the gates still stand.

Copy this promotion contract

Keep one file in the repo: PROMOTION.yaml. Deploy jobs should refuse to start unless every gate is pass and every evidence path exists on disk. Do not type pass by hand. Generate the files, then let a checker read both.

# PROMOTION.yaml — proposal for a local clone, not an executed audit
version: 1
service: billing-adapter
lab:
  allowed_hosts:
    - "127.0.0.1"
    - "lab.internal.example"
  must_not_appear_in:
    - deploy/
    - infra/
    - clients/
gates:
  - id: HOST_SPLIT
    question: "Is production a different host than the lab?"
    evidence: evidence/host-split.txt
    fail_closed: "Refuse deploy if a lab hostname appears under deploy/, infra/, or clients/."
  - id: SECRET_MATERIAL
    question: "Were lab tokens, cookies, and debug keys rotated out?"
    evidence: evidence/secret-scan.json
    fail_closed: "Refuse deploy if any lab secret pattern still matches."
  - id: MODEL_PIN
    question: "Is every model call pinned to a named contract, not the lab default?"
    evidence: evidence/model-pin.md
    fail_closed: "Refuse deploy if the client still reads LAB_MODEL or an unset MODEL_ID."
  - id: DATA_CLASS
    question: "Is production data blocked from the lab host?"
    evidence: evidence/data-class.md
    fail_closed: "Refuse deploy if the lab can still reach prod datastores."
  - id: OBSERVABILITY
    question: "Can you name the prod service in logs, traces, and alerts without the lab project name?"
    evidence: evidence/obs-names.txt
    fail_closed: "Refuse deploy if service.name still equals the assistant project."
  - id: BLAST_RADIUS
    question: "Is there a one-command rollback that does not SSH into the lab?"
    evidence: evidence/rollback.md
    fail_closed: "Refuse deploy if rollback docs mention the lab host."

Six gates. Six files. If a file is missing, the answer is no.

Fail-closed checker you can run locally

The script below is a proposal. Run it in a clone you control. It does not open a network connection. It fails if evidence is missing, if a gate is not pass, or if a forbidden lab host string appears under the listed trees.

#!/usr/bin/env python3
"""promotion_gate.py — fail closed when lab evidence is missing."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("install pyyaml in your local venv before running", file=sys.stderr)
    sys.exit(2)

REQUIRED_STATUS = {"pass"}


def load_contract(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or "gates" not in data:
        raise ValueError("PROMOTION.yaml must contain a gates list")
    return data


def scan_forbidden_hosts(root: Path, hosts: list[str], trees: list[str]) -> list[str]:
    hits: list[str] = []
    for tree in trees:
        base = root / tree
        if not base.exists():
            continue
        for path in base.rglob("*"):
            if not path.is_file():
                continue
            text = path.read_text(encoding="utf-8", errors="ignore")
            for host in hosts:
                if host and host in text:
                    hits.append(f"{path}: contains {host}")
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=Path, default=Path("."))
    parser.add_argument("--config", type=Path, default=Path("PROMOTION.yaml"))
    args = parser.parse_args()

    contract = load_contract(args.config)
    failures: list[str] = []

    for gate in contract["gates"]:
        evidence = args.root / gate["evidence"]
        if not evidence.is_file():
            failures.append(f"{gate['id']}: missing {evidence}")
            continue
        marker = evidence.read_text(encoding="utf-8", errors="ignore")
        if "STATUS: pass" not in marker:
            failures.append(f"{gate['id']}: {evidence} has no STATUS: pass line")

    lab = contract.get("lab", {})
    hits = scan_forbidden_hosts(
        args.root,
        lab.get("allowed_hosts", []),
        lab.get("must_not_appear_in", []),
    )
    failures.extend(hits)

    if failures:
        print("PROMOTION FAILED")
        for item in failures:
            print(f"- {item}")
        return 1

    print(json.dumps({"result": "ok", "gates": [g["id"] for g in contract["gates"]]}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire it so deploy never starts without the checker:

python3 -m pip install pyyaml
python3 promotion_gate.py --root . --config PROMOTION.yaml

.PHONY: promotion-gate
promotion-gate:
    python3 promotion_gate.py --root . --config PROMOTION.yaml

If your pipeline cannot see evidence/, that is a fail, not a skip.

What each evidence file has to prove

Write short files. A novel in evidence/ is how teams hide a missing gate.

HOST_SPLIT

Prove the production runtime is a different machine, cluster, or account than the lab. A useful evidence/host-split.txt looks like this:

STATUS: pass
lab_host: 127.0.0.1
prod_runtime: ecs://billing-adapter-prod
checked_trees: deploy/, infra/, clients/
command: rg -n "127.0.0.1|lab.internal.example" deploy infra clients

If rg prints a line, you do not have a split. You have a rename.

SECRET_MATERIAL

Rotate anything the lab touched. Demo cookies, assistant tokens, copied Authorization headers, default DB passwords. Then record the scan, not the secret.

rg -n "LAB_|DEBUG_TOKEN|sk-live|BEGIN OPENSSH" --glob '!.git' . \
  && echo "still dirty" && exit 1
printf 'STATUS: pass\nscanner: rg\npatterns: LAB_, DEBUG_TOKEN\n' > evidence/secret-scan.json

Adjust the patterns for your repo. Do not paste live credentials into the evidence file. The file is a receipt, not a backup.

MODEL_PIN

Lab defaults are not a contract. Pin the caller so production cannot silently follow whatever the assistant used that afternoon.

STATUS: pass
caller: src/llm_client.py
required_env:
  - MODEL_ID
  - MODEL_ENDPOINT
  - MAX_OUTPUT_TOKENS
forbidden_env:
  - LAB_MODEL
unset_means: fail closed, do not fall back

If the client still does os.getenv("MODEL_ID", "whatever-the-lab-used"), the gate is red. A default is a second lab.

DATA_CLASS

Keep production records off the lab host. That includes logs that embed payloads, copied fixtures, and "just one customer row."

STATUS: pass
prod_datastores:
  - postgres://prod-billing
lab_may_read:
  - fixtures/synthetic/
lab_may_not_read:
  - customers
  - invoices
egress_check: no prod DSN in lab compose files

If your lab compose file still contains the prod DSN "for realism," throw the compose file away. Realism is not a reason to mix classes.

OBSERVABILITY

Rename the service before the first prod request. Lab project names in service.name, dashboard folders, or alert titles will page the wrong people.

rg -n "service.name|OTEL_SERVICE_NAME|job:" deploy infra \
  | tee evidence/obs-names.txt
# then append:
# STATUS: pass
# prod_name: billing-adapter
# lab_name_absent: yes

BLAST_RADIUS

Rollback must not require SSH into the machine where you vibe-coded the demo. If the runbook says "just restart the lab process," you have no blast-radius control.

STATUS: pass
rollback: kubectl rollout undo deploy/billing-adapter
does_not_use: ssh lab.internal.example
owner_alert: #billing-oncall

Decision table: promote, rebuild, or throw away

Signal	Promote	Rebuild in prod-shaped infra	Throw the lab away
Lab hostname in clients	No	Yes, after stripping	If clients cannot be found
Model id unset at runtime	No	Yes, after pinning	If nobody can name the contract
Prod data reached the lab	No	Only after rotation and review	Default when customer data moved
Rollback needs lab SSH	No	Yes	If no other runtime exists
Evidence file missing	No	No	Yes, until the file exists

Read the table left to right. Missing evidence is not "unknown." Missing evidence is "do not promote."

A 20-minute dry run

Label this unexecuted until you run it on your clone.

Create PROMOTION.yaml and empty evidence/ files without STATUS: pass.
Run python3 promotion_gate.py. Confirm it exits 1.
Put http://127.0.0.1:8080 into clients/demo.env. Run it again. Confirm the host scan fails.
Remove the lab URL. Fill each evidence file with a STATUS: pass line and the commands you actually ran.
Run the checker again. Only then allow the deploy job to see the artifact.

If step 2 passes, the checker is wrong. A first run with empty evidence must fail.

Limitations

This checklist does not prove load, correctness, or latency. It does not replace threat modeling, IAM review, or a real staging environment. String search will miss encoded hostnames, image layers, and secrets injected at runtime. STATUS: pass is only as honest as the person who wrote it.

Free servers and free model access also have limits you cannot inventory from a YAML file: capacity, tenancy, retention, and how long the option remains available. Do not treat a lab uptime number as a production SLO. Do not treat a convenience default as a vendor contract.

Who should not use this

Skip this approach if you already promote through a locked-down path that forbids third-party assistants on any host that can see production data. Skip it if you need a compliance certificate; a repo checklist is not one. Skip it if the service never leaves the laptop. And skip it if you were about to point production DNS at the same process that runs the assistant.

In that last case the fix is not a better YAML file. The fix is a different host.

If you need a disposable lab before you run this checker, MonkeyCode's free model access and free server option are one way to keep that scratch work off the production cluster. Use it as a lab. Then make the checker fail until the lab disappears from deploy/.

Green Tests Are Not a Runtime: A Fail-Closed Production-Readiness Checklist

Taylor Zhu — Thu, 17 Sep 2026 06:06:07 +0000

Green unit tests do not mean the change can take traffic. If an AI-assisted PR exposes a route, a job, or a migration, you block merge until rollback, capacity, and probe evidence exist as files your pipeline can fail closed on.

That is the whole rule. The rest of this article is a copy-paste checklist, a schema, and a small checker that treats TBD as a red gate.

Why "tests passed" is the wrong merge signal

AI-assisted diffs arrive faster than review habits. The suite goes green because the same session that wrote the handler also wrote the mocks. You get a PR that looks finished. Production does not care.

You need evidence that is boring and named. Not a screenshot of a local 200. A file in the repo that CI can parse, and a command that actually rolls the change back.

If a field is missing, you do not negotiate. You fail closed.

What this checklist covers

This is a runtime production-readiness gate. It answers one question: if this lands on a server that receives real requests, can you stop it, bound it, and see it?

It does not replace load tests in a prod-like environment. It does not certify a security review. It does not prove the model that drafted the code was correct. It only proves that someone named the operational contract before merge.

Do not use this as a substitute for an SRE handbook you already trust. Use it when AI-authored changes start skipping the messy parts: rollback, flags, probes, and numbers with units.

The fail-closed gates

Copy these nine gates. Each gate has required evidence and a fail-closed rule. Empty strings, TODO, TBD, and n/a without a reason code all fail.

1. Rollback is a command

Evidence: ops/rollback.md contains one fenced command block tagged rollback that a pager-holder can paste.
Fail closed: no tagged block, or the block is kubectl apply -f . with no version pin, or it says "revert the PR" with no SHA strategy.

A paragraph is not a rollback. If the on-call cannot paste one command, you do not have a rollback.

2. Probes have failure semantics

Evidence: ops/probes.yml names liveness and readiness paths, success codes, and the exact condition that must flip readiness to false.
Fail closed: readiness equals liveness, or either probe hits a dependency that can deadlock the process.

Liveness answers "should we kill this process?" Readiness answers "should we send it traffic?" If those are the same URL, you will either flap or serve errors with a smile.

3. Capacity is a number with a unit

Evidence: ops/capacity.yml sets max_rps, max_payload_bytes, and max_concurrency as integers.
Fail closed: any value is 0, negative, or a string like plenty.

Models love adjectives. Production wants integers. You can be conservative. You cannot be vague.

4. Timeouts are hard deadlines

Evidence: inbound request timeout and outbound client timeout are integers in milliseconds, and inbound must exceed outbound.
Fail closed: missing timeouts, or inbound shorter than outbound.

This is not a retry policy. Retries without a deadline just move the fire. You pin the wall-clock budget for one attempt so a hung dependency cannot hold a worker forever.

5. Flags default closed

Evidence: every new path is behind a flag whose default in config/flags.yml is false.
Fail closed: default true, or no flag name on a newly exported route.

Default-on is how a "small demo" becomes Friday traffic. Keep the flag name in the PR title so reviewers can grep it.

6. Migrations reverse

Evidence: db/forward.sql and db/reverse.sql both exist, and reverse is not a comment that says "restore from backup".
Fail closed: forward-only DDL, or reverse drops data with no expand/contract note.

Backups are a disaster plan. A reverse migration is a merge requirement. If you cannot unwind the schema, you cannot unwind the release.

7. Correlation IDs on every new path

Evidence: the new handler reads or mints x-request-id and logs it as a structured field.
Fail closed: print debugging, or logs without a request key.

When the page fires, you will search one ID across gateway, app, and store. If that ID was never written, you are reading poetry.

8. On-call owner is a person-shaped string

Evidence: ops/owner.yml has team and pager fields that match your roster file.
Fail closed: owner: ai or owner: whoever-merged.

Models do not carry pagers. A team name that cannot be resolved in the roster is the same as no owner.

9. Secrets stay out of the diff

Evidence: CI runs a scan; ops/secret_scan.txt is the last clean report path.
Fail closed: any private key PEM, AKIA prefixes, or .env files in the change set.

AI assistants paste from local env files with alarming confidence. The gate does not "trust the model." It trusts a scan artifact.

How to pick the numbers without lying

You do not need a week of benchmarking to fill max_rps. You need a ceiling you will defend at 2 a.m.

Start from last week's production for the parent service, not from a model suggestion. If you have no parent, start from the smallest integer that still lets a canary through: max_concurrency: 2, max_rps: 5, max_payload_bytes equal to the documented API limit.

Write the source of the number in a comment above the field. Comments are not parsed. They are for the reviewer who will ask "why 40?"

If you cannot name a source, the number is fiction. Leave the field out and let the checker fail. That failure is cheaper than a silent 500 storm.

Evidence schema you can commit

Put this at ops/prod_gates.yml. The checker below reads it. Do not keep a second copy in a wiki.

version: 1
service: payments-api
change_id: pr-1842
rollback:
  command: |
    kubectl set image deploy/payments-api payments-api=ghcr.io/example/payments-api:sha-9f3c1a
  verified_in: staging
probes:
  liveness: /healthz
  readiness: /readyz
  readiness_fails_when: "dependency_ping != ok"
capacity:
  max_rps: 40
  max_payload_bytes: 65536
  max_concurrency: 16
deadlines_ms:
  inbound: 800
  outbound: 300
flags:
  - name: payments_api_v2_charge
    default: false
migration:
  forward: db/forward.sql
  reverse: db/reverse.sql
observability:
  request_id_header: x-request-id
owner:
  team: payments-runtime
  pager: payments-oncall
secret_scan: ops/secret_scan.txt

Those numbers are placeholders. Replace them with your service's real ceilings. The gate does not care that 40 RPS is right. It cares that you typed an integer you are willing to own.

Keep reverse SQL next to forward SQL:

-- db/forward.sql
ALTER TABLE charges ADD COLUMN IF NOT EXISTS idempotency_key TEXT;

-- db/reverse.sql
ALTER TABLE charges DROP COLUMN IF EXISTS idempotency_key;

If reverse would destroy rows, stop. Split the change into expand, migrate, contract. The checker cannot see data loss. You can.

Checker that fails closed

Save as scripts/check_prod_gates.py. Run it in CI. Treat exit code 1 as a merge block, not a warning.

#!/usr/bin/env python3
"""Fail-closed production-readiness checker. Missing or TBD fields block merge."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this gate\n")
    raise SystemExit(2)

BAD = {"", "tbd", "todo", "n/a", "none", "unknown", "ai"}


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def load(path: Path) -> dict:
    if not path.is_file():
        fail(f"missing {path}")
    data = yaml.safe_load(path.read_text()) or {}
    if not isinstance(data, dict):
        fail("prod_gates.yml must be a mapping")
    return data


def need(v, label: str) -> None:
    if v is None:
        fail(f"{label} is missing")
    if isinstance(v, str) and v.strip().lower() in BAD:
        fail(f"{label} is a placeholder")


def main() -> None:
    root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
    cfg = load(root / "ops" / "prod_gates.yml")

    rb = cfg.get("rollback") or {}
    need(rb.get("command"), "rollback.command")
    cmd = str(rb.get("command"))
    if "revert the pr" in cmd.lower() and "sha" not in cmd.lower():
        fail("rollback.command must pin an image tag or git SHA")

    probes = cfg.get("probes") or {}
    need(probes.get("liveness"), "probes.liveness")
    need(probes.get("readiness"), "probes.readiness")
    if probes.get("liveness") == probes.get("readiness"):
        fail("readiness must not equal liveness")
    need(probes.get("readiness_fails_when"), "probes.readiness_fails_when")

    cap = cfg.get("capacity") or {}
    for key in ("max_rps", "max_payload_bytes", "max_concurrency"):
        val = cap.get(key)
        if not isinstance(val, int) or val <= 0:
            fail(f"capacity.{key} must be a positive int")

    dl = cfg.get("deadlines_ms") or {}
    inbound, outbound = dl.get("inbound"), dl.get("outbound")
    if not isinstance(inbound, int) or not isinstance(outbound, int):
        fail("deadlines_ms.inbound and outbound must be ints")
    if inbound <= 0 or outbound <= 0:
        fail("deadlines must be positive")
    if inbound <= outbound:
        fail("inbound deadline must exceed outbound deadline")

    flags = cfg.get("flags") or []
    if not flags:
        fail("flags must list at least one default-closed flag")
    for flag in flags:
        need(flag.get("name"), "flag.name")
        if flag.get("default") is not False:
            fail(f"flag {flag.get('name')} must default to false")

    mig = cfg.get("migration") or {}
    for key in ("forward", "reverse"):
        rel = mig.get(key)
        need(rel, f"migration.{key}")
        if not (root / str(rel)).is_file():
            fail(f"migration.{key} file missing: {rel}")

    obs = cfg.get("observability") or {}
    need(obs.get("request_id_header"), "observability.request_id_header")

    owner = cfg.get("owner") or {}
    need(owner.get("team"), "owner.team")
    need(owner.get("pager"), "owner.pager")

    scan = cfg.get("secret_scan")
    need(scan, "secret_scan")
    if not (root / str(scan)).is_file():
        fail(f"secret_scan report missing: {scan}")

    print("PASS: production-readiness gates are named and parseable")


if __name__ == "__main__":
    main()

Run it locally until it fails for the right reason:

python3 -m pip install pyyaml
python3 scripts/check_prod_gates.py .

Catch leftover fiction before CI does:

grep -nEi 'tbd|todo|n/a|plenty|revert the pr' ops/prod_gates.yml && echo 'placeholders found' && exit 1

Makefile target you can teach the team in one line:

.PHONY: prod-gates
prod-gates:
    python3 scripts/check_prod_gates.py .

CI sketch:

# .github/workflows/prod-gates.yml
name: prod-gates
on: [pull_request]
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python3 scripts/check_prod_gates.py .

If the workflow is continue-on-error: true, you do not have a gate. You have a diary.

Decision table for reviewers

Signal in the PR	Merge?	Why
Tests green, `ops/prod_gates.yml` missing	No	No runtime contract
Gates file present, `rollback.command` is prose	No	Cannot execute a paragraph
Capacity numbers copied from another service	No until edited	Copied ceilings are fiction
Flag default true "because demo"	No	Demo defaults leak
Reverse migration is "restore backup"	No	Disaster plan, not a reverse
Checker PASS, load test still pending	Yes, behind the flag only	Gate is necessary, not sufficient

Paste this table into the PR template. Reviewers should not re-litigate the rules in comments. The file either parses or it does not.

Where a scratch environment helps

You want the checker to run somewhere that is not your laptop and not production. A short dry run catches YAML that looks valid and still fails the integer rules.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a throwaway box plus model access to draft ops/prod_gates.yml from a diff and then execute scripts/check_prod_gates.py, MonkeyCode's free model access and free server option can host that loop. Do not treat that box as staging. Do not paste production secrets into it. Use it to get a red/green signal on the evidence files before you open the PR.

Limitations

The checker believes files. A determined author can write max_rps: 40 with no benchmark behind it. Fail-closed schema is not a load test. It is an honesty prompt with teeth.

Placeholder detection is string-based. max_rps: 999999 passes. You still need a human who knows the service.

Probe paths are not probed. The script does not HTTP-get /readyz. Hook that in staging separately.

Deadline math is local. It does not see the mesh timeout in front of you. If the proxy is 200ms and you wrote 800ms inbound, production will disagree. Add the proxy number to the YAML when you know it.

Who should not use this

Skip this approach if you already enforce production contracts in a service catalog with real owners and automated rollbacks. You would be duplicating a stronger system.

Skip it if your compliance process requires signed evidence in a GRC tool. A YAML file in git is not that artifact.

Skip it if the change cannot receive traffic — docs-only, comment-only, or lockfile bumps with no runtime surface. Forcing nine gates on a README change trains people to ignore gates.

Do not use a scratch-server path for regulated data, customer payloads, or production credentials. The checklist assumes public-ish metadata: probe paths, integer ceilings, flag names.

What you do tomorrow morning

Pick one service that recently merged an AI-assisted route. Add ops/prod_gates.yml with honest numbers, even if they are conservative. Run the checker locally until it fails for the right reasons, then until it passes.

Then attach the workflow. The first red CI is the point. After that, "tests passed" stops being the whole story, and traffic gets a named budget before it arrives.

Name the Mutation or Don't Merge: A Fail-Closed Side-Effect Checklist

Taylor Zhu — Wed, 16 Sep 2026 04:13:54 +0000

If a pull request can mutate anything outside the process, you do not merge it until every write has an owner, a dry-run, an idempotency key, and a fail-closed abort. Green tests on fixtures are not that evidence. AI-assisted diffs hide mutations in helpers, generated clients, and "temporary" scripts that survive review.

This is a copy-paste gate, not a vibe. You inventory writes, attach an evidence pack, and block merge when the pack is incomplete. The checklist below is for services that can touch databases, queues, files, webhooks, mail, or money.

Why tests miss the write

Unit tests mock the client. Integration tests hit a disposable schema. Neither proves what happens when the same code runs against a shared queue or a live billing endpoint. AI tools are especially good at making the happy path compile. They are not responsible for the side effect.

You need a different question than "does it pass?" Ask: what can this diff mutate, and what happens if that mutation runs twice?

What counts as a mutation

Treat any of the following as a write surface. If you cannot name it, you cannot merge it.

SQL or ORM INSERT / UPDATE / DELETE, including migrations that backfill rows
Queue publish, stream append, scheduled job enqueue
HTTP POST / PUT / PATCH / DELETE to a system you do not own
File, object-store, or cache writes that other services read
Mail, SMS, webhook, or notification dispatch
Billing, quota, or feature-flag mutations
Secret or config writes (vault, SSM, .env generation)

Reads that can trigger lazy writes (cache fill, analytics fire-and-forget, "log this to the warehouse") count too. If the process can change the world, it is a mutation.

Gate 1: Inventory before review

Do not start the human review until the PR names every write surface. Put the inventory in the description, not in a chat thread.

## Side-effect inventory
| Surface | Direction | Owner | Dry-run | Idempotency | Abort |
|---|---|---|---|---|---|
| `billing.capture` | POST /v1/charges | payments | `--dry-run` flag | `Idempotency-Key` | fail closed if key missing |
| `jobs.enqueue(invoice)` | queue | billing-worker | local redis only | job id = invoice_id | no enqueue if dry-run |

Fail closed if the table is empty and the diff still contains write verbs. Empty because "nothing writes" is allowed only after a scanner says the same thing.

A scanner you can run locally

Label this as a starting template, not a complete static analyzer. Run it on the changed files. Extend the patterns for your language. Exit non-zero when a write verb has no matching inventory row.

#!/usr/bin/env bash
# sidefx-scan.sh — fail closed if write-like tokens appear without inventory
set -euo pipefail
ROOT="${1:-.}"
INVENTORY="${2:-SIDEFX.md}"
PATTERNS='INSERT |UPDATE |DELETE |\.create\(|\.save\(|\.destroy\(|enqueue\(|publish\(|put_object|POST |PATCH |send_email|capture\('

if [[ ! -f "$INVENTORY" ]]; then
  echo "FAIL: $INVENTORY missing" >&2
  exit 2
fi

hits=$(git diff --unified=0 origin/main...HEAD -- "$ROOT" \
  | grep -E '^\+' | grep -Ev '^\+\+\+' | grep -E "$PATTERNS" || true)

if [[ -z "$hits" ]]; then
  echo "OK: no write-like tokens in diff"
  exit 0
fi

missing=0
while IFS= read -r line; do
  token=$(echo "$line" | grep -Eo 'enqueue\(|publish\(|capture\(|put_object|send_email|INSERT |UPDATE |DELETE ' | head -n1 || true)
  if [[ -n "$token" ]] && ! grep -Fqi "$token" "$INVENTORY"; then
    echo "UNOWNED: $line" >&2
    missing=1
  fi
done <<< "$hits"

if [[ "$missing" -ne 0 ]]; then
  echo "FAIL: write token not named in $INVENTORY" >&2
  exit 1
fi
echo "OK: every write-like token is named"

Wire it as a required check. Do not let the model "fix" the inventory by deleting the table. The inventory is the contract.

# .github/workflows/sidefx.yml
name: side-effect-gate
on: pull_request
jobs:
  inventory:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: bash scripts/sidefx-scan.sh . SIDEFX.md

Gate 2: Evidence pack per surface

For each row, attach four artifacts. Missing any one is a no-merge.

Dry-run path. A flag, header, or environment that executes the code without committing the write. Print what would happen. If the library cannot dry-run, wrap it and fail closed when the wrapper is absent in staging.
Idempotency key. A stable key derived from the business id, not from uuid4() at call time. Replay must not double-charge, double-mail, or double-enqueue.
Audit line. One structured log or table row that records actor, surface, key, and outcome. Logs that dump payloads with secrets fail this gate.
Abort. If the key, owner, or dry-run flag is missing, the process stops. Retries do not invent a key. Fallbacks do not switch to a live endpoint.

Proposed helper (unexecuted example — adapt to your stack):

# proposed: fail-closed write wrapper
from dataclasses import dataclass

@dataclass(frozen=True)
class WriteGate:
    surface: str
    owner: str
    dry_run: bool
    idempotency_key: str

    def allow(self) -> None:
        if not self.owner:
            raise RuntimeError(f"{self.surface}: owner required")
        if not self.idempotency_key:
            raise RuntimeError(f"{self.surface}: idempotency key required")
        if self.dry_run:
            return

def mutate(gate: WriteGate, fn):
    gate.allow()
    if gate.dry_run:
        print({"would_call": gate.surface, "key": gate.idempotency_key})
        return {"status": "dry-run"}
    return fn(gate.idempotency_key)

You review the wrapper, not the model's confidence. If the PR calls the raw client beside the wrapper, that is an unowned mutation.

Gate 3: Staging proof, not fixture proof

Fixtures lie about fan-out. Before merge, run the write path once in a staging environment that cannot reach production credentials.

Minimum proof:

Dry-run output captured in the PR (command + snippet, not a screenshot of a chat)
One replay with the same idempotency key showing no second mutation
Abort test: unset the key or the owner and show the process exits non-zero
Credential check: production hostnames and live keys are absent from staging config

# proposed staging probe — replace endpoints with yours
export DRY_RUN=1
export IDEMPOTENCY_KEY="invoice:1842"
./bin/app enqueue-invoice --id 1842
# expect: would_call=jobs.enqueue key=invoice:1842

unset IDEMPOTENCY_KEY
if ./bin/app enqueue-invoice --id 1842; then
  echo "FAIL: abort did not fire" >&2
  exit 1
fi

If you cannot run staging, you do not merge a new write surface. Local Docker is acceptable only when it uses the same wrapper and cannot resolve production DNS.

Decision table you can paste into the PR template

Question	Pass evidence	Fail closed when
What mutates?	Inventory table matches scanner hits	Diff has write tokens, table omitted
Who owns it?	Named team or on-call in the row	"TBD" or model-generated owner
Can it dry-run?	Command + output in the PR	Flag ignored in non-dev env
Is replay safe?	Same key, second call is a no-op	New UUID per attempt
Does abort work?	Unset key → non-zero exit	Catch-and-continue
Can it reach prod?	Staging deny-list / pinned roots	Prod URL in default config
Are secrets quiet?	Redacted audit line	Token or PAN in logs

Keep the table short. If a row cannot be filled, the gate is closed. Do not negotiate "just this once" for generated code. Generated code is why the gate exists.

Using an assistant without handing it the merge button

A coding assistant is useful for enumeration, not for approval. You can point it at a diff and ask it to list write surfaces you missed. You still require the scanner, the inventory, and the staging abort.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already run reviews on a laptop that should not keep paid inference on all day, MonkeyCode's free model access and free server option are enough to host that enumeration step. Feed it the diff and the inventory template. Reject any suggestion that adds a write client without a dry-run path. The server does not replace CI. It is a place to draft the table before the required check runs.

Do not paste production secrets into that session. Do not let the assistant "complete" the abort test by deleting it.

Limitations

This checklist does not prove correctness of business logic. It does not replace load tests, schema review, or legal hold. The scanner is pattern-based; a renamed helper will dodge it until you update the patterns. Idempotency keys do not help if two surfaces share no common id.

Who should not use this as written:

Docs-only or comment-only PRs (skip the gate; do not fake an inventory)
Throwaway prototypes with no shared data and no network writes
Teams without a staging environment that is actually isolated — fix that first
Pipelines that cannot fail closed (warnings-only checks train people to ignore them)

If your agent can call tools, pin those contracts separately. This article is only about mutations that escape the process after merge.

What you do on Monday

Add SIDEFX.md to the repo. Add the scanner as a required status. Paste the decision table into the PR template. For the next AI-assisted change that touches a queue or an HTTP client, refuse merge until the four artifacts exist.

You are not slowing the model down. You are refusing to ship an unowned write. That is the whole job.

If you want a second pass over a noisy diff, run the inventory draft on MonkeyCode's free server with free model access, then throw away anything that cannot survive the scanner.

Every Gate Leaves a Receipt: A Ship-Readiness Matrix for AI-Assisted PRs

Taylor Zhu — Mon, 14 Sep 2026 21:48:33 +0000

A checklist that produces no evidence is a wish with bullet points. When a model writes most of a diff, the review question stops being "does this code look right?" and becomes "did the checks that would catch this class of bug actually run, on this commit, recently?" Those are different questions, and only the second one is answerable from files on disk.

This post gives you a copyable gate matrix, a ~60-line checker that fails closed, and a negative test plan that proves the checker actually blocks. Every part of it is useful with zero AI in the loop.

The failure shape

The scenario below is illustrative, not a specific incident. The shape, though, is common.

An agent opens a pull request. CI is green. The green run reused a cached dependency directory, skipped the integration suite because a path filter stopped matching months ago, and never ran the migration dry-run at all. Nothing in the PR says which checks were skipped. Green is green, so you merge. The first real signal arrives three days later in production.

The fix is not a better prompt. It is making every gate emit a receipt: a small JSON file bound to a commit SHA, containing the command that produced it, the result, and a timestamp. A checker then fails closed when a receipt is missing, stale, tampered with, or belongs to a different commit.

Gate classes: hard, soft, advisory

Not every check deserves the same blocking power. Split them:

Hard gates block merge. Missing receipt means no merge — the override, if you allow one, is a separate logged commit, not a checkbox.
Soft gates require a named human acknowledgement attached to the PR. They still emit a receipt, but status: "ack" is acceptable.
Advisory gates report only. Their purpose is to generate data until you trust them enough to promote them.

The matrix

Gate	Question it answers	Receipt must contain	Fail-closed rule
`tests.unit`	Does the changed function still do what its name says?	commit, cmd, status, finished_at	missing, or older than 2h → block
`tests.integration`	Do the service boundaries still hold?	commit, cmd, status, run_id	skipped by path filter → block
`deps.lockfile`	Did the resolved dependency graph change in this diff?	before/after lockfile hash	hash changed with no note → block
`migration.dryrun`	Did the schema change run against prod-shaped data?	cmd, row counts before/after, status	dry-run absent → block
`secrets.scan`	Did the diff introduce a credential?	scanner name + version, commit	scanner older than policy → block
`rollback.proof`	Can you undo this inside the agreed window?	revert commit, measured time	no revert commit → block
`owner.ack`	Did whoever owns this path read the diff?	name, commit, timestamp	ack predates last push → block

Two rules carry the whole design. The receipt is bound to the SHA, and the checker reads receipts rather than the CI badge.

The artifact: a receipt checker

Node 18+, no dependencies. Save as shipgate.mjs.

#!/usr/bin/env node
// shipgate.mjs — fail closed unless every required gate left a fresh receipt.
import { readFileSync, readdirSync } from "node:fs";
import { join } from "node:path";

const DIR = process.env.RECEIPT_DIR ?? "artifacts/receipts";
const HEAD = process.env.GIT_SHA ?? "";

const REQUIRED = [
  { id: "tests.unit",        maxAgeH: 2  },
  { id: "tests.integration", maxAgeH: 6  },
  { id: "secrets.scan",      maxAgeH: 24 },
  { id: "deps.lockfile",     maxAgeH: 24 },
  { id: "migration.dryrun",  maxAgeH: 24 },
  { id: "rollback.proof",    maxAgeH: 24 },
];

function load(dir) {
  const byGate = new Map();
  for (const file of readdirSync(dir)) {
    if (!file.endsWith(".json")) continue;
    try {
      const r = JSON.parse(readFileSync(join(dir, file), "utf8"));
      if (r && typeof r.gate === "string") byGate.set(r.gate, r);
    } catch {
      // An unparseable receipt is a missing receipt.
    }
  }
  return byGate;
}

const receipts = load(DIR);
const blockers = [];
const now = Date.now();

for (const { id, maxAgeH } of REQUIRED) {
  const r = receipts.get(id);
  if (!r) { blockers.push(`${id}: no receipt`); continue; }
  if (!r.commit || r.commit !== HEAD) {
    blockers.push(`${id}: receipt is for ${String(r.commit).slice(0, 8)}, HEAD is ${HEAD.slice(0, 8)}`);
  }
  if (r.status !== "pass") blockers.push(`${id}: status=${r.status}`);
  if (!r.cmd) blockers.push(`${id}: no command recorded, so it is not reproducible`);
  const ageH = (now - Date.parse(r.finished_at)) / 36e5;
  if (!Number.isFinite(ageH) || ageH > maxAgeH) {
    blockers.push(`${id}: receipt is ${Math.round(ageH)}h old (max ${maxAgeH}h)`);
  }
}

if (blockers.length) {
  console.error("SHIP GATE: BLOCKED");
  for (const b of blockers) console.error("  - " + b);
  process.exit(1);
}
console.log(`SHIP GATE: PASS — ${REQUIRED.length} receipts verified against ${HEAD.slice(0, 8)}`);

Emitting a receipt from CI

Wrap the command; never let a failing step write pass.

emit_receipt() {
  local gate="$1" status="$2" cmd="$3"
  mkdir -p artifacts/receipts
  jq -n --arg gate "$gate" --arg commit "${GIT_SHA:?}" \
        --arg status "$status" --arg cmd "$cmd" \
        --arg finished_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
        '{gate:$gate, commit:$commit, status:$status, cmd:$cmd, finished_at:$finished_at}' \
    > "artifacts/receipts/${gate}.json"
}

set -euo pipefail
npm ci --ignore-scripts

if npm run test:unit; then
  emit_receipt tests.unit pass "npm run test:unit"
else
  emit_receipt tests.unit fail "npm run test:unit"
  exit 1
fi

Prove the checker blocks

An untested gate is a gate you do not have. Four commands, run locally:

node shipgate.mjs; echo "baseline, expect 0, got $?"

cp artifacts/receipts/tests.unit.json /tmp/r.json
rm artifacts/receipts/tests.unit.json
node shipgate.mjs; echo "missing receipt, expect 1, got $?"

jq '.commit="deadbeef"' /tmp/r.json > artifacts/receipts/tests.unit.json
node shipgate.mjs; echo "wrong SHA, expect 1, got $?"

jq '.finished_at="2020-01-01T00:00:00Z"' /tmp/r.json > artifacts/receipts/tests.unit.json
node shipgate.mjs; echo "stale receipt, expect 1, got $?"

mv /tmp/r.json artifacts/receipts/tests.unit.json

The baseline case matters as much as the failures. If the checker never exits 0, your team will route around it within a week.

Debugging a blocked gate in fifteen minutes

Read the blocker line, not the log. Each line names a gate and a reason: missing, stale, wrong SHA, or non-pass.
Wrong SHA? The gate ran, but on an older commit. Re-run it on HEAD instead of re-emitting a receipt with a patched SHA — the second option turns your gate into decoration.
Stale? Look at the age limit for that gate. Unit tests get 2 hours because they are cheap. Integration tests get 6. Raise the limit only with a reason you would write in the PR.
Missing? Check the CI path filter first. Filters outlive the file layout they were written for.
Everything passes locally, nothing passes in CI? Reproduce in a clean container before touching the checker.

Where a hosted free tier fits

Step 5 is where a scratch environment earns its place. MonkeyCode is an open-source project that, per its own outreach material, provides free model access and a free server option, with an advertised allowance of 10 million tokens. Treat those figures as vendor-stated and verify the current terms and quotas yourself before you build anything around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two places it genuinely helps this workflow, and one place it does not:

Scaffolding the boring part. Hand a model your repo layout and the matrix above, and ask it to generate the REQUIRED list plus the per-gate age limits. You still review every line; the value is skipping the blank-page step.
A clean box for step 5. When a gate blocks and your laptop cannot reproduce the failure, run the checker and the failing command in a fresh hosted environment instead of debugging a polluted local cache.
Not a replacement for CI. A hosted free tier has no required-status-check integration, no build provenance, and no promise that it will be there next quarter. Keep the authoritative run inside your own pipeline.

Do not paste production data, credentials, or customer records into shared infrastructure. If a gate needs prod-shaped data, generate a synthetic fixture and record the fixture version in the receipt.

Who should not use this

Solo prototypes. Seven gates for a weekend project is ceremony. Start with tests.unit and rollback.proof.
Repos without merge protection. A checker that nothing is required to run is documentation theatre. Wire it in as a required status check or skip the whole idea.
Teams with no prod-shaped test data. A migration.dryrun receipt against an empty database is worse than no receipt, because it looks like evidence.
Anyone who needs the free tier to be permanent. Plan for it to change; the matrix does not depend on it.

The one thing to take away

If a gate cannot produce a file that names the commit it ran against, you are not enforcing a gate — you are trusting a colour. Start with two gates, make the checker exit non-zero, and test that it blocks before you trust a single green run.

If you want to try the scaffolding and the clean-box steps end to end, the project's repository and its current free-tier documentation are the place to start — and measure the checker's real cost on your own pipeline before you add gate number three.

Cap the Retry: A Fail-Closed Loop Budget Checklist

Taylor Zhu — Sun, 13 Sep 2026 20:28:18 +0000

If an agent can retry a write, three numbers must live in the repo before you merge: a hard iteration cap, a wall-clock cap, and a side-effect budget. Missing any one of them is a fail-closed event, not a warning. Soft limits in a README do not count.

You already review the prompt. You already argue about the model. That is not the production problem. The production problem is an unbounded loop that talks to a queue, a billing API, or a ticket system and then tries again.

Why this checklist exists

Agent loops fail in a boring way. They do not always hallucinate a function name. They retry.

A tool call times out. The process dies mid-JSON. The orchestrator restarts the same step. You now have two refunds, two deploys, or two “urgent” issues with the same title. The model did not need to be clever. It needed a ceiling.

If you cannot point at the ceiling in git, you do not merge.

What “unbounded” looks like in a PR

Scan the diff for these shapes. Any one of them is enough to block.

while True / for _ in range(999) around a tool dispatcher
retries with no idempotency key on a non-GET side effect
a timeout that only lives in a chat message, not in config
a “max steps” comment that the runtime never reads
a cron or webhook that re-enters the same agent with no run id

You do not need a research paper to classify this. You need a file the CI job can parse.

The merge rule

Treat loop configuration as a production contract. The PR must add or update that contract. The checker must fail closed. Reviewers do not get to “remember to watch it in staging.”

Copy this as the team rule:

Iteration cap — integer, required, runtime-enforced, below the team ceiling.
Wall-clock cap — seconds, required, includes tool I/O, not just model time.
Side-effect budget — per non-idempotent tool, required, counted on attempt not on success.
Retry identity — every retry of a write carries a stable idempotency key.
Dry-run fixture — one recorded transcript that hits the caps without touching prod.

If the evidence is missing, the gate returns non-zero. Warnings are for linters. This is a merge gate.

Gate 1: Iteration cap

An iteration is one model turn plus the tool batch it emitted. Say that in the contract. Do not let “step” mean three different things in three files.

Fail closed when:

max_iterations is absent, zero, or above the team ceiling
the runtime reads a different key than the contract (MAX_STEPS vs max_iterations)
the cap is only applied in “strict mode”

Evidence you attach to the PR:

the contract file path
the runtime log line that prints the remaining iterations
a unit test that asserts the loop stops on the cap, not on an exception

A cap that is not tested is a comment.

Gate 2: Wall-clock cap

Model latency is not the budget. Tool I/O is the budget. A “30s LLM timeout” with a five-minute HTTP client is how you page the on-call.

Fail closed when:

there is no max_wall_clock_seconds
the HTTP client timeout is greater than the remaining wall clock
a child process can outlive the parent agent

You want one clock. Process start is t0. Every tool call checks now - t0. When the remaining time cannot cover the next client timeout, you stop. You do not start a write you cannot bound.

Gate 3: Side-effect budget

Reads can be chatty. Writes cannot. Classify every tool as read, idempotent_write, or unsafe_write. Only the last two consume the side-effect budget, and unsafe_write consumes it on attempt.

That last clause matters. A timeout after the server applied the write still spent the budget. If you count only HTTP 200, you will double-apply on retry.

Fail closed when:

a tool that creates, charges, pages, deploys, or emails is not classified
unsafe_write has no budget
the budget resets inside a single run
classification lives in prose instead of the contract

Gate 4: Idempotency keys on retries

Retries without identity are duplicate work. You generate the key before the first attempt and reuse it. The key is a function of run_id + step_name + input_hash. It is not a random UUID minted inside the retry loop.

Fail closed when:

unsafe_write retries with a fresh UUID
the provider’s idempotency header is optional
a crashed worker restarts the step with a new run_id and no recovery map

If the downstream API has no idempotency support, the tool is not retryable. Put it behind a human approval or a single-shot executor. Do not pretend a sleep-and-repeat loop is safe.

Gate 5: Dry-run fixture must pass

A contract you never execute is a wish. You keep one fixture that drives the real loop code against recorded tool responses. The fixture must prove three stops: iteration cap, wall-clock cap, and side-effect budget. All three. One happy-path transcript is not enough.

Fail closed when:

there is no fixture path in the contract
the fixture talks to a live network
only the happy path is asserted
the fixture is skipped on CI because “it needs a GPU”

The dry-run can run on a small CPU box. That is the point.

Copy-paste contract

Commit this as agent/loop-budget.yml. Keep it boring. Boring parses.

# agent/loop-budget.yml
version: 1
team_ceilings:
  max_iterations: 8
  max_wall_clock_seconds: 90
  max_unsafe_writes: 2
run:
  max_iterations: 6
  max_wall_clock_seconds: 60
  max_unsafe_writes: 1
  dry_run_fixture: tests/fixtures/loop_budget_dryrun.jsonl
tools:
  search_docs:
    class: read
  upsert_ticket:
    class: idempotent_write
    idempotency_header: Idempotency-Key
  charge_customer:
    class: unsafe_write
    retryable: false
  page_oncall:
    class: unsafe_write
    retryable: false

If a new tool lands without a class, the checker fails. Unknown is not read.

The checker

This script is the artifact. Run it locally, then in CI. It does not call a model. It refuses to guess.

#!/usr/bin/env python3
"""Fail closed if the agent loop contract is missing, over-ceiling, or untested."""
from __future__ import annotations

import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit  = 2
    raise SystemExit(2)

ROOT = Path(__file__).resolve().parents[1]
CONTRACT = ROOT / "agent" / "loop-budget.yml"
ALLOWED = {"read", "idempotent_write", "unsafe_write"}


def die(msg: str) -> None:
    sys.stderr.write(f"FAIL-CLOSED: {msg}\n")
    raise SystemExit(1)


def load_contract() -> dict:
    if not CONTRACT.is_file():
        die(f"missing {CONTRACT}")
    data = yaml.safe_load(CONTRACT.read_text()) or {}
    if data.get("version") != 1:
        die("unsupported contract version")
    return data


def require_int(node: dict, key: str, hi: int) -> int:
    if key not in node:
        die(f"missing {key}")
    val = node[key]
    if not isinstance(val, int) or isinstance(val, bool) or val < 1:
        die(f"{key} must be a positive int")
    if val > hi:
        die(f"{key}={val} exceeds team ceiling {hi}")
    return val


def check_tools(tools: dict) -> None:
    if not tools:
        die("tools map is empty")
    for name, spec in tools.items():
        if not isinstance(spec, dict):
            die(f"tool {name} is not a map")
        klass = spec.get("class")
        if klass not in ALLOWED:
            die(f"tool {name} has unknown class {klass!r}")
        if klass == "idempotent_write" and not spec.get("idempotency_header"):
            die(f"tool {name} retries writes without an idempotency header")
        if klass == "unsafe_write" and spec.get("retryable", False):
            die(f"tool {name} is unsafe_write and must not be retryable")


def check_fixture(path_str: str, tools: dict) -> None:
    path = ROOT / path_str
    if not path.is_file():
        die(f"dry-run fixture missing: {path}")
    stops = {"iteration_cap": 0, "wall_clock_cap": 0, "side_effect_budget": 0}
    unsafe = {n for n, s in tools.items() if s.get("class") == "unsafe_write"}
    for line_no, line in enumerate(path.read_text().splitlines(), 1):
        if not line.strip():
            continue
        try:
            ev = json.loads(line)
        except json.JSONDecodeError as exc:
            die(f"fixture line {line_no}: {exc}")
        kind = ev.get("stop_reason")
        if kind in stops:
            stops[kind] += 1
        if ev.get("network") is True:
            die(f"fixture line {line_no} sets network=true")
        tool = ev.get("tool")
        if tool in unsafe and ev.get("attempted") and ev.get("retry_with_new_key"):
            die(f"fixture line {line_no} retries unsafe_write with a new key")
    missing = [k for k, n in stops.items() if n < 1]
    if missing:
        die("fixture never demonstrated stops: " + ", ".join(missing))


def main() -> None:
    data = load_contract()
    ceilings = data.get("team_ceilings") or {}
    run = data.get("run") or {}
    tools = data.get("tools") or {}
    hi_iter = require_int(ceilings, "max_iterations", 10**6)
    hi_wall = require_int(ceilings, "max_wall_clock_seconds", 10**6)
    hi_writes = require_int(ceilings, "max_unsafe_writes", 10**6)
    require_int(run, "max_iterations", hi_iter)
    require_int(run, "max_wall_clock_seconds", hi_wall)
    require_int(run, "max_unsafe_writes", hi_writes)
    fixture = run.get("dry_run_fixture")
    if not fixture:
        die("run.dry_run_fixture is required")
    check_tools(tools)
    check_fixture(str(fixture), tools)
    print("loop-budget: ok")


if __name__ == "__main__":
    main()

Install and run:

pip install pyyaml
python tools/check_loop_budget.py
echo $?
# 0 means the contract and fixture both exist and close the loop.

Label the snippet as a starting checker, not a production agent. It does not execute tools. It only refuses missing evidence.

Fixture shape

Keep the dry-run as JSONL so you can grep it in review.

{"stop_reason":"iteration_cap","iterations":6,"network":false}
{"stop_reason":"wall_clock_cap","elapsed_ms":60010,"network":false}
{"stop_reason":"side_effect_budget","tool":"charge_customer","attempted":true,"retry_with_new_key":false,"network":false}

Three lines. Three stops. If a later PR deletes a line to “clean up noise,” CI goes red. That is the desired failure.

CI gate

Wire it so main cannot move without the checker. Example GitHub Actions job:

name: loop-budget
on:
  pull_request:
    paths:
      - "agent/**"
      - "tools/check_loop_budget.py"
      - "tests/fixtures/loop_budget_dryrun.jsonl"
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python tools/check_loop_budget.py

Path filters are optional. If your agents live all over the repo, drop the paths: block and run it on every PR. Cheap checks should be loud.

Decision table

Use this in review. If two people disagree, they are arguing the table, not vibes.

Signal in the PR	Evidence required	Merge if missing?
New tool that writes	`class` + budget impact	No
Retry wrapper added	Stable idempotency key	No
Timeout only on the model client	Wall-clock covers tools	No
`max_iterations` raised	Still `<= team_ceilings` + fixture updated	No
Fixture talks to the network	Recorded responses only	No
Unsafe write marked retryable	Redesign or human approval	No
Docs say “be careful”	None. Docs are not a gate	No

Print the table in the PR template if your team skips YAML. The checker still wins.

Where a free model box fits

You still need a place to exercise the real loop against the fixture when the unit checker is not enough. That is a dry-run host, not a production identity.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a throwaway environment that is not your laptop and not prod, MonkeyCode’s free model access and free server option can host that dry-run. Keep the same contract file. Point the runtime at recorded tools first. Only then let a free model fill the “what would the next thought be” slot. Do not point that box at live billing APIs. The gate above exists so a convenient model host cannot become an unbounded worker.

That is optional. The checklist stands if you never touch the product.

What this does not prove

This gate does not prove the agent is correct. It does not prove the model followed policy. It does not prove the tool schemas are right. It only proves the loop cannot run forever, cannot ignore the clock, and cannot retry unsafe writes as if they were reads.

It also does not replace:

authz on the tools themselves
egress allowlists
secret scanning of prompts and traces
a rollback plan for the writes that did land

If you already have those gates, keep them. Stack this one next to them. Do not fold everything into a single “agent linter” that nobody trusts.

Who should skip this

Do not use this approach if any of the following is true:

the “agent” is a single model call with no tools and no retry
a cluster autoscaler already kills the job at a hard wall clock and the job cannot emit side effects after SIGTERM
you are in a notebook proving an idea, with credentials that cannot reach prod
every write already goes through a human approval queue with no automatic retry

If you are in that last bucket, a YAML contract is ceremony. Stay there until someone adds a retry.

Ship the ceiling

Put agent/loop-budget.yml in the PR that introduces the loop, not in a follow-up. Run the checker on that PR. Attach the fixture that demonstrates all three stops. If a reviewer says “we will monitor it,” that is a no.

Unbounded retries are not an AI problem. They are a missing number in git. Cap the retry, or do not merge.

If It Can Write, Pin the Contract: A Fail-Closed Agent Tool Checklist

Taylor Zhu — Sat, 12 Sep 2026 18:28:31 +0000

If an agent can call a tool that writes, deletes, pays, or talks to a network, you do not merge on a demo. You merge on a pinned tool contract, a budget, and evidence that the run stayed inside both. Everything else is a live experiment wearing a pull request.

Tool calling is the part of agent work that actually changes production. Prompts do not. Completions do not. A JSON blob that becomes rm, transfer, or deploy does. Treat that blob as an API, not as a suggestion.

This checklist is for teams shipping agents that invoke tools—local functions, HTTP actions, or MCP-style servers. Copy it. Fail closed. Do not soften a red gate because the trace “looked reasonable.”

What you are actually shipping

You are not shipping a chat loop. You are shipping a remote-control surface with a probabilistic driver.

That surface needs the same things any other production API needs: a schema, a version, an owner, a timeout, a retry policy, and a recorded decision about side effects. If you cannot point to those artifacts in the repo, you do not have a product. You have a script that sometimes works.

Write the contract first. Then let the model talk to the contract. Never the other way around.

Gate 0 — Inventory before intelligence

List every tool the agent may invoke in this change. If the list is generated at runtime from a directory scrape, a plugin folder, or “whatever the MCP server advertised today,” the gate fails.

Required evidence:

A committed tools/inventory.yaml with a frozen name set
A content hash of each tool descriptor
An owner and on-call path per tool
A side-effect class: read, write, delete, network, payment, or deploy

Fail closed when:

A runtime tool name is missing from inventory
A descriptor hash drifted without a contract bump
Any tool has side-effect class unknown

Unknown is not a category. Unknown is a merge blocker.

Gate 1 — Pin the contract, not the vibe

Every inventoried tool gets a contract file. One file per tool. No “see README.” No “the model knows the shape.”

# tools/contracts/create_invoice.contract.yaml
name: create_invoice
version: 3
owner: billing-platform
side_effects: [write, network]
allow_write: true
timeout_ms: 4000
max_calls_per_run: 2
retry:
  max: 1
  backoff_ms: 250
  retry_on: [timeout]
idempotency_key_required: true
input_schema:
  type: object
  additionalProperties: false
  required: [account_id, amount_cents, currency, idempotency_key]
  properties:
    account_id: { type: string, pattern: "^acc_[a-z0-9]{8,}$" }
    amount_cents: { type: integer, minimum: 1, maximum: 50000000 }
    currency: { type: string, enum: [USD, EUR] }
    idempotency_key: { type: string, minLength: 16, maxLength: 64 }
output_schema:
  type: object
  additionalProperties: false
  required: [status, invoice_id]
  properties:
    status: { type: string, enum: [created, duplicate, rejected] }
    invoice_id: { type: ["string", "null"] }
    error: { type: ["string", "null"] }
untrusted_output: true
forbidden_args:
  - raw_sql
  - shell
  - file_path

Fail closed when the contract is missing, when additionalProperties is true on a write tool, or when a write tool has no idempotency key. Soft schemas are how “create invoice” becomes “create invoice and also whatever field the model invented.”

Gate 2 — Arguments are data, not instructions

Validate arguments before the tool runs. Validate outputs before they re-enter the model context. Both checks are mechanical. Neither is a prompt.

You want three failures, not one vague exception:

Schema miss: wrong types, extra keys, broken enums
Policy miss: amount over limit, cross-tenant id, missing idempotency key
Shape miss on the way back: tool returned prose, a stack trace, or a field you never allowed

If output is untrusted—and tool output is untrusted—strip it down to the schema. Do not concatenate raw tool text into the next prompt. That is how a search result becomes an instruction.

Gate 3 — Budgets are merge gates, not dashboards

A loop without a budget is an unbounded job. Unbounded jobs do not belong in CI or in prod.

Pin all four limits in tools/budget.yaml:

Wall clock for the whole run
Model-call count
Tool-call count per tool and in total
Max payload bytes in and out of each tool

The run dies when any limit trips. Dying is success for the gate. Continuing “just to finish the task” is the failure mode.

Record the counters in the evidence packet. A green CI job with no counters is not green. It is unmeasured.

Gate 4 — Writes need a dry-run path you actually execute

Read tools can run against fixtures. Write tools need a dry-run that cannot mutate the real system, plus one recorded mutation against a disposable fixture.

Fail closed when:

allow_write: true and there is no dry-run implementation
Dry-run and real path share no schema
The only proof of the write is a screenshot or a chat log
The fixture is the staging database you share with humans

If you cannot replay the write against a throwaway store, you cannot review it. If you cannot review it, you do not merge it.

Gate 5 — Trace or it did not happen

Every tool call in the PR needs a trace row you can grep. Not a marketing “observability story.” A row.

Minimum columns:

run_id, step, tool, contract_version
argument digest (not raw secrets)
output digest and schema-validation result
latency_ms, retry_count, budget remaining
decision: allowed, blocked_schema, blocked_policy, blocked_budget

Fail closed when a write has no row, when the argument digest does not match the validated payload, or when the model saw a blocked output anyway. Hidden retries are still retries. Hidden writes are incidents.

Reproducible artifact: fail the build if the contract is theater

Save this as scripts/check_tool_contracts.py. It is a proposal you can run locally. It does not call a model. It only checks that the repo can tell the truth about tools.

#!/usr/bin/env python3
"""Fail closed if agent tool contracts are missing, writable, or unpinned."""
from __future__ import annotations

import hashlib, json, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("Install pyyaml before running this check.", file=sys.stderr)
    sys.exit = 2

ROOT = Path("tools")
REQUIRED = {
    "name", "version", "owner", "side_effects", "allow_write",
    "timeout_ms", "max_calls_per_run", "input_schema", "output_schema",
    "untrusted_output",
}
WRITE_CLASSES = {"write", "delete", "payment", "deploy"}


def load(path: Path):
    return yaml.safe_load(path.read_text()) or {}


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def fail(msg: str, errors: list[str]) -> None:
    errors.append(msg)


def main() -> int:
    errors: list[str] = []
    inv_path = ROOT / "inventory.yaml"
    if not inv_path.exists():
        print("missing tools/inventory.yaml", file=sys.stderr)
        return 1
    inventory = load(inv_path)
    tools = inventory.get("tools") or []
    if not tools:
        print("inventory tools[] is empty", file=sys.stderr)
        return 1

    budget_path = ROOT / "budget.yaml"
    if not budget_path.exists():
        errors.append("missing tools/budget.yaml")
    else:
        budget = load(budget_path)
        for key in ("wall_clock_ms", "max_model_calls", "max_tool_calls", "max_payload_bytes"):
            if not isinstance(budget.get(key), int) or budget[key] <= 0:
                fail(f"budget.{key} must be a positive int", errors)

    seen = set()
    for item in tools:
        name = item.get("name")
        expected_hash = item.get("descriptor_sha256")
        contract_path = ROOT / "contracts" / f"{name}.contract.yaml"
        if not name:
            fail("inventory entry missing name", errors)
            continue
        seen.add(name)
        if not contract_path.exists():
            fail(f"{name}: missing {contract_path}", errors)
            continue
        digest = sha256(contract_path)
        if expected_hash != digest:
            fail(f"{name}: hash mismatch (inventory={expected_hash} file={digest})", errors)
        spec = load(contract_path)
        missing = REQUIRED - set(spec)
        if missing:
            fail(f"{name}: missing fields {sorted(missing)}", errors)
        effects = set(spec.get("side_effects") or [])
        if effects & WRITE_CLASSES:
            if not spec.get("allow_write"):
                fail(f"{name}: write-class tool must set allow_write true", errors)
            if not spec.get("idempotency_key_required"):
                fail(f"{name}: write-class tool needs idempotency_key_required", errors)
            if spec.get("input_schema", {}).get("additionalProperties") is not False:
                fail(f"{name}: write tools must set additionalProperties false", errors)
        if spec.get("untrusted_output") is not True:
            fail(f"{name}: untrusted_output must be true", errors)
        if int(spec.get("timeout_ms") or 0) <= 0:
            fail(f"{name}: timeout_ms must be positive", errors)
        if int(spec.get("max_calls_per_run") or 0) <= 0:
            fail(f"{name}: max_calls_per_run must be positive", errors)

    extra = sorted(p.stem.replace(".contract", "") for p in (ROOT / "contracts").glob("*.contract.yaml") if p.stem.split(".")[0] not in seen)
    # filenames are name.contract.yaml so stem is name.contract
    extra = []
    for p in (ROOT / "contracts").glob("*.contract.yaml"):
        n = p.name.removesuffix(".contract.yaml")
        if n not in seen:
            extra.append(n)
    if extra:
        fail(f"contracts not in inventory: {extra}", errors)

    if errors:
        print("TOOL CONTRACT CHECK FAILED", file=sys.stderr)
        for e in errors:
            print(f"- {e}", file=sys.stderr)
        return 1
    print(json.dumps({"ok": True, "tools": sorted(seen)}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire it so a missing hash is a red build, not a warning:

pip install pyyaml
python3 scripts/check_tool_contracts.py

Add a tiny inventory so the check has something honest to chew on:

# tools/inventory.yaml
tools:
  - name: create_invoice
    descriptor_sha256: REPLACE_WITH_SHA256_OF_THE_CONTRACT_FILE
    owner: billing-platform

Compute the hash the same way CI will:

python3 - <<'PY'
from pathlib import Path
import hashlib
p = Path("tools/contracts/create_invoice.contract.yaml")
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY

Paste that digest into inventory. If someone “just tweaks” an enum, the hash moves, and merge stops. That is the point.

Decision table you can paste into the PR template

Question	Evidence	Fail closed if
Which tools can this agent call?	`tools/inventory.yaml`	Runtime discovery or globbing
Is the descriptor pinned?	SHA-256 in inventory matches file	Hash missing or drifted
Can it write?	`side_effects` + `allow_write`	Write with no idempotency key
Are args constrained?	`additionalProperties: false` + enums	Open objects on write tools
Is output hostile?	`untrusted_output: true` + schema strip	Raw tool text in next prompt
What stops the loop?	`tools/budget.yaml` counters	Any counter absent in the trace
Can you replay the write?	Dry-run + fixture mutation log	Staging-as-fixture or screenshot
Did CI see the same contracts?	checker exit code 0	Warning-only lint

If any row is “we’ll add it later,” the answer is no merge. Later is how contracts rot.

Where a free eval box helps—and where it does not

You still need the checker in your CI. Contracts live in your repo. A hosted loop does not replace Gate 0 through Gate 5.

It can help when you want a throwaway process that hammers fixture tools without burning a paid API on every contract tweak. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option; those are the only product claims here. Use them, if they fit your constraints, to run the checker and a fixture tool server in a box you can destroy. Do not use that box as production. Do not treat a successful playground run as evidence. Evidence is the inventory hash, the budget counters, and the trace rows in the PR.

If you try it, keep the same fail-closed rule: no pinned contract, no write.

Limitations

This checklist does not prove the model chose the right tool. It proves the tool could not silently change shape, retry forever, or write without a key. Those are different problems. Do not pretend a schema gate is an eval suite.

It also does not replace IAM. A contract that says account_id must match a pattern does not bind the credential the tool process holds. If the process can reach every tenant, the model only needs one bad id. Put real authz in the tool implementation. Put the contract in front of it anyway.

Hashes pin files, not behavior. A server behind an MCP endpoint can change while the local descriptor stays frozen. If you call remote tools, pin the server identity and fail on advertisement drift. Local YAML is necessary. It is not sufficient for a moving endpoint.

Who should not use this

Skip this if your agent is read-only over public docs and cannot call anything with side effects. A search loop with no tools is not this problem.

Skip it if you cannot fail the build. A checklist that reviewers “keep in mind” is not a gate. It is a blog post.

Skip it if your tools are one-off shell aliases on a laptop. Pinning contracts for a personal scratchpad is ceremony. The moment that scratchpad can charge a card or deploy, it stops being a scratchpad.

Merge rule

You merge when inventory, hashes, schemas, budgets, dry-run proof, and traces are in the packet, and the checker exits 0. You do not merge because the agent completed the happy path once.

Pin the contract. Treat output as hostile. Kill the loop on budget. If it can write, those are not style preferences. They are the difference between a tool and an incident.

Pin the Roots or Don't Merge: A Fail-Closed Agent Egress Checklist

Taylor Zhu — Fri, 11 Sep 2026 16:00:18 +0000

You should not merge an agent change until every filesystem root, network host, and executable it can reach is named in a committed allowlist. If that file is missing, the pipeline fails. That is the whole policy.

An LLM in a pull request is not the risk by itself. The risk is the tools you wired to it. Shell. HTTP. MCP. Those tools turn a suggestion into a process with credentials, a working directory, and a network stack.

This checklist is for teams shipping coding agents into shared CI or a shared box. Copy it. Fail closed. Do not treat a green unit test as proof that the agent stayed inside the repo.

What you are actually shipping

You are not shipping a prompt. You are shipping a runtime.

If the agent can run a command, it can read files the reviewer never opened. If it can fetch a URL, it can leave your VPC with a token sitting in an environment variable. If it can call an MCP server, it can grow new verbs the next time that server updates.

Name the boundary. Then prove the run stayed inside it.

The merge rule

Treat agent egress like a production firewall change:

Default deny.
Explicit allow.
Evidence on the PR.
Missing evidence fails the job.

Do not negotiate item 4 in Slack. If the packet is incomplete, the merge is incomplete.

Copy-paste checklist

Use this as a PR template. Every box needs an artifact, not a vibe.

1. Identity of the agent

[ ] agent_id is stable across PRs (not tmp or test).
[ ] Model, tool host, and MCP server versions are pinned in the same commit as the code.
[ ] A human owner is named. Bots do not own allowlists.

If the id changes, treat it as a new service. New service, new review.

2. Filesystem roots

[ ] workspace_roots lists every directory the agent may read or write.
[ ] Roots are repo-relative. No $HOME. No /tmp unless a job-scoped directory is created in CI and destroyed after.
[ ] Write roots are a subset of read roots.
[ ] Symlink escape is denied: the gate resolves paths before compare.

A root of . is acceptable for a docs bot. It is not acceptable for an agent that also mounts secrets.

3. Binaries and argv

[ ] binaries is an allowlist of executable basenames, not $PATH.
[ ] Shell wrappers (bash -lc, sh -c, python -c) are either forbidden or require a second reviewer.
[ ] Package managers (npm, pip, curl | sh) are denied unless the PR is specifically about dependency changes.

You do not need a perfect sandbox to start. You need a list you can grep.

4. Network hosts

[ ] network.mode is deny or allowlist. Never open.
[ ] Hosts are exact names, not *.cloud.
[ ] Redirects to a non-listed host fail the run.
[ ] Metadata endpoints and link-local addresses are denied by default.

Host allowlists are not a full network policy. They are the minimum you can enforce in application CI.

5. Environment and secrets

[ ] env_allow names every variable the process may read.
[ ] Names matching SECRET, TOKEN, PASSWORD, PRIVATE are denied unless explicitly listed.
[ ] The job prints variable names, never values.
[ ] Tokens used by the agent are scoped to that job and expire with it.

If a secret appears in a tool argument, the run is a failed run. Rotate. Then fix the allowlist.

6. Loop bounds

[ ] max_steps is set.
[ ] max_wall_clock_sec is set.
[ ] max_tool_calls_per_step is set.
[ ] On any limit, the agent stops. It does not retry with a wider tool.

Unbounded loops are not “research mode” in a merge pipeline. They are an open invoice and an open shell.

7. MCP and plugin graphs

[ ] Every MCP server is pinned by name, URL or path, and digest.
[ ] The tool list is frozen in the same file. A new tool is a new PR.
[ ] Servers that can execute arbitrary commands inherit the binary and root rules above.

If the server can add a tool without a digest change, it does not belong in this pipeline.

Evidence the PR must attach

A checklist without files is theater. Require these paths, or fail:

Gate	Required file	Fail closed when
Manifest present	`agent-egress.yml`	file missing or empty
Schema valid	CI validator log	unknown keys, empty lists while a capability is enabled
Trace exists	`artifacts/agent-trace.jsonl`	no file, or zero tool events while tools were enabled
Trace in bounds	CI comparator log	path, host, binary, or env outside the manifest
Bounds held	trace summary	`max_steps` or wall clock exceeded
Secret scan	CI log	secret-like names in argv or stdout

Store the trace next to the manifest. Reviewers should be able to open one folder and see both the policy and the run.

Artifact: a fail-closed validator

Label: this is a proposed local gate. Run it on a sample manifest before you trust it in a protected branch.

agent-egress.yml:

version: 1
agent_id: docs-triage
owner: platform-ci
workspace_roots:
  - .
write_roots:
  - ./artifacts
binaries:
  - git
  - python3
network:
  mode: allowlist
  hosts:
    - api.github.com
env_allow:
  - GITHUB_TOKEN
  - CI
max_steps: 20
max_wall_clock_sec: 180
max_tool_calls_per_step: 3
mcp_servers: []

ci/check_agent_egress.py:

#!/usr/bin/env python3
"""Fail closed if the agent egress manifest is missing or incomplete."""
from __future__ import annotations

import argparse
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("MISSING_DEP: PyYAML is required", file=sys.stderr)
    sys.exit(2)

REQUIRED = (
    "version",
    "agent_id",
    "owner",
    "workspace_roots",
    "write_roots",
    "binaries",
    "network",
    "env_allow",
    "max_steps",
    "max_wall_clock_sec",
    "max_tool_calls_per_step",
    "mcp_servers",
)
FORBIDDEN_ROOT_MARKERS = ("$HOME", "~", "/tmp", "/var", "C:\\")


def fail(msg: str) -> None:
    print(f"FAIL: {msg}", file=sys.stderr)
    sys.exit(1)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--manifest", default="agent-egress.yml")
    parser.add_argument("--fail-closed", action="store_true", default=True)
    args = parser.parse_args()

    path = Path(args.manifest)
    if not path.is_file():
        fail(f"manifest not found: {path}")

    raw = path.read_text(encoding="utf-8").strip()
    if not raw:
        fail("manifest is empty")

    data = yaml.safe_load(raw)
    if not isinstance(data, dict):
        fail("manifest must be a mapping")

    missing = [k for k in REQUIRED if k not in data]
    if missing:
        fail(f"missing keys: {missing}")

    if not data["agent_id"] or data["agent_id"] in {"tmp", "test", "default"}:
        fail("agent_id must be stable and non-placeholder")

    roots = data["workspace_roots"]
    if not roots:
        fail("workspace_roots must not be empty")
    for root in roots:
        if not isinstance(root, str) or any(m in root for m in FORBIDDEN_ROOT_MARKERS):
            fail(f"refusing root: {root!r}")

    writes = data["write_roots"]
    if not writes:
        fail("write_roots must not be empty")

    binaries = data["binaries"]
    if not binaries:
        fail("binaries allowlist must not be empty")
    if any(b in {"bash", "sh", "zsh", "cmd", "powershell"} for b in binaries):
        fail("shell binaries require a separate exception PR")

    network = data["network"]
    if network.get("mode") not in {"deny", "allowlist"}:
        fail("network.mode must be deny or allowlist")
    if network["mode"] == "allowlist" and not network.get("hosts"):
        fail("allowlist mode requires hosts")

    if int(data["max_steps"]) < 1 or int(data["max_wall_clock_sec"]) < 1:
        fail("loop bounds must be positive")

    print(f"PASS: {path} agent_id={data['agent_id']}")


if __name__ == "__main__":
    main()

Run it locally the same way CI will:

python3 -m pip install pyyaml
python3 ci/check_agent_egress.py --manifest agent-egress.yml --fail-closed
echo $?
# expected: 0 on a complete file, 1 on any gap

CI job sketch:

name: agent-egress-gate
on:
  pull_request:
    paths:
      - "agent-egress.yml"
      - "ci/check_agent_egress.py"
      - "**/*agent*"
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 -m pip install pyyaml
      - run: python3 ci/check_agent_egress.py --manifest agent-egress.yml --fail-closed

Keep the path filter honest. If your agent code lives outside *agent*, drop the filter. A skipped gate is an open gate.

Trace comparator, minimum version

The manifest is policy. The trace is evidence. You still need a comparator.

Proposed event shape (one JSON object per line):

{"ts": "2026-09-11T12:00:00Z", "type": "tool", "name": "run_terminal_cmd", "binary": "git", "argv": ["status"], "cwd": ".", "host": null}

Comparator rules you can implement in a short script:

cwd must resolve under workspace_roots.
binary must be in binaries.
host must be in network.hosts when not null.
Event count must be <= max_steps.
Unknown type fails. Do not ignore fields you do not understand.

If you cannot produce a trace, you cannot merge. “The vendor UI does not export logs” is a vendor problem, not a reason to skip the gate.

Where a coding assistant actually helps

Drafting the first allowlist from a recorded trace is tedious. A coding assistant is useful there: it can turn a JSONL file into a candidate YAML. It is not useful as the gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a throwaway workspace to generate those traces without pointing the agent at production, MonkeyCode’s free model access and free server option are enough to iterate on the manifest. Keep the validator in your CI. The model does not get a vote.

Limitations

This checklist does not replace OS sandboxes, seccomp, or a real egress proxy. Path allowlists lose if the process can follow a symlink you did not resolve. Host allowlists lose if a listed host issues a redirect you do not follow in the comparator. MCP pins lose if the server mutates tools behind the same digest.

Loop bounds do not stop a single dangerous command. They stop a runaway. You still need binary and argv policy for the dangerous command.

The validator above does not parse traces. Ship the comparator before you claim production readiness. Until then, label the gate manifest-only in the PR so reviewers know what they are not seeing.

Who should not use this

Teams that need the agent to browse the open web. Use a dedicated browsing account and a different pipeline.
Teams with no human owner for the allowlist. An unowned file will rot, then someone will set network.mode: open.
Regulated workloads that require a named control framework. This is an engineering gate, not an audit certificate.
Local experiments on a throwaway clone with no secrets. Do not block research with production ceremony. Do block the merge.

Decision you make on every PR

Ask one question: if this agent process is still running in ten minutes, which roots, hosts, and binaries can it still touch?

If you cannot answer from a file in the repo, do not merge. Pin the roots. Attach the trace. Fail closed.

Freeze the Model Graph Before Merge: A Fail-Closed Identity Checklist

Taylor Zhu — Wed, 09 Sep 2026 16:16:35 +0000

If you cannot name the exact model identifier, prompt bytes, and tool schemas that produced a passing eval, you do not have a releasable agent. Aliases are not versions. A green CI run against latest is a screenshot of a moving target.

You already gate on tests, timeouts, and rollback windows. This checklist covers a different failure: identity drift. The agent still compiles. The fixtures still parse. The model behind the friendly name moved.

The failure unit tests will not show you

Agent PRs fail in production for boring reasons. A provider retargets an alias. A free endpoint swaps the checkpoint behind the same label. A teammate edits a system prompt and forgets that CI loads a different path.

Your unit tests keep passing. They never asked who answered.

Tool-calling stacks make this worse. The model is only one node. Prompts, tool JSON schemas, retrieval snapshots, and decoding knobs are the rest of the graph. If any node is unpinned, the eval is not replayable. If it is not replayable, it is not evidence. Merge evidence, not vibes.

Freeze this graph, not a dashboard default

Treat the runtime the way you treat a lockfile. You would not ship lodash@latest next to a payment charge. Do not ship chat-model@latest next to a tool that can write.

Pin all five:

Model identity — a concrete model id. Not default, auto, latest, or an undocumented alias.
Prompt bytes — SHA-256 of the exact system and developer prompts loaded at eval time.
Tool schemas — SHA-256 of the serialized allowlist the model was allowed to see.
Decoding knobs — temperature, top_p, max output tokens, and seed if the API exposes one.
Corpus snapshot — a git SHA or object digest. Not “whatever is in the bucket today.”

Missing any one of those is a merge blocker. Do not warn. Fail closed.

Copy-paste gates, evidence, and fail-closed rules

Use this as a PR template. Every row is a gate. “We’ll watch it in staging” is not evidence.

Gate A — Identity file exists

Evidence: agent-lock.json is committed in the same PR as the agent change.

Fail closed if: the file is missing, untracked, or generated after the eval instead of before it.

Gate B — No alias tokens

Evidence: model_id is a concrete vendor identifier, not a moving label.

Fail closed if: the value is empty or matches latest, default, auto, free, or current.

Gate C — Digests match the git tree

Evidence: CI recomputes hashes from the checkout and diffs them against the lock file.

Fail closed if: lock hash and tree hash disagree, even by one byte.

Gate D — Eval ran against the lock

Evidence: artifacts/eval-report.json echoes model_id, hashes, and knobs verbatim.

Fail closed if: the report omits a lock field, or the echoed values do not match.

Gate E — Replay fixture stored

Evidence: one redacted golden request/response plus the tool-call trace for the critical path.

Fail closed if: you cannot replay the eval locally with the lock file and the fixture alone.

Gate F — Tool allowlist is explicit

Evidence: the tool list in the lock file is an allowlist. Runtime cannot inject unknown names.

Fail closed if: the agent discovers tools from the environment without a lock-file match.

A reproducible CI artifact

Label this as a proposed gate. Wire it to your runner. It does not call a vendor. It only refuses to merge an unpinned graph.

Create agent-lock.json before you run evals:

{
  "model_id": "REPLACE_WITH_CONCRETE_ID",
  "prompt_sha256": "",
  "tools_sha256": "",
  "temperature": 0,
  "max_output_tokens": 1024,
  "seed": 7,
  "corpus_git_sha": "none",
  "eval_suite": "tests/agent_eval"
}

Empty strings are illegal. If you have no retriever, write none on purpose. Implicit nothing is how aliases sneak back in.

Hash the prompts and tools from the tree, not from memory:

#!/usr/bin/env bash
set -euo pipefail

prompt_sha=$(cat prompts/system.md prompts/developer.md | sha256sum | awk '{print $1}')

tools_sha=$(python - <<'PY'
import hashlib, json, pathlib
raw = pathlib.Path("tools/allowlist.json").read_text()
data = json.loads(raw)
canonical = json.dumps(data, sort_keys=True, separators=(",", ":"))
print(hashlib.sha256(canonical.encode()).hexdigest())
PY
)

echo "prompt_sha256=${prompt_sha}"
echo "tools_sha256=${tools_sha}"

Paste those digests into the lock file, commit it, then fail closed in CI:

#!/usr/bin/env bash
# proposed gate: reject unpinned agent graphs
set -euo pipefail

lock="agent-lock.json"
test -f "${lock}" || { echo "missing ${lock}"; exit 1; }

python - "${lock}" <<'PY'
import hashlib, json, pathlib, re, sys

lock = json.loads(pathlib.Path(sys.argv[1]).read_text())
required = [
    "model_id", "prompt_sha256", "tools_sha256",
    "temperature", "max_output_tokens", "corpus_git_sha", "eval_suite",
]
missing = [k for k in required if lock.get(k) in (None, "")]
if missing:
    raise SystemExit(f"lock missing fields: {missing}")

model = str(lock["model_id"])
if re.search(r"latest|default|auto|free|current", model, re.I):
    raise SystemExit(f"alias refused: {model}")

prompt = (
    pathlib.Path("prompts/system.md").read_bytes()
    + pathlib.Path("prompts/developer.md").read_bytes()
)
if hashlib.sha256(prompt).hexdigest() != lock["prompt_sha256"]:
    raise SystemExit("prompt hash mismatch")

tools = json.dumps(
    json.loads(pathlib.Path("tools/allowlist.json").read_text()),
    sort_keys=True,
    separators=(",", ":"),
).encode()
if hashlib.sha256(tools).hexdigest() != lock["tools_sha256"]:
    raise SystemExit("tools hash mismatch")

print("agent-lock.json: pinned and matched")
PY

Add a contract so the eval cannot “pass” while talking to a different model:

# tests/test_eval_echoes_lock.py
# proposed test: the eval runner must echo the lock, not a UI default
import json
from pathlib import Path


def test_eval_report_echoes_lock():
    lock = json.loads(Path("agent-lock.json").read_text())
    report = json.loads(Path("artifacts/eval-report.json").read_text())
    for key in ("model_id", "prompt_sha256", "tools_sha256", "temperature"):
        assert report.get(key) == lock[key], f"{key} drifted between lock and eval"

Run the same sequence on every PR:

mkdir -p artifacts
# your eval runner must write artifacts/eval-report.json
# including the lock fields it actually used
pytest tests/test_eval_echoes_lock.py tests/agent_eval -q

If the eval runner cannot print the model id it used, it is not a runner you can merge with. Throw it out.

Decision table you can paste into the PR

Symptom	What you probably did not pin	Merge?
Pass locally, fail in CI	prompt path or working directory	No
Pass Monday, fail Thursday, no code change	model alias	No
Same model id, different tool calls	tool schema or retrieval corpus	No
Golden trace will not replay	temperature, seed, or hidden preamble	No
New tool appeared at runtime	allowlist not enforced	No

Print the table. If the author cannot point to a lock field for the symptom, the PR stays closed.

Where a free eval lane belongs

You still need somewhere to run the suite when a laptop cannot. That is an eval lane. It is not production. Keep customer data, production credentials, and write-capable tools off it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Point the same pin-and-prove job at that lane if you want hashes computed and eval-report.json published off your laptop. Do not promote that server to a write-enabled agent. If the lane cannot echo a concrete model id into the report, fail closed. Steal the lock file even if you never use that lane. The gate is the point.

Limitations, and who should skip this

This approach assumes you can read a durable model identifier and that your eval is cheaper than a production incident. Some hosted APIs never expose a stable id. If that is your vendor, this checklist cannot save you. Switch, or wrap the call behind a proxy you control.

A free eval lane can also disagree with production. Different hardware, different batching, different hidden preambles. Pinning does not make two endpoints identical. It only makes which endpoint you tested an explicit merge fact. If you cannot afford a production-shaped replay, do not pretend the free lane is one.

Do not use this as a substitute for:

security review of tool side effects
PII redaction in traces
load tests
a rollback plan

A pinned bad prompt is still a bad prompt. Hashing it only makes the mistake reproducible.

Skip the full lock file if your “agent” is a single offline classifier with no tools and no network. You are not in the failure mode this gate exists for. Add it the moment the model can call anything with a side effect.

Merge rule

No lock file, no echoed eval, no merge. Aliases are a paging incident you scheduled for yourself. Pin the graph. Prove the eval saw that graph. Then ship.

Fifteen-Minute Rollback or No Merge: A Reversibility Checklist

Taylor Zhu — Tue, 08 Sep 2026 03:40:11 +0000

If an AI-assisted change cannot be reversed in fifteen minutes, it is not production-ready. Green CI does not prove that. A passing demo does not prove that either.

You need four named proofs before merge: who owns the blast radius, what switch stops the new path, which command restores the old path, and what you will watch for the first hour. Missing any one of those is a fail. Not a comment. A fail.

Why agent diffs skip the unmerge clock

Agents optimize for "make the tests pass." They do not feel pager duty. They will add a consumer, a cron, a new index, or a side-effecting webhook because the prompt said implement. You review the happy path. Production reviews the undo path.

The cheap-code problem is not style. It is volume. When a patch is cheap to generate, reversible design becomes the scarce resource. Your gate has to encode that scarcity, or a Friday deploy will teach it to you.

This article is a copy-paste production-readiness checklist for that gap. It is not a model bake-off. It is not a claim that any assistant understands your traffic.

Four proofs, not an essay

Treat reversibility as a contract the PR must attach. Keep it short. If someone needs a narrative, the change is too large.

1. Owner of record

A human pager target. Not "the AI." Not a Slack channel with fifty people. One primary, one backup. If both are out, the gate fails.

2. Blast radius

What user journeys, tables, queues, and regions can this break? Write names, not "backend." If you cannot list them, you do not understand the change.

3. Stop switch

A feature flag, config kill, traffic weight, or process you can flip without a rebuild. Schema expansions that old binaries cannot ignore fail this proof unless you document an expand/contract sequence in the same packet.

4. Restore command

The exact command or runbook step that returns traffic or data to the prior behavior. "Revert the PR" is acceptable only when the change is stateless and any migration is additive and unused. If you wrote data, reverting git is not a restore command.

Copy-paste packet

The YAML below is a proposed template, not a completed incident. Swap team names and metric names for yours. Leave the structure.

# .reversibility.yml — proposed production gate
apiVersion: reversibility/v1
change:
  id: "pr-${PULL_REQUEST_NUMBER}"
  summary: "Add retry wrapper around billing capture"
  generated_by: "agent-assisted"
owner:
  primary: "oncall-payments"
  backup: "oncall-platform"
  timezone: "America/Los_Angeles"
blast_radius:
  journeys: ["checkout/capture", "checkout/refund-lookup"]
  datastores: ["payments.captures"]
  queues: ["captures.outgoing"]
  regions: ["us-west-2"]
stop_switch:
  type: "feature_flag"
  name: "payments.retry_wrapper"
  default: false
  disable_command: "flagctl disable payments.retry_wrapper --env prod"
restore:
  unmerge_minutes: 15
  data_written: false
  command: "kubectl rollout undo deploy/payments-api"
  verifies: "GET /internal/flag/payments.retry_wrapper returns false"
watch_first_hour:
  - "capture_success_rate"
  - "capture_latency_p99"
  - "retry_storm_count"
limits:
  max_files: 40
  max_migration_files: 0

If data_written is true, command cannot be git revert alone. The validator below enforces that.

Fail the build when the packet is theater

Run this locally or in CI. It deploys nothing. It fails closed on missing fields, empty blast-radius lists, a rebuild-shaped stop switch, and the git-revert lie. Treat it as an unexecuted starting point until you wire it to your flag CLI.

#!/usr/bin/env python3
"""Fail closed if .reversibility.yml cannot be undone in 15 minutes."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit  = 2  # labeled example; fix locally if you paste this

REQUIRED = [
    ("owner", "primary"),
    ("owner", "backup"),
    ("stop_switch", "name"),
    ("stop_switch", "disable_command"),
    ("restore", "command"),
    ("restore", "verifies"),
]

BANNED_OWNERS = {"ai", "agent", "bot", "copilot", "nobody"}
REBUILD_HINTS = ("docker build", "terraform apply", "helm upgrade", "migrate up")


def fail(msg: str) -> None:
    print(f"reversibility: FAIL: {msg}", file=sys.stderr)
    raise SystemExit(1)


def main(path: Path) -> None:
    if not path.is_file():
        fail(f"missing {path}")
    data = yaml.safe_load(path.read_text()) or {}
    for section, key in REQUIRED:
        value = (data.get(section) or {}).get(key)
        if not value or not str(value).strip():
            fail(f"{section}.{key} is empty")

    owner = str(data["owner"]["primary"]).strip().lower()
    if owner in BANNED_OWNERS:
        fail("owner.primary must be a human pager target")

    radius = data.get("blast_radius") or {}
    for key in ("journeys", "datastores", "regions"):
        items = radius.get(key) or []
        if not items:
            fail(f"blast_radius.{key} must list real names")

    restore = data["restore"]
    minutes = int(restore.get("unmerge_minutes") or 0)
    if minutes <= 0 or minutes > 15:
        fail("restore.unmerge_minutes must be 1..15")

    command = str(restore["command"]).lower()
    disable = str(data["stop_switch"]["disable_command"]).lower()
    if any(h in disable for h in REBUILD_HINTS):
        fail("stop_switch requires a rebuild; that is not a kill switch")

    data_written = bool(restore.get("data_written"))
    if data_written and "git revert" in command:
        fail("data_written=true cannot use git revert as the restore command")

    watch = data.get("watch_first_hour") or []
    if len(watch) < 2:
        fail("watch_first_hour needs at least two metrics")

    print("reversibility: PASS")


if __name__ == "__main__":
    main(Path(".reversibility.yml"))

Wire it as a required check. A suggested snippet, not a vendor-specific workflow:

python3 check_reversibility.py
# exit 1 blocks merge; do not allow a skip label for agent-assisted PRs

Decision table you can paste into the PR template

Condition	Gate	Why
Owner is a model, bot, or rotating channel	Fail	Nobody gets paged
Blast radius is "all services" or blank	Fail	No one can scope the incident
Stop switch needs a rebuild or migration	Fail	You cannot stop it in minutes
`data_written: true` and restore is `git revert`	Fail	Git does not unwrite rows
`unmerge_minutes` > 15	Fail	You chose a change that is too large
Additive, unused column + flag default off	Pass	Expand/contract is intact
Stateless config with `kubectl rollout undo`	Pass	Old ReplicaSet is still present
Dual-write with a documented drain command	Pass	You can stop the new writer first

Print the table in the PR body. Reviewers should argue with the row, not with vibes.

How teams fake the four proofs

You will see these. Reject them.

Owner set to a bot. The packet compiles. The pager does not.
Blast radius "platform." That is a department, not a journey.
Stop switch that ships in the same binary as the bug. You cannot disable what you must rebuild to change.
Restore command that is a wiki URL. A link is not a command. If the wiki is down, you are down twice.
unmerge_minutes: 15 next to a blocking backfill. Arithmetic is not a runbook.
Watch list copied from another service. You will alert on the wrong graph while the real queue melts.

A useful review question: "If this ships at 16:55, what do I type at 17:02?" If the author hesitates, the packet is incomplete.

Fill the packet without making the agent invent prod

Do not ask an assistant to guess your flag name. Feed it inventory you already trust: service list, flag catalog, dashboard UIDs, on-call schedule. Then ask it only to draft the YAML against that inventory.

Proposed prompt, not a magic spell:

Using ONLY the inventory pasted below, fill .reversibility.yml.
If a field is not in the inventory, write UNKNOWN and stop.
Do not invent flag names, metric names, or pager targets.
Inventory:
- services: ...
- flags: ...
- dashboards: ...
- oncall: ...

If the draft contains UNKNOWN, the gate should fail. Unknown is information. A plausible fake name is not.

Keep the change small enough for a fifteen-minute undo. Split schema expand from schema contract. Split dual-write from cutover. Agents will gladly one-shot all three. You should not.

Run the gate before shared CI is the bottleneck

You can execute the validator on a laptop against a branch. That is the whole point: catch a missing restore command before reviewers debate style.

When you want the same check on a throwaway remote box while an assistant fills the YAML, you do not need a dedicated cluster. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are one way to draft the packet and run check_reversibility.py on a machine you can delete. They do not replace your on-call roster, your flag service, or your production metrics. If those are missing, skip the product and fix the roster.

What this will not save you from

Wrong physics. A fifteen-minute undo does not repair a unique constraint you already enforced on live rows.
Silent correctness. Rollback can restore an old bug you were trying to leave.
Multi-region drift. If only one region has the flag default, your packet is lying about blast radius.
Vendor runtimes you cannot roll back. If the change is a managed resource with no undo API, this checklist should fail until you have a compensating action.
Load you never measured. watch_first_hour is useless if the metrics do not exist yet.

The script does not talk to Kubernetes, feature flags, or billing. It only refuses empty theater. You still have to run the disable command in a staging environment once, on purpose, and paste the output into the PR.

Who should not use this

Skip this approach if you ship only documentation, or if every change is already behind a global dark-launch switch with a tested disable path. Skip it if your "production" is a single developer database with no users. Skip it if you will add a skip label on every agent PR; a gate that is optional is not a gate.

Do not use a reversibility packet as permission to merge unbounded agent diffs. The packet is a shrinking device. If you cannot name the undo, shrink the change until you can.

Ship the owner, the radius, the switch, and the restore command. Then merge. If you cannot, keep the branch. Fifteen minutes is the test. Everything else is a story about tests that were green.

Don't Ship the Sandbox: A Fail-Closed Promotion Checklist

Taylor Zhu — Sun, 06 Sep 2026 21:33:07 +0000

A green agent run on free compute is not a merge decision. It is a draft produced on an untrusted host. You promote the change only after isolation, provenance, and a trusted re-run all pass. If any of those are missing, you fail closed.

Agents assume. Free sandboxes hide that. Your job is to make promotion boring, labeled, and refusible.

The merge you should refuse

You let an agent edit a service on a laptop, a shared VM, or a complimentary cloud box. Tests pass there. The PR looks complete. Then you merge from that host’s word.

That is the bug. The sandbox is not your CI. It is not your production identity provider. It is not allowed to mint a release.

Treat every agent workspace as hostile until a trusted pipeline repeats the same git SHA with your real secrets policy, your real network policy, and your real artifact store.

What “untrusted host” actually means

Untrusted does not mean “malware.” It means you cannot prove four things:

Who ran it. A free box may be shared, preempted, or reused.
What it could see. Prompt context, .env files, and cloud metadata are easy to leak.
Where it could send bytes. Egress is the silent failure.
Whether you can replay it. If you cannot re-run the SHA, you cannot defend the merge.

You do not argue with the model. You refuse the promotion path.

Copy this promotion checklist

Use it as a gate file in the repo. Empty fields fail the job. “Not applicable” is not a field. Delete a gate only when the change cannot touch prod, data, or credentials.

Gate 0 — Label the environment

Every agent job exports an environment class. No label, no artifacts.

# proposal: required env for any agent runner
AGENT_ENV_CLASS=sandbox   # sandbox | trusted_ci | prod
AGENT_RUN_ID=$(uuidgen)
AGENT_GIT_SHA=$(git rev-parse HEAD)

Fail closed if AGENT_ENV_CLASS=prod on a developer laptop, a free VM, or any host missing your CI OIDC identity. Sandbox may propose. Only trusted_ci may attest.

Gate 1 — Data and secrets stay out of the sandbox

You do not load production dumps “just this once.” You do not mount the org’s cloud keys so the agent can “debug faster.”

Fail closed when any of these are true:

Production connection strings are present in the sandbox process environment.
The workspace can read ~/.aws, ~/.kube, or CI OIDC tokens.
Logs may contain live customer payloads.
The prompt pack includes real tickets with PII.

Replacement rule: synthetic fixtures, scrubbed snapshots, or a dedicated sandbox account with no prod IAM trust.

Gate 2 — Network and writes are allowlisted

If the agent can hit the internet, it can exfiltrate the repo. If it can write anywhere, it can persist a hook you will not review.

Minimum allowlist:

Direction	Allow	Deny by default
Egress	Package registry, your git host, your artifact store	Everything else
Writes	Repo worktree, job temp dir	`$HOME`, Docker socket, extra volumes
Reads	Declared fixture paths	Secret stores, sibling repos

A missing allowlist is a deny. Do not “open it for now.”

Gate 3 — Trusted re-run is mandatory

Sandbox green is a hint. Merge green is a second execution.

You pin the SHA. You rebuild. You re-run tests and scanners on trusted_ci. You compare artifact digests. If the sandbox cannot produce a digest, it cannot propose a merge.

# proposal: promotion.json (checked into the PR, not a screenshot)
{
  "git_sha": "REPLACE_WITH_FULL_SHA",
  "sandbox_run_id": "",
  "sandbox_artifact_sha256": "",
  "trusted_ci_run_id": "",
  "trusted_ci_artifact_sha256": "",
  "env_class_attested": "trusted_ci",
  "feature_flag": "agent-change-TICKET",
  "owner": "oncall-handle",
  "rollback": "gh pr revert --yes"
}

Fail closed if sandbox_artifact_sha256 equals trusted_ci_artifact_sha256 but env_class_attested is still sandbox. Same bytes from the wrong host are not an attestation.

Gate 4 — Blast radius is numeric

“Small PR” is not a control. You cap what the change may touch.

Max paths changed (example: 20 files, 2 services).
Max IAM actions added (example: zero in sandbox-origin PRs).
Max regions or clusters (example: one staging namespace).
Feature flag default off.
No schema drop, no authz change, no secret rotation from an agent sandbox.

If the diff exceeds the cap, the promotion job fails. A human splits the work. The agent does not get a bigger blast radius because the model was confident.

Gate 5 — Owner and rollback exist before merge

No owner, no merge. No rollback command that has been typed once in staging, no merge.

You need a named human, a chat channel, and a revert that does not depend on the agent being available. If rollback is “ask the model to undo it,” you do not have rollback.

A fail-closed workflow you can paste

Label the following as a proposal. Wire it to your CI. Do not run it as a clever local alias that can be skipped with --no-verify.

# proposal: .github/workflows/promotion-gate.yml
name: promotion-gate
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  refuse-sandbox-attestation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Fail closed on missing promotion evidence
        env:
          AGENT_ENV_CLASS: ${{ vars.AGENT_ENV_CLASS }}
        run: bash scripts/promotion_gate.sh

# proposal: scripts/promotion_gate.sh
set -euo pipefail
FILE="promotion.json"

if [[ ! -f "$FILE" ]]; then
  echo "fail-closed: $FILE missing"
  exit 1
fi

python3 - <<'PY'
import json, sys
required = [
    "git_sha", "sandbox_run_id", "sandbox_artifact_sha256",
    "trusted_ci_run_id", "trusted_ci_artifact_sha256",
    "env_class_attested", "feature_flag", "owner", "rollback"
]
data = json.load(open("promotion.json"))
missing = [k for k in required if not str(data.get(k, "")).strip()]
if missing:
    print("fail-closed: empty fields:", ", ".join(missing))
    sys.exit(1)
if data["env_class_attested"] != "trusted_ci":
    print("fail-closed: sandbox cannot attest a merge")
    sys.exit(1)
if data["sandbox_artifact_sha256"] == data["trusted_ci_artifact_sha256"] and not data["trusted_ci_run_id"]:
    print("fail-closed: digest copied without a trusted run id")
    sys.exit(1)
print("promotion evidence present; human review still required")
PY

Keep the script stupid. Clever exceptions become merge holes.

Decision table: sandbox result vs merge action

Sandbox result	Trusted CI result	Secrets/egress gates	Action
Pass	Pass, same SHA	Pass	Human review, flag off
Pass	Fail	Pass	Do not merge; sandbox was a false green
Fail	Pass	Pass	Do not auto-merge; explain the delta
Pass	Not run	Anything	Fail closed
Pass	Pass	Missing allowlist	Fail closed
Pass on `prod` label from a laptop	Anything	Anything	Fail closed, rotate credentials

The interesting row is sandbox pass / trusted fail. That is the whole reason this checklist exists. You will see it. You should want to see it.

Where a free sandbox still helps

You still want a cheap place to let an agent thrash. Draft diffs. Throwaway fixtures. Prompt iteration. That host should be labeled sandbox and stripped of prod trust.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a place to iterate without burning paid CI minutes, MonkeyCode’s free model access and free server option can sit on the sandbox side of this split. They do not replace trusted_ci. They must not write env_class_attested.

Use whatever sandbox you already have. The gate is the product. The host is disposable.

Limitations

This checklist does not prove the change is correct. It proves you refused a class of unsafe promotions.

It will not catch a wrong business rule that still compiles. It will not catch a model that omitted a test you never wrote. It will not survive a team that grants themselves continue-on-error: true.

Digests can match and still be wrong if both runs used the same poisoned fixture. Environment labels can be spoofed if your CI does not mint them. Feature flags do not help if the flag wraps only the happy path.

Free sandboxes also disappear, throttle, or change under you. That is another reason they cannot be the system of record. Pin the SHA on infrastructure you operate.

Who should not use this

Skip the full gate if you are:

Working in a throwaway repo with no secrets, no customers, and no deploy.
Pairing on a spike that will be deleted before it touches main.
Already running agents only inside a locked-down trusted_ci image with no sandbox hop.

Do not skip it if the agent can open a PR, talk to a package registry, or see a staging cluster. That is already production-adjacent.

What you do Monday

Pick one service. Add promotion.json as required evidence. Add the script. Break a PR on purpose by leaving trusted_ci_run_id empty. Confirm the job fails.

Then break it again by labeling a laptop run as trusted_ci. If that merge is possible, your identity story is the incident, not the model.

Sandbox green is allowed. Sandbox authority is not. Fail closed, then promote.