DEV Community: Taylor Zhu

No Complete Bundle, No Promote: A Fail-Closed Rehearsal Checklist

Taylor Zhu — Sun, 11 Oct 2026 21:46:33 +0000

Do not promote an agent change from a rehearsal until the evidence bundle is complete. A green log line is not that bundle. If any required field is missing, empty, or inconsistent, the gate fails closed and the change stays put.

You already know a fluent answer can still hide a bad diff. What you need is a record you can re-check without trusting the chat scroll. This checklist is that record. It does not grade model quality, and it does not deploy anything.

Why a rehearsal is not a release

A shared or free server is a convenient place to run the loop once. It is not your production host, and it is not a signed release.

You can learn which tools fired, which files changed, and which command exited non-zero. You cannot learn your latency budget, your data-handling rules, or your rollback path from that single run. Treat the rehearsal as an input to a gate, not as the gate itself.

The failure mode is ordinary. Someone pastes a tidy summary into the ticket and calls the work done. The summary omits the model identifier the API actually returned, the lockfile hash, or the secret-scan result. Next week nobody can replay the decision. Fail closed before that summary becomes the only artifact.

Fields the bundle must hold

Keep one JSON file per run. Name it after the run id, not after a branch nickname.

Required fields:

run_id: a unique string you generate before the run starts
started_at and finished_at: UTC timestamps with a timezone
rehearsal_env and target_env: labels you define, not marketing names
model_id: the identifier string the API returned, copied verbatim
prompt_sha256, lock_sha256, diff_sha256: 64 lowercase hex characters
tool_calls: an integer count recorded by your harness
commands: a non-empty list of {cmd, exit_code} actually executed
secret_scan: exactly clean or findings
reviewer and reviewed_at: empty until a person signs
action: hold or promote

Do not add a field called looks_good. A boolean without a source file is how checklists rot. If you need a comment, put it in the ticket, not in a field the gate ignores.

Eight gates that fail closed

Walk these in order. Stop at the first failure. Do not collect a warning and continue.

Presence. Every required key exists and is non-empty, except reviewer fields while action is hold.
Time order. finished_at is strictly after started_at. Equal timestamps fail. A zero-duration run is not evidence.
Hash shape. Each *_sha256 matches 64 lowercase hex characters. Uppercase, prefixes, and truncated digests fail.
Model pin. model_id is the raw API string. A nickname in your notes does not fill an empty field.
Environment split. rehearsal_env must differ from target_env unless waiver_reason is present and a reviewer has signed. A rehearsal labeled prod fails.
Command evidence. commands is a non-empty list, and every exit code is an integer. A missing test command is a failure, not a skip.
Secret scan. Only clean may proceed. findings, skipped, or any other string fails closed. Do not encode an exception in this file.
Human promote. action=promote requires reviewer and reviewed_at at or after finished_at. The script never flips the action for you.

Automation can refuse. It should not approve. If you want a ninth gate later, add it as another hard failure, not as a score.

Fill the bundle before you argue about the diff

You fill fields from artifacts, in this order.

Write run_id and started_at before the first tool call. If you reconstruct them afterward, say that in the ticket and expect the times to be distrusted.
Save the prompt and the lockfile you actually used. Hash those files. Store the hex digest only.
Copy model_id from the response your client already logs. If the client does not log it, fix the client. Do not guess a name.
Record every command the agent or you ran, with the exit code the shell returned.
Run the secret scanner your team already trusts. Store clean only when that tool reported no findings. This checklist does not include a scanner.
Hash the diff. Set action to hold.
Ask a reviewer to read the diff. Only that person may set promote, reviewer, and reviewed_at.

Three of those steps are clerical. They are still the ones that get skipped when the summary sounds confident. The script exists so a skipped step cannot hide inside a cheerful paragraph.

A validator you can copy

The script below is a proposal. It was not executed against a live host for this article, and it does not call any model API. It checks shape and a few inequalities. Drop it in a repo and point it at a bundle you produced.

#!/usr/bin/env python3
"""Fail closed unless a rehearsal bundle is complete. Proposal, not a benchmark."""
import json, re, sys
from datetime import datetime

HEX = re.compile(r"^[0-9a-f]{64}$")
REQUIRED = [
    "run_id", "started_at", "finished_at", "rehearsal_env", "target_env",
    "model_id", "prompt_sha256", "lock_sha256", "diff_sha256",
    "tool_calls", "commands", "secret_scan", "action",
]

def parse_time(value):
    return datetime.fromisoformat(str(value).replace("Z", "+00:00"))

def fail(msg):
    print(f"FAIL: {msg}")
    sys.exit(1)

def main(path):
    with open(path, encoding="utf-8") as fh:
        bundle = json.load(fh)
    for key in REQUIRED:
        if key not in bundle or bundle[key] in ("", None, []):
            fail(f"missing {key}")
    start, end = parse_time(bundle["started_at"]), parse_time(bundle["finished_at"])
    if end <= start:
        fail("finished_at must be after started_at")
    for key in ("prompt_sha256", "lock_sha256", "diff_sha256"):
        if not HEX.match(str(bundle[key])):
            fail(f"{key} is not 64 lowercase hex chars")
    if bundle["rehearsal_env"] == "prod":
        fail("refusing a rehearsal labeled prod")
    if bundle["rehearsal_env"] == bundle["target_env"] and not bundle.get("waiver_reason"):
        fail("rehearsal_env equals target_env without waiver_reason")
    if not isinstance(bundle["tool_calls"], int) or bundle["tool_calls"] < 0:
        fail("tool_calls must be a non-negative int")
    if not isinstance(bundle["commands"], list) or not bundle["commands"]:
        fail("commands empty")
    for i, step in enumerate(bundle["commands"]):
        if "cmd" not in step or not isinstance(step.get("exit_code"), int):
            fail(f"commands[{i}] needs cmd and integer exit_code")
    if bundle["secret_scan"] != "clean":
        fail("secret_scan must be clean")
    if bundle["action"] not in ("hold", "promote"):
        fail("action must be hold or promote")
    if bundle["action"] == "promote":
        if not bundle.get("reviewer") or not bundle.get("reviewed_at"):
            fail("promote requires reviewer and reviewed_at")
        if parse_time(bundle["reviewed_at"]) < end:
            fail("reviewed_at is earlier than finished_at")
    print("PASS: complete enough to review, not proof of production fitness")
    sys.exit(0)

if __name__ == "__main__":
    if len(sys.argv) != 2:
        fail("usage: check_bundle.py <bundle.json>")
    main(sys.argv[1])

A hold bundle, with fake hashes

Replace every digest with real sha256sum output from your tree. The hex below is only there so the shape checker has something to parse. It is not a measurement, and it is not your run.

{
  "run_id": "2026-10-12T09-14-02Z-west",
  "started_at": "2026-10-12T09:14:02Z",
  "finished_at": "2026-10-12T09:21:40Z",
  "rehearsal_env": "free-server-rehearsal",
  "target_env": "staging-app",
  "model_id": "<paste the id your API returned>",
  "prompt_sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "lock_sha256": "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210",
  "diff_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "tool_calls": 4,
  "commands": [
    {"cmd": "python -m unittest discover -s tests", "exit_code": 0}
  ],
  "secret_scan": "clean",
  "action": "hold"
}

A bundle that must fail

Save this as bad.json when you want to see the gate refuse a promote with no reviewer. You should get a non-zero exit. If your wrapper turns that into success, fix the wrapper.

{
  "run_id": "missing-reviewer",
  "started_at": "2026-10-12T09:14:02Z",
  "finished_at": "2026-10-12T09:21:40Z",
  "rehearsal_env": "free-server-rehearsal",
  "target_env": "staging-app",
  "model_id": "<paste the id your API returned>",
  "prompt_sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "lock_sha256": "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210",
  "diff_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "tool_calls": 1,
  "commands": [{"cmd": "true", "exit_code": 0}],
  "secret_scan": "clean",
  "action": "promote"
}

Commands

Hash the inputs you actually sent, then run the gate. Keep the exit code.

sha256sum prompt.txt > prompt.sha
sha256sum requirements.lock > lock.sha
git diff --binary > candidate.diff
sha256sum candidate.diff > diff.sha
python3 check_bundle.py rehearsal.json
echo "gate_exit=$?"
python3 check_bundle.py bad.json
echo "bad_exit=$?"

Exit 0 means the file is complete enough for a person to read. Exit 1 means stop. Do not catch that code and continue the pipeline.

Compare the stored diff digest with the file on disk before you discuss the patch:

sha256sum candidate.diff
python3 -c 'import json; print(json.load(open("rehearsal.json"))["diff_sha256"])'

If the two digests differ, fix the bundle. Do not edit the digest so the story matches.

Decision table

What you see	Gate result	What you do
Any required field missing	Fail closed	Hold. Fill the field from logs, not from memory.
`secret_scan` is not `clean`	Fail closed	Stop. Follow your leak process. Do not promote.
Rehearsal env equals target env	Fail closed	Split the labels, or add a reviewed waiver.
Rehearsal env is `prod`	Fail closed	Relabel. A production host is not a rehearsal.
Commands recorded, action `hold`, file check passes	File check only	A human may still reject the diff.
`action=promote` without reviewer	Fail closed	The script will not invent approval.
Bundle passes and a reviewer signed	Still not prod proof	Use your normal change window. This file is evidence, not a deploy.

Where a free model and a free server fit

You can rehearse a non-production loop where a free model and a free server are available, then freeze the bundle before anyone talks about promotion. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a possible place to run that rehearsal. This article does not state quotas, hardware, duration, model names, uptime, or that the free option will remain as it is. Read the current product terms before you depend on them.

Map rehearsal_env to the label you use for that server. Map model_id from the response you received, not from a nickname in a status page. If the client cannot show an identifier you can store, do not invent one. Hold.

Use one small task that contains no customer data. Run it, write the bundle by hand the first time, and only then decide whether the checker belongs in CI. If the data does not belong on that server, do not send it there.

Limitations

This checker looks at shape and a few inequalities. It does not execute your tests, scan for secrets, or judge whether the diff is correct.

A passing exit code can sit next to a wrong fix. Hash fields can be honest and still describe a bad prompt. Timestamps lie if the host clock is skewed. You still need log retention, access control, and a rollback you have practiced on the target environment.

No latency numbers, cost figures, or accuracy scores are claimed here. Placeholder hashes are not results. A free rehearsal that passes this file check is still not a capacity plan, a security review, or a production certification.

Who should not use this

Skip the approach if you need attested hardware, a private network, or a data boundary the rehearsal server does not provide. Skip it if the run would include secrets, production credentials, or personal data. Skip it if your release process already requires a signed artifact this JSON would only duplicate. Skip it if you want a script to approve production. This one refuses that job.

If the bundle is incomplete, you hold. That is the whole policy.

No Lease Card, No Domain: Fail-Closed Gates for a Borrowed Agent Host

Taylor Zhu — Sat, 10 Oct 2026 18:20:47 +0000

You do not attach a domain, a webhook, or a customer queue to a host you do not own. A free model route and a free server are a lease for a spike, not a runtime you quietly promote. If the lease card is missing, stale, or silent on exit, the change fails closed and the spike stays in quarantine.

That rule is narrower than "the tests are green." A green job can still pass on a box that disappears when an allowance ends or a terms page changes. You are gating ownership and exit. You are not gating the color of CI.

Name the blast radius before you name a tool

You are trying to stop one accident. A teammate prototypes an agent on a borrowed host, the demo works, and someone points real traffic at it because the URL already exists. The host was never promised as production. When the lease ends, you lose the process, the disk, or both, and the rollback is a chat thread.

Treat model headlines the same way. A nickname in a news post is not a pin you can deploy. You re-read the offer page on the day of the cutover, and you store that date in the repo. If you cannot point at the page you read, you do not promote.

Keep the spike. Refuse the binding. Those are different changes, and they should not share a commit.

What you may assume

Two availability claims are enough to run the spike, and nothing more. Free model access exists. A free server option exists.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one place you can use those two options while you fill out the lease card below. This article does not claim a quota, a hardware shape, a duration, a model catalog, a benchmark, or that the offer stays up.

You copy live fields from the current product page into the card. An empty field is a failed gate, not a guess you fill from memory or from an old chat. If you delete the product name, the gates still apply to any borrowed model route and any borrowed host.

Six gates, each with evidence

Copy this list into the pull request that would attach traffic. Each gate needs an artifact in the repo. A verbal "we checked" does not count.

1. Lease card is present and fresh

Evidence is ops/lease-card.yaml with terms_read_on, allowance_note, host_offer, model_route, and reader. Fail closed if the file is absent, if terms_read_on is outside your window, or if allowance_note is blank.

Do not invent a number to make the field pass. A stale figure is worse than an empty one, because reviewers trust it. Write what you read today, including "I could not find a published allowance" if that is the truth.

2. Model route is an identifier, not a headline

Evidence is a route string that matches the page you read, plus a one-line note of where you read it. Fail closed if the route is latest, free, default, or a name you only saw in a roundup.

You can change the pin later. You cannot promote on a moving nickname. Headlines are topic signals, not deploy config.

3. The borrowed host holds no long-lived secret

Evidence is a secret scan log, or a CI check that denies .env files and cloud keys on that path. Fail closed if customer tokens or deploy keys sit on the host.

A free server is a spike box. It is not a vault. A demo login is still a secret, and it does not belong in a notebook you might share.

4. Data exit is rehearsed

Evidence is a dry-run log that copies prompts, logs, and any user content to a place you own, then deletes the spike copy. Fail closed if the only copy lives on the borrowed disk.

"We can export later" is not a rehearsal. Later is when the disk is already gone. If the drill has not been run, the gate is red even if the demo looks fine.

5. Traffic attachment is a separate change

Evidence is a DNS, webhook, or queue diff in its own pull request, and that request links the lease card. Fail closed if one commit both creates the demo and binds production traffic.

Reviewers cannot see the blast radius in a mixed diff. Split it. The demo can merge on Monday. The binding waits for the card.

6. A person owns rollback, and the kill switch was timed

Evidence is a rollback_owner who is a named person, plus a drill note with a duration. Fail closed if the owner is "the team" or the drill has never been run.

You want a human who can unbind the route without asking who has the password. A rotation alias is not a person. Name someone for this cutover.

The checker you run before review

The script below is a proposal. It uses only the standard library. It does not call a vendor, and it does not know your quota. Run it locally, then wire a non-zero exit to the pull request that binds traffic.

# ops/lease-card.yaml
# Example only. Replace every value after you read the live page.
terms_read_on: "2026-10-10"
reader: "oncall-name"
host_offer: "free-server"
model_route: "pin-from-the-page-you-read"
allowance_note: "copied from the current offer page, not from memory"
data_exit: "rehearsed"
secrets_on_host: "no"
traffic_pr: "separate"
rollback_owner: "sam"
rollback_drill_on: "2026-10-09"

#!/usr/bin/env python3
"""Fail closed unless a borrowed-host lease card is complete and fresh.
Proposal only: stdlib, no network, no vendor client.
"""
from __future__ import annotations

import sys
from datetime import date, datetime
from pathlib import Path

REQUIRED = [
    "terms_read_on",
    "reader",
    "host_offer",
    "model_route",
    "allowance_note",
    "data_exit",
    "secrets_on_host",
    "traffic_pr",
    "rollback_owner",
    "rollback_drill_on",
]
BANNED_ROUTES = {"", "latest", "free", "default", "auto"}
MAX_AGE_DAYS = 7


def parse_flat_yaml(text: str) -> dict[str, str]:
    data: dict[str, str] = {}
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if not line or ":" not in line:
            continue
        key, value = line.split(":", 1)
        data[key.strip()] = value.strip().strip('"').strip("'")
    return data


def age_days(iso_day: str, today: date) -> int:
    parsed = datetime.strptime(iso_day, "%Y-%m-%d").date()
    return (today - parsed).days


def main(path: str) -> int:
    card = parse_flat_yaml(Path(path).read_text(encoding="utf-8"))
    errors: list[str] = []
    for key in REQUIRED:
        if not card.get(key):
            errors.append(f"missing or blank: {key}")
    route = card.get("model_route", "").lower()
    if route in BANNED_ROUTES:
        errors.append("model_route is a nickname, not a pin")
    if card.get("secrets_on_host", "").lower() != "no":
        errors.append("secrets_on_host must be no")
    if card.get("data_exit", "").lower() != "rehearsed":
        errors.append("data_exit must be rehearsed")
    if card.get("traffic_pr", "").lower() != "separate":
        errors.append("traffic attachment must be a separate change")
    owner = card.get("rollback_owner", "").lower()
    if owner in {"", "team", "oncall"}:
        errors.append("rollback_owner must be a named person")
    today = date.today()
    for key in ("terms_read_on", "rollback_drill_on"):
        raw = card.get(key, "")
        if not raw:
            continue
        try:
            delta = age_days(raw, today)
            if delta > MAX_AGE_DAYS or delta < 0:
                errors.append(f"{key} is outside the {MAX_AGE_DAYS}-day window")
        except ValueError:
            errors.append(f"{key} is not YYYY-MM-DD")
    if errors:
        print("FAIL closed:")
        for item in errors:
            print(f"- {item}")
        return 1
    print("PASS: lease card is complete enough to review the traffic PR")
    return 0


if __name__ == "__main__":
    target = sys.argv[1] if len(sys.argv) > 1 else "ops/lease-card.yaml"
    sys.exit(main(target))

python3 ops/check_lease_card.py ops/lease-card.yaml
echo $?

A missing card is not a warning. It is a stop. Do not add a bypass label because the demo is on a call in twenty minutes.

What the drill log must contain

You keep the drill log next to the card, in ops/exit-drill.md. Four lines are enough. Each line is evidence a reviewer can diff.

Start time and end time, in UTC, so the duration is not a vibe.
Source path on the borrowed host, and the destination path you own.
A checksum or file count that matches on both sides before you delete the spike copy.
The command or console step that unbinds DNS, the webhook, or the queue, plus who ran it.

If any line is missing, gate 4 or gate 6 fails. You do not finish the drill in a ticket and leave the proof in a screenshot nobody can diff. A screenshot can support the log. It cannot replace it.

Also record what you refused to copy. Customer payloads that should never have been on the spike box are an incident note, not a successful exit. The checker will not catch that. A human reading the log will, which is why gate 6 names a person.

Review the traffic pull request in this order

Open the lease card. Reject it if terms_read_on is outside your window.
Read allowance_note. If it looks remembered, send the author back to the live offer page.
Diff secrets. The borrowed host path must not gain a key, a token, or a .env.
Read the exit log. A path you own must show the copy, and the spike copy must be deleted in the drill.
Confirm the DNS, webhook, or queue diff is the only production binding in the request.
Require the named rollback owner to acknowledge the pull request.
Run the checker. Non-zero means no merge.

Short requests still get the full list. A small webhook is still production traffic if a customer can hit it.

A small decision table

Situation	Result	What you do
Card fresh, no secrets, exit rehearsed, traffic change is separate	Review may proceed	Attach traffic only behind the kill switch you already timed
Demo works, card blank	Fail closed	Stay in quarantine
Allowance copied from memory or an old post	Fail closed	Re-read the offer page and restamp the date
One commit creates the app and the DNS record	Fail closed	Split the change
Rollback owner is a rotation name	Fail closed	Name a person for this cutover
You need a durability promise the page does not make	Do not use this path	Pick a host whose contract you can cite

Use the table in the pull request comment. Paste the row you matched, and paste the checker output under it. Reviewers should not have to reconstruct your reasoning from a green demo GIF.

Limitations, and who should skip this

This checklist does not make a free server production-grade. It stops you from pretending a spike is a cutover. Skip it for regulated data, long-lived customer secrets, or any workload that needs an uptime promise.

Skip it if nobody will own rollback. A passing script is not a penetration test, a backup proof, or a legal reading of the offer. The 7-day window is a team choice in the proposal, not a vendor rule. Shorten it if the page you depend on changes often.

The checker parses a flat file on purpose so a reviewer can see every failure. It will not understand nested YAML. Keep the card flat. Do not publish quota figures, hardware names, or "unlimited" language in the card. Those claims go stale, and a stale card is how a borrowed host becomes an unowned dependency.

Keep the spike cheap enough to throw away

You can still learn on the borrowed box. Break the prompt. Time your own requests. Write down what failed. Those notes belong in a place you own.

Free model access and a free server are useful because the spike should be cheap enough to discard. Discard only works if exit is rehearsed before the URL leaves the notebook. Cheap is not the same as safe to bind.

If you want a concrete place to try that spike, use MonkeyCode's free model access and free server, copy the current offer into the lease card, and run the checker before any domain change. Stop there until a person reviews the traffic pull request.

Promotion Packets for Agent Branches: Four Gates That Fail Closed

Taylor Zhu — Fri, 09 Oct 2026 14:48:07 +0000

You do not promote an agent branch because it compiles. You promote it only when a fail-closed gate can show four pieces of evidence: a clean workspace record, a secret-clean diff, a human-owned lockfile decision, and a rollback pointer that already exists.

A cheap draft is not a merge. A shared scratch host can carry a leftover env file, a warm package cache, or a prompt from the previous session. Green tests on your laptop do not prove the patch was born clean. Hold the branch until the packet is complete.

The failure you are actually preventing

The interesting bug is not a red test. The interesting bug is a green branch that was born on a borrowed machine.

You asked an assistant to fix a flaky client. It edited three files. CI passed. Nobody asked whether the worktree started clean, whether the patch touched a credential path, or whether a lockfile moved because that host already had a different registry cache.

That gap is a promotion problem, not a model-quality problem. Treat the agent branch as untrusted input. Untrusted input does not get a protected remote until the evidence packet is complete. If a required file is missing, the answer is no. A warning is not a pass.

What you copy, and what you refuse to assume

Copy the gates, the evidence filenames, and the exit codes. Do not copy a quota, a hardware spec, or a permanence claim from outreach copy or from an old post.

This checklist is not a general production-readiness review, and it is not a substitute for code review. It answers one question: may this branch leave the scratch host for the protected remote?

You also do not hard-code a token budget from memory. Plans change. If a number is not on the current product page you opened today, it is not an input to this gate.

Four gates

1. Workspace record

Before the assistant edits anything, record the host and the base. A human wrapper writes this file. The model's final chat message is not evidence.

Require these fields:

host_id: a name you assigned to the scratch machine
started_clean: true only if git status --porcelain was empty
base_sha: the commit you will diff against
recorded_at: an ISO timestamp from that host

No file, no promote. A record that says started_clean: false is also a fail, unless you are in a deliberate dirty spike and a separate human override file exists. Do not flip the boolean by hand to get a green run.

2. Secret-clean diff

List every path in git diff --name-only <base_sha>. Reject the branch if any path hits a denylist.

Start boring:

.env and any .env.*
id_rsa, id_ed25519, *.pem, *.p12
credentials.json and anything under a secrets directory

A filename check is a tripwire, not a secret scanner. Keep the scanner you already trust. Do not let the tripwire be the only control, and do not let a missing scanner become a silent pass. Missing evidence fails closed.

3. Lockfile decision

If the diff touches a lockfile, a human owns that change in writing.

Watch the usual names: package-lock.json, pnpm-lock.yaml, yarn.lock, poetry.lock, uv.lock, Pipfile.lock, go.sum, Cargo.lock, and Gemfile.lock.

An assistant on a scratch host will often regenerate these from whatever index that host can see. That is how you ship a different tree than the one your team reviewed. If none of those names changed, this gate passes. If any changed, require evidence/lockfile-decision.json with human_owner, reason, and the same base_sha. An empty reason is a fail.

4. Rollback pointer

Create the rollback pointer before you push, not after the incident channel opens.

evidence/rollback.txt holds one rev the team already knows: a tag or a full SHA. The checker runs git rev-parse --verify on it. An empty file, or a rev this clone cannot resolve, stops the push.

You are not proving the rollback will be painless. You are proving you named a known rev before the new branch moved. If your remote deletes tags, add a remote check you control. Local resolution alone is not a remote guarantee.

Decision table

Use this as the review card. Do not negotiate it in the moment.

Evidence	Valid	Missing or false	Action
Workspace record, clean start	Pass	Fail	Stop. Recreate the worktree.
Denylist against diff paths	No hits	Any hit	Stop. Remove the secret path from the branch.
Lockfile decision	Present if lockfiles changed	Changed, no owner	Stop. Revert the lockfile or get a human owner.
Rollback rev	Resolves in this clone	Empty or unknown	Stop. Tag or record the base first.
Checker process	Exit 0	Exit 2, crash, or skip	Do not push. A skipped checker is a fail.

Fail closed means a non-zero exit. A crashed checker is a failed promotion, not an exception you ignore because the diff looks small.

The promotion sequence

Run the steps in order. Skipping a step to save time is how the packet rots.

Clone onto a disposable worktree. Confirm git status --porcelain is empty.
Write evidence/workspace.json and evidence/rollback.txt from the base SHA.
Let the assistant edit only that worktree. Keep protected-remote credentials off the scratch host.
Diff against base_sha. If a lockfile moved, add a human decision file or revert it.
Run the checker. Exit 0 is necessary, not sufficient. A human still reads the diff.
Push only from a machine that is allowed to hold the protected remote.

A checker you can copy

The script below is an unexecuted example. It is not a benchmark, and it is not a log from a run on your repo. Read it, then point it at a scratch clone. The intended contract is simple: missing evidence prints FAIL and exits 2. That is how the branches are written. It is not a captured measurement.

#!/usr/bin/env python3
"""Fail-closed promotion gate. Unexecuted example. Exit 0 only if evidence holds."""
import json, subprocess, sys
from pathlib import Path

ROOT = Path(".").resolve()
EVID = ROOT / "evidence"
DENY_NAMES = {".env", "id_rsa", "id_ed25519", "credentials.json"}
DENY_SUFFIXES = (".pem", ".p12")
LOCKS = {
    "package-lock.json", "pnpm-lock.yaml", "yarn.lock",
    "poetry.lock", "uv.lock", "Pipfile.lock",
    "go.sum", "Cargo.lock", "Gemfile.lock",
}

def fail(msg: str) -> None:
    print(f"FAIL: {msg}", file=sys.stderr)
    sys.exit(2)

def git(*args: str) -> str:
    proc = subprocess.run(["git", *args], cwd=ROOT, text=True, capture_output=True)
    if proc.returncode != 0:
        fail("git " + " ".join(args) + ": " + proc.stderr.strip())
    return proc.stdout.strip()

def denied(path: str) -> bool:
    name = Path(path).name
    if name in DENY_NAMES or name.startswith(".env"):
        return True
    if name.endswith(DENY_SUFFIXES):
        return True
    return "secrets" in Path(path).parts

def main() -> None:
    record_path = EVID / "workspace.json"
    if not record_path.is_file():
        fail("missing evidence/workspace.json")
    record = json.loads(record_path.read_text())
    for key in ("host_id", "started_clean", "base_sha", "recorded_at"):
        if key not in record:
            fail("workspace.json missing " + key)
    if record["started_clean"] is not True:
        fail("workspace did not start clean")
    base = str(record["base_sha"])
    git("rev-parse", "--verify", base)

    paths = [p for p in git("diff", "--name-only", base).splitlines() if p]
    hits = [p for p in paths if denied(p)]
    if hits:
        fail("denylist paths in diff: " + ", ".join(hits))

    lock_hits = [p for p in paths if Path(p).name in LOCKS]
    if lock_hits:
        decision_path = EVID / "lockfile-decision.json"
        if not decision_path.is_file():
            fail("lockfile changed with no lockfile-decision.json")
        decision = json.loads(decision_path.read_text())
        if not decision.get("human_owner") or not decision.get("reason"):
            fail("lockfile decision needs human_owner and reason")
        if decision.get("base_sha") != base:
            fail("lockfile decision base_sha does not match workspace")

    rollback_path = EVID / "rollback.txt"
    if not rollback_path.is_file():
        fail("missing evidence/rollback.txt")
    rev = rollback_path.read_text().strip()
    if not rev:
        fail("rollback.txt is empty")
    git("rev-parse", "--verify", rev)
    print("PASS: promotion evidence holds for", record["host_id"])

if __name__ == "__main__":
    main()

Collect the packet before the assistant edits, then run the checker before any push:

mkdir -p evidence
if [ -z "$(git status --porcelain)" ]; then clean=true; else clean=false; fi
sha=$(git rev-parse HEAD)
stamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)
cat > evidence/workspace.json <<EOF
{
  "host_id": "scratch-host-a",
  "started_clean": ${clean},
  "base_sha": "${sha}",
  "recorded_at": "${stamp}"
}
EOF
printf '%s\n' "$sha" > evidence/rollback.txt
python3 promote_gate.py

If started_clean is false, fix the worktree. Do not edit the JSON to force a pass. Break the gates on purpose once: delete rollback.txt, touch a lockfile, add a fake .env. You want exit code 2 each time, before you trust the script enough to put it next to an assistant.

Where a free model and a free server fit

Use the cheap workspace for the draft. Do not treat it as the source of truth.

MonkeyCode offers free model access and a free server option you can use to try this gate on a scratch branch. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims were supplied for this draft. They are not a model list, a token quota, a hardware spec, a duration, or a promise that the offer stays fixed. This article deliberately does not state a token figure, because no primary plan page was attached and an old number would be the wrong kind of evidence. Open the current docs and confirm limits before you depend on them.

The free server is a reasonable place to run promote_gate.py against a disposable clone. It is a bad place to store long-lived credentials, production remotes, or the only copy of a rollback tag. Keep protected-remote credentials off that host. Push only after the checker exits 0 and a human has read the diff.

If you delete every product name from this page, the gate still stands. The product is a convenient scratch host, not the evidence.

Limitations

The denylist misses secrets pasted into ordinary source files. Pair it with a scanner you already trust.
git rev-parse proves the rev exists in this clone. It does not prove the remote still has it.
A human name in JSON is not an approval system. If you need SSO-backed review, use your forge's pull request rules. This file is a local tripwire.
Clock skew can make recorded_at look fresh on a drifted host. Do not treat the timestamp as a legal audit log.
The script matches lockfiles by basename. A rename that hides a lockfile under a new path needs a human look.
Fail-closed scripts fail open the day someone aliases them to true. Pin the command in CI, or do not pretend you have the gate.

Who should not use this

Skip the checker if the spike will be deleted the same day and will never be pushed. Also skip it if your platform already blocks secret files, lockfile drift, and unsigned pushes with controls you have tested. A second local script will not make a mature control plane safer.

Do not park customer data, production keys, or regulated traces on a free shared server because the draft was cheap. This checklist assumes the scratch host is disposable and untrusted. If that assumption is false, stop and use an isolated runner your security team already accepts.

One next step

Pick one repo you already maintain. Add the checker on a branch that has no assistant in it. Break each gate on purpose and confirm exit code 2. Only then point an assistant at a scratch clone.

If you want that clone on a disposable host, MonkeyCode's free server option is enough to start, after you read the current limits yourself. Leave the protected remote for the machine that passed the gate.

Stamp a Capacity Contract Before the Agent Leaves the Notebook

Taylor Zhu — Thu, 08 Oct 2026 10:21:39 +0000

You should refuse the route until a capacity contract is stamped. A free model path that answered your demo is not capacity. A free server that stayed up for an afternoon is not a host contract either.

Ship the endpoint only after four facts are written down, dated, and checked. Those facts are the allowance text, the server identity, the data class you will send, and the date the source page was fetched. If any fact is missing, stale, or copied from memory, the gate fails closed and the route stays dark.

The failure you are actually gating

You can prototype an agent in an evening now. That speed is useful. It is also how a notebook path becomes a public URL before anyone records what the free tier promised.

Free access fails differently from a paid outage. The call may succeed, then throttle, then vanish when a quota window resets. The host may stay reachable while the model name behind it changes. You feel that as flaky product behavior. Your users feel it as a broken feature.

A green local run does not close this gap. You need evidence you can re-read next week, not a screenshot of one lucky reply.

What you stamp

Keep one JSON file next to the service. Treat it as release evidence, not as configuration the app should trust at runtime for secrets. The checker below only reads it.

Required fields:

doc_url: the page you actually opened
fetched_on: ISO date of that read
allowance_text: a short quote of the current allowance, or the exact words not stated
server_id: a concrete host or project id, not the word free
data_class: public-demo, synthetic, or customer — pick one
expires_on: when you will re-read the docs
fallback: refuse or a named paid path you already operate
probe_note: what a local smoke call showed, including the clock time

Do not put a token count in this file because a chat, a social post, or an old draft said a round number. Paste what the project page says on fetched_on. If it says nothing numeric, write not stated and let the gate fail.

Round numbers age badly. A figure that was true in a launch note can be gone by the time you promote. Your job is to cite the page you opened, not to preserve a memorable total.

A stdlib checker you can run today

This script is a proposal. It does not call a vendor, and it is not a benchmark. It exits 0 only when the file is complete, fresh, and internally strict. Anything else exits 2.

#!/usr/bin/env python3
"""Fail closed unless a capacity contract is stamped and fresh."""
import argparse, json, sys
from datetime import date, datetime

VAGUE = {
    "unlimited", "plenty", "free forever",
    "as much as you want", "tbd", "todo",
}

def parse_day(raw):
    return datetime.strptime(raw, "%Y-%m-%d").date()

def main():
    p = argparse.ArgumentParser()
    p.add_argument("contract")
    p.add_argument("--max-age-days", type=int, default=7)
    p.add_argument("--today", default=date.today().isoformat())
    args = p.parse_args()
    today = parse_day(args.today)
    reasons = []
    try:
        with open(args.contract, encoding="utf-8") as handle:
            doc = json.load(handle)
    except (OSError, json.JSONDecodeError) as exc:
        print(f"FAIL unreadable contract: {exc}")
        return 2
    required = [
        "doc_url", "fetched_on", "allowance_text", "server_id",
        "data_class", "expires_on", "fallback", "probe_note",
    ]
    for key in required:
        if not str(doc.get(key, "")).strip():
            reasons.append(f"missing {key}")
    if reasons:
        print("FAIL")
        print("\n".join(reasons))
        return 2
    try:
        fetched = parse_day(doc["fetched_on"])
        expires = parse_day(doc["expires_on"])
    except ValueError:
        print("FAIL dates must be YYYY-MM-DD")
        return 2
    age = (today - fetched).days
    if age < 0 or age > args.max_age_days:
        reasons.append(f"fetched_on age {age}d outside 0..{args.max_age_days}")
    if expires < today:
        reasons.append("expires_on is in the past; re-read the source")
    allowance = doc["allowance_text"].strip().lower()
    if allowance in VAGUE or allowance == "not stated":
        reasons.append("allowance_text is not a citable current limit")
    if not str(doc["doc_url"]).startswith("https://"):
        reasons.append("doc_url must be https")
    if doc["server_id"].strip().lower() in {"free", "localhost", "server"}:
        reasons.append("server_id is a label, not an identity")
    if doc["data_class"] not in {"public-demo", "synthetic", "customer"}:
        reasons.append("data_class must be public-demo, synthetic, or customer")
    if doc["data_class"] == "customer" and doc["fallback"] == "refuse":
        reasons.append("customer data needs a named fallback, not only refuse")
    if len(doc["probe_note"]) < 20:
        reasons.append("probe_note is too thin to audit")
    if reasons:
        print("FAIL")
        print("\n".join(f"- {item}" for item in reasons))
        return 2
    print("PASS capacity contract is stamped")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Pair it with a contract you fill by hand. The sample below is intentionally bad so you can see a fail before you trust a pass.

{
  "doc_url": "https://example.invalid/docs/free-access",
  "fetched_on": "2026-10-08",
  "allowance_text": "not stated",
  "server_id": "free",
  "data_class": "synthetic",
  "expires_on": "2026-10-15",
  "fallback": "refuse",
  "probe_note": "smoke pending"
}

Run it the same way every time you promote:

python3 check_capacity_contract.py contract.json --max-age-days 7 --today 2026-10-08
echo "exit=$?"

That sample must fail. allowance_text is not stated, and server_id is only the word free. Fix the file from a live doc read, then run again. A pass means the file is strict. It does not mean the vendor reserved anything for you.

Where a free model path fits

You still need somewhere to exercise the smoke note. MonkeyCode, as supplied for this draft, offers free model access and a free server option you can use for that probe. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it as the thing you measure, not as a number this page invents. Open the current project documentation, copy the allowance wording into allowance_text, and copy a real server or project id into server_id. If the page lists no quota, no hardware shape, and no duration, leave those claims out. The checker is supposed to fail until the page speaks.

Do not treat a pass on that free server as permission to send customer content. Keep data_class at synthetic or public-demo unless a separate review names a fallback you already run.

Model names belong in the same file only when the docs name them on the fetch date. If you hardcode a model id from memory, you have recreated the stale-contract bug in a different field. Add model_id later, with the same freshness rule, once you are ready to fail the job when it drifts.

Decide with the table, not with vibes

Evidence you have	Route decision
Fresh quote, concrete server id, synthetic data, fallback `refuse`	Demo route only
Same, but `fetched_on` older than your max age	Refuse until you re-read
Allowance text is a remembered round number with no URL	Refuse
`data_class` is `customer` and fallback is `refuse`	Refuse
Docs changed and your quote no longer matches	Refuse and restamp
You need an SLA, concurrency floor, or retention promise	Do not use this gate; get a written plan

Read the row before you open the port. A demo route means a token-gated URL, a banner that the path is uncommitted, and no customer payload. It does not mean quietly leave it up.

Wire the gate into promotion

Fetch the provider page and fill contract.json the same day.
Send one synthetic prompt to the free server. Write the status, a latency ballpark, and the timestamp into probe_note. Do not paste secrets or customer text into that note.
Run the checker in CI on the promote job. Fail the job on exit 2.
Set expires_on no later than seven days out. When that date passes, the job fails even if nothing else changed.
On fail, keep the previous route dark. Do not fall through to a silent default model.

python3 check_capacity_contract.py contract.json --max-age-days 7 || exit 2
# only then: your existing promote command

If you have no promote command yet, stop. The checker is not a deployer. It will not create a hostname, rotate a key, or drain traffic. Those steps stay in the pipeline you already review.

Pin the checker and the contract in the same commit as the route change. A pass on your laptop against an uncommitted file is not evidence the job will see. Reviewers should be able to open the JSON and the doc URL without asking you what the limit was.

Limits you should say out loud

This gate checks a file you wrote. It cannot see a quota change that happened after fetched_on. A seven-day max age only bounds how long you trust your own notes.

It does not load-test the free server. One smoke call in probe_note can hide a concurrency collapse. It does not review retention, training use, or region. Those are separate reads of the same docs, and you should add fields only when you will actually enforce them.

Free access can be withdrawn. A stamped file is a refusal rule, not a reservation. If your product cannot tolerate a sudden refuse, this path is the wrong host.

The vague-phrase list is short on purpose. You can sneak a fuzzy sentence past it. That is why a human still reads allowance_text in review. Automation catches empty and stale files. It does not replace judgment about whether the quote is specific enough to operate on.

Who should not use this

Skip the approach if you need a contractual SLA, a fixed model build, or a promise about hardware. Those are procurement problems. A checklist will not create them.

Skip it if you cannot name the data class. Customer content does not belong on a free server by default, and the script will not discover that for you.

Skip it if you want a score, a leaderboard, or a claim that one free host is faster than another. This artifact has no benchmark and should not be stretched into one.

Also skip it for a hackathon demo that will be deleted the same night, if nobody else can hit the URL. The stamp earns its keep when a route might outlive the person who created it.

One next step

Fill contract.json from the documentation you can open today, run the checker, and keep the route dark until you get PASS. If you are trying MonkeyCode's free model access and free server for that probe, copy the limits from the project page into the file first, then let the exit code decide, not the demo.

Stamp Every Write or Don't Share the Host: A Fail-Closed Attribution Checklist

Taylor Zhu — Thu, 08 Oct 2026 08:00:01 +0000

If you cannot name the principal, the tool, and the rollback owner for each write, do not share the agent host. A single-owner lab can be loose. A shared writer cannot.

Free model access helps you draft the change. A free server gives you a place to rehearse it. Neither one draws a tenancy line. You draw that line with gates, stored evidence, and a fail-closed default.

Share is a privilege, not a side effect

You can boot an agent service in an afternoon. That speed hides the boundary. While you are the only caller, a bad write stays local, and you are the person who notices it.

Add a second workflow and the same process becomes a shared runtime. Two jobs now share a credential, a working directory, and a log stream. A changed row no longer points at anyone. A vanished route takes both jobs down, and the record still has no owner.

Stop before that second writer is enabled. Isolation is the release gate. Missing fields are a stop, not a warning you park for later.

Six gates you can copy

A gate passes only when the evidence under it sits in the run record. Empty, null, or unverifiable means fail closed. The host stays single-tenant until every gate for that tenant is true.

Gate 1: Bind a principal

Every mutating call carries principal_id, workflow_id, and purpose. A shared API key is not an identity. If two jobs present the same key, you have not separated them.

File the request log line and the inventory row that maps that key to exactly one workflow. No inventory row, no share.

Gate 2: Deny tools by default

The host may call only tools on that workflow's allowlist. A new tool is a reviewed config change. A prompt that claims it needs a tool is not approval.

File the allowlist and its SHA-256 beside the run id. Start from a file like this, then replace the names with your own:

# Proposed allowlist. Not a live policy until you load it.
workflows:
  invoice-draft-v3:
    principal_id: team-billing/bot-invoices
    deny_by_default: true
    tools:
      - name: draft_invoice
        mutating: true
        rollback: revert_invoice_draft

Gate 3: Prove rollback on a fixture

Each write record holds resource_id, a before-image reference, and a rollback command or a compensating action. If you cannot reverse the write on a fixture, the tool stays disabled for that tenant.

File the command output from a rollback you actually ran. A comment in the ticket does not count. A fixture that only prints "would revert" is not enough unless your real runner uses that same command.

Gate 4: Keep logs from mixing

Every log line includes tenant_id. A query for tenant A must return zero lines for tenant B. Strip secrets and prompt bodies that contain credentials before the line leaves the host.

File a fixture result that shows the cross-tenant count is zero, plus the exact command. This probe is a proposal. Point it at your real log layout:

grep -c 'tenant_id=tenant-b' logs/tenant-a.log

A count other than 0 fails the gate. Do not explain the hit away in chat. Fix the split, then run the probe again.

Gate 5: Lose the free path, stop the writes

Free model access and a free server can throttle, change, or disappear. You do not control that window. Do not hard-wire a shared writer to a route you cannot replace. If the configured route or host identity is missing, refuse the mutation.

File durable_route_accepted only after a named owner reads the provider's current terms and records a fallback. Do not invent a quota, a box size, or an expiry date. If the durable route is not accepted, shared_writer_enabled stays false.

Gate 6: Cut the wire in one command

You need one switch that disables every mutating tool on that host. Practice it on a lab target. An untested switch is a paragraph, not a control.

File timestamped output of the disable command, then a follow-up call that shows the write was rejected. A local drill can be this small:

mkdir -p ./lab/run
touch ./lab/run/writes.disabled
test -f ./lab/run/writes.disabled && date -u +%Y-%m-%dT%H:%M:%SZ

Your real runner must check that flag, or an equivalent switch, before any mutate tool runs. The file alone does nothing.

How to read the results

Do not average the gates. One miss blocks the share.

Any required field missing or empty: do not share the host.
Fields present, but shared_writer_enabled is false: do not share the host.
Kill switch not tested in this change window: do not share the host.
All marks true, owner named, raw logs read: you may enable that one tenant. You may not silently add another.

A checker you can run before the second writer

This script is a proposed release gate. It does not call a model, and it does not open a socket. It reads a JSON evidence file and exits non-zero unless every required mark is present. Treat it as unexecuted until you run it on your files.

#!/usr/bin/env python3
"""Proposed fail-closed tenancy checker. Not a runtime sandbox."""
import json
import sys

REQUIRED = (
    "principal_id",
    "workflow_id",
    "tool_allowlist_sha256",
    "rollback_dry_run_ok",
    "log_isolation_ok",
    "durable_route_accepted",
    "kill_switch_tested_at",
)

def fail(msg: str) -> int:
    print("FAIL closed.", msg)
    return 1

def main(path: str) -> int:
    with open(path, encoding="utf-8") as handle:
        evidence = json.load(handle)
    missing = [key for key in REQUIRED if not evidence.get(key)]
    if missing:
        return fail("missing or empty: " + ", ".join(missing))
    if evidence.get("shared_writer_enabled") is not True:
        return fail("shared_writer_enabled is not true")
    print("PASS. Fields are present. Read the raw artifacts before you share.")
    return 0

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: python3 tenancy_gate.py evidence.json")
        raise SystemExit(2)
    raise SystemExit(main(sys.argv[1]))

Save a starter record that should fail. The timestamp is a placeholder, not a drill log.

{
  "principal_id": "team-billing/bot-invoices",
  "workflow_id": "invoice-draft-v3",
  "tool_allowlist_sha256": "replace-with-real-hash",
  "rollback_dry_run_ok": true,
  "log_isolation_ok": true,
  "durable_route_accepted": false,
  "kill_switch_tested_at": "2026-10-08T00:00:00Z",
  "shared_writer_enabled": false
}

python3 tenancy_gate.py evidence.json
echo "exit=$?"
sha256sum allowlist.json

You want a non-zero exit on that file. false is a failing value because the checker treats it as empty. That is deliberate. Flip a mark only after the artifact exists, paste the real hash, then run the checker again.

A pass means the record is complete. It does not mean the host is safe. Go read the log probe and the kill-switch output before you change your mind.

Use a free lab, then leave the share off

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is enough to draft allowlist notes, rollback steps, and fixture prompts. The free server option is enough to host the single-owner drill: run the checker, run the kill switch, and keep the second workflow disconnected. That is the slice that matches this method. The gates still work if you use another lab.

Do not paste production secrets, customer records, or live tokens into that draft path. A lab model is still a copy destination. Use synthetic fixtures.

Do not treat the offer as a capacity plan. This article names no model, no token quota, no hardware size, no duration, and no benchmark.

Those details change, and they are not assumed here. Read the current terms, write what you read next to durable_route_accepted, and date the note. If you cannot replace the route, leave the share off.

If you already have that free server, use it for this drill and keep shared_writer_enabled false until a named owner accepts the evidence. Do the drill on synthetic data. Stop when the checker fails.

Limits to say out loud

The checker inspects a file. It does not watch the process. A complete JSON record can still sit beside a tool that ignores the allowlist, a document that hides a write, or a pipeline that drops tenant_id after the test.

Enforcement belongs in the tool runner. This script only blocks a release that has no record.

It also does not recompute the hash. A non-empty string passes the field check. Compare sha256sum output in review, or extend the script before you call it strict.

There is no measured miss rate here. Do not invent one for a status slide.

Clocks skew. Keep the kill-switch command output, not only kill_switch_tested_at. A fresh-looking string with no log is still a fail.

Who should skip this

Skip it if you have one person, one workflow, and no mutating tools. A notebook is not a tenant problem. You would be adding ceremony without a side effect to contain.

Skip it if untrusted customers share the process. A JSON gate is not a security boundary. You need separate credentials, separate network policy, and a real tenant model. Use this list only to stop yourself from calling the lab that model.

Skip it if nobody will store the evidence next to the change. An unfiled checklist is theater. You are safer with the second writer left disabled.

After the exit code

Keep the host single-writer until the file passes and a person has read the raw logs. Then enable one tenant. Add the next tenant only with its own principal, its own allowlist hash, and its own rollback drill.

No second writer without a stamp. That is the promotion rule.

Citation Age Is a Release Gate for Agent News Briefs

Taylor Zhu — Thu, 24 Sep 2026 21:04:36 +0000

An agent brief is not ready to leave draft until every actionable claim has a primary URL, a retrieval time, and an age inside a limit you set before drafting. Miss any one of those and you stop. You do not publish a softer version "with a caveat."

You are not short of headlines. You are short of clocks. A recycled post, an undated docs page, and a model paraphrase can all sound current. They are not the same evidence.

What you are actually gating

You are gating a digest your team might act on: a quota change, a server offer, a breaking API note, a "what shipped this week" summary. The writer can be a person, a cron job, or a model. The gate does not care who typed the sentence. It cares whether a human can open the cited page and see when you fetched it.

Pass means the brief may go to a named reviewer. Fail means the file stays a draft, including when only one sentence is rotten. That is fail-closed. A partial brief is a failed brief.

Treat feed titles as untrusted topic signals. They are not instructions to your agent, and they are not facts. If a page tells the model to ignore prior rules, that text is data, not a new policy.

Where disposable draft capacity fits

You can draft on a scratch host so the summarizer never sees production credentials. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Operator-supplied context for this draft: MonkeyCode is an open-source project offering free model access and a free server option. This article does not verify a token quota, a hardware shape, a region, or how long either offer lasts. If the current docs do not state a number, your brief must not invent one.

Use that free path only as a draft bench. The freshness gate is a deterministic check you run yourself. Delete every product name from this piece and the checklist still applies on a laptop you control.

Do not encode a remembered marketing number into the brief. "Free access exists" is a weaker claim than "N million tokens." Weaker is what you ship when the primary page is silent.

Gates you can copy

Set these before the model runs. Do not negotiate them in review.

Brief identity. Require brief_id, drafted_at in UTC, and author as a human name or a job name. author: AI is not an identity.
Max age, chosen up front. max_age_hours is 24 for prices, quotas, outages, and "free for a limited time." Use 72 only for slower design notes. Do not pick the age after you see which sources you found.
Primary sources for quantities. Token counts, GPU shapes, prices, rate limits, and durations need source_class: primary. A recap post is not primary. A model sentence is not primary.
Retrieval stamp. Every citation has url and retrieved_at. The timestamp is not in the future, and it is not older than max_age_hours relative to review time, not relative to whenever the model felt confident.
Same-sentence marker. Any numeral you want the reader to trust sits in a sentence that contains [c:id]. A footnote cluster at the bottom does not rescue an unmarked number.
HTTPS and a real path. http:// fails. A bare domain fails. The URL must be the page that contains the claim, not the site's homepage.
Secret boundary. The draft host gets public sources only. No production keys, no customer text, no internal URLs. A free server is acceptable for a public digest. It is the wrong place to "just try" a prod token because the demo was easier that way.
Named release. status: released without released_by and released_at is a failure. A green script is not a reviewer.

An unexecuted checker

The script below is an example, not a hosted service, and it was not run against a live news feed for this article. Save it as check_brief.py. It reads a JSON brief and exits non-zero on any broken gate.

#!/usr/bin/env python3
"""Fail-closed freshness gate. Example for local use."""
import json, sys, re
from datetime import datetime, timezone, timedelta

NUM = re.compile(
    r"\b\d[\d,]*(?:\.\d+)?\s*(?:tokens|hours|days|GB|ms|%|USD)?\b", re.I
)
CITE = re.compile(r"\[c:([a-z0-9_-]+)\]")

def parse(ts):
    return datetime.fromisoformat(ts.replace("Z", "+00:00")).astimezone(timezone.utc)

def main(path):
    doc = json.load(open(path, encoding="utf-8"))
    errors = []
    now = datetime.now(timezone.utc)
    for key in ("brief_id", "drafted_at", "max_age_hours", "body", "citations"):
        if key not in doc:
            errors.append("missing %s" % key)
    if errors:
        print("\n".join(errors))
        return 1
    drafted = parse(doc["drafted_at"])
    if drafted > now + timedelta(minutes=5):
        errors.append("drafted_at is in the future")
    max_age = timedelta(hours=float(doc["max_age_hours"]))
    cites = {c["id"]: c for c in doc["citations"]}
    for cid, c in cites.items():
        url = c.get("url", "")
        if not url.startswith("https://") or url.rstrip("/") == "https://example.com":
            errors.append("%s: url must be a specific https page" % cid)
        fetched = parse(c["retrieved_at"])
        if fetched > now + timedelta(minutes=5):
            errors.append("%s: retrieved_at in the future" % cid)
        age = now - fetched
        if age > max_age:
            errors.append("%s: age %s exceeds max_age" % (cid, age))
        if c.get("kind") == "number" and c.get("source_class") != "primary":
            errors.append("%s: numeric claim needs source_class=primary" % cid)
    for sent in re.split(r"(?<=[.!?])\s+", doc["body"]):
        if not NUM.search(sent):
            continue
        ids = CITE.findall(sent)
        if not ids:
            errors.append("number without citation: %s" % sent[:80])
            continue
        for i in ids:
            if i not in cites:
                errors.append("unknown citation %s" % i)
    if doc.get("status") == "released" and not doc.get("released_by"):
        errors.append("released brief needs released_by")
    if errors:
        print("FAIL")
        print("\n".join(errors))
        return 1
    print("PASS")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Here is a fixture shape, not evidence of a live offer. Replace the URL with a page you opened today. Leave the sample host in place and the checker should fail on purpose, because example.com is not a source.

{
  "brief_id": "ai-brief-2026-09-25",
  "drafted_at": "2026-09-25T09:00:00Z",
  "max_age_hours": 24,
  "status": "draft",
  "author": "job:digest-draft",
  "body": "Free model access is an availability claim, not a measured quota [c:docs1].",
  "citations": [
    {
      "id": "docs1",
      "url": "https://example.com/docs/pricing",
      "retrieved_at": "2026-09-25T08:40:00Z",
      "source_class": "primary",
      "kind": "number"
    }
  ]
}

date -u +%Y-%m-%dT%H:%M:%SZ
python3 check_brief.py brief.json
echo $?

Exit 0 is the only freshness pass. Exit 1 means you fix the source or you delete the sentence. You do not average the two.

Debug a failure in this order

Work top down. Later steps lie if the clock is wrong.

Compare date -u on the draft host with a known NTP source. A skewed clock makes a fresh page look expired, or an old page look new.
If the error is missing, stop. Do not let the model "fill the JSON." You add fields from the fetch log.
If the error is age, refetch. Do not edit retrieved_at by hand to go green.
If the error is number without citation, split the sentence. Keep the qualitative clause. Drop the numeral until a primary page supports it.
If source_class is anything but primary on a quantity, downgrade the sentence. "A secondary post mentioned a quota" is not a number you can plan capacity on.
If status is released and released_by is empty, the script is telling you the process failed, not the prose. Put it back to draft.

A model line such as "I checked the docs" never satisfies step 4. Confidence is not a URL.

Decision table

Evidence in the file	Gate	What you do
Primary URL, fetched inside max age, numeral marked `[c:id]`	Pass freshness	Human reviews, then maybe release
Recap blog states a quota, vendor page not fetched	Fail	Delete the number
URL present, `retrieved_at` absent	Fail	Refetch and stamp
Docs page says access exists, no quantity	Pass only with no numeral	Write the weaker sentence
Model claims it verified a headline	Fail	Model text is not a citation
Checker passed, reviewer field empty	Fail on release	Keep the draft unpublished

Limits, and who should skip this

A fresh primary page can still be wrong. This gate checks structure and age, not truth. You still read the page before you cite a number in a ticket.

It will not notice a paraphrase that outruns the paragraph you fetched. Keep claims narrower than the heading you linked. It will not strip prompt-injection text from a fetched page. That is a fetcher problem. Do not pretend this script fetches anything.

Free model access can change without this article changing. Do not put a quota, a duration, or a hardware guess in a runbook because a draft once said so. Pin the Python you used in a one-line note next to the script. "Works on the free server today" is not a pin.

Skip the approach if you need a legal, medical, or financial sign-off. Skip it if no named person will set released_by. Skip it if the only host that can reach the sources would require production credentials. In that last case you do not have a digest pipeline. You have an incident shaped like a paragraph.

What you ship instead of a hype line

Write the sentence that can pass: free model access and a free server option are operator-supplied availability claims, and you confirm both on the project docs before you depend on them. A sentence that names a token grant, a forever window, or a specific box does not ship unless you retrieved a primary page that says so, this week, and the marker sits in that sentence.

If you already draft on MonkeyCode, point the job at public docs only and run check_brief.py before anyone pastes the brief into a ticket. Keep the checker even if that draft host disappears. The host is optional. The clock is not.

Budget the Loop or Don't Dispatch: A Fail-Closed Tool-Call Checklist

Taylor Zhu — Wed, 23 Sep 2026 17:02:14 +0000

If a model can invoke tools, an open-ended loop is a production bug, not a feature. You fail closed on iteration count, wall-clock budget, per-tool timeout, allowlist, and result schema—or you do not dispatch the first call.

Tool-calling demos make the happy path look finished. The model picks a function. Your runtime executes it. The model “thinks” again. That path hides failure modes unit tests never see: a mutating endpoint retried, a search tool hung, JSON that is almost valid, a loop that burns the afternoon because nobody set max_steps.

You do not need another agent framework to stop that. You need gates that can refuse the next call, plus evidence that they actually fired.

The problem you actually ship

A tool loop is a state machine you did not write. Each step can call a side-effecting API, block on a network you do not own, return text that is not the schema you documented, or ask for another step. Forever.

Green local traces do not prove any of that is bounded. A checklist that cannot fail the run is commentary. It is not a control.

Fail-open here looks friendly. The model gets “one more try.” The hung tool gets a longer timeout. Invalid JSON gets pasted into the next prompt as “context.” That is how a demo becomes an unbounded distributed job.

Four gates, then evidence

Copy these. If a gate cannot produce evidence, it is not a gate. Do not reorder them to be nice to the model.

1. Iteration cap

Gate: steps_used < MAX_STEPS before every model turn and every tool dispatch.

Evidence: a structured log line with run_id, step, and max_steps.

Fail closed: when step >= MAX_STEPS, skip the model, skip tools, return a terminal budget_exceeded error. No hidden extra turn.

2. Wall-clock budget

Gate: now - started_at < MAX_WALL_MS on a monotonic clock.

Evidence: start timestamp on the run record, plus the remaining budget at each step.

Fail closed: expire the run even if the model is mid-token. Cancel in-flight tools you own. Do not start new ones.

3. Per-tool timeout and allowlist

Gate: tool name is in a pinned allowlist, and the call is wrapped in a hard timeout shorter than the remaining wall budget.

Evidence: allowlist version or hash in the run record; timeout value per tool name.

Fail closed: unknown tool name is a hard error, not a prompt to invent another function. Timeout returns a typed error. The model may see that error only if budget remains.

4. Result schema

Gate: tool output is parsed against an explicit schema before it re-enters the model context.

Evidence: schema id, parse ok/fail, byte size after truncation.

Fail closed: invalid JSON, missing required keys, or oversized payloads never reach the next prompt. Raw text is not “close enough.”

Check budget first. Then allowlist. Then invoke under timeout. Then schema. Unknown tool beats timeout. Budget beats schema. You are not scoring elegance. You are stopping the loop.

Copy-paste checklist for the PR that binds tools

Use this on the change that wires tools into a runtime, not on the notebook that demoed them.

Named run — every dispatch carries a run_id. No anonymous loops in prod.
Hard MAX_STEPS — integer, committed in the same change as the tool bind. Not a dashboard default someone can raise without review.
Hard MAX_WALL_MS — includes model time and tool time.
Tool allowlist — exact names. Not a regex over the whole registry.
Per-tool timeout — shorter than remaining wall budget at dispatch time.
Failed steps still count — a timeout or schema miss increments the step counter. Otherwise retries are free and unbounded.
Schema on the way back — parse before concatenate. Truncate before parse if you must bound memory.
Terminal error types — budget_exceeded, tool_timeout, unknown_tool, schema_invalid stay distinct so operators can grep them.
No silent continue — catching a gate error and stuffing it into the prompt as a normal tool result is a bypass. Do not ship it.
Human resume is a new run — a retry gets a new run_id and a recorded parent. It is not a hidden extra iteration on the dead run.

Print the evidence or the run does not ship. A green UI screenshot is not evidence.

Example evidence line you can demand in logs:

{
  "run_id": "run-20260923-01",
  "step": 3,
  "max_steps": 8,
  "wall_ms_used": 4120,
  "max_wall_ms": 15000,
  "tool": "search",
  "allowlist_version": "tools-v4",
  "schema_id": "search.hits.v1",
  "gate": "ok"
}

If that object cannot be produced, the gate is theater.

Artifact: a fail-closed dispatcher you can run

The following is a proposal you can execute locally. It is not production telemetry. It does not call a vendor. It shows gate order and fail-closed errors.

# tool_loop_gate.py
from __future__ import annotations

import json
import time
from dataclasses import dataclass, field
from typing import Any, Callable, Dict, Mapping

class GateError(Exception):
    def __init__(self, code: str, message: str) -> None:
        super().__init__(message)
        self.code = code

@dataclass
class ToolSpec:
    name: str
    timeout_s: float
    schema_required_keys: tuple[str, ...]

@dataclass
class RunBudget:
    run_id: str
    max_steps: int
    max_wall_s: float
    started_monotonic: float = field(default_factory=time.monotonic)
    steps_used: int = 0

    def remaining_wall_s(self) -> float:
        return self.max_wall_s - (time.monotonic() - self.started_monotonic)

    def assert_can_step(self) -> None:
        if self.steps_used >= self.max_steps:
            raise GateError(
                "budget_exceeded",
                f"{self.run_id}: max_steps={self.max_steps}",
            )
        if self.remaining_wall_s() <= 0:
            raise GateError(
                "budget_exceeded",
                f"{self.run_id}: wall clock exhausted",
            )

def parse_tool_result(raw: str, spec: ToolSpec) -> Dict[str, Any]:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError as exc:
        raise GateError("schema_invalid", f"{spec.name}: not json ({exc})") from exc
    if not isinstance(data, dict):
        raise GateError("schema_invalid", f"{spec.name}: expected object")
    missing = [k for k in spec.schema_required_keys if k not in data]
    if missing:
        raise GateError("schema_invalid", f"{spec.name}: missing {missing}")
    return data

def dispatch(
    budget: RunBudget,
    allowlist: Mapping[str, ToolSpec],
    tool_name: str,
    invoke: Callable[[], str],
) -> Dict[str, Any]:
    budget.assert_can_step()
    spec = allowlist.get(tool_name)
    if spec is None:
        raise GateError("unknown_tool", f"not in allowlist: {tool_name}")

    timeout = min(spec.timeout_s, max(budget.remaining_wall_s(), 0.0))
    if timeout <= 0:
        raise GateError("budget_exceeded", "no time left for tool")

    # Proposal only: replace with a real kill (subprocess, worker, sandbox).
    deadline = time.monotonic() + timeout
    raw = invoke()
    if time.monotonic() > deadline:
        budget.steps_used += 1
        raise GateError("tool_timeout", f"{tool_name}: exceeded {timeout}s")

    parsed = parse_tool_result(raw, spec)
    budget.steps_used += 1
    return parsed

Dry-run it:

python - <<'PY'
from tool_loop_gate import RunBudget, ToolSpec, dispatch, GateError

allow = {"search": ToolSpec("search", timeout_s=2.0, schema_required_keys=("hits",))}
budget = RunBudget(run_id="run-demo-1", max_steps=2, max_wall_s=5.0)

def ok():
    return '{"hits": []}'

print(dispatch(budget, allow, "search", ok))

try:
    dispatch(budget, allow, "shell", ok)
except GateError as e:
    print(e.code, str(e))

budget.steps_used = 2
try:
    dispatch(budget, allow, "search", ok)
except GateError as e:
    print(e.code, str(e))
PY

You should see one parsed object, then unknown_tool, then budget_exceeded. If a second tool still runs after the cap, the gate is not fail closed.

The timeout in this snippet is a check after return. That is labeled on purpose. A shippable timeout must kill the worker. Do not take the post-hoc check to production by itself.

Decision table

Condition	Evidence you store	Action
`tool_name` missing from allowlist	name + allowlist version	raise `unknown_tool`; do not offer extra tools
`steps_used >= MAX_STEPS`	step counters	raise `budget_exceeded`; end run
wall clock exhausted	start + now	cancel in-flight work; same budget error
tool exceeds timeout	tool, timeout_s	raise `tool_timeout`; count the step
payload not JSON / missing keys	schema id	raise `schema_invalid`; do not stuff raw text into context
all gates pass	run_id, step, latency	increment step; continue

If two rows could apply, pick the stricter one. Do not let the model vote.

Proposed test plan (unexecuted)

Treat these as tests you still have to run in CI. They are not results.

Allowlist miss — dispatch rm when the allowlist is {search}. Expect unknown_tool and zero side effects.
Step cap — MAX_STEPS=1, two sequential tools. The second call never invokes invoke().
Failed steps consume budget — first tool returns invalid JSON; second dispatch is refused when MAX_STEPS=1.
Wall clock — inject a clock that jumps past max_wall_s between steps. Expect budget_exceeded.
Schema isolation — return not-json from a tool. The next model prompt must not contain that string.
Timeout ownership — hang the tool past timeout_s. The worker is killed, not joined.

Wire assertions to process exit codes. A skipped test is an open gate.

Bypasses that look like product behavior

Watch for these in review. They reopen the loop while sounding reasonable.

Catching GateError and sending str(error) back as a normal tool payload. That teaches the model to keep talking.
Counting only successful tools. Timeouts then become free retries.
Raising MAX_STEPS in a config map to “unblock” an incident. That is how the unbounded loop returns.
Sharing one budget across tenants or across users in the same process. One runaway run should not steal another run’s steps; it also should not inherit a huge shared cap.
Letting the model add a tool name to the allowlist mid-run. The allowlist is an operator artifact. It is not a conversation.

If you need a longer job, start a new run with a parent id. Do not mutate the cap on a live run.

Where a scratch model path fits

You will want a non-prod loop to prove the gates fire. Do not use the production model endpoint for that. Fake tools, tiny prompts, and the dispatcher above are enough to see unknown_tool and budget_exceeded on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a throwaway place to exercise the same gate against a model that is not your production path, MonkeyCode's free model access and free server option can host that dry run. Keep production credentials out of it. The checklist does not depend on that environment.

Limitations — who should not use this

This dispatcher is not an authorization system. An allowlist of tool names does not replace user-scoped auth, rate limits, or audit logs.

It does not measure answer quality. A loop that stops in three steps can still be wrong. Pair quality checks with a separate eval path; do not overload this runtime gate with scoring.

It is the wrong shape for batch ETL and for human-in-the-loop research that is supposed to run for hours. Those jobs need a workflow engine with checkpoints, not a request-scoped MAX_STEPS=8.

Do not use a post-return timeout as your only kill switch. Do not treat this list as a substitute for idempotency on mutating tools. If a tool can move money, inventory, or identity, bounding the loop is necessary and still insufficient. You still need authz, an idempotency key, and a human approval path you can name in the run record.

Teams that already dispatch tools through a workflow engine with SLAs, kill switches, and stored step evidence can skip this snippet. Use their controls. Do not run a second unbounded loop beside them “just for the agent.”

Ship the bound, not the demo

Tool calling is easy to demo and easy to leave unbounded. Put the four gates in the runtime that actually dispatches. Store the evidence. Fail closed.

If a PR binds a new tool without MAX_STEPS, wall clock, allowlist, timeout, and schema, you do not merge it. The model can wait.

No Key, No Write: Fail-Closed Idempotency Gates for AI-Touched APIs

Taylor Zhu — Tue, 22 Sep 2026 15:18:09 +0000

A retry without an idempotency key is not resilience. It is a second charge, a second ticket, or a second deploy. If an assistant, a worker, or a model-backed endpoint can fire a mutating call twice, you fail closed until the key, the replay window, and the evidence exist.

You do not need a new platform for this. You need a gate that refuses unsigned writes.

Start from the failure, not the model

AI-assisted paths retry more than humans. Timeouts look like failures. Tool calls get re-issued after a reconnect. A coding agent hits apply twice because the first patch looked incomplete.

Your unit tests can stay green while two refunds land. That is not an edge case. That is the default once anything in the loop can auto-retry.

Treat every mutating tool call as a write. Reads can be noisy. Writes need a key.

What “fail-closed” means here

Fail-closed is a refusal rule, not a dashboard color. If the request cannot prove uniqueness, the write does not leave your process.

Copy these criteria as-is:

No key, no write. Missing Idempotency-Key or idempotency_key is a 4xx from your gate, not a “best effort” log line.
Key is caller-owned. The model, agent, or retry loop does not invent a new key on each attempt. The key is minted before the first attempt and reused.
Replay window is explicit. Same key plus same payload inside the window returns the original receipt. Same key plus different payload is a conflict, not a merge.
Side effects are named. Create, charge, page, deploy, email. If the handler cannot name the mutation, it does not run.
Evidence is stored before ack. Request hash, key, actor, tool name, and result id land in an append-only receipt. No receipt, no success.

If any row above is “we will add that later,” the path is not production-ready. Later is how duplicates ship.

A gate you can paste into CI

Label the following as a proposal you must adapt. It is a local contract check, not a distributed consensus service.

# idempotency_gate.py
# Proposal: reject mutating tool calls that cannot prove replay safety.
from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from typing import Any

MUTATING_TOOLS = {"create_ticket", "refund_charge", "deploy_service", "send_page"}
REQUIRED = ("tool", "idempotency_key", "payload")


@dataclass(frozen=True)
class GateResult:
    ok: bool
    code: str
    detail: str


def canonical_hash(payload: Any) -> str:
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()


def gate_tool_call(call: dict, seen: dict[str, str]) -> GateResult:
    for field in REQUIRED:
        if not call.get(field):
            return GateResult(False, "missing_field", field)

    tool = call["tool"]
    if tool in MUTATING_TOOLS and call.get("dry_run") is True:
        return GateResult(True, "dry_run", tool)

    if tool in MUTATING_TOOLS:
        key = str(call["idempotency_key"]).strip()
        if len(key) < 16:
            return GateResult(False, "weak_key", "idempotency_key must be >= 16 chars")

        digest = canonical_hash({"tool": tool, "payload": call["payload"]})
        prior = seen.get(key)
        if prior is None:
            seen[key] = digest
            return GateResult(True, "accepted", digest)
        if prior != digest:
            return GateResult(False, "key_payload_conflict", key)
        return GateResult(True, "replay", digest)

    return GateResult(True, "read_path", tool)

Run it as a test, not as a story.

# test_idempotency_gate.py
from idempotency_gate import gate_tool_call


def test_missing_key_fails_closed():
    seen = {}
    call = {"tool": "refund_charge", "payload": {"charge_id": "ch_1"}}
    result = gate_tool_call(call, seen)
    assert result.ok is False
    assert result.code == "missing_field"


def test_replay_with_same_payload_is_ok():
    seen = {}
    call = {
        "tool": "refund_charge",
        "idempotency_key": "req_7f3c91a0b2dd44e1",
        "payload": {"charge_id": "ch_1", "amount": 1400},
    }
    first = gate_tool_call(call, seen)
    second = gate_tool_call(call, seen)
    assert first.code == "accepted"
    assert second.code == "replay"


def test_same_key_different_body_conflicts():
    seen = {}
    key = "req_7f3c91a0b2dd44e1"
    a = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 1400}}
    b = {"tool": "refund_charge", "idempotency_key": key, "payload": {"amount": 2800}}
    assert gate_tool_call(a, seen).ok is True
    conflict = gate_tool_call(b, seen)
    assert conflict.ok is False
    assert conflict.code == "key_payload_conflict"

python -m pytest test_idempotency_gate.py -q

Green tests here only prove the gate. They do not prove your payment provider honors the same key. Wire the provider’s idempotency header in a separate contract test, or keep the path closed.

Copy-paste production checklist

Print this. Fill the evidence column. Empty evidence is a no-ship.

Gate	Evidence you must attach	Fail-closed if missing
Mutating tools are listed	Checked-in allowlist, not a prompt	Unknown tool name can write
Key minted before first attempt	Client/trace code that creates the key once	Retry loop hashes a new UUID each time
Key travels on every hop	Header or envelope field in logs	Worker retries with a naked body
Payload hash is stable	Canonical JSON (sorted keys, no wall-clock)	Timestamp inside the hash
Replay window documented	TTL in hours, plus storage name	“Until restart” or “in memory for now”
Conflict is a 409	Status + metric + example	Silent overwrite
Dry-run is default for new tools	Flag in the tool schema	First call in prod is live
Timeout ≠ “maybe it wrote”	Distinct codes: timeout, unknown, committed	Timeout auto-retries a write
Human abort exists	Kill switch or feature flag	Only the agent can stop itself
Receipt is queryable	Request id → key → result id	Success body discarded after 200

Do not “approximately” pass a row. Approximately is how double pages reach on-call.

Commands that make the gate visible

You want the refusal in logs, not in a wiki. Start with boring shell checks.

# Proposal: inspect a captured tool-call log for unsigned writes.
# Replace the file with your own capture. Do not paste secrets.
jq -r 'select(.tool_call.mutating==true) | [.ts, .tool, .idempotency_key, .result] | @tsv' \
  ./captures/toolcalls.jsonl

# Fail CI if any mutating line lacks a key.
if jq -e 'select(.mutating==true and ((.idempotency_key|type)!="string" or (.idempotency_key|length)<16))' \
  ./captures/toolcalls.jsonl >/dev/null; then
  echo "fail-closed: mutating call without a usable idempotency key" >&2
  exit 1
fi

If you cannot produce captures/toolcalls.jsonl, you cannot audit retries. Capture is part of the gate, not a nice-to-have.

Where a scratch coding environment fits

You can draft the wrapper, the allowlist, and the tests in a throwaway workspace. That is useful when you do not want the first sketch sitting on a production laptop path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option. Those two things are enough to generate the gate file and the pytest above, then throw the workspace away. They do not replace the checklist. They do not prove your real payment or ticketing API honors keys. Keep the fail-closed rules on your side of the boundary.

If you try that path, export the files and run pytest on your machine. A generated test that never executed is not evidence.

Decision table for retries

Use this when someone says “just retry it.”

Observed outcome	Safe next step	Forbidden next step
HTTP 409 / key conflict	Inspect payload drift; do not retry with a new key	Mint a fresh key to “force it through”
HTTP 200 with receipt id	Stop. Store the receipt.	Fire the same body with a new key
Timeout, no receipt	Lookup by key. Only retry if lookup is empty and the API documents “timeout means not committed.”	Blind retry because the UI spun
5xx after a write-looking call	Lookup by key first	Exponential backoff on the write itself
Agent disconnected mid-tool	Resume with the original key	Let the agent start a sibling call
Dry-run succeeded	Flip dry-run off behind the same key only if the API treats dry-run as non-committing	Assume dry-run locked the resource

Read that last row twice. Dry-run semantics are vendor-specific. If the vendor doc is silent, you fail closed.

How this shows up in agent tool schemas

If a model or coding agent is allowed to call tools, the schema is part of production. A free-form arguments: object is not a contract.

{
  "name": "refund_charge",
  "mutating": true,
  "dry_run_default": true,
  "required": ["idempotency_key", "charge_id", "amount_cents"],
  "properties": {
    "idempotency_key": {"type": "string", "minLength": 16},
    "charge_id": {"type": "string", "pattern": "^ch_"},
    "amount_cents": {"type": "integer", "minimum": 1}
  }
}

Reject extra fields you do not understand. Agents like to add reason, note, and force. force is how keys get bypassed.

Pin the schema in git. Diff it in review. If the agent can widen the schema without a human receipt, you do not have a gate.

Limitations

This checklist does not give you exactly-once delivery across two data centers. It gives you at-most-one intended mutation per key inside one system that honors the key.

It will not save you if:

the downstream API ignores idempotency headers
your canonical hash includes clocks, randoms, or unordered maps
keys are derived from user text the model can rephrase
retries hop to a second cluster with a cold store
“success” is a UI checkbox instead of a stored receipt

In-memory seen dicts die with the process. Use them in unit tests. Do not use them as the production store.

Who should not use this approach

Skip this if your path is strictly read-only. Skip it if the binary is a one-shot CLI with no retry and no agent. Skip it if you already have a transactional outbox and the tool layer cannot emit writes except through that outbox.

Do not bolt this onto a system whose source of truth is a spreadsheet. The gate will look formal. The duplicates will still land in the sheet.

Ship rule

If you cannot answer “what happens on the second identical write?” with a receipt id, you do not ship the tool. Fail closed. Then fill the evidence column until a duplicate is boring.

A scratch assistant can help you type the wrapper. Your gate, your store, and your conflict code still have to exist after the chat ends.

Expire the Eval or Don't Enable: A Fail-Closed Measurement Checklist

Taylor Zhu — Mon, 21 Sep 2026 13:34:35 +0000

A model-backed endpoint is not ready because the unit suite is green. It is ready when the measurement that can stop enablement is named, hashed, dated, owned, and fail-closed. If that measurement is missing, you are not shipping a feature. You are hoping.

This is not another reminder that tests exist. You already have those. This is the gate that decides whether a prompt change, a retrieval tweak, or a tool-schema edit may take production traffic.

What you are actually gating

Unit tests check deterministic code. Model-backed paths are not deterministic in the way your CI likes. You still need a number. You still need a cutoff. You still need a clock on the set that produced that number.

Without a clock, last quarter's goldens keep blessing this week's behavior. That is how a "measured" system drifts into an unmeasured one. Quietly.

You will copy four things:

A fail-closed rule.
A gate matrix you can paste into a PR template.
A YAML manifest the job must refuse to run without.
A small runner that exits non-zero when the evidence is incomplete.

None of this requires a vendor. A free model path and a spare server help when you want the job to run without burning a paid key. They are optional. The gate is not.

The fail-closed rule

If any of the following is true, the enablement does not happen. No silent skip. No "we'll backfill the eval."

The eval set has no eval_id.
The eval set has no content hash recorded in the repo.
The eval set is older than its own max_age_days.
The eval set has no human owner.
The job cannot load the set from the path the manifest names.
The score is below threshold and there is no waiver file with an expiry.
The waiver is past expiry, or it does not name a ticket.

Read that again. A missing eval is a failed eval. An expired eval is a failed eval. An owned-by-nobody eval is a failed eval.

You do not "degrade to best effort" on the measurement that authorizes traffic. Best effort is for retries. Measurement is for permission.

Missing is red, not yellow

Teams love a skip. Skip looks kind in a dashboard. Skip is how a prompt PR lands on Friday and the goldens catch up "next sprint." Next sprint is where stale answers become product behavior.

If the job cannot run, the flag stays off. That is the whole method. Write it on the PR template so nobody has to remember it under pressure.

Gate matrix you can copy

Paste this into the PR that touches prompts, tools, retrieval, or the model-backed route.

Gate	Evidence you must attach	Fail-closed if missing
Identity	`eval_id` in `eval_gate.yaml`	Enablement blocked
Integrity	SHA-256 of the dataset file	Hash mismatch or empty hash
Freshness	`generated_at` + `max_age_days`	Age exceeded
Ownership	GitHub handle or team alias	Owner blank
Execution	CI log of the runner, same commit SHA	Job skipped or ran on another SHA
Threshold	`min_pass_rate` compared to scored output	Score absent or below cutoff
Waiver	`waivers/<eval_id>.md` with ticket + expiry	Waiver missing, expired, or unnamed
Rollback	Kill switch / flag name in the PR	No named flag, no enablement

You do not need all eight to be fancy. You need all eight to be present. Fancy comes later.

Treat the matrix as a receipt, not a vibe. If a cell is empty, the row failed. If the row failed, the flag stays down.

Manifest the job must refuse to invent

Create eval_gate.yaml next to the route you are protecting. Keep it boring.

# Proposal: commit this beside the model-backed route.
# Unexecuted example — wire paths to your repo.
eval_id: checkout_assist_v3
dataset_path: evals/checkout_assist_v3.jsonl
dataset_sha256: REPLACE_WITH_REAL_SHA256
generated_at: "2026-09-14"
max_age_days: 14
owner: "checkout-ml"
min_pass_rate: 0.86
blocking: true
feature_flag: "checkout_assist_v3"
model_path_env: MODEL_ENDPOINT

Two notes. First: blocking: true means CI red is the product. If you set it false, you are collecting telemetry, not a gate. Second: model_path_env names an environment variable, not a model. You pin the path in deploy config. You do not hide it in a notebook.

Hash the dataset in CI so a silent edit cannot ride along:

# Label: local helper, run from repo root.
python - <<'PY'
from pathlib import Path
import hashlib, sys
p = Path("evals/checkout_assist_v3.jsonl")
if not p.is_file():
    sys.exit("eval dataset missing")
digest = hashlib.sha256(p.read_bytes()).hexdigest()
print(digest)
PY

If that digest does not match dataset_sha256, the job fails. You do not "update the hash later." You update it in the same PR that changed the set, or you do not change the set.

Refresh without laundering the gate

Goldens have to move. Domains shift. That is not a license to move the cutoff in the same diff as the prompt.

Use a split:

Freeze the old hash in git history. Do not rewrite it.
Add rows in a dataset-only PR. New generated_at. New hash. Same min_pass_rate.
Land prompt or tool changes in a second PR that must beat the already-merged set.
Raise min_pass_rate only after the new set has survived a week of blocking CI.

If you change the prompt, the set, and the cutoff together, you did not measure a regression. You authored a new story and called it a score.

Keep the files dull and findable:

evals/checkout_assist_v3.jsonl
evals/checkout_assist_v3.README.md
waivers/
eval_gate.yaml
eval_gate.py

The README names who labels a row, what "correct" means, and which ticket last rebuilt the set. If that file is empty, the owner field in YAML is theater.

Runner that fails closed

The runner below is a proposal. It does not call a vendor SDK. It scores a JSONL file of {id, input, expect} rows against a callable you already own. Wire your own client. Keep the control flow.

# eval_gate.py — proposal / unexecuted example
from __future__ import annotations

import hashlib
import json
import os
import sys
from datetime import date, datetime, timezone
from pathlib import Path

import yaml


def fail(msg: str) -> None:
    print(f"FAIL-CLOSED: {msg}", file=sys.stderr)
    raise SystemExit(1)


def load_manifest(path: Path) -> dict:
    if not path.is_file():
        fail(f"manifest missing: {path}")
    data = yaml.safe_load(path.read_text()) or {}
    for key in (
        "eval_id",
        "dataset_path",
        "dataset_sha256",
        "generated_at",
        "max_age_days",
        "owner",
        "min_pass_rate",
        "blocking",
        "feature_flag",
    ):
        if data.get(key) in (None, ""):
            fail(f"manifest field empty: {key}")
    return data


def assert_fresh(generated_at: str, max_age_days: int) -> None:
    born = date.fromisoformat(generated_at)
    age = (date.today() - born).days
    if age > int(max_age_days):
        fail(f"eval expired: age={age}d max={max_age_days}d")


def assert_hash(dataset: Path, expected: str) -> None:
    digest = hashlib.sha256(dataset.read_bytes()).hexdigest()
    if digest != expected:
        fail(f"dataset hash mismatch: got={digest}")


def assert_waiver_if_needed(eval_id: str, pass_rate: float, min_pass: float) -> None:
    if pass_rate >= min_pass:
        return
    waiver = Path("waivers") / f"{eval_id}.md"
    if not waiver.is_file():
        fail(f"score {pass_rate:.3f} < {min_pass} and no waiver")
    text = waiver.read_text()
    if "ticket:" not in text.lower() or "expires:" not in text.lower():
        fail("waiver missing ticket: or expires:")
    # Keep parsing strict and local. Do not fetch the ticket tracker here.


def score_rows(dataset: Path, call) -> float:
    rows = [json.loads(line) for line in dataset.read_text().splitlines() if line.strip()]
    if not rows:
        fail("dataset empty")
    hits = 0
    for row in rows:
        got = call(row["input"])
        # Exact match on purpose. Fuzzy scoring belongs in a named scorer file.
        if str(got).strip() == str(row["expect"]).strip():
            hits += 1
    return hits / len(rows)


def main() -> None:
    manifest = load_manifest(Path("eval_gate.yaml"))
    if not manifest["blocking"]:
        fail("blocking is false; this job is not a gate")
    dataset = Path(manifest["dataset_path"])
    if not dataset.is_file():
        fail(f"dataset missing: {dataset}")
    assert_hash(dataset, manifest["dataset_sha256"])
    assert_fresh(manifest["generated_at"], manifest["max_age_days"])
    if not os.environ.get(manifest.get("model_path_env", "MODEL_ENDPOINT")):
        fail("model path env is unset; refuse to score against an implicit default")

    def call(prompt: str) -> str:
        # Wire your client here. Do not silently fall back to a different model.
        raise RuntimeError("plug in your model client")

    pass_rate = score_rows(dataset, call)
    assert_waiver_if_needed(manifest["eval_id"], pass_rate, float(manifest["min_pass_rate"]))
    print(json.dumps({
        "eval_id": manifest["eval_id"],
        "pass_rate": pass_rate,
        "feature_flag": manifest["feature_flag"],
        "commit": os.environ.get("GIT_SHA", ""),
        "scored_at": datetime.now(timezone.utc).isoformat(),
    }, indent=2))


if __name__ == "__main__":
    main()

Notice what the runner does not do. It does not default the model path. It does not treat blocking: false as success. It does not accept an empty dataset as 100%. Those are the holes teams leave "for later." Later is how stale goldens become production truth.

Exact match is a feature here. If your task is generative, do not sprinkle if "close enough" into this file. Add scorer.py, hash that file too, and name it in the manifest. Hidden fuzzy logic is a second model you forgot to version.

CI shape, not CI poetry

Keep the workflow stupid. One job. Same commit SHA as the merge.

# .github/workflows/eval-gate.yml — proposal
name: eval-gate
on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "eval_gate.yaml"
      - "src/checkout_assist/**"
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python eval_gate.py
        env:
          MODEL_ENDPOINT: ${{ secrets.MODEL_ENDPOINT }}
          GIT_SHA: ${{ github.sha }}

If the path filter never matches your prompt files, the gate never runs. That is a hole. Put the prompt files where the filter looks, or drop the filter.

Print the JSON on success and archive it. A gate without a receipt is a story you tell in standup. You already know how those rot.

Store the receipt next to the flag name. When someone asks "why is v3 on?" you point at a commit SHA, a hash, a pass rate, and a date. You do not point at a Slack memory.

Where a free model path actually helps

You need a real MODEL_ENDPOINT for the runner. A laptop that is asleep is not a runtime. A paid key that someone rotated is not a runtime either.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you are already using MonkeyCode, the useful part here is mundane: free model access and a free server option give you a place to keep MODEL_ENDPOINT pointed at something that answers when CI asks. That is the whole integration. The checklist does not care which product serves the bytes. It cares that the path is explicit, the job is blocking, and the goldens can expire.

Do not point production traffic at the same endpoint you use to score goldens unless that is a decision you have written down. Scoring and serving are different jobs. Mixing them is how a load spike looks like an eval outage, or the reverse.

One practical split:

CI eval uses the free server and free model access.
Production serving uses the pinned path your deploy config already named.
Both paths show up as env values. Neither is a default in code.

If you do not use MonkeyCode, run the same split against whatever endpoint you already trust. The artifact above still applies. Fork the checklist first. The product is optional.

Limitations

Exact string match will punish valid paraphrases. If your task is generative, replace score_rows with a named scorer file and version that file like the dataset. Do not hide the scorer in a prompt.

This checklist does not measure fairness, latency, or cost. If those can block a release, they need their own manifests. Do not overload min_pass_rate with three meanings.

max_age_days: 14 is an example, not a standard. Set the clock from how often your domain shifts. A tax table and a chitchat bot do not share a freshness budget.

The waiver path can become a culture hole. If every PR carries a waiver, you no longer have a gate. You have a diary. Cap concurrent waivers at one per eval_id, and make expiry shorter than max_age_days.

A hashed JSONL file is not a sampling strategy. If production traffic is 90% language A and your goldens are 90% language B, the gate is precise and wrong. Fix the set. Do not celebrate the score.

Who should not use this

Skip this if you do not have a model-backed route in production, and no plan to enable one. A checklist without a flag to protect is ceremony.

Skip this if your "eval" is a handful of vibe checks in a chat window. Write the JSONL first. Then gate.

Skip this if a human must approve every response at runtime. Your measurement problem is different. You need an audit log, not a pass rate.

Skip this if you cannot fail closed — regulated change windows, vendor-owned models you cannot pin, or an org that treats red CI as optional. The runner will only annoy you.

What you do on Monday

Pick one endpoint. One. Write eval_gate.yaml with a real hash or refuse to merge. Put blocking: true. Name the feature flag. Run the job on the PR that would have enabled it.

If the job cannot run, you do not enable. Stale goldens are not a vibe problem. They are an authorization problem. Treat them that way.

Cut Over the Model Path or Don't Ship: A Fail-Closed Inference Checklist

Taylor Zhu — Sat, 19 Sep 2026 09:39:15 +0000

If production can still reach the model endpoint you used while drafting, you did not cut over. A green deploy does not prove that. It only proves the process started.

Treat the inference path as a promotion surface. Pin the base URL, the model identity, the timeout, and the fallback. If any of those still point at a lab or shared drafting host, fail the gate. Do not “try prod first and fall back to the sandbox.” That pattern turns an outage into a data leak.

This is not a vibe-coding essay. It is a copy-paste checklist plus a small policy checker you can run in CI. Use it when an AI feature is about to leave the branch.

The failure you are actually shipping

You added a chat box, a summarizer, or a “explain this diff” button. During development the client pointed at whatever answered quickly. That is normal. It is also how lab hosts become silent production dependencies.

The bug is not “we used a model.” The bug is an unnamed path. Someone pasted a base URL into a helper. A generated client baked it into a default. A retry wrapper treats HTTP 429 from prod as a reason to call the drafting host. None of that shows up in a unit test that mocks complete().

You need evidence that prod traffic cannot reach the lab path. You also need evidence that when prod is unhealthy, the feature fails closed instead of wandering.

What “cut over” means here

Cut over is not “we created a prod API key.” Cut over means all four of these are true at once:

The production base URL is an allowlisted origin, not an environment default inherited from .env.local.
The model identifier is an explicit production contract, not default, latest, or an empty string.
Timeouts, token ceilings, and retry counts are set in prod config and are smaller than “keep hammering.”
Fallback is deny-by-default: no lab host, no alternate vendor you have not reviewed, no “best effort” second try against the drafting server.

If you cannot show those four, you are still in the sandbox. Ship the feature flag off.

Six gates, each with a receipt

Every gate needs an artifact. A Slack “looks good” is not a receipt.

Gate 1 — Named production origin

Pass: INFERENCE_BASE_URL in the production secret store matches a committed allowlist. The value is HTTPS, has no path wildcards, and is not a personal tunnel.

Fail closed: any prod manifest that still contains localhost, 127.0.0.1, ngrok, trycloudflare, or a hostname tagged lab, dev, sandbox, or draft.

Receipt: the commit SHA of inference-allowlist.json plus the secret-store version id.

Gate 2 — Model identity is not a moving alias

Pass: production sets INFERENCE_MODEL_ID to a pinned id your vendor or self-hosted gateway documents. The same id appears in the runbook.

Fail closed: latest, auto, empty, or a name that only exists on the drafting server.

Receipt: a one-line contract in the repo: model id, max tokens, and who owns rotation.

Gate 3 — Lab host denylist in CI

Pass: CI greps deployable config (Helm, Terraform, Docker Compose prod overlay, sealed secrets templates) and fails on denylisted hosts.

Fail closed: denylist bypassed with “temporary” comments, or the check only scans src/ and ignores deploy/.

Receipt: the CI job log URL for the merge commit.

Gate 4 — Budgets are numbers, not folklore

Pass: production config sets request timeout, connect timeout, max retries, and a per-request token ceiling. Retries do not change the destination.

Fail closed: unlimited retries, exponential backoff with no cap, or a retry that swaps base_url.

Receipt: the config snippet and a test that asserts the client is constructed once with prod settings.

Gate 5 — Fallback cannot re-enter the lab

Pass: on timeout, 5xx, or quota errors, the feature returns a user-visible failure and an internal metric. No second client.

Fail closed: “if prod fails, call the free server so the demo still works.”

Receipt: a failing integration test that stubs prod errors and asserts the lab host is never dialed.

Gate 6 — Traffic identity

Pass: production requests send a stable X-Service / User-Agent and an environment tag prod. Logs can answer “which app, which model id, which base URL” without reading source.

Fail closed: the drafting client and the prod client are the same binary with the same defaults.

Receipt: one redacted log line from a staging call that already uses the prod origin.

Copy-paste checklist

Use this in the PR. Check a box only when the artifact exists.

[ ] inference-allowlist.json lists the single prod origin (or the exact set of regional origins).
[ ] inference-denylist.json lists lab, draft, and tunnel hosts.
[ ] Prod overlay sets INFERENCE_BASE_URL and INFERENCE_MODEL_ID; neither is blank.
[ ] Client construction test fails if base_url is missing.
[ ] Error-path test fails if any HTTP call target is on the denylist.
[ ] Timeout, retry cap, and token ceiling are set in prod, not only in a README.
[ ] Feature flag default is off until Gate 1–5 receipts are linked.
[ ] Runbook names the owner of model-id rotation.
[ ] Rollback is “flag off,” not “point DNS back at the sandbox.”

If a box depends on “we will add it after launch,” the gate failed.

Artifact: a policy file CI can reject

Commit this as inference-policy.json. Keep it boring. Boring is reviewable.

{
  "allow_base_urls": [
    "https://inference.prod.example.internal"
  ],
  "deny_host_substrings": [
    "localhost",
    "127.0.0.1",
    "ngrok",
    "trycloudflare",
    "lab.",
    "sandbox.",
    "draft.",
    "dev-inference"
  ],
  "require_env": [
    "INFERENCE_BASE_URL",
    "INFERENCE_MODEL_ID",
    "INFERENCE_TIMEOUT_MS",
    "INFERENCE_MAX_RETRIES",
    "INFERENCE_MAX_OUTPUT_TOKENS"
  ],
  "max_retries": 1,
  "forbid_fallback_base_url": true
}

Replace the allow URL with yours. Do not add the drafting host “just for staging” in the same file that production loads. Staging gets its own overlay.

Artifact: a fail-closed checker

The script below is a proposal you can run locally and in CI. It does not call any model. It only inspects env and text files you pass as deploy roots. Label it unproven against your repo until you execute it once and keep the log.

#!/usr/bin/env python3
"""Fail closed if prod config can still reach a lab inference host."""

from __future__ import annotations

import json
import os
import sys
from pathlib import Path
from urllib.parse import urlparse

POLICY = Path("inference-policy.json")
SCAN_ROOTS = [Path("deploy"), Path("k8s"), Path("infra"), Path(".")]
SCAN_SUFFIXES = {".yml", ".yaml", ".json", ".tf", ".env", ".toml"}


def load_policy() -> dict:
    if not POLICY.exists():
        print("FAIL: inference-policy.json missing")
        sys.exit(2)
    return json.loads(POLICY.read_text())


def host_of(url: str) -> str:
    parsed = urlparse(url if "://" in url else f"https://{url}")
    return (parsed.hostname or "").lower()


def denied(host: str, needles: list[str]) -> str | None:
    for needle in needles:
        if needle.lower() in host:
            return needle
    return None


def check_env(policy: dict) -> list[str]:
    errors = []
    for key in policy["require_env"]:
        if not os.environ.get(key):
            errors.append(f"missing env {key}")
    base = os.environ.get("INFERENCE_BASE_URL", "")
    if base:
        host = host_of(base)
        hit = denied(host, policy["deny_host_substrings"])
        if hit:
            errors.append(f"INFERENCE_BASE_URL host matches denylist '{hit}': {host}")
        allowed_hosts = {host_of(u) for u in policy["allow_base_urls"]}
        if host not in allowed_hosts:
            errors.append(f"INFERENCE_BASE_URL host not allowlisted: {host}")
    model = os.environ.get("INFERENCE_MODEL_ID", "")
    if model.lower() in {"", "latest", "auto", "default"}:
        errors.append(f"INFERENCE_MODEL_ID is not pinned: {model!r}")
    try:
        retries = int(os.environ.get("INFERENCE_MAX_RETRIES", "99"))
    except ValueError:
        retries = 99
        errors.append("INFERENCE_MAX_RETRIES is not an int")
    if retries > int(policy["max_retries"]):
        errors.append(f"retries {retries} exceed policy max {policy['max_retries']}")
    return errors


def check_files(policy: dict) -> list[str]:
    errors = []
    needles = policy["deny_host_substrings"]
    for root in SCAN_ROOTS:
        if not root.exists():
            continue
        for path in root.rglob("*"):
            if not path.is_file() or path.suffix.lower() not in SCAN_SUFFIXES:
                continue
            if path.name == POLICY.name:
                continue
            # Skip local-only samples; prod overlays must still pass.
            if ".local." in path.name or path.name.endswith(".example"):
                continue
            text = path.read_text(errors="ignore")
            lower = text.lower()
            for needle in needles:
                if needle.lower() in lower and "prod" in path.parts:
                    errors.append(f"{path}: denylist hit {needle!r}")
    return errors


def main() -> int:
    policy = load_policy()
    errors = check_env(policy) + check_files(policy)
    if errors:
        print("FAIL-CLOSED: inference cutover incomplete")
        for item in errors:
            print(f" - {item}")
        return 1
    print("PASS: inference path looks cut over (config scan only)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Wire it so merge is impossible when it exits 1:

# .github/workflows/inference-cutover.yml
name: inference-cutover
on:
  pull_request:
    paths:
      - "deploy/**"
      - "k8s/**"
      - "infra/**"
      - "inference-policy.json"
      - ".github/workflows/inference-cutover.yml"
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Require prod env in the CI job (from secrets, not the PR)
        env:
          INFERENCE_BASE_URL: ${{ secrets.INFERENCE_BASE_URL }}
          INFERENCE_MODEL_ID: ${{ secrets.INFERENCE_MODEL_ID }}
          INFERENCE_TIMEOUT_MS: ${{ secrets.INFERENCE_TIMEOUT_MS }}
          INFERENCE_MAX_RETRIES: ${{ secrets.INFERENCE_MAX_RETRIES }}
          INFERENCE_MAX_OUTPUT_TOKENS: ${{ secrets.INFERENCE_MAX_OUTPUT_TOKENS }}
        run: python3 scripts/check_inference_cutover.py

Secrets belong in the store, not in the PR description. If CI has no prod URL, the job must fail. A skipped check is an open gate.

Decision table

Symptom	What you might tell yourself	Fail-closed action
Drafting host still in prod overlay	“Staging needs it”	Split overlays. Prod overlay cannot parse the lab hostname.
Model id is `latest`	“We want improvements automatically”	Pin an id. Rotate with a ticket and a replay test.
Prod 429, client calls lab	“Users should still get an answer”	Return an error. Page the owner. Never change `base_url` in retry.
Unit tests mock the SDK	“Coverage is high”	Add one test that inspects the constructed URL.
Tunnel URL in a hotfix	“Only for this incident”	Incident flag off. Do not hot-patch origin.
Free drafting server is fast today	“We can launch on it”	Launch is a contract plus capacity you control. Speed is not a contract.

Where a free drafting path belongs

Drafting against a shared or free inference endpoint is fine. Shipping that endpoint is not.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a place to sketch clients, prompts, and the checker itself, MonkeyCode’s free model access and free server option can stay on the drafting side of the line. Keep that origin on the denylist for production overlays. Generate the policy file there if you want. Then run the checker against the repo you actually deploy. The product mention ends here because the gate does not care which editor you used. It cares whether prod can still dial the lab.

Who should not use this approach

Do not use this checklist as a substitute for load tests, eval suites, or a real vendor contract. It will not tell you the model is accurate. It will not size GPUs. It will not prove GDPR. It only proves you did not leave a drafting path wired into the live service.

Skip it if you have no production inference origin yet. In that case the honest gate is “do not ship the feature,” not “scan a policy file that allowlists a wish.”

Also skip it if your app does not call a model at runtime. Generated code that never performs inference is a different review: dependency pins, tests, and side effects. This article is only for the path that answers complete() after deploy.

Limitations

The scanner is string-based. Obfuscated URLs, runtime service discovery, and hosts injected by a sidecar can evade it. Pair it with egress policy in the mesh or network layer. If the cluster can resolve the lab hostname, add a network deny.

The allowlist does not prove capacity. A pinned origin can still 429. Fail closed on that too: user-visible error, metric, page. Do not reopen the sandbox to absorb overflow.

Aliases will drift. Re-run the job when you rotate model ids. If nobody owns rotation, you do not have a contract. You have a default.

Ship rule

Name the production origin. Pin the model id. Deny the lab host in CI. Cap retries without changing destination. Link the job log on the PR. If you cannot, leave the flag off.

That is the whole method. The drafting environment can be free, shared, or noisy. Production cannot inherit it.

The Lab Host Is Not Prod: A Fail-Closed Promotion Checklist

Taylor Zhu — Fri, 18 Sep 2026 08:00:54 +0000

You can prototype on a free model and a free server. You cannot promote that host. Treat the lab as disposable, pin a promotion contract, and fail closed when the evidence files are missing.

That is the rule. Everything below is a copy-paste checklist for the cutover, not another merge ritual.

The leak you will actually ship

AI-assisted services rarely blow up because a generated loop is ugly. They blow up because the lab leaked.

You scaffold a handler. You run it on the same box that hosts the assistant. You paste the lab URL into a client. You leave a debug token in .env. A teammate then deploys "the same thing."

Unit tests will not catch that. A PR checklist about tools or retries will not catch it either. Promotion is a different gate. If the process that built the service still owns the runtime, you do not have production. You have a shared notebook with a port.

What counts as a lab

Call it a lab when all of these are true:

The coding assistant, the app process, and scratch data can share a host.
Model access is convenience-first, not contract-first.
Secrets are whatever unblocked the demo.
Nobody can name the production identity of the service.

Free model access and a free server option are useful in that phase. They become a problem only when you skip the cutover and ship the box you happened to be using.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a scratch environment with free model access and a free server option, MonkeyCode is an open-source project some teams park that work on. The checklist does not depend on it. Strip the product name out and the gates still stand.

Copy this promotion contract

Keep one file in the repo: PROMOTION.yaml. Deploy jobs should refuse to start unless every gate is pass and every evidence path exists on disk. Do not type pass by hand. Generate the files, then let a checker read both.

# PROMOTION.yaml — proposal for a local clone, not an executed audit
version: 1
service: billing-adapter
lab:
  allowed_hosts:
    - "127.0.0.1"
    - "lab.internal.example"
  must_not_appear_in:
    - deploy/
    - infra/
    - clients/
gates:
  - id: HOST_SPLIT
    question: "Is production a different host than the lab?"
    evidence: evidence/host-split.txt
    fail_closed: "Refuse deploy if a lab hostname appears under deploy/, infra/, or clients/."
  - id: SECRET_MATERIAL
    question: "Were lab tokens, cookies, and debug keys rotated out?"
    evidence: evidence/secret-scan.json
    fail_closed: "Refuse deploy if any lab secret pattern still matches."
  - id: MODEL_PIN
    question: "Is every model call pinned to a named contract, not the lab default?"
    evidence: evidence/model-pin.md
    fail_closed: "Refuse deploy if the client still reads LAB_MODEL or an unset MODEL_ID."
  - id: DATA_CLASS
    question: "Is production data blocked from the lab host?"
    evidence: evidence/data-class.md
    fail_closed: "Refuse deploy if the lab can still reach prod datastores."
  - id: OBSERVABILITY
    question: "Can you name the prod service in logs, traces, and alerts without the lab project name?"
    evidence: evidence/obs-names.txt
    fail_closed: "Refuse deploy if service.name still equals the assistant project."
  - id: BLAST_RADIUS
    question: "Is there a one-command rollback that does not SSH into the lab?"
    evidence: evidence/rollback.md
    fail_closed: "Refuse deploy if rollback docs mention the lab host."

Six gates. Six files. If a file is missing, the answer is no.

Fail-closed checker you can run locally

The script below is a proposal. Run it in a clone you control. It does not open a network connection. It fails if evidence is missing, if a gate is not pass, or if a forbidden lab host string appears under the listed trees.

#!/usr/bin/env python3
"""promotion_gate.py — fail closed when lab evidence is missing."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("install pyyaml in your local venv before running", file=sys.stderr)
    sys.exit(2)

REQUIRED_STATUS = {"pass"}


def load_contract(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or "gates" not in data:
        raise ValueError("PROMOTION.yaml must contain a gates list")
    return data


def scan_forbidden_hosts(root: Path, hosts: list[str], trees: list[str]) -> list[str]:
    hits: list[str] = []
    for tree in trees:
        base = root / tree
        if not base.exists():
            continue
        for path in base.rglob("*"):
            if not path.is_file():
                continue
            text = path.read_text(encoding="utf-8", errors="ignore")
            for host in hosts:
                if host and host in text:
                    hits.append(f"{path}: contains {host}")
    return hits


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=Path, default=Path("."))
    parser.add_argument("--config", type=Path, default=Path("PROMOTION.yaml"))
    args = parser.parse_args()

    contract = load_contract(args.config)
    failures: list[str] = []

    for gate in contract["gates"]:
        evidence = args.root / gate["evidence"]
        if not evidence.is_file():
            failures.append(f"{gate['id']}: missing {evidence}")
            continue
        marker = evidence.read_text(encoding="utf-8", errors="ignore")
        if "STATUS: pass" not in marker:
            failures.append(f"{gate['id']}: {evidence} has no STATUS: pass line")

    lab = contract.get("lab", {})
    hits = scan_forbidden_hosts(
        args.root,
        lab.get("allowed_hosts", []),
        lab.get("must_not_appear_in", []),
    )
    failures.extend(hits)

    if failures:
        print("PROMOTION FAILED")
        for item in failures:
            print(f"- {item}")
        return 1

    print(json.dumps({"result": "ok", "gates": [g["id"] for g in contract["gates"]]}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire it so deploy never starts without the checker:

python3 -m pip install pyyaml
python3 promotion_gate.py --root . --config PROMOTION.yaml

.PHONY: promotion-gate
promotion-gate:
    python3 promotion_gate.py --root . --config PROMOTION.yaml

If your pipeline cannot see evidence/, that is a fail, not a skip.

What each evidence file has to prove

Write short files. A novel in evidence/ is how teams hide a missing gate.

HOST_SPLIT

Prove the production runtime is a different machine, cluster, or account than the lab. A useful evidence/host-split.txt looks like this:

STATUS: pass
lab_host: 127.0.0.1
prod_runtime: ecs://billing-adapter-prod
checked_trees: deploy/, infra/, clients/
command: rg -n "127.0.0.1|lab.internal.example" deploy infra clients

If rg prints a line, you do not have a split. You have a rename.

SECRET_MATERIAL

Rotate anything the lab touched. Demo cookies, assistant tokens, copied Authorization headers, default DB passwords. Then record the scan, not the secret.

rg -n "LAB_|DEBUG_TOKEN|sk-live|BEGIN OPENSSH" --glob '!.git' . \
  && echo "still dirty" && exit 1
printf 'STATUS: pass\nscanner: rg\npatterns: LAB_, DEBUG_TOKEN\n' > evidence/secret-scan.json

Adjust the patterns for your repo. Do not paste live credentials into the evidence file. The file is a receipt, not a backup.

MODEL_PIN

Lab defaults are not a contract. Pin the caller so production cannot silently follow whatever the assistant used that afternoon.

STATUS: pass
caller: src/llm_client.py
required_env:
  - MODEL_ID
  - MODEL_ENDPOINT
  - MAX_OUTPUT_TOKENS
forbidden_env:
  - LAB_MODEL
unset_means: fail closed, do not fall back

If the client still does os.getenv("MODEL_ID", "whatever-the-lab-used"), the gate is red. A default is a second lab.

DATA_CLASS

Keep production records off the lab host. That includes logs that embed payloads, copied fixtures, and "just one customer row."

STATUS: pass
prod_datastores:
  - postgres://prod-billing
lab_may_read:
  - fixtures/synthetic/
lab_may_not_read:
  - customers
  - invoices
egress_check: no prod DSN in lab compose files

If your lab compose file still contains the prod DSN "for realism," throw the compose file away. Realism is not a reason to mix classes.

OBSERVABILITY

Rename the service before the first prod request. Lab project names in service.name, dashboard folders, or alert titles will page the wrong people.

rg -n "service.name|OTEL_SERVICE_NAME|job:" deploy infra \
  | tee evidence/obs-names.txt
# then append:
# STATUS: pass
# prod_name: billing-adapter
# lab_name_absent: yes

BLAST_RADIUS

Rollback must not require SSH into the machine where you vibe-coded the demo. If the runbook says "just restart the lab process," you have no blast-radius control.

STATUS: pass
rollback: kubectl rollout undo deploy/billing-adapter
does_not_use: ssh lab.internal.example
owner_alert: #billing-oncall

Decision table: promote, rebuild, or throw away

Signal	Promote	Rebuild in prod-shaped infra	Throw the lab away
Lab hostname in clients	No	Yes, after stripping	If clients cannot be found
Model id unset at runtime	No	Yes, after pinning	If nobody can name the contract
Prod data reached the lab	No	Only after rotation and review	Default when customer data moved
Rollback needs lab SSH	No	Yes	If no other runtime exists
Evidence file missing	No	No	Yes, until the file exists

Read the table left to right. Missing evidence is not "unknown." Missing evidence is "do not promote."

A 20-minute dry run

Label this unexecuted until you run it on your clone.

Create PROMOTION.yaml and empty evidence/ files without STATUS: pass.
Run python3 promotion_gate.py. Confirm it exits 1.
Put http://127.0.0.1:8080 into clients/demo.env. Run it again. Confirm the host scan fails.
Remove the lab URL. Fill each evidence file with a STATUS: pass line and the commands you actually ran.
Run the checker again. Only then allow the deploy job to see the artifact.

If step 2 passes, the checker is wrong. A first run with empty evidence must fail.

Limitations

This checklist does not prove load, correctness, or latency. It does not replace threat modeling, IAM review, or a real staging environment. String search will miss encoded hostnames, image layers, and secrets injected at runtime. STATUS: pass is only as honest as the person who wrote it.

Free servers and free model access also have limits you cannot inventory from a YAML file: capacity, tenancy, retention, and how long the option remains available. Do not treat a lab uptime number as a production SLO. Do not treat a convenience default as a vendor contract.

Who should not use this

Skip this approach if you already promote through a locked-down path that forbids third-party assistants on any host that can see production data. Skip it if you need a compliance certificate; a repo checklist is not one. Skip it if the service never leaves the laptop. And skip it if you were about to point production DNS at the same process that runs the assistant.

In that last case the fix is not a better YAML file. The fix is a different host.

If you need a disposable lab before you run this checker, MonkeyCode's free model access and free server option are one way to keep that scratch work off the production cluster. Use it as a lab. Then make the checker fail until the lab disappears from deploy/.

Green Tests Are Not a Runtime: A Fail-Closed Production-Readiness Checklist

Taylor Zhu — Thu, 17 Sep 2026 06:06:07 +0000

Green unit tests do not mean the change can take traffic. If an AI-assisted PR exposes a route, a job, or a migration, you block merge until rollback, capacity, and probe evidence exist as files your pipeline can fail closed on.

That is the whole rule. The rest of this article is a copy-paste checklist, a schema, and a small checker that treats TBD as a red gate.

Why "tests passed" is the wrong merge signal

AI-assisted diffs arrive faster than review habits. The suite goes green because the same session that wrote the handler also wrote the mocks. You get a PR that looks finished. Production does not care.

You need evidence that is boring and named. Not a screenshot of a local 200. A file in the repo that CI can parse, and a command that actually rolls the change back.

If a field is missing, you do not negotiate. You fail closed.

What this checklist covers

This is a runtime production-readiness gate. It answers one question: if this lands on a server that receives real requests, can you stop it, bound it, and see it?

It does not replace load tests in a prod-like environment. It does not certify a security review. It does not prove the model that drafted the code was correct. It only proves that someone named the operational contract before merge.

Do not use this as a substitute for an SRE handbook you already trust. Use it when AI-authored changes start skipping the messy parts: rollback, flags, probes, and numbers with units.

The fail-closed gates

Copy these nine gates. Each gate has required evidence and a fail-closed rule. Empty strings, TODO, TBD, and n/a without a reason code all fail.

1. Rollback is a command

Evidence: ops/rollback.md contains one fenced command block tagged rollback that a pager-holder can paste.
Fail closed: no tagged block, or the block is kubectl apply -f . with no version pin, or it says "revert the PR" with no SHA strategy.

A paragraph is not a rollback. If the on-call cannot paste one command, you do not have a rollback.

2. Probes have failure semantics

Evidence: ops/probes.yml names liveness and readiness paths, success codes, and the exact condition that must flip readiness to false.
Fail closed: readiness equals liveness, or either probe hits a dependency that can deadlock the process.

Liveness answers "should we kill this process?" Readiness answers "should we send it traffic?" If those are the same URL, you will either flap or serve errors with a smile.

3. Capacity is a number with a unit

Evidence: ops/capacity.yml sets max_rps, max_payload_bytes, and max_concurrency as integers.
Fail closed: any value is 0, negative, or a string like plenty.

Models love adjectives. Production wants integers. You can be conservative. You cannot be vague.

4. Timeouts are hard deadlines

Evidence: inbound request timeout and outbound client timeout are integers in milliseconds, and inbound must exceed outbound.
Fail closed: missing timeouts, or inbound shorter than outbound.

This is not a retry policy. Retries without a deadline just move the fire. You pin the wall-clock budget for one attempt so a hung dependency cannot hold a worker forever.

5. Flags default closed

Evidence: every new path is behind a flag whose default in config/flags.yml is false.
Fail closed: default true, or no flag name on a newly exported route.

Default-on is how a "small demo" becomes Friday traffic. Keep the flag name in the PR title so reviewers can grep it.

6. Migrations reverse

Evidence: db/forward.sql and db/reverse.sql both exist, and reverse is not a comment that says "restore from backup".
Fail closed: forward-only DDL, or reverse drops data with no expand/contract note.

Backups are a disaster plan. A reverse migration is a merge requirement. If you cannot unwind the schema, you cannot unwind the release.

7. Correlation IDs on every new path

Evidence: the new handler reads or mints x-request-id and logs it as a structured field.
Fail closed: print debugging, or logs without a request key.

When the page fires, you will search one ID across gateway, app, and store. If that ID was never written, you are reading poetry.

8. On-call owner is a person-shaped string

Evidence: ops/owner.yml has team and pager fields that match your roster file.
Fail closed: owner: ai or owner: whoever-merged.

Models do not carry pagers. A team name that cannot be resolved in the roster is the same as no owner.

9. Secrets stay out of the diff

Evidence: CI runs a scan; ops/secret_scan.txt is the last clean report path.
Fail closed: any private key PEM, AKIA prefixes, or .env files in the change set.

AI assistants paste from local env files with alarming confidence. The gate does not "trust the model." It trusts a scan artifact.

How to pick the numbers without lying

You do not need a week of benchmarking to fill max_rps. You need a ceiling you will defend at 2 a.m.

Start from last week's production for the parent service, not from a model suggestion. If you have no parent, start from the smallest integer that still lets a canary through: max_concurrency: 2, max_rps: 5, max_payload_bytes equal to the documented API limit.

Write the source of the number in a comment above the field. Comments are not parsed. They are for the reviewer who will ask "why 40?"

If you cannot name a source, the number is fiction. Leave the field out and let the checker fail. That failure is cheaper than a silent 500 storm.

Evidence schema you can commit

Put this at ops/prod_gates.yml. The checker below reads it. Do not keep a second copy in a wiki.

version: 1
service: payments-api
change_id: pr-1842
rollback:
  command: |
    kubectl set image deploy/payments-api payments-api=ghcr.io/example/payments-api:sha-9f3c1a
  verified_in: staging
probes:
  liveness: /healthz
  readiness: /readyz
  readiness_fails_when: "dependency_ping != ok"
capacity:
  max_rps: 40
  max_payload_bytes: 65536
  max_concurrency: 16
deadlines_ms:
  inbound: 800
  outbound: 300
flags:
  - name: payments_api_v2_charge
    default: false
migration:
  forward: db/forward.sql
  reverse: db/reverse.sql
observability:
  request_id_header: x-request-id
owner:
  team: payments-runtime
  pager: payments-oncall
secret_scan: ops/secret_scan.txt

Those numbers are placeholders. Replace them with your service's real ceilings. The gate does not care that 40 RPS is right. It cares that you typed an integer you are willing to own.

Keep reverse SQL next to forward SQL:

-- db/forward.sql
ALTER TABLE charges ADD COLUMN IF NOT EXISTS idempotency_key TEXT;

-- db/reverse.sql
ALTER TABLE charges DROP COLUMN IF EXISTS idempotency_key;

If reverse would destroy rows, stop. Split the change into expand, migrate, contract. The checker cannot see data loss. You can.

Checker that fails closed

Save as scripts/check_prod_gates.py. Run it in CI. Treat exit code 1 as a merge block, not a warning.

#!/usr/bin/env python3
"""Fail-closed production-readiness checker. Missing or TBD fields block merge."""
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("install pyyaml before running this gate\n")
    raise SystemExit(2)

BAD = {"", "tbd", "todo", "n/a", "none", "unknown", "ai"}


def fail(msg: str) -> None:
    print(f"FAIL: {msg}")
    raise SystemExit(1)


def load(path: Path) -> dict:
    if not path.is_file():
        fail(f"missing {path}")
    data = yaml.safe_load(path.read_text()) or {}
    if not isinstance(data, dict):
        fail("prod_gates.yml must be a mapping")
    return data


def need(v, label: str) -> None:
    if v is None:
        fail(f"{label} is missing")
    if isinstance(v, str) and v.strip().lower() in BAD:
        fail(f"{label} is a placeholder")


def main() -> None:
    root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
    cfg = load(root / "ops" / "prod_gates.yml")

    rb = cfg.get("rollback") or {}
    need(rb.get("command"), "rollback.command")
    cmd = str(rb.get("command"))
    if "revert the pr" in cmd.lower() and "sha" not in cmd.lower():
        fail("rollback.command must pin an image tag or git SHA")

    probes = cfg.get("probes") or {}
    need(probes.get("liveness"), "probes.liveness")
    need(probes.get("readiness"), "probes.readiness")
    if probes.get("liveness") == probes.get("readiness"):
        fail("readiness must not equal liveness")
    need(probes.get("readiness_fails_when"), "probes.readiness_fails_when")

    cap = cfg.get("capacity") or {}
    for key in ("max_rps", "max_payload_bytes", "max_concurrency"):
        val = cap.get(key)
        if not isinstance(val, int) or val <= 0:
            fail(f"capacity.{key} must be a positive int")

    dl = cfg.get("deadlines_ms") or {}
    inbound, outbound = dl.get("inbound"), dl.get("outbound")
    if not isinstance(inbound, int) or not isinstance(outbound, int):
        fail("deadlines_ms.inbound and outbound must be ints")
    if inbound <= 0 or outbound <= 0:
        fail("deadlines must be positive")
    if inbound <= outbound:
        fail("inbound deadline must exceed outbound deadline")

    flags = cfg.get("flags") or []
    if not flags:
        fail("flags must list at least one default-closed flag")
    for flag in flags:
        need(flag.get("name"), "flag.name")
        if flag.get("default") is not False:
            fail(f"flag {flag.get('name')} must default to false")

    mig = cfg.get("migration") or {}
    for key in ("forward", "reverse"):
        rel = mig.get(key)
        need(rel, f"migration.{key}")
        if not (root / str(rel)).is_file():
            fail(f"migration.{key} file missing: {rel}")

    obs = cfg.get("observability") or {}
    need(obs.get("request_id_header"), "observability.request_id_header")

    owner = cfg.get("owner") or {}
    need(owner.get("team"), "owner.team")
    need(owner.get("pager"), "owner.pager")

    scan = cfg.get("secret_scan")
    need(scan, "secret_scan")
    if not (root / str(scan)).is_file():
        fail(f"secret_scan report missing: {scan}")

    print("PASS: production-readiness gates are named and parseable")


if __name__ == "__main__":
    main()

Run it locally until it fails for the right reason:

python3 -m pip install pyyaml
python3 scripts/check_prod_gates.py .

Catch leftover fiction before CI does:

grep -nEi 'tbd|todo|n/a|plenty|revert the pr' ops/prod_gates.yml && echo 'placeholders found' && exit 1

Makefile target you can teach the team in one line:

.PHONY: prod-gates
prod-gates:
    python3 scripts/check_prod_gates.py .

CI sketch:

# .github/workflows/prod-gates.yml
name: prod-gates
on: [pull_request]
jobs:
  fail-closed:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python3 scripts/check_prod_gates.py .

If the workflow is continue-on-error: true, you do not have a gate. You have a diary.

Decision table for reviewers

Signal in the PR	Merge?	Why
Tests green, `ops/prod_gates.yml` missing	No	No runtime contract
Gates file present, `rollback.command` is prose	No	Cannot execute a paragraph
Capacity numbers copied from another service	No until edited	Copied ceilings are fiction
Flag default true "because demo"	No	Demo defaults leak
Reverse migration is "restore backup"	No	Disaster plan, not a reverse
Checker PASS, load test still pending	Yes, behind the flag only	Gate is necessary, not sufficient

Paste this table into the PR template. Reviewers should not re-litigate the rules in comments. The file either parses or it does not.

Where a scratch environment helps

You want the checker to run somewhere that is not your laptop and not production. A short dry run catches YAML that looks valid and still fails the integer rules.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you need a throwaway box plus model access to draft ops/prod_gates.yml from a diff and then execute scripts/check_prod_gates.py, MonkeyCode's free model access and free server option can host that loop. Do not treat that box as staging. Do not paste production secrets into it. Use it to get a red/green signal on the evidence files before you open the PR.

Limitations

The checker believes files. A determined author can write max_rps: 40 with no benchmark behind it. Fail-closed schema is not a load test. It is an honesty prompt with teeth.

Placeholder detection is string-based. max_rps: 999999 passes. You still need a human who knows the service.

Probe paths are not probed. The script does not HTTP-get /readyz. Hook that in staging separately.

Deadline math is local. It does not see the mesh timeout in front of you. If the proxy is 200ms and you wrote 800ms inbound, production will disagree. Add the proxy number to the YAML when you know it.

Who should not use this

Skip this approach if you already enforce production contracts in a service catalog with real owners and automated rollbacks. You would be duplicating a stronger system.

Skip it if your compliance process requires signed evidence in a GRC tool. A YAML file in git is not that artifact.

Skip it if the change cannot receive traffic — docs-only, comment-only, or lockfile bumps with no runtime surface. Forcing nine gates on a README change trains people to ignore gates.

Do not use a scratch-server path for regulated data, customer payloads, or production credentials. The checklist assumes public-ish metadata: probe paths, integer ceilings, flag names.

What you do tomorrow morning

Pick one service that recently merged an AI-assisted route. Add ops/prod_gates.yml with honest numbers, even if they are conservative. Run the checker locally until it fails for the right reasons, then until it passes.

Then attach the workflow. The first red CI is the point. After that, "tests passed" stops being the whole story, and traffic gets a named budget before it arrives.