DEV Community: Emery Lin

A Red Model Check Is Not a Rerun Signal: Triage, Repair Once, Then Merge

Emery Lin — Sat, 07 Nov 2026 18:38:30 +0000

Rerunning a red model check until it turns green is how a merge gate becomes decoration. The fix is not a better prompt. It is a policy that decides what kind of red you have before anything is allowed to run a second time.

Below is the gate I would wire into an existing pipeline: a JSONL ledger, a five-class triage table, one bounded repair attempt, and a quarantine rule that expires. Total new code is about 120 lines. No agent framework required.

The three reds you keep mixing together

A model-backed check can fail for three unrelated reasons, and a rerun answers none of them. Your diff genuinely broke something, your fixture or prompt moved underneath the check, or the endpoint answered differently for identical input. Only one of those is a code problem.

When you hit Rerun failed jobs without evidence, you convert the second and third cases into a green check. That is how a team ends up with a gate that has never caught anything in six months — and no one can prove it either way.

The rule: a rerun is only legal on an unchanged commit, and it produces a record, not just a new status badge.

What every attempt must record

You need five fingerprints before a verdict means anything. The prompt hash and the fixture hash are separate on purpose — collapsing them is what makes "the test started failing" impossible to diagnose.

Field	Why it exists
`diff_fp`	Hash of the patch, not the commit SHA. Survives rebases.
`prompt_sha`	Which review instructions produced this verdict.
`fixture_sha`	Which cases the check was evaluated against.
`verdict`	`pass` / `fail`, parsed, not eyeballed.
`attempt`	How many times this exact diff has been judged.

Step 1 — Fingerprint the diff, not the commit

Commit SHAs lie to you. An empty commit, a rebase, or a merge from main changes the SHA and changes nothing about the code under review. Hash the patch instead.

# scripts/model_gate.py (part 1)
import hashlib, json, pathlib, subprocess

ROOT = pathlib.Path(__file__).resolve().parents[1]

def sh(*args: str) -> str:
    return subprocess.run(args, cwd=ROOT, capture_output=True,
                          text=True, check=True).stdout

def sha(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def diff_fingerprint(base: str) -> tuple[str, list[str]]:
    patch = sh("git", "diff", "--binary", f"{base}...HEAD")
    files = sh("git", "diff", "--name-only", f"{base}...HEAD").split()
    return hashlib.sha256(patch.encode()).hexdigest()[:16], files

Step 2 — Classify the red before touching it

The classifier compares the new record against the last record for the same check. That is the whole trick: you are not comparing against yesterday, you are comparing against the last judgment of this check.

# scripts/model_gate.py (part 2)
def triage(rec: dict, prev: dict | None) -> str:
    if prev is None:
        return "first-run"
    if rec["diff_fp"] != prev["diff_fp"]:
        return "diff-changed"          # your code moved: rerun is legitimate
    if rec["fixture_sha"] != prev["fixture_sha"]:
        return "fixture-drift"         # the check moved, not the code
    if rec["verdict"] != prev["verdict"]:
        return "nondeterministic"      # identical inputs, different answer
    return "reproducible-red"          # identical inputs, same failure

Read the branches carefully. nondeterministic can only be reached when every fingerprint matches — meaning you re-ran the same commit with no edits in between. If you edit anything between runs, you land in diff-changed and you have learned nothing about flakiness. You have learned about your edit.

Class	Trigger	Action	Blocks merge
`first-run`	No earlier record	Report, don't repair	Yes
`diff-changed`	`diff_fp` differs	One rerun allowed	Yes
`fixture-drift`	`fixture_sha` differs, diff same	Stop, fix inputs	Yes
`nondeterministic`	All fingerprints equal, verdict flipped	Quarantine + ledger note	No, but time-boxed
`reproducible-red`	All fingerprints equal, stable fail	Exactly one repair attempt	Yes

Implement the decision as data, not as if chains scattered through YAML:

def decide(cls: str, verdict: str) -> tuple[str, int]:
    if cls == "fixture-drift":
        return "block:fixture", 1
    if cls == "nondeterministic":
        return "quarantine", 0        # advisory, and it expires
    if verdict == "pass":
        return "merge", 0
    if cls == "reproducible-red":
        return "repair", 1            # budget: one attempt
    return "report", 1

Step 3 — Bound the repair loop to one attempt

An unbounded agent that keeps patching until the check passes is a machine for producing unrelated diffs. Give the repair step a written contract and enforce it with a shell guard, not with trust.

# scripts/repair_guard.sh — a repair may not leave the original diff
BASE="${BASE:-origin/main}"
allowed=$(git diff --name-only "$BASE...HEAD" | sort -u)
touched=$(git diff --name-only HEAD | sort -u)
extra=$(comm -13 <(echo "$allowed") <(echo "$touched"))
if [ -n "$extra" ]; then
  echo "repair escaped the diff: $extra" >&2
  exit 1
fi

The contract, as a comment at the top of your repair prompt, is three lines and no more: only edit files already in the diff; do not touch fixtures, prompts, or CI config; if the failure cannot be fixed inside that boundary, output ESCALATE and stop. Anything else and the agent will "fix" the gate instead of the code — which passes review, because the diff looks confident.

Step 4 — Wire the hooks so the policy is not optional

Two entry points, one script. Local runs replay the fixture lane and never touch the network, so git push stays fast.

#!/usr/bin/env bash
# .git/hooks/pre-push
set -euo pipefail
python scripts/model_gate.py --mode local

# .github/workflows/gate.yml (trimmed)
- name: Fixture lane
  run: python scripts/model_gate.py --mode local
- name: Model lane
  run: |
    python scripts/model_gate.py --mode ci --base "origin/${{ github.base_ref }}" \
      | tee .ci/last-run.txt

The fixture lane is the part that keeps your local hook honest: recorded verdicts plus expected results, checked in, replayed in milliseconds. When someone edits a fixture case without updating the expectation, fixture-drift fires in CI and the local hook stays green — which is exactly the asymmetry you want.

Step 5 — Make quarantine expire

nondeterministic is the class most likely to become a permanent hiding place. A check that flips is a check you cannot use, so give it a deadline measured in merges, not vibes.

Write the entry to .ci/quarantine.txt with the merge count at entry time, then fail a nightly job when the entry is older than your agreed window. Two rules keep this honest: a quarantined check must be labelled on the PR by a human, and removing the label requires a new ledger record with matching fingerprints. Without the second rule, quarantine is just a quieter rerun button.

Where a free endpoint actually helps

The gate above needs two things teams stall on: an endpoint to call and somewhere to run the job. MonkeyCode documents free model access and a free server option, which maps onto exactly those two blockers — the model lane gets an endpoint, and the fixture lane plus ledger can run on the provided server instead of waiting on a runner procurement ticket.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free allowance as a way to start measuring, not as a capacity plan. Check the project's current terms for what is included, because token allowances and server specs change and a gate built on a stale number is worse than no gate. If you want the second lane running this week without a billing conversation, that is the shortest path I can describe honestly.

Limitations, and who should skip this

Temperature 0 is not determinism. Providers change routing and quantisation behind a stable endpoint name. The ledger detects this; it does not prevent it.
You are sending code to an endpoint. If your repository cannot leave your network, run --mode local only and accept that the model lane stays advisory.
Large diffs degrade the check. Truncate at a stated byte limit and record the truncation in the ledger, or your reproducible-red is really input-too-big.
The ledger grows forever. Rotate monthly; keep the last record per prompt_sha plus everything in quarantine.
Skip this entirely if your team cannot commit to reading the ledger. A gate nobody reads is a rerun button with extra steps.

Merge checklist

diff_fp is computed from the patch, not the commit SHA.
Prompt and fixture hashes are recorded separately.
Reruns are only permitted on unchanged fingerprints.
Repair is bounded to files already in the diff, enforced by repair_guard.sh.
Quarantine entries carry an entry timestamp and expire on a merge count.
Local hook replays fixtures and touches no network.
The ledger is rotated, not deleted, and the quarantine slice survives rotation.

If you only adopt one item, adopt item three. Everything else is bookkeeping around the single decision that keeps a model-backed gate worth having: you may rerun, but never to hide what you just learned.

Fixtures Rot: Put a Staleness Gate in Front of Your Model-Backed CI

Emery Lin — Fri, 06 Nov 2026 17:16:13 +0000

Fixtures Rot: Put a Staleness Gate in Front of Your Model-Backed CI

A recorded model response is a cache entry with no expiry date. That is the entire bug. You record it once, replay it forever, and six weeks later your pipeline is confidently green against a prompt nobody remembers editing.

The merge queue is usually not the weak point. Teams that adopt recorded fixtures tend to get the gate right and the lifecycle wrong. The fixture becomes a binary blob that everyone is afraid to touch, so it never gets refreshed, and its assertions slowly stop describing anything real.

Below is the lifecycle I use: a decision table for where each check belongs, a local hook that never touches the network, a manifest with a hash and a deadline, and a nightly job that opens a reviewable diff instead of quietly rewriting history.

1. Put each check where it can actually be trusted

Not every check deserves the same home. Local hooks and server-side CI have different failure modes, and mixing them is how you get developers bypassing --no-verify as a reflex.

Here is the placement table I start from. The latency budgets are example defaults, not measured numbers — set your own after one week of real data.

Check	Runs in	Network	Example budget	On failure
Manifest schema, prompt hash, fixture age	PR job	no	< 2 s	Block merge
Offline replay of recorded fixtures	pre-push hook + PR job	no	120 s	Warn locally, block in CI
Live model smoke test	Nightly only	yes	unbounded	Open an issue, never block merges
Re-record drifted fixtures	Nightly only	yes	unbounded	Open a PR for human review

Two rules fall out of that table. Anything requiring a network call lives off the pull-request path, because network-dependent checks are the ones that produce random red builds. And any check that can fail for a reason unrelated to the diff must not block the merge — it reports instead.

2. Make the pre-push hook refuse to touch the network

The hook has one job: catch the deterministic failures in under two minutes, then get out of the way. It should be boring, and it should be safe to run on a plane.

Create the hook file inside the repo so it is versioned, not per-machine.
Force the offline project selection through an environment variable.
Add a hard timeout, and fail open when the timeout fires.
Register the hooks directory once per clone.

mkdir -p .githooks
cat > .githooks/pre-push <<'HOOK'
#!/usr/bin/env bash
set -euo pipefail

# Only the offline project. Live model calls never run from a hook.
export MODEL_TESTS=offline

if ! command -v node >/dev/null 2>&1; then
  echo "pre-push: node not found, skipping offline suite" >&2
  exit 0
fi

timeout 120 npx --no-install vitest run --project offline || {
  status=$?
  if [ "$status" -eq 124 ]; then
    echo "pre-push: offline suite exceeded 120s, deferring to CI" >&2
    exit 0
  fi
  exit "$status"
}
HOOK
chmod +x .githooks/pre-push
git config core.hooksPath .githooks

Failing open on a timeout is deliberate. A slow laptop should not be able to block a push, and the same suite runs again in CI where timeouts are real failures. What the hook buys you is a fast local signal, not authority.

The part to enforce in review: no hook may read MODEL_ENDPOINT or any API key. If a model call appears in a hook, it will eventually hang on someone's flaky hotel wifi, and they will start pushing with --no-verify and never come back.

3. Give every fixture an owner, a hash, and a deadline

A fixture without a deadline is documentation. The manifest is what turns it into a check.

{
  "schema": 1,
  "fixtures": [
    {
      "id": "triage-missing-index",
      "path": "fixtures/triage/missing-index.json",
      "prompt_sha256": "REPLACE_WITH_REAL_SHA256",
      "model": "REPLACE_WITH_MODEL_ID",
      "recorded_at": "2026-11-01T02:14:00Z",
      "max_age_days": 14,
      "owner": "@your-team"
    }
  ]
}

Three fields do the heavy lifting. prompt_sha256 ties the recording to the exact prompt text that produced it. max_age_days is the point where an engineer has to look at the thing on purpose. owner means the staleness alert has a destination other than a shared channel nobody reads.

The verifier is short enough to read in full, and it has no dependencies beyond the standard library:

#!/usr/bin/env python3
"""Fail if a fixture is stale, its prompt changed, or it is not in the manifest."""
import hashlib
import json
import pathlib
import sys
from datetime import datetime, timezone

ROOT = pathlib.Path(__file__).resolve().parent.parent
MANIFEST = ROOT / "fixtures" / "manifest.json"
PROMPTS = ROOT / "prompts"
NOW = datetime.now(timezone.utc)


def sha256(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    manifest = json.loads(MANIFEST.read_text())
    errors: list[str] = []
    declared: set[str] = set()

    for fx in manifest["fixtures"]:
        declared.add(fx["path"])
        body = ROOT / fx["path"]
        prompt = PROMPTS / f'{fx["id"]}.md'

        if not body.exists():
            errors.append(f'{fx["id"]}: recorded body missing')
            continue
        if not prompt.exists():
            errors.append(f'{fx["id"]}: prompt file missing')
            continue
        if sha256(prompt) != fx["prompt_sha256"]:
            errors.append(f'{fx["id"]}: prompt edited since recording')

        recorded = datetime.fromisoformat(fx["recorded_at"].replace("Z", "+00:00"))
        age = (NOW - recorded).days
        if age > fx["max_age_days"]:
            errors.append(f'{fx["id"]}: {age}d old, budget {fx["max_age_days"]}d')

    for orphan in sorted((ROOT / "fixtures").rglob("*.json")):
        rel = orphan.relative_to(ROOT).as_posix()
        if rel != "fixtures/manifest.json" and rel not in declared:
            errors.append(f"{rel}: on disk but not in the manifest")

    print(f"checked {len(declared)} fixtures, {len(errors)} problem(s)")
    for err in errors:
        print(f"  - {err}")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main())

The orphan check is the one people skip and later regret. Fixtures deleted from the manifest but left on disk keep passing locally long after they stop being maintained, which is worse than deleting them.

Wire it as the first step of the PR job:

name: model-checks
on: pull_request
jobs:
  offline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 scripts/verify-fixtures.py
      - run: npm ci
      - run: npx vitest run --project offline

Two seconds of hashing before two minutes of tests saves the team from debugging an assertion that was never about their change.

4. Flake control means separating drift from randomness

Most "flaky model test" reports are one of three things: the fixture is stale, the prompt changed, or the assertion is genuinely non-deterministic. They need different responses, and conflating them trains the team to retry blindly.

Prompt hash mismatch — not a flake. The prompt moved; re-record deliberately.
Age over budget — not a flake. Refresh or raise max_age_days with a written reason.
Same fixture, different result on replay — a real bug in your harness or a genuinely unstable assertion.
Live smoke test failing intermittently — the only case that earns a retry.

Keep a quarantine file with a mandatory expiry, and make expiry itself a CI failure:

{
  "quarantine": [
    {
      "id": "live-smoke-summarizer",
      "owner": "@your-team",
      "reason": "endpoint timeouts under load",
      "expires_on": "2026-11-21"
    }
  ]
}

If a quarantined entry reaches its expiry date, the gate blocks and someone has to either fix it or extend it in a visible diff. Silent permanent quarantine is how a suite ends up with twelve disabled tests and a green badge.

The retry policy I recommend: zero automatic retries on the offline suite, up to two on the nightly live smoke, and both attempts recorded in the job log. If you cannot see both attempts, you cannot tell drift from noise.

5. Re-record on a schedule, and make the job produce a diff

Refreshing fixtures by hand does not survive a busy quarter. Put it on a schedule, and make the output a pull request rather than a commit to main.

The loop below is complete and runnable; the model call is a single stub you point at whatever endpoint you use. Labeling it that way is honest — endpoint shapes differ, and faking a client would only hide the one line you need to change.

#!/usr/bin/env python3
"""Nightly re-record. Signals drift by exiting non-zero so CI opens a PR."""
import hashlib
import json
import pathlib
import sys
from datetime import datetime, timezone

ROOT = pathlib.Path(__file__).resolve().parent.parent
MANIFEST = ROOT / "fixtures" / "manifest.json"


def call_model(prompt: str) -> dict:
    """Your only integration point: wire your model client here."""
    raise NotImplementedError("connect the endpoint you use")


def main() -> int:
    manifest = json.loads(MANIFEST.read_text())
    changed: list[str] = []

    for fx in manifest["fixtures"]:
        prompt_path = ROOT / "prompts" / f'{fx["id"]}.md'
        body_path = ROOT / fx["path"]
        old = json.loads(body_path.read_text())
        new = call_model(prompt_path.read_text())

        if new == old:
            continue

        body_path.write_text(json.dumps(new, indent=2, sort_keys=True) + "\n")
        fx["prompt_sha256"] = hashlib.sha256(prompt_path.read_bytes()).hexdigest()
        fx["recorded_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
        changed.append(fx["id"])

    if changed:
        MANIFEST.write_text(json.dumps(manifest, indent=2) + "\n")
        print("drift:", *changed, sep="\n  ")
        return 1

    print("no fixture drift")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Returning non-zero on drift is intentional. In most CI systems that turns the workflow into a failed run you can convert into a PR, which means a human reads the new outputs before they become the baseline. A job that silently commits is a job that launders regressions into expectations.

This worker is where a hosted environment actually helps, because the job needs two things a laptop cannot provide: a stable schedule and credentials that never sit in a developer shell. MonkeyCode is an open-source project that offers free model access and a free server option, so you can run the loop without provisioning anything first. The operator's current offer states 10 million tokens and a free server; treat that as the offer at the time of writing and check the live terms before you plan a schedule around it.

Do the arithmetic before you schedule anything. Tokens per day is roughly fixtures × average prompt tokens × runs per day, and a nightly run over a few dozen fixtures is a very different number from an hourly run over hundreds. My advice is to start nightly, measure one real week, then decide whether you need a second run per day.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One operational note: run this on a schedule or on prompt changes only. Never run it as part of a pull-request workflow, and never in a context where fork PRs can read the credentials.

6. Where this approach breaks, and who should skip it

This is a lifecycle for recorded fixtures, not a quality argument. A green replay proves your harness is deterministic and your fixtures are current — it says nothing about whether the model is good.

High-variance outputs. Free-form generation with byte-level comparison will churn on every run and teach the team to ignore the job. Use schema or rubric assertions instead of equality.
Sensitive prompts. If prompts or outputs contain customer data, do not record them at all. Write synthetic prompts and accept the coverage loss.
Fast-moving models. If model versions change faster than your max_age_days, the staleness gate becomes background noise. Shorten the budget or move the whole check to nightly reporting.
Very small suites. Under roughly ten fixtures, the manifest and verifier are more machinery than you need. A single test file with inline expectations is fine until it hurts.
Teams that want to block merges on model quality. Don't. Block on staleness and determinism, report on quality, and let a human read the drift diff.

If you already have recorded fixtures and no manifest, the cheapest first step is the verifier alone. Add hashes and ages for your existing files, run it once, and count the failures. That number tells you how much of your current green is earned.

If you want to try the nightly loop without a provisioning conversation, the free server option and free model access are enough to run it at small scale — start with ten fixtures and a seven-day budget. Keep one thing in mind as the suite grows: green should mean "not stale", and nothing more.

Test the Merge Result, Not the Branch: A Green-to-Merge Contract for CI

Emery Lin — Thu, 05 Nov 2026 15:53:34 +0000

Most CI systems answer the wrong question. They tell you whether your branch passes tests on a base commit that has already moved on. The tree you actually ship is the merge result, and nobody tested it.

This post is about closing that gap with three small pieces: a local merge check, a gate receipt that the merge queue can verify, and a flake budget that expires. None of it requires a new CI vendor.

1. Three ways a green PR head lies to you

A green check is a claim about head_sha on top of base_sha_at_push_time. That pair is already history by the time you merge.

Semantic conflict. PR A renames a function's parameter; PR B adds a caller using the old name. Both diffs are clean, the textual merge is clean, the merged tree does not compile.
Merge-only resolution. Lockfiles, generated clients, and changelogs resolve differently in the merged tree than in either branch. Your branch tested the lockfile it wrote, not the one that survived the merge.
Order dependence. A is green before B, B is green before A, A+B is red. Neither author sees the failure; the default branch does.

If your team merges more than a few PRs a day into one branch, at least one of these is happening weekly. The fix is not more tests on the branch. It is one test on the merge.

2. Rebuild the merge locally in one script

Before you change CI, prove the gap exists. This script builds the merge in a throwaway worktree and runs your existing gate inside it. Git 2.5+, no plugins.

#!/usr/bin/env bash
# scripts/merge-check.sh <base-ref> [head-ref]
set -euo pipefail
base="${1:-origin/main}"
head="${2:-HEAD}"
tmp="$(mktemp -d)"

git fetch --quiet origin "$base" || true
git worktree add --detach "$tmp" "$base" >/dev/null
trap 'git worktree remove --force "$tmp" >/dev/null 2>&1 || true' EXIT

git -C "$tmp" merge --no-ff --no-edit "$head" >/dev/null
merge_tree="$(git -C "$tmp" rev-parse 'HEAD^{tree}')"
head_sha="$(git rev-parse "$head")"
echo "merge_tree=$merge_tree head_sha=$head_sha"

( cd "$tmp" && ./scripts/gate.sh )   # your existing lint + unit + integration

Run it against your last twenty merges. If you want a cheaper pre-check without touching the working tree, git 2.38+ gives you the tree object directly:

git merge-tree --write-tree origin/main HEAD || echo 'textual conflict, stop here'

Step order matters: textual conflicts are cheap to detect and should fail first. Only spend CI minutes on a tree that at least merges.

3. Make each gate emit a receipt, not a badge

A badge says "green". A receipt says what was green, on which tree, for how long. The merge queue needs the second one, because a stale receipt is worse than no receipt.

{
  "schema": 1,
  "gate": "unit",
  "base_sha": "9f1c2ab",
  "head_sha": "4d0e77c",
  "merge_tree": "b31a8e0",
  "result": "pass",
  "quarantined": ["tests/test_clock.py::test_dst_boundary"],
  "duration_s": 412,
  "started_at": "2026-11-05T09:12:44Z"
}

Each gate writes one file into receipts/. The queue verifies them before it merges, and it refuses receipts that disagree on merge_tree:

# scripts/verify_receipts.py
import json, pathlib, sys

REQUIRED = {"lint", "unit", "integration"}
receipts = {}
for path in pathlib.Path("receipts").glob("*.json"):
    r = json.loads(path.read_text())
    receipts.setdefault(r["gate"], []).append(r)

missing = REQUIRED - receipts.keys()
if missing:
    sys.exit(f"missing gates: {sorted(missing)}")

trees = {r["merge_tree"] for group in receipts.values() for r in group}
if len(trees) != 1:
    sys.exit(f"receipts disagree on merge tree: {sorted(trees)} — stale gate, rerun")

failed = [g for g, group in receipts.items() if any(r["result"] != "pass" for r in group)]
if failed:
    sys.exit(f"failed gates: {sorted(failed)}")
print("all receipts agree on", trees.pop())

That single len(trees) != 1 check is the highest-value line in this article. It catches the classic race where one job retried on an older base and still reported green.

4. Label the failure, and don't trust the label

When the merged tree fails but both branches were green, the first question is always the same: is this a real defect or a flake? Getting that wrong in either direction costs an hour.

A model is decent at this as a first pass, because failure logs are long and repetitive and the signal is usually in ten lines. It is not good enough to be the decision. So: the label goes into the receipt as an advisory field, and the human keeps the merge button.

I use MonkeyCode's free model access for exactly this narrow step — a short classification call over the trimmed failure log, output constrained to a small enum. The operator also offers a free server option, which is enough to run the merge-check job for a small repository without tying up a laptop; treat that machine as ephemeral, with no SLA and no assumption that state survives.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability claims are the operator's, not a benchmark: at the time of writing the operator describes a free token allowance on the order of ten million and a free server option. Quotas, model rosters, and terms change without notice, so check the project's current terms before you put a number in a budget.

Keep the call small and constrain the output:

PROMPT = """Classify this CI failure log. Reply with JSON only:
{"label": "product"|"flake"|"infra", "confidence": 0.0-1.0, "evidence": "one line"}
Rules: product = deterministic defect in the diff; flake = timing/order/network nondeterminism;
infra = runner, disk, registry, or credentials. Quote the log line you relied on.
"""

Store the log and the returned label side by side as a fixture under test/fixtures/triage/. Your test asserts the schema and the enum — never string equality against a model response. When a human overrides a label, append both labels to the fixture. After a few weeks you have a labeled set you can reason about, which is the part that actually improves triage.

5. A flake budget that expires

Quarantine without an expiry date is deletion with extra steps. Encode both limits in a file the CI reads:

# .ci/flake-budget.yaml
budget: 3                  # max simultaneously quarantined tests
window_days: 7
quarantine_max_age_days: 14

Now the check is boring and absolute: more than three entries fails the build, and any entry older than fourteen days fails the build. Either someone fixes the test or someone deletes it, and both outcomes are honest. A quarantined test that still appears in the receipt's quarantined array is visible to reviewers, so "we skipped it" never becomes invisible.

6. When this pays off, and when it doesn't

Situation	Run the merge check?	Why
5+ PRs/day into one branch	Yes	Order dependence is near-certain
Generated files or lockfiles in the repo	Yes	Merge-only resolution is common
Docs-only changes	No	Skip by path filter, save the minutes
Monorepo with 40-minute gates	Partial	Merge check on affected packages only
Solo repo, one PR at a time	No	The branch is the merge result

7. Limitations, and who should skip this

This adds a second CI run per PR. If your gates take longer than about fifteen minutes, run the merge check only at queue time, not on every push.

Receipts are an audit trail, not a security control. Anyone with write access to the artifact store can forge one; if you need signed provenance, use your platform's attestation mechanism instead of a homegrown JSON file.

And if your platform already ships a merge queue that tests the merged result, use it. Writing this yourself is for teams whose queue exists but whose gates report on the wrong tree — a narrower problem than it sounds.

Finally, do not let an advisory model label gate a release. If a failure needs a compliance decision, a human reads the log.

8. Twenty-minute rollout

Run scripts/merge-check.sh against your last twenty merges; count how many merged trees would have failed.
If the count is zero, stop. You have the data and you saved yourself a project.
If it is non-zero, add the receipt fields to your three main gates.
Wire verify_receipts.py into the merge step, not the test step.
Add .ci/flake-budget.yaml with budget: 0 for a week. Whatever breaks first is your real flake inventory.
Raise the budget to 3 and start the fourteen-day clock.

Test the tree you ship. Everything else is a progress bar.

A Model-Backed Test That Passed Once Is a Sample Size of One

Emery Lin — Tue, 27 Oct 2026 13:24:38 +0000

A single green run of a model-backed test tells you almost nothing. It is one draw from a distribution you have never measured. Until you know that distribution, your merge button is a coin flip with extra steps.

Here is the fix in one sentence: sample each model-backed fixture N times, write a ledger, and let the merge rule read the ledger instead of the last exit code.

This post gives you the fixture contract, a reference sampler, a decision table, and two hooks — a cheap one locally, a strict one in CI.

Why reruns make the problem worse

Most teams handle a red model test by rerunning it. It goes green. Everyone moves on. Repeat that three times and you have trained the team to ignore the check.

There are two failure modes and they look identical in the UI. A real regression that failed once and passed on retry. Or a genuinely unstable case that passed this time and will fail after lunch. Reruns cannot separate them, because a rerun only gives you a second sample you throw away.

So stop rerunning to green. Start recording samples, then decide.

Step 1: Make the fixture assert on invariants, not prose

Most false flakes are self-inflicted. If your assertion is "output equals this paragraph," every run is a coin flip. Assert on structure, ranges, and required fields instead.

# tests/model/cases/summarize_changelog.yaml
id: summarize_changelog
input: fixtures/changelog.txt
expect:
  format: json
  required_keys: [breaking, added, removed]
  max_items_per_key: 5
  forbid_substrings: ["I cannot", "As an AI"]
  max_output_chars: 1200

Four rules keep this honest. Assert JSON shape, not wording. Assert counts and bounds, not exact strings. Never assert on whitespace or markdown formatting. And keep one "golden" case per feature, not per fixture — golden cases are for humans to read, not for the gate.

Now write each case outcome to a log so a sampler can read it.

# tests/model/conftest.py
import json, os
from pathlib import Path
import pytest

LOG = Path(os.environ.get("MODEL_RUN_LOG", "artifacts/run.jsonl"))
RUN_ID = os.environ.get("MODEL_RUN_ID", "local")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when != "call":
        return
    LOG.parent.mkdir(parents=True, exist_ok=True)
    with LOG.open("a") as fh:
        fh.write(json.dumps({"run": RUN_ID, "case": item.nodeid, "passed": report.passed}) + "\n")

Step 2: Sample, tally, and write a ledger

The sampler below is a reference implementation. Trim it, rename it, and run it against your own suite — the numbers it prints will be yours, which is the only kind worth having.

#!/usr/bin/env python3
# tools/sample_model_tests.py
import hashlib, json, os, subprocess, sys
from collections import defaultdict
from pathlib import Path

RUNS = int(os.environ.get("SAMPLE_RUNS", "5"))
FLAKY_BUDGET = int(os.environ.get("FLAKY_BUDGET", "0"))
CASES = Path("tests/model/cases")
LEDGER = Path("artifacts/model-ledger.json")


def fixture_digest() -> str:
    h = hashlib.sha256()
    for p in sorted(CASES.glob("*.yaml")):
        h.update(p.name.encode())
        h.update(p.read_bytes())
    return h.hexdigest()[:12]


def main() -> int:
    digest = fixture_digest()
    log = Path("artifacts/run.jsonl")
    log.unlink(missing_ok=True)

    for i in range(RUNS):
        env = {**os.environ, "MODEL_RUN_ID": f"{digest}-{i}", "MODEL_RUN_LOG": str(log)}
        subprocess.run([sys.executable, "-m", "pytest", "tests/model", "-q"], env=env, check=False)

    tally = defaultdict(lambda: [0, 0])  # case -> [passes, runs]
    for line in log.read_text().splitlines():
        row = json.loads(line)
        tally[row["case"]][1] += 1
        tally[row["case"]][0] += int(row["passed"])

    cases, flaky = {}, []
    for case, (passes, runs) in sorted(tally.items()):
        rate = passes / runs
        state = "stable-pass" if rate == 1 else "stable-fail" if rate == 0 else "flaky"
        if state == "flaky":
            flaky.append(case)
        cases[case] = {"passes": passes, "runs": runs, "pass_rate": round(rate, 2), "state": state}

    ledger = {"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky, "cases": cases}
    LEDGER.parent.mkdir(parents=True, exist_ok=True)
    LEDGER.write_text(json.dumps(ledger, indent=2) + "\n")
    print(json.dumps({"fixture_digest": digest, "runs_per_case": RUNS, "flaky": flaky}, indent=2))

    if len(flaky) > FLAKY_BUDGET:
        print(f"BLOCK: {len(flaky)} flaky case(s) exceed budget {FLAKY_BUDGET}", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

Two details matter more than they look. The fixture_digest ties a ledger to an exact set of fixtures, so a stale ledger cannot silently bless new cases. And the budget is a number you set deliberately, not a vibe.

Step 3: Turn the ledger into a merge rule

A pass rate is only useful if it maps to an action. This table is the whole policy — copy it into your PR template so reviewers see the same rule the gate uses.

Ledger state	Pass rate	Merge decision
stable-pass	1.00 over 5 runs	Merge. Keep it as a required check.
stable-fail	0.00 over 5 runs	Block. This is a regression, not noise.
flaky, mostly passing	≥ 0.60	Quarantine to nightly, open a ticket, block only if the case guards a high-blast-radius path.
flaky, mostly failing	< 0.60	Block. Treat it as failing until proven otherwise.
fixtures changed	digest mismatch vs. baseline	Block pending an explicit re-baseline in the PR body.

The "mostly passing" row is where teams cheat. A case at 0.6 passes four times out of five, so it will merge bad code eventually. Quarantine is a debt ticket, not a fix.

Step 4: Two hooks, two different jobs

Run a cheap sample before you push, and a strict one in CI. The local hook should be fast enough that nobody disables it.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
SAMPLE_RUNS=3 FLAKY_BUDGET=0 python tools/sample_model_tests.py

Install it once per clone:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push

The CI job runs more samples and uploads the ledger as an artifact, so a reviewer can read the evidence instead of trusting the checkmark.

model-flake-gate:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - name: Sample model-backed fixtures
      env:
        MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
        MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
        SAMPLE_RUNS: "5"
        FLAKY_BUDGET: "0"
      run: python tools/sample_model_tests.py
    - uses: actions/upload-artifact@v4
      if: always()
      with:
        name: model-ledger
        path: artifacts/model-ledger.json

Pin action versions to commit SHAs in a real repository, and keep the API key out of logs — the sampler never needs to echo it.

Where MonkeyCode fits in this workflow

Sampling multiplies your run count by five, which is exactly the workload you do not want to pay per call for while you are still calibrating fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source project offering free model access — currently stated as 10 million free tokens — plus a free server option. For this workflow that combination is a reasonable place to run the sampler: point MODEL_BASE_URL at the free endpoint, and use the free server to host the sampler job so five-times runs do not eat your own CI minutes. Check the project's current terms yourself before you build a budget or a contract around them.

One caveat that is not a caveat but a requirement: temperature 0 reduces variance, it does not remove it. Ledgers are endpoint-scoped. If you switch providers or models, re-baseline, or you will compare two different distributions and call it a regression.

Who should not use this approach

If you need contractual latency or availability guarantees, data residency, or audited per-request logs, a free tier is the wrong foundation for your gate — use a provider you can hold to terms. If your model tests are a demo rather than a required check, the sampler is overhead; one run is fine. And if your fixtures assert on exact prose, fix the fixtures first, because sampling will only tell you what you already know.

The useful habit is smaller than the tooling: stop asking whether the test passed, and start asking how often. If you want to try the sampler before wiring a billing account, the free endpoint and free server are enough to get your first ledger written.

Your Review Prompt Is a Dependency — Pin It Before It Reaches the Merge Queue

Emery Lin — Mon, 26 Oct 2026 11:55:09 +0000

Most model-backed review jobs do not fail because the model is bad. They fail because someone edited prompts/review.md at 4pm on a Friday, and nothing in the pipeline noticed.

AI-authored diffs keep arriving faster than human review capacity grows, so more teams are wiring a model call into the PR path. That is a reasonable move. The problem is that the prompt, the decoding settings, and the fixture set now behave like production code, while being versioned like a scratch note.

This article gives you a small, reproducible artifact for that gap: a review lockfile plus a checker that fails CI when the prompt drifts without an explicit lock bump. It is boring on purpose. Boring is what you want sitting between a git push and a merge queue.

Three failure classes, three different fixes

When a model-backed review job goes red, people reach for the same reflex: rerun it. That reflex is only correct for one of the three failure classes.

Symptom	Class	Wrong fix	Right fix
`429`, timeout, DNS blip	Transport	none needed — rerun it	Backoff, request budget, alerting
Same diff, different verdict	Nondeterminism	Retry until green	k-of-n agreement, quarantine on split
Verdict flips after an unreviewed prompt edit	Drift	Usually invisible, so nothing	Lockfile + lock-bump PR

The third row is the expensive one. It does not produce a flaky red. It produces a confident green on a rule nobody approved.

So the goal is not "make the model deterministic." The goal is to make the inputs to the model reviewable, and to make disagreement loud instead of flaky.

Step 1 — Separate the inputs you can pin

Pin exactly three things. Anything else is configuration you should not silently inherit.

The prompt text — one file, no inlined strings scattered across the codebase.
The decoding settings — temperature, max tokens, top-p, stop sequences, in a committed JSON file.
The fixture set — a handful of diffs with the verdict label you expect.

The model handle itself belongs in settings too, but treat it as a named slot your operator resolves, not a hardcoded identifier you scatter across jobs. When the slot's resolution changes, your lock should change with it — loudly.

prompts/review.md              # the instruction text
review.settings.json           # decoding params + model slot name
fixtures/review/*.json         # diff + expected label
review.lock.json               # generated hashes, committed

Step 2 — Generate the lock with a canonical hash

Hashing raw file bytes is a trap. A line-ending change or a trailing space from your editor will produce a spurious drift signal, and your team will start ignoring the check within a week.

Canonicalize first. Normalize line endings, strip trailing whitespace per line, strip the trailing blank lines, then hash. Sort settings keys before serializing. This is the whole trick, and it is the difference between a check people trust and a check people --no-verify around.

#!/usr/bin/env python3
# tools/lock_review.py
import argparse, hashlib, json, pathlib, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
LOCK = ROOT / "review.lock.json"

def canonical_text(path: pathlib.Path) -> str:
    raw = path.read_text(encoding="utf-8").replace("\r\n", "\n")
    lines = [line.rstrip() for line in raw.split("\n")]
    return "\n".join(lines).strip() + "\n"

def sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def digest_prompt() -> str:
    return sha(canonical_text(ROOT / "prompts/review.md"))

def digest_settings() -> str:
    settings = json.loads(canonical_text(ROOT / "review.settings.json"))
    blob = json.dumps(settings, sort_keys=True, separators=(",", ":"))
    return sha(blob)

def digest_fixtures() -> str:
    files = sorted((ROOT / "fixtures/review").glob("*.json"))
    blob = "".join(f"{f.name}:{sha(canonical_text(f))}\n" for f in files)
    return sha(blob)

def current() -> dict:
    return {
        "schema": 1,
        "prompt_sha256": digest_prompt(),
        "settings_sha256": digest_settings(),
        "fixtures_sha256": digest_fixtures(),
    }

def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--check", action="store_true")
    ap.add_argument("--update", action="store_true")
    args = ap.parse_args()

    now = current()
    if args.update:
        LOCK.write_text(json.dumps(now, indent=2) + "\n", encoding="utf-8")
        print("lock written — open a lock-bump PR, do not push this alone")
        return 2

    if not LOCK.exists():
        print("missing review.lock.json; run --update once and commit it")
        return 1

    locked = json.loads(LOCK.read_text(encoding="utf-8"))
    drift = {k: (locked.get(k), v) for k, v in now.items() if locked.get(k) != v}
    if not drift:
        print("review lock OK")
        return 0
    for key, (was, is_) in drift.items():
        print(f"DRIFT {key}: locked={was} current={is_}")
    return 1

if __name__ == "__main__":
    sys.exit(main())

Run it once locally to create the baseline, then treat the output as read-only:

python tools/lock_review.py --update && git add review.lock.json
python tools/lock_review.py --check
echo $?   # expect 0

Now edit one word in prompts/review.md and rerun --check. You want exit code 1 and a printed prompt_sha256 drift line. If you get a clean pass after editing the prompt, stop and fix the canonicalization before continuing.

Step 3 — Catch drift before it leaves a laptop

The CI check is the backstop. The hook is what saves your reviewers from a red build they did not cause.

#!/usr/bin/env bash
# .git/hooks/pre-push
set -euo pipefail
if ! python tools/lock_review.py --check; then
  echo "review.lock.json is stale."
  echo "Fix: python tools/lock_review.py --update, then open a lock-bump PR."
  exit 1
fi

Keep the CI side narrow. You only need it to run when the pinned inputs change:

name: review-lock
on:
  pull_request:
    paths:
      - "prompts/**"
      - "review.settings.json"
      - "fixtures/review/**"
      - "tools/lock_review.py"
      - "review.lock.json"
jobs:
  lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python tools/lock_review.py --check

The lock-bump PR is the whole point. A prompt change now shows up as a reviewable diff of hashes plus the prompt body, owned by a human, instead of a silent behavioral change buried in a squash merge.

Step 4 — Make disagreement quarantine, not fail

Here is where most teams get flaky. They call the model once, get approve on one run and needs_human on the next, and the job flaps. Retrying until green is not a fix; it is a coin flip with extra billing.

Run the verdict k times per fixture and require agreement. On a split, emit quarantine — a distinct state that never masquerades as pass or fail.

# tools/fixture_check.py  (sketch — call_model is your own adapter)
import json, pathlib

K = 3
LABELS = {"approve", "block", "needs_human"}

def verdict_for(diff: str) -> str:
    # your transport, your auth, your timeout handling
    return call_model(prompt=PINNED_PROMPT, diff=diff, settings=SETTINGS)

rows = []
for fx in sorted(pathlib.Path("fixtures/review").glob("*.json")):
    data = json.loads(fx.read_text())
    runs = [verdict_for(data["diff"]) for _ in range(K)]
    bad = [r for r in runs if r not in LABELS]
    if bad:
        state = "quarantine"          # unparseable output is not a verdict
    elif len(set(runs)) > 1:
        state = "quarantine"          # split decision, human decides
    else:
        state = "pass" if runs[0] == data["expected_label"] else "fail"
    rows.append((fx.name, state, runs))

for name, state, runs in rows:
    print(f"{state:10s} {name:32s} {runs}")

Two rules keep this honest. First, quarantine must be a first-class state in your reporting, not a relabeled failure. Second, never let the fixture runner write to review.lock.json — generating the lock and validating behavior are separate jobs with separate permissions.

Step 5 — The green-to-merge decision table

Write this down and put it next to the merge queue config. Ambiguity is where broken gates live.

Lock state	Fixture state	Merge queue action
drift	not run	Block. Needs a lock-bump PR.
OK	any `fail`	Block. The prompt no longer matches your expected labels.
OK	any `quarantine`	Hold. Route to a human, keep the diff out of the queue.
OK	all `pass`, verdict `block`	Block on content, not on infrastructure.
OK	all `pass`, verdict `approve`	Admit to queue.
OK	all `pass`, verdict `needs_human`	Route to a human. Do not treat as green.

That last row matters more than it looks. A model that says "I cannot judge this" is not a passing check, and a pipeline that treats it as one has quietly converted uncertainty into approval.

Running the whole loop without a budget fight

All five steps are runnable end to end before you ask anyone for spend, because none of them require a large sample. You are checking lock stability and label agreement on a dozen fixtures, not benchmarking quality.

For that bootstrap phase I used MonkeyCode's free model access and its free server option to host the fixture runner, which meant the gate existed before the procurement conversation started. Check the current terms on the official MonkeyCode project page before you plan around them — free-tier scope and availability are exactly the kind of thing that changes, and I am not going to quote you numbers I cannot verify today.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If your team already has a paid endpoint and a runner, skip the product entirely. Nothing above depends on it. The lockfile is the artifact; the vendor is a convenience.

Limitations and who should not use this

The lock proves provenance, not quality. A pinned, reviewed, thoroughly wrong prompt passes the lock check every time.
k = 3 is a starting point, not a measurement. I have not measured disagreement rates on your diffs. Log the run counts for two weeks and pick k from your own data.
Skip this if your review is fully deterministic (linters, type checkers, policy engines). You already have this property for free; a hash lock adds nothing.
Skip this if you cannot version prompts in the same repo as the fixtures. A lockfile that hashes files living in three different systems is a maintenance tax with no payoff.
Do not use quarantine as a soft pass. If humans never triage the quarantine queue, you have built an approval machine with extra steps.

The honest version of the pitch: this adds a small amount of friction to prompt edits and removes a class of silent behavior change. If your prompt changes weekly and nobody has ever been surprised by a verdict flip, you may not need it yet. When you do get surprised, come back — the checker is forty lines and it will still be here.

A 90-Minute Context-Budget Workshop Your Team Can Run This Week

Emery Lin — Tue, 20 Oct 2026 10:03:34 +0000

Most prompt bugs are arithmetic bugs wearing a personality. A team that cannot state what currently sits inside its context window cannot debug what comes out of it. So before anyone rewrites another instruction, run a short internal workshop that makes the arithmetic visible, repeatable, and shared.

What follows is a plan for a ninety-minute session with five numbered steps, one small runnable calculator, three exercises with checkable answers, and a decision table for the moment the numbers go negative. The calculator is ordinary Python that I have run locally; the exercises use illustrative constants rather than measured production figures, so verify them against your own tokenizer before quoting them in a design review.

The failure this session is designed to prevent

The recurring shape is unglamorous. Retrieved chunks grow as the corpus grows, conversation history accumulates because deleting it feels rude, and the system prompt never shrinks because it was carefully written. Eventually the oldest instructions fall silently out of the window, and the model keeps answering with confidence from a context you never intended.

Nobody notices, because nothing errors. The output stays fluent, the latency stays flat, and the only symptom is a slow drift in behaviour that gets blamed on the model.

A budget written on one page turns that drift into a diff. The workshop exists to produce that page and to make sure five people, not one, can read it.

Materials and room setup

You need a prompt repository people can actually open, one machine per pair with Python 3.9 or newer, and a tokenizer library installed. For the third exercise you also need a place to run a handful of behaviour checks, and that is the only part that touches a model.

That last part is the usual reason internal workshops never happen, because twenty participants poking a paid endpoint is a budget conversation. MonkeyCode's outreach includes free models and a free server option, which is enough to run the behaviour checks in this session without opening a procurement ticket. I have not benchmarked either, so size the room conservatively and read the current terms on the project page yourself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Book a room with a shared screen, and ask each pair to bring one real prompt file. Abstract examples produce abstract arguments.

Step 1 — Agree on one measuring stick

Tokens are the only unit your provider bills and truncates in, so characters and words are not substitutes. Pick the tokenizer your provider publishes, and if none is published, pick one library and admit in writing that it is an approximation.

python -m pip install tiktoken

# count.py — prints the token length of a file, per encoding
import os, sys, tiktoken

name = os.environ.get('TOKENIZER', 'cl100k_base')
enc = tiktoken.get_encoding(name)
text = open(sys.argv[1], encoding='utf-8').read()
print(f'{name}\t{len(enc.encode(text))}\t{sys.argv[1]}')

The teaching point here is not the number, it is the disagreement. Ask two pairs to count the same file with different encodings and put both results on the board, because a three to eight percent gap across families is normal and worth seeing once.

Record the choice in the repository, next to the prompts. A budget measured with an undocumented tokenizer is not a budget, it is a rumour.

Step 2 — Do the subtraction before the writing

Hand out the calculator and make every pair run it against their own prompt file. Deliberately keep it boring: no network, no SDK, plain arithmetic that anyone in the room can audit line by line.

#!/usr/bin/env python3
# context_budget.py — offline arithmetic, exit 1 when over budget
import json, sys
from pathlib import Path

def compute(window, reserve, parts):
    used = sum(parts.values())
    return {'window': window, 'reserved': reserve, 'used': used,
            'headroom': window - reserve - used, 'parts': parts}

cfg = json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
result = compute(cfg['window'], cfg['reserve'], cfg['parts'])
print(json.dumps(result, indent=2, sort_keys=True))
if result['headroom'] < 0:
    print(f"OVER BUDGET by {abs(result['headroom'])} tokens", file=sys.stderr)
    sys.exit(1)

{"window": 32000, "reserve": 2000,
 "parts": {"system": 1400, "tools": 2600, "retrieved": 18000,
           "history": 9000, "scratch": 600}}

Run it and the session gets its first useful moment. This configuration is over by 1600 tokens, which the script reports on stderr with a non-zero exit code, and the room immediately starts negotiating instead of writing prose.

That negotiation is the workshop. Every suggestion is now a number someone can accept or reject on the record.

Step 3 — Three exercises, ordered by how badly they fail

Exercise one checks agreement rather than correctness. Each pair counts the same prompt file twice, on two machines, and the pass condition is that both counts match exactly; if they do not, the encodings differ and the room has found a real inconsistency.

Exercise two is the JSON above. Pairs must restore positive headroom with the smallest single change, and the expected answer is to drop roughly two retrieved chunks of about 800 tokens each, or trim history by an equivalent amount. Any pair that trims the system prompt first should explain why, because that is usually the wrong lever.

Exercise three is a behaviour check, and it needs a model. Pairs rewrite a six-sentence system prompt into three sentences without losing a constraint, then run five fixed inputs through both versions and diff the outputs. The check is not whether the shorter prompt is better, it is whether the constraints survived, and five inputs are enough to catch the obvious losses.

Step 4 — The over-budget decision table

Agreement matters more than precision here, so give the room one table and ask them to argue about the ranking rather than the wording.

Lever	Tokens recovered	Quality risk	Latency effect	Owner
Fewer retrieved chunks	High	Medium, can drop evidence	Lower	Retrieval owner
Trim oldest history turns	High	Low for chat, real for tasks	Lower	Product
Compress history into a summary	Medium	Medium, lossy	Slight increase	Prompt author
Shorten the system prompt	Low	Low if constraints kept	Negligible	Prompt author
Drop tool schemas not used	Medium	Low	Lower	Platform
Split into two sequential calls	Large per call	Low, adds complexity	Higher total	Architect

The single most common mistake in the room is treating the system prompt as the big lever. In the configuration above it is 1400 tokens out of 31600, so cutting it in half recovers less than the reserve margin.

Put that sentence on the slide. It saves a week of pointless rewriting.

Step 5 — Ship one page, not a slide deck

The deliverable is a file committed next to the prompts, roughly fifteen lines, listing the window, the reserve, the parts, and the tokenizer name. It should fail loudly in the repository's checks when the parts stop summing to something sane.

The page also names the person who owns each line, because a budget without an owner is a comment. Finish the session by having each pair read their page aloud for thirty seconds; anything confusing gets fixed while the author is still in the room.

What this workshop will not fix

Arithmetic will not tell you whether your retrieval is any good, and a perfectly budgeted context full of irrelevant chunks is still irrelevant. It also will not survive a provider change on its own, since tokenizers differ and a budget written for one family drifts for another.

Finally, it will not make anyone write better instructions. It only removes one class of invisible failure so that instruction quality becomes the visible problem.

Skip the session entirely if your prompts live in scattered chat threads, because there is nothing to budget yet. Skip it as well if your context problems are really retrieval-quality problems, and skip it if you are a solo developer who already knows what is in the window.

Assignment for next week

Each pair leaves with one task: add the budget page for their own prompt file and open a pull request that touches nothing else. Review those pull requests together at the next team meeting, and accept them only when the numbers are reproducible by a second person.

Teams that want the behaviour check from exercise three in their own workflow can point it at the free models and free server option while the exercise is still an experiment. Keep the arithmetic local, keep the page in the repository, and let the model be the least interesting dependency in the room.

Put a Token Bucket Between Your CI Jobs and the Model Endpoint

Emery Lin — Mon, 19 Oct 2026 08:45:33 +0000

If two or three CI jobs share one model endpoint, the failure you notice first is not a bad answer. It is a 429 that your retry loop converts into three slow jobs, one starved job, and a queue that reads like flakiness. Admission control fixes that. More retries make it worse.

This post builds a small gate you can run in front of a shared endpoint, a decision table for what to retry, and a local harness that reproduces the collision without spending a single token in CI.

The shape of the failure

A retry loop assumes the bottleneck is transient. With a shared endpoint, the bottleneck is your own concurrency. Job A retries, so it holds its slot longer; job B retries, so it overlaps A's second attempt; job C waits on a runner that is now billed for idle time. Nobody's prompt changed, but the pipeline got slower and less predictable.

Free tiers make this shape common because capacity is shared rather than reserved. That is a fair trade for prototyping, and a bad one for an unguarded merge path.

MonkeyCode is an open-source project that offers free model access and a free server option, which is exactly the situation where a gate earns its keep. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free allowance as a budget to be spent deliberately, not as an unmetered pipe. Check the project page for the current allowance and terms before you size anything below — free-tier numbers change, and a stale figure in a comment block is worse than no figure.

What to measure before you build anything

You need two numbers: how many model calls a pipeline run makes, and how many of those came back rate-limited. Both are already in your logs.

# calls per run
gh run view "$RUN_ID" --log | grep -c 'model_call_ok'

# rate-limited responses in the same run
gh run view "$RUN_ID" --log | grep -c 'status=429'

# how long jobs waited on the endpoint
grep 'model_wait_seconds' ci.log | awk -F= '{s+=$2} END {print s"s total wait"}'

Write the three numbers into the repository, next to the gate. If rate-limited responses are zero across a week of runs, stop here — you do not need this. If they are above zero, the ratio of wait time to wall-clock time tells you whether to raise capacity or lower concurrency.

Step 1 — One bucket, one lock file

One bucket for the whole pipeline, not one per job. A per-job bucket multiplies your concurrency by the number of jobs, which is the bug you were trying to fix.

# ci_gate/bucket.py
import fcntl, json, os, random, time
from dataclasses import dataclass

@dataclass
class Bucket:
    path: str
    capacity: int = 6          # in-flight calls you allow
    refill_per_sec: float = 0.2  # recovered calls per second

    def _load(self):
        try:
            with open(self.path) as f:
                return json.load(f)
        except FileNotFoundError:
            return {"tokens": float(self.capacity), "ts": time.time()}

    def _write(self, state):
        tmp = self.path + ".tmp"
        with open(tmp, "w") as f:
            json.dump(state, f)
        os.replace(tmp, self.path)

    def acquire(self, timeout=90.0):
        deadline = time.time() + timeout
        while True:
            with open(self.path + ".lock", "w") as lk:
                fcntl.flock(lk, fcntl.LOCK_EX)
                state = self._load()
                now = time.time()
                tokens = min(
                    self.capacity,
                    state["tokens"] + (now - state["ts"]) * self.refill_per_sec,
                )
                if tokens >= 1:
                    self._write({"tokens": tokens - 1, "ts": now})
                    return True
                self._write({"tokens": tokens, "ts": now})
                wait = (1 - tokens) / self.refill_per_sec
            if time.time() > deadline:
                return False
            time.sleep(min(wait, 5.0) * (1 + random.random() * 0.3))

Two properties matter here. The lock makes the read-modify-write atomic, and the timeout means a job fails loudly instead of hanging until the workflow timeout kills it. The default numbers are placeholders — set capacity from your measured concurrency, not from optimism.

Step 2 — Separate admission from retry

Admission asks may I call now. Retry asks should I call again. Mixing them produces the loop that starves your other jobs.

# ci_gate/call.py
import time
from .bucket import Bucket

BUCKET = Bucket(path="/var/ci/model_bucket.json", capacity=6, refill_per_sec=0.2)

def call_model(payload, attempts=3):
    for i in range(attempts):
        if not BUCKET.acquire(timeout=90):
            raise SystemExit("admission timeout: not calling, not retrying")
        resp = post(payload)
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code != 429:
            raise RuntimeError(f"non-retryable status={resp.status_code}")
        time.sleep(min(2 ** i, 20) * (1 + random.random()))
    raise RuntimeError("exhausted attempts after admission")

Every attempt re-enters the bucket, so a retry costs a token like any other call. If you skip that, retries bypass the only mechanism you built.

Step 3 — Classify before you retry

The gate stops the pile-up. The table stops you from retrying things that cannot succeed.

Signal	Retry?	Where the fix actually belongs
429 with `Retry-After`	Yes, after admission	Honour the header; lower `capacity`
429 with no header	Yes, capped at 3, jittered	Your `capacity` is above real capacity
500 / 502 / 503	Once	Endpoint health, not your prompt
400 / 422	Never	Prompt or fixture is wrong
Timeout after admission	Once, and log the duration	Your latency budget, not the bucket
Output differs from fixture	Never	This is a test failure, not infra

That last row is the one teams get wrong. A model answer that no longer matches a recorded fixture is a signal about the model or your prompt, and retrying it politely until it agrees is how a green check stops meaning anything.

Step 4 — Reproduce the collision locally, not in CI

You cannot debug concurrency in a pipeline that takes eleven minutes to start. Use a fake endpoint that rate-limits the first few requests and run the real gate against it.

# tests/fake_endpoint.py
import itertools, json
from http.server import BaseHTTPRequestHandler, HTTPServer

seen = itertools.count()

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = next(seen)
        code = 429 if n < 4 else 200
        body = {"ok": True} if code == 200 else {"error": "rate_limited"}
        raw = json.dumps(body).encode()
        self.send_response(code)
        self.send_header("Retry-After", "1")
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(raw)

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8787), Handler).serve_forever()

Run three workers against it and assert on behaviour, not on elapsed time:

python tests/fake_endpoint.py &
python -m pytest tests/test_gate.py -k parallel -q
# assert: requests_in_flight <= capacity at all times
# assert: total calls stayed within the attempt budget
# assert: exactly one job exited on admission timeout

Keep this test deterministic: no network, no sleeps longer than the retry cap, no wall-clock assertions. A gate whose own tests are flaky will be deleted by the second person who sees it go red.

Step 5 — Where the gate process lives

The lock file needs a filesystem shared by every job that calls the endpoint. On ephemeral hosted runners, /var/ci/ disappears between jobs, and your bucket silently resets to full every time — the worst kind of green.

Two honest options. Run the gate as a small service on one always-on host, and have jobs ask it for a slot over HTTP. Or run the whole pipeline, gate included, on a single machine you control. MonkeyCode's free server option is one place to put that process while you are evaluating the workflow; a lint-checked systemd unit beside the repo keeps it reproducible either way.

The pre-commit hook that keeps the gate honest

A hook cannot test the endpoint, but it can test the gate.

# .git/hooks/pre-commit
#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/test_gate.py -q --no-header
python -m ruff check ci_gate/

If the hook and CI run the same two commands, a broken bucket fails in seconds instead of after a queue. That is the whole green-to-merge path: same gate locally, same gate in CI, one shared budget.

Limitations and who should skip this

fcntl.flock is POSIX-only; on Windows runners you need a different primitive or the service model. Token buckets also cannot discover the endpoint's real capacity — you supply the number, and a wrong number just fails more quietly. And if your model call is not on the merge path at all, this is infrastructure for a problem you do not have.

Skip it if you have a dedicated quota, if a single job calls the endpoint once per run, or if your team has no owner for the gate's configuration. An unowned limit is a limit that gets raised on the first red build.

Merge checklist

Logged calls-per-run and 429s-per-run for a full week.
Set capacity from the measured peak, not from the highest number that once went green.
Verified retries re-enter the bucket.
Confirmed fixture mismatches are classified as test failures, never retried.
Ran the fake-endpoint test in the pre-commit hook and in CI.
Documented which host owns the lock file, and what happens when it moves.

If you try the workflow, run the fake-endpoint suite against MonkeyCode's free model access first and compare your measured 429 rate with the table above — that is the cheapest way to find out whether your concurrency was ever the problem.

Your CI Gate Should Not Call a Hosted Model: Record, Replay, Then Merge

Emery Lin — Sun, 18 Oct 2026 07:23:27 +0000

A model call inside a required check is an uncontrolled dependency. The lane that blocks merge should read fixtures from disk. Only the lane that cannot block merge should touch the network.

That single split removes most of the flakiness people blame on "AI in CI". It also makes review cheaper, because every model answer your gate saw is a file you can diff.

Below is the adapter, the contract test, the two-job workflow, and the triage order I use when the gate goes red.

Why the live call breaks the gate

Three failure shapes show up almost immediately once a model call sits behind a required check.

Latency variance. A 900 ms call and a 14 s call in the same job produce different timeouts on different days. Your retry logic then hides a real prompt regression behind a slow afternoon.

Sampling drift. Even at low temperature, output shape changes between runs. A gate that asserts on prose will pass on Monday and fail on Tuesday with no code change in between.

Outage equals red. When the provider is down, your PR shows as failing. The author sees a red check and starts editing code that was never broken.

There is a fourth problem that gets less attention: secrets. If the blocking job holds a model key, that key is now in the same process as pull-request code you do not control. Moving live calls out of the blocking lane is a security decision, not only a stability one.

The anti-pattern looks like this:

# do not do this in a required check
try:
    return call_model(prompt)
except TimeoutError:
    return load_cassette(prompt)   # silent fallback hides drift

A silent fallback is worse than no fallback. You cannot tell a replayed answer from a live one in the logs, so nobody knows which runs actually tested anything.

Two lanes, one adapter

Before writing code, agree on which lane does what.

Lane	Trigger	Network	Blocks merge	Purpose
`gate`	every PR	none	yes	prove your code handles the recorded contract
`live-drift`	nightly schedule	yes	no	detect provider or model drift, open an issue
`record`	manual, local	yes	n/a	write or refresh one cassette, reviewed in the PR

If a lane cannot block merge, it cannot hold a secret that PR code can reach. Put the model key on live-drift only, and keep the gate job at contents: read.

Build the adapter in five steps

1. Make the call hashable

Hash every input that can change the answer, including the template identity. If you hash only the prompt text, an edit to the system instruction can silently reuse a stale cassette.

# ci/model_gate.py
from __future__ import annotations

import hashlib
import json
import os
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable

ADAPTER_VERSION = "3"  # bump when hashing or request shape changes
CASSETTE_DIR = Path(os.environ.get("CASSETTE_DIR", "ci/cassettes"))
MODE = os.environ.get("MODEL_MODE", "replay")  # replay | record | live


class MissingCassette(RuntimeError):
    pass


@dataclass(frozen=True)
class Call:
    template_id: str
    template_version: int
    prompt: str
    temperature: float = 0.0
    max_tokens: int = 512


def cassette_key(call: Call) -> str:
    blob = json.dumps(
        {"adapter": ADAPTER_VERSION, **asdict(call)},
        sort_keys=True,
        separators=(",", ":"),
    )
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:20]

2. Fail closed in replay mode

Missing fixture means missing fixture. Raise, do not call out.

Transport = Callable[[Call], dict]


def run(call: Call, transport: Transport | None = None) -> dict:
    path = CASSETTE_DIR / f"{cassette_key(call)}.json"

    if MODE == "replay":
        if not path.exists():
            raise MissingCassette(
                f"no cassette for {call.template_id} v{call.template_version} "
                f"({path.name}); record it with MODEL_MODE=record"
            )
        return json.loads(path.read_text())["response"]

    if MODE == "record":
        if path.exists():
            return json.loads(path.read_text())["response"]
        if transport is None:
            raise ValueError("record mode needs a transport")
        response = transport(call)
        path.parent.mkdir(parents=True, exist_ok=True)
        path.write_text(
            json.dumps(
                {"request": asdict(call), "response": response},
                indent=2,
                sort_keys=True,
            )
            + "\n"
        )
        return response

    if transport is None:
        raise ValueError("live mode needs a transport")
    return transport(call)

Store the request inside the cassette. Reviewers need to see what was asked, not only what came back.

3. Assert contracts, not prose

Your gate exists to prove the shape your code consumes. Validate structure and bounds, then stop.

# ci/contracts.py
PATCH_PLAN = {
    "type": "object",
    "required": ["files", "reason"],
    "properties": {
        "files": {"type": "array", "items": {"type": "string"}, "maxItems": 20},
        "reason": {"type": "string", "minLength": 10},
    },
    "additionalProperties": False,
}

# tests/test_patch_plan_contract.py
import pytest
from jsonschema import ValidationError, validate

from ci.contracts import PATCH_PLAN
from ci.model_gate import Call, run


def test_plan_is_mergeable(cassette_prompt):
    call = Call(template_id="patch_plan", template_version=4, prompt=cassette_prompt)
    result = run(call)
    try:
        validate(result, PATCH_PLAN)
    except ValidationError as exc:
        pytest.fail(f"recorded output breaks the contract: {exc.message}")

Do not assert on wording. The moment you test prose, you have a test that fails for reasons you cannot reproduce.

4. Split the workflow

The gate job gets no key. The nightly job gets the key and is allowed to fail.

# .github/workflows/model-gate.yml
name: model-gate
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  gate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      MODEL_MODE: replay
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q

  live-drift:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    continue-on-error: true
    env:
      MODEL_MODE: live
      MODEL_BASE_URL: ${{ secrets.MODEL_BASE_URL }}
      MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r ci/requirements.txt
      - run: pytest tests/ -q -m live

This is also where a free endpoint earns its place. The nightly lane makes a small, bounded number of real calls, so it is the right consumer of a free model allowance. MonkeyCode's free model access and free server option fit that lane: you can run the drift job without opening a paid budget line first, and the free server option gives the schedule somewhere to live.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms for free offerings change. Check the current terms on the project page before you let a nightly job depend on them, and keep a continue-on-error job non-blocking until you have watched it for a week.

5. Keep the fixtures small

One cassette per template version. When you bump template_version, the old file stops matching and you should delete it in the same commit. If ci/cassettes/ grows without bound, nobody reviews it, and an unreviewed fixture is a test with a comment that lies.

When the gate goes red, triage in this order

Read the exception type. MissingCassette is a fixture problem. ValidationError is a contract problem. A KeyError on response means a hand-edited or truncated cassette.
MissingCassette: did you change the prompt? Record once with MODEL_MODE=record, then read the cassette diff. If the new answer surprises you, that is the review you wanted.
ValidationError: fix the fixture, not the assertion. The recorded output never satisfied the contract. Either the prompt is underspecified or the contract is wrong. Decide which, in the PR.
Contract passes but production breaks. Your schema is looser than your consumer. Tighten maxItems, add additionalProperties: false, and add a bound you actually enforce downstream.

Track one number: cassette miss rate per week. Above roughly one in ten PR runs means your templates are churning, and the gate is measuring your edit speed rather than your code.

Limits worth stating out loud

Replay proves your code handles a known answer set. It cannot prove the model still behaves. That is the nightly lane's job, and it needs a human to read the failure.

Cassettes go stale when a provider swaps a model behind an endpoint name. Pin the model identifier in the request and re-record deliberately, not on a schedule that erases the diff.

A single recorded answer is not an evaluation. If you need quality numbers, you need multiple samples and a scoring rule, which is a different pipeline with different costs.

And the drift job is a smoke alarm, not a gate. Once it blocks merges, you have rebuilt the problem this article removes.

Who should not do this

Skip the replay lane if your prompts are assembled at runtime from unbounded user input, because the cassette space explodes and you will spend more time pruning than testing.

Skip it if you need a per-PR quality score on model output. Replay gives you a contract check, and a contract check is not a quality signal.

And skip it if nobody on the team owns the prompts. Record/replay makes prompt changes visible, which is only useful if someone is accountable for them.

Start with one template. Record it, wire the gate, and watch the miss rate for a week before you add the second. If you want to stand up the nightly lane before you commit a budget, the free model access and free server option in the MonkeyCode project are a reasonable place to try the loop, terms permitting.

A Green Pipeline That Ran Three of Five Shards Is Not Green

Emery Lin — Sat, 17 Oct 2026 06:25:04 +0000

Your merge queue trusts a color. It should trust a coverage claim.

A sharded CI run where shard 3 died during setup, shard 5 collected zero tests, and shard 1 passed on its third attempt produces exactly the same green check as a clean sweep of the whole matrix. Nothing in the check name tells you which one you got. The fix is not a smarter retry policy — it is a per-shard manifest plus a gate that refuses to merge anything the manifest cannot prove.

Four ways a shard quietly disappears

The matrix shrinks. --splits is computed from a test count that changed between the plan job and the test job. Fewer jobs get scheduled, needs is satisfied, the workflow is green.
A partition is empty. Shard 4 is assigned no tests. pytest exits with code 5, and someone already appended || true to keep the pipeline moving.
A step is allowed to fail. continue-on-error: true left over from a migration week means a crashed shard reports success at the job level.
A retry hides the first attempt. pytest-rerunfailures turns a flake into a pass, and the report contains no field that says "this took three tries."

All four are coverage bugs, not flake bugs. Treat them that way.

The contract: one manifest per shard

Stop reading the job summary. Have every shard write a small JSON file next to its report, and let the gate read only those files.

The manifest needs six fields and nothing more: index, commit, collected, node_ids, status, attempts. Add quarantined if you run a quarantine lane. If a field is missing, the gate blocks — absence is not consent.

Step 1 — Discover once, split deterministically

Collect node IDs in a single job, persist them, and split from that file. Never recompute the split inside the test job.

python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
wc -l < nodes.txt   # the number every later step must agree with

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      shards: ${{ steps.plan.outputs.shards }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - id: plan
        run: |
          python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
          n=$(wc -l < nodes.txt)
          echo 'shards='$(( (n + 199) / 200 )) >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with: { name: nodes, path: nodes.txt }

Step 2 — Emit the manifest from a hook

Do not parse the terminal output. Use pytest_sessionfinish and derive everything from session.items.

# conftest.py
import json, os, pathlib

def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path(os.environ.get('SHARD_DIR', 'shard-reports'))
    out.mkdir(parents=True, exist_ok=True)
    nodes = sorted({item.nodeid for item in session.items})
    payload = {
        'index': int(os.environ['SHARD_INDEX']),
        'commit': os.environ['GITHUB_SHA'],
        'collected': len(nodes),
        'node_ids': nodes,
        'status': 'passed' if exitstatus == 0 else 'failed',
        'attempts': int(os.environ.get('SHARD_ATTEMPT', '1')),
        'quarantined': os.environ.get('QUARANTINE') == '1',
    }
    (out / f'shard-{payload["index"]}.json').write_text(json.dumps(payload))

Expose SHARD_INDEX from the matrix, not from a counter inside the process. Counter drift is how two shards end up writing shard-0.json and the gate sees four files where five were expected.

Step 3 — Gate on the manifest, not the color

Stdlib only, no dependencies, no network. It exits non-zero on the first violation it can explain.

# gate_shards.py
import json, os, pathlib, sys

head = os.environ['GITHUB_SHA']
expected = int(os.environ['SHARD_COUNT'])
files = sorted(pathlib.Path(sys.argv[1]).glob('shard-*.json'))

def block(msg):
    print(f'::error::{msg}')
    sys.exit(1)

if len(files) != expected:
    block(f'found {len(files)} manifests, expected {expected}')

indexes, owner, retried = set(), {}, []
for path in files:
    m = json.loads(path.read_text())
    commit, index, status = m['commit'], m['index'], m['status']
    if commit != head:
        block(f'{path.name} tested {commit[:8]}, head is {head[:8]}')
    if index in indexes:
        block(f'shard {index} reported twice')
    indexes.add(index)
    if status != 'passed':
        block(f'shard {index} finished as {status}')
    if m['collected'] == 0:
        block(f'shard {index} collected zero tests')
    if m['attempts'] > 1 and not m['quarantined']:
        retried.append(index)
    for node in m['node_ids']:
        if node in owner:
            block(f'{node} ran in shards {owner[node]} and {index}')
        owner[node] = index

missing = sorted(set(range(expected)) - indexes)
if missing:
    block(f'no result for shard(s) {missing}')
if retried:
    block(f'green only after a retry in shard(s) {retried}')
print(f'OK: {len(owner)} node ids across {expected} shards on {head[:8]}')

Note what the duplicate check buys you: a split bug that sends the same 40 tests to two shards now fails the gate instead of inflating your pass count.

Step 4 — Wire it as its own required check

  gate:
    needs: [plan, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: shard-*
          path: shard-reports
          merge-multiple: true
      - env:
          SHARD_COUNT: ${{ needs.plan.outputs.shards }}
          GITHUB_SHA: ${{ github.event.pull_request.head.sha }}
        run: python3 gate_shards.py shard-reports

Two details earn their keep. if: always() stops the gate from being skipped when test fails, and a skipped gate reads like a passed one in most UIs. Overriding GITHUB_SHA with the PR head SHA means the manifest is compared against the code you reviewed, not the merge commit.

Step 5 — Make the retry ledger boring

The attempts > 1 rule is the flake policy, and it is deliberately blunt: a test that only goes green on the second attempt blocks the merge until someone either fixes it or moves it to the quarantine lane with an owner and an expiry date. Rerunning until green is not a decision, it is a coin flip with a paper trail.

Track the top five quarantined files weekly. If the list does not shrink, your quarantine is where tests go to be forgotten.

Where a free model actually fits

Not in the gate. The gate must stay deterministic, offline, and explainable in one line of output.

Triage is a different job: turning twenty scattered failure snippets into three probable root causes is exactly the kind of summarization that saves a reviewer ten minutes. MonkeyCode is an open-source project that offers free model access and a free server option, which makes it a low-friction place to run that advisory step before you decide to pay for anything. The project advertises a 10M-token allowance and a free server tier — treat both as vendor claims and check the current numbers in its console before you build a dependency on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# triage.py — advisory only; never writes status
import json, os, urllib.request, pathlib

fails = []
for path in pathlib.Path('shard-reports').glob('shard-*.json'):
    fails.extend(json.loads(path.read_text()).get('failures', []))
if not fails:
    raise SystemExit('nothing to triage')

prompt = 'Group by shared root cause. Markdown, max 200 words.\n\n' + '\n'.join(fails[:20])
body = json.dumps({'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0}).encode()
req = urllib.request.Request(
    os.environ['MONKEYCODE_BASE_URL'] + '/chat/completions',
    data=body,
    headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY'],
             'Content-Type': 'application/json'},
)
print(urllib.request.urlopen(req, timeout=60).read().decode())

Read the base URL and model ID from the project's own docs; endpoints move, and a hardcoded one in a tutorial rots the week after publication. Scrub secrets and customer identifiers from fails before it leaves the runner — failure logs are the most common accidental leak in this pattern.

Hard rule: triage output lands in a PR comment. It never sets status, never flips quarantined, never authorizes a merge.

Limits, and who should skip this

Small suites. Under roughly 200 tests, sharding overhead exceeds the wall-clock win, and one manifest beats five.
Non-hermetic runners. If your runner image digest can change mid-run, a passed shard is not reproducible and the manifest proves less than you think.
Regulated audit trails. Don't put a compliance gate behind a free tier whose quota and retention you have not verified in writing.
Anyone tempted to parse model prose. If you cannot keep the machine summary out of the decision path, don't add the summary.

The checker above is stdlib Python 3.12 and untested against your report format — adapt the status mapping to your runner before you make the gate required. Once it is required, the interesting metric stops being pass rate and becomes coverage: how many node IDs, from how many shards, on which commit.

If you want to try the triage step before wiring a billing account, the project's free model access and free server option are the shortest path to finding out whether it earns a slot in your pipeline.

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

Emery Lin — Fri, 16 Oct 2026 05:25:39 +0000

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

A retry counts as evidence only when it replays from a cold cache and leaves a record a later job can verify. A warm rerun proves that your cache works. It does not prove that your branch builds.

Most advice about flaky CI ends at "just retry the job". That advice quietly hands your merge queue a pass that was never produced under the same conditions as a fresh clone, and it leaves nothing behind for the merge job to inspect. This post fixes the boring half of the problem: make one replay emit a machine-checkable admission record, then make the merge path refuse anything else.

The workflow below is a proposal you can adapt, not a transcript of a run I am handing you as a benchmark. Label your own numbers when you measure them.

The failure mode, stated precisely

A pull request fails on a cold runner because a dependency resolves differently without the warm module cache. Someone clicks retry. The job runs on the same runner, reuses the toolchain, and goes green.

The green check is real, but its subject is different. It tested the tree plus the warm cache, and your merge queue only knows about the tree. When main is built from scratch later, the original failure returns, now attributed to a merge that "passed CI".

So the rule is short: a replay resets the environment and writes down what the environment was. Everything else is a cache hit wearing a check mark.

Write the admission record first

Define the record before you write the workflow. You want fields that the merge job can compare against current reality, not a prose summary. Keep it to one small JSON artifact.

{
  "schema": "admission/v1",
  "tree_sha": "9f1c4e2",
  "base_sha": "3ab07d1",
  "runner_image": "ghcr.io/acme/ci-runner@sha256:91a0...",
  "hook_version": 7,
  "cache": "cold",
  "seed": 4211,
  "toolchain": { "node": "20.11.1", "python": "3.12.3" },
  "result": "pass",
  "started_at": "2026-10-16T09:12:04Z"
}

Four fields do the heavy lifting. cache must equal cold, or the record is void by definition. base_sha catches the case where main moved after your replay. runner_image catches a silent runner rotation. hook_version lets you invalidate every old record in one commit when you change the test harness itself.

If you have already learned not to let a cancelled run count as a pass, the same logic applies here: a cancelled or timed-out replay writes no record, and a missing record is a failure, never a neutral state.

Step 1: purge, then replay

The workflow has one job that runs on a clean runner and deliberately destroys every cache it can reach. Note that the retry path and the first attempt use the same job; there is no separate "fast" path for reruns.

name: replay
on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  checks: write

jobs:
  cold-replay:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0

      - name: Record the base commit
        run: echo BASE_SHA=${{ github.event.pull_request.base.sha }} >> $GITHUB_ENV

      - name: Record the tree commit
        run: echo TREE_SHA=$(git rev-parse HEAD^{tree}) >> $GITHUB_ENV

      - name: Purge every warm artifact
        run: rm -rf node_modules .venv ~/.npm ~/.cache/pip ~/.cargo/registry

      - name: Read the runner image digest
        run: echo IMAGE_DIGEST=$(jq -r .image /etc/runner-manifest.json) >> $GITHUB_ENV

      - name: Install pinned toolchain
        run: |
          corepack enable
          npm ci --no-audit --no-fund

      - name: Run the suite with a fixed seed
        run: npm test -- --seed=${{ github.run_number }}4211 --reporter=json > test-report.json

      - name: Emit the admission record
        if: always()
        run: python3 scripts/emit_record.py

      - uses: actions/upload-artifact@v4
        with:
          name: admission-record
          path: admission.json
          retention-days: 14

Two details matter more than the rest. The seed is derived from the run number so a retry is a genuinely different ordering, not a repeat of the same lucky interleaving. And the record upload uses if: always(), so a failing replay still produces an artifact that says "result: fail" — invisible failures are how teams end up re-running until green.

Step 2: gate the merge on the record

The merge job never runs the test suite. It downloads the record and checks it against the repository as it exists right now. If any input moved, the record is stale and the merge is refused.

#!/usr/bin/env python3
# scripts/admit.py
import json, subprocess, sys

REQUIRED = {
    'schema', 'tree_sha', 'base_sha', 'runner_image',
    'hook_version', 'cache', 'seed', 'result',
}

def git(*args):
    return subprocess.check_output(['git', *args], text=True).strip()

def admit(record, head_ref, base_sha, image_digest, hook_version):
    missing = REQUIRED - record.keys()
    if missing:
        return False, f'missing fields: {sorted(missing)}'
    if record['schema'] != 'admission/v1':
        return False, 'unknown record schema'
    if record['cache'] != 'cold':
        return False, 'record came from a warm cache'
    if record['result'] != 'pass':
        return False, 'record is not a pass'
    if record['tree_sha'] != git('rev-parse', head_ref + '^{tree}'):
        return False, 'tree changed after the replay'
    if record['base_sha'] != base_sha:
        return False, 'base moved; rebase and replay'
    if record['runner_image'] != image_digest:
        return False, 'runner image rotated; replay on the new image'
    if str(record['hook_version']) != str(hook_version):
        return False, 'hook version changed'
    return True, 'admitted'

if __name__ == '__main__':
    record = json.load(open(sys.argv[1]))
    ok, reason = admit(
        record,
        head_ref=sys.argv[2],
        base_sha=sys.argv[3],
        image_digest=sys.argv[4],
        hook_version=sys.argv[5],
    )
    print(reason)
    sys.exit(0 if ok else 1)

Keep this script in the default branch and call it from a required check that the pull request author cannot edit. A gate that lives in the same diff it approves is decoration.

Step 3: classify the flake before you spend a retry

Not every red run deserves a replay. Split failures into two buckets before you rerun anything, because a product failure replayed three times is three times the noise.

Infrastructure failures look like exit code 143, OOM kills, DNS resolution errors, or a registry 5xx in the log. These are the only ones eligible for a replay, and each replay voids the previous record. Product failures — an assertion, a snapshot diff, a type error — block the branch and go to a human.

Budget the replays per pull request, not per job. Two replays is a common ceiling; if a third is needed, the branch has a real ordering dependency and you should fix it instead of paying for it.

Step 4: the green-to-merge decision table

Situation	Action
Record passes; tree, base, image and hook version all match	Auto-merge
Any infrastructure replay happened	Require a fresh cold replay before merge
Failure is a product assertion	Block; human triage
Base branch moved after the replay	Rebase, then cold replay
Runner image rotated since the replay	Cold replay on the new image
No record (cancelled, timeout, OOM)	Treat as failing, never neutral
Hook version changed in this diff	Reject the diff; harness changes land separately

That last row is the one people skip, and it is the one that keeps the gate honest. If a pull request can change the harness and the code it tests at the same time, your record describes a moving target.

Where a free model and a free server fit

If your patches come from an agent, the same record shape still applies — the agent is just another producer of a tree hash. In that setup the operator states that MonkeyCode offers free model access plus a free server option, which is enough to try this loop before committing paid CI minutes to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the part you can control: generate the patch with the free model access, run the cold replay on the free server option, and keep the admission record identical to the one your paid runners emit. Do not hard-code quota or hardware numbers into your workflow — free-tier terms change, so read the current page before you depend on a specific allowance, and never treat a free runner as a compliance or isolation boundary.

Limitations, and who should skip this

Cold replay costs time. A suite that takes twelve minutes warm can take twenty-five or more cold, and monorepos with large native builds will feel it first. If your build already requires a licensed artifact you cannot download on a clean machine, this approach will block you until you solve caching deliberately — which is a different project.

Skip it if your team has no required check that pull request authors cannot edit, because the record will not be enforceable. Skip it if every failure is already a product failure and your flake rate is near zero. The overhead only pays for itself when a warm cache is a plausible explanation for a green run.

A five-line checklist for the next pull request

Does the failing job emit an admission record even when it fails?
Does the record include the tree, the base, the runner image, and the hook version?
Is the merge gate reading that record from a path the author cannot modify?
Is the replay budget per pull request, not per job?
When the base moves, does the old record become invalid automatically?

If you can answer yes to all five, a green check means something you can defend after the outage.

A Cancelled Review Run Is Not a Passing Review

Emery Lin — Thu, 15 Oct 2026 11:30:30 +0000

The rule, in one line. A cancelled run is missing evidence, and a missing verdict must never be read as a pass.

That sentence is the article. The rest is the machinery that enforces it in a pipeline where one non-deterministic lane — an AI review step, a fuzz run, an end-to-end suite — sits next to the checks that actually block a merge.

Why the cancelled lane is the expensive one

Most teams bolt on an AI review step because it is cheap and advisory. Then they do two reasonable things that combine badly.

They set concurrency: cancel-in-progress: true, so a fresh push cancels the review of the previous commit. That is the correct cost decision.
They either require the review check in branch protection, or they treat the bot's comment as the verdict. Both make a cancelled run load-bearing.

Put those together and you get the worst kind of bug: a lane that has never once emitted "no evidence", merged into a system where nobody has tested what "no evidence" means. A check whose failure path has never run is not a check. It is a habit.

The same shape appears in browser suites cancelled on the third retry, scanners that time out, and any job with a 10-minute timeout sitting next to a 6-minute model call.

Bind the verdict to fields, not to a comment

A comment is not evidence. Evidence is a small artifact with fields the gate can verify without trusting the producer. Four fields carry the weight:

head_sha — the commit the verdict is about. Without it, a review of commit A authorizes a merge of commit B.
input_digest — a hash of exactly what the lane read: tracked file modes and contents plus lockfiles. Without it, a review of last week's lockfile authorizes today's dependency bump.
run_attempt — which attempt produced this. A verdict from attempt 1 must not authorize attempt 3.
status: complete — set only on normal completion of the lane. A cancelled or killed job writes nothing.

That last point is the design: a cancelled lane produces no artifact. Absence is the signal. You never have to teach the gate to interpret cancelled, because there is nothing to interpret.

Step 1 — the advisory workflow

Reference implementation, not something I have run against your repository. Treat it as a starting point, and verify the cancellation behaviour on your own runner first, because step-level conditions during cancellation have changed between runner versions.

name: advisory-review
on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: advisory-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Compute input digest
        id: digest
        run: |
          DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Run review lane
        id: lane
        env:
          MONKEYCODE_API_KEY: ${{ secrets.MONKEYCODE_API_KEY }}
        run: ./ci/run-review.sh > review.md

      - name: Write verdict
        if: success() || failure()
        run: |
          STATUS="fail"
          [ "${{ steps.lane.outcome }}" = "success" ] && STATUS="pass"
          jq -n \
            --arg sha "${{ github.event.pull_request.head.sha }}" \
            --arg digest "${{ steps.digest.outputs.digest }}" \
            --arg attempt "${{ github.run_attempt }}" \
            --arg status "$STATUS" \
            '{status: "complete", conclusion: $status, head_sha: $sha,
              input_digest: $digest, run_attempt: $attempt}' > verdict.json

      - name: Upload verdict
        if: success() || failure()
        uses: actions/upload-artifact@v4
        with:
          name: verdict-${{ github.event.pull_request.number }}-${{ github.run_attempt }}
          retention-days: 7
          path: verdict.json

Note the asymmetry: the artifact is uploaded on both success and failure of the lane, but not on cancellation. A red verdict is a verdict; a missing verdict is not green.

Step 2 — the required gate

Keep the gate in a separate workflow triggered by workflow_run, or as a job that downloads the artifact. It reads the repository and nothing else. No secrets, no tokens, no write access.

#!/usr/bin/env bash
# ci/gate.sh — required check. Fails closed on any missing or stale evidence.
set -euo pipefail

VERDICT="${1:-verdict.json}"
[[ -f "$VERDICT" ]] || { echo "No verdict artifact: fail closed." >&2; exit 1; }

HEAD_SHA="$(git rev-parse HEAD)"
DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"

jq -e --arg sha "$HEAD_SHA" --arg digest "$DIGEST" '
  .status            == "complete"
  and .conclusion    == "pass"
  and .head_sha      == $sha
  and .input_digest  == $digest
' "$VERDICT" >/dev/null \
  || { echo "Verdict rejected: stale, partial, or mismatched input." >&2; exit 1; }

echo "Evidence accepted for $HEAD_SHA"

The digest formula has to be identical in both places. Copy it from one file into the other, or export it from a shared script — the moment the two commands drift by one sort flag, every verdict is silently rejected, and someone "fixes" it by loosening the gate.

Step 3 — a pre-push hook that keeps the deterministic subset honest

Hooks do not replace CI. They keep the cheap, deterministic checks off the critical path, so the required lanes stay short enough that cancelling them is never tempting.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
BASE="${BASE_REF:-origin/main}"

CHANGED="$(git diff --name-only "$BASE"...HEAD || true)"

if echo "$CHANGED" | grep -qE '^(ci/|\.github/workflows/)'; then
  echo "Pipeline config changed — running the gate self-test locally." >&2
  ./ci/gate.sh --self-test
fi

./ci/test.sh --subset=hermetic --no-retry

Enable it with git config core.hooksPath .githooks, and add one CI job that asserts the hook path is still set — otherwise the hook is decoration on everyone's machine but yours.

Step 4 — quarantine flakes with an expiry date, not a retry count

A retry nobody records is a false green with extra steps. Keep a small ledger, and make the gate read it.

# ci/flakes.yaml
- id: "tests/test_upload.py::test_resume_after_timeout"
  first_seen: 2026-09-02
  expires: 2026-10-02
  owner: "@emery"
  reason: "S3 mock intermittently returns 500"

Two rules make the ledger work:

The gate fails if an entry is expired. Not warns. Fails. An expiry you can ignore is a permanent exemption with a nice date on it.
Any retry increments retry_count in the verdict, and a non-zero retry_count on a required lane opens a tracking issue automatically. The retry stays visible instead of dissolving into a green check.

Choosing what is allowed to block

Lane	Required?	Can write to the repo?	Verdict artifact?
Advisory AI review	No	No (comments only)	Yes, complete-only
Hermetic unit tests	Yes	No	No
Integration suite	Yes, after quarantine review	No	No
Gate / evidence check	Yes	No	Reads only
Deploy preview	No	Preview environment only	No

The rule behind the table: anything that can produce a verdict must not be able to produce the input it is judging, and anything that can write to the repository must not be required.

Limitations, and who should skip this

This costs roughly a day of setup and one more artifact to keep alive. If you merge a handful of PRs a month, or your tests are not hermetic yet, the quarantine ledger becomes a place where failures go to be forgotten. Fix flakiness first.

Three more cases where this is the wrong trade:

Repositories where the model cannot legally see the code. If your data policy forbids sending source to a hosted model, the advisory lane does not exist and neither does most of this article.
Teams without artifact retention. If artifacts expire in a day and PRs live for three, the gate fails on old PRs for the wrong reason.
Pipelines where the expensive lane is genuinely required. A required non-deterministic lane is a queue you cannot merge through. Make it deterministic or make it advisory; do not make it required and cancellable.

Where a free lane actually fits

The advisory review lane is the right place to experiment, because it is the one lane in this design that cannot authorize a merge, cannot write to the repository, and can be cancelled without losing evidence.

The operator states that the MonkeyCode open-source project offers free model access and a free server option. Those are the two claims I am passing along. I am deliberately not quoting a token figure, a model name, or a duration here, because free-tier quotas and terms are exactly the kind of data that goes stale between writing and reading. Check the project's current terms before you size a lane around them.

If you run the lane on a free server, treat the box as ephemeral: no durable caches, no long-lived secrets, and rotate anything you put there. The gate above assumes the advisory lane is untrusted, which is the correct assumption for a free host.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One way to use it: point the advisory lane at the free server on a scratch fork, then push a second commit while the first review is still running and watch what your gate does. If it accepts the cancelled verdict, you have found a real bug — and the bug was never in the model.

Do Not Let the Explain Job Authorize the Retry

Emery Lin — Tue, 13 Oct 2026 02:29:21 +0000

A model can explain a red job. It must not authorize the retry, and it must not mint the merge. You freeze a local decision record before any assistant runs, and you point branch protection at that record alone.

This is a practical split you can add beside checks you already trust. The scripts are a proposed example. They were not executed for this draft. Run them on a branch, read the diff, and only then consider requiring the new check.

Stop treating a summary as an exit code

Logs are easier to summarize than they were a year ago. Your merge rule did not get stricter by itself. A paragraph that says "this looks flaky" invites a rerun. The rerun sometimes goes green for a reason the paragraph never named, and the paragraph becomes the story the team remembers.

You break that habit by refusing to parse prose in the gate. The classifier sees an exit class, a fixture digest, and a remaining flake budget. The explain job sees a redacted log. Those readers do not share a pen.

Give each artifact one writer

You want three files, and you want three different write permissions.

ci/decision.json is the merge evidence. The trusted test job writes it once per attempt, then the file becomes read-only.

ci/notes/failure.md is a comment. An explain job may create it. No required check scores its wording.

ci/retry-ledger.jsonl is an append-only list of rerun reasons. The allowed reasons are a short local vocabulary. "The note sounded flaky" is not in that vocabulary.

If one token can write all three, you do not have a gate. You have a notebook with permission to go green. Split the tokens before you split the workflow file.

Seal the record before any network call

Run the classifier on the trusted runner, immediately after the suite, with no model call and no socket. Exit code 75 is a convention you define in the harness for a known flake signature. It is not a platform default. If that signature does not exist in code, set retry_allowed to false and leave it there.

GITHUB_SHA below is the GitHub Actions name for the commit under test. Map it to your forge's own SHA variable. The rule is the seal, not the vendor.

#!/usr/bin/env bash
set -euo pipefail
# Proposed post-test hook. Unexecuted. No network.
rec=ci/decision.json
digest="$(sha256sum fixtures/manifest.json | awk '{print $1}')"
budget="$(jq -r '.flake_budget' policy/ci.json)"
used="$(wc -l < ci/retry-ledger.jsonl || true)"
used="${used:-0}"
remain=$((budget - used))
class="fail"
retry=false
if [[ "${TEST_EXIT}" == "0" ]]; then
  class="pass"
elif [[ "${TEST_EXIT}" == "75" && "${remain}" -gt 0 ]]; then
  class="flake_candidate"
  retry=true
fi
umask 022
jq -n \
  --arg head "${GITHUB_SHA}" \
  --arg digest "${digest}" \
  --arg class "${class}" \
  --argjson remain "${remain}" \
  --argjson retry "${retry}" \
  '{schema:1,job:"trusted-test",head:$head,fixture_sha256:$digest,exit_class:$class,flake_budget_remaining:$remain,retry_allowed:$retry,note_status:"absent"}' \
  > "${rec}"
sha256sum "${rec}" > ci/decision.json.sha256
chmod a-w "${rec}" ci/decision.json.sha256

note_status starts as absent and is not a slot for the model to fill in later. A rewrite of exit_class or retry_allowed must fail the hash check. You are freezing a small tuple, not collecting a richer story.

Keep policy/ci.json in git, with flake_budget as a plain integer. Review a budget change the way you review a production flag. A budget edited in the same pull request that is currently red is a smell. It is not a fix.

Walk the red job in five steps

Run the suite on the trusted runner. Save the raw exit code. Do not hide a retry inside the test process, and do not let a wrapper convert a timeout into success.
Run the hook above. It writes and freezes ci/decision.json for this attempt. If the hook fails, the attempt fails. You do not continue to an explain job without a seal.
If retry_allowed is true, append one ledger line with reason=local_flake_signature, then rerun once under a new attempt id. Write a new sealed record. Do not edit the first file in place.
Start the explain job only after a terminal failure. It may draft ci/notes/failure.md. It must not call the status API, and it must not push to the branch.
Require a pinned check that runs the verifier in the next section. Do not require the explain job. A missing note leaves the merge decision unchanged.

The ledger line is supposed to look boring. One reason, one attempt, no prose:

printf '%s attempt=%s reason=local_flake_signature head=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${ATTEMPT}" "${GITHUB_SHA}" >> ci/retry-ledger.jsonl

A second automatic retry means you do not have a budget. You have a loop. If you want another attempt, a person changes the policy in git and reviews that change in the open.

Host the note where it cannot set status

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The explain job is the only step where a hosted assistant belongs. MonkeyCode's free model access and free server option can host that optional job when you want a drafted note and you do not want to stand up another build machine. Treat both as current product terms you must read, not as facts this article measured. No token quota, machine size, time limit, or permanence is claimed here. If the docs you can cite disagree with the button you clicked, skip the remote job until they match.

Put a read-only redacted log on that runner. Limit its write path to ci/notes/. Do not mount the decision record read-write. Do not inject a credential that can create check runs. A free server is still someone else's CPU. You would not paste production secrets into a chat box. Do not upload them in a log either.

Stamp the note before any prose:

---
role: advisory
writer: explain-job
may_affect_status: false
---

Redact first. Remove Authorization headers, echoed environment files, signed URLs, and customer identifiers. If you cannot redact with confidence, write a local note that says the log was withheld, and do not call the model. The gate does not need a guess. It already has an exit class.

When the explain call times out, returns empty text, or never starts, you leave the sealed record alone. A missing comment is not a missing test. People get this backwards when a red X from the note job sits next to a real failure. Fix the job so it cannot publish a commit status. Do not "fix" the X by teaching reviewers to ignore red.

Verify the seal and ignore the tone

This verifier is the required check. It is a proposed, unexecuted script. It does not grade sentences. It assumes a Unix mode bit, which matches the chmod line above.

#!/usr/bin/env python3
"""Merge gate. Reads the sealed record. Does not score note prose."""
import hashlib, json, os, sys
from pathlib import Path

rec = Path("ci/decision.json")
seal = Path("ci/decision.json.sha256").read_text().split()[0]
body = rec.read_bytes()
if hashlib.sha256(body).hexdigest() != seal:
    sys.exit("decision record hash mismatch")
if rec.stat().st_mode & 0o222:
    sys.exit("decision record still has a write bit")
data = json.loads(body)
if data.get("schema") != 1:
    sys.exit("unexpected schema")
expected = os.environ.get("EXPECTED_SHA", "")
if not expected or data.get("head") != expected:
    sys.exit("head mismatch")
if data.get("job") != "trusted-test":
    sys.exit("unexpected writer")
if data.get("exit_class") != "pass":
    sys.exit("not green: %s" % data.get("exit_class"))
if data.get("retry_allowed") is True:
    sys.exit("final record still allows retry")
note = Path("ci/notes/failure.md")
if note.exists():
    text = note.read_text(encoding="utf-8")
    if "may_affect_status: false" not in text or "role: advisory" not in text:
        sys.exit("note missing advisory stamp")
print("merge-gate ok")

Give this check a stable name and protect that name. Do not let the explain workflow reuse it. The success line is there so you can grep the log. The script does not print the note. Once the note is in the required log, someone will screenshot it into the merge thread and treat the screenshot as evidence.

An empty EXPECTED_SHA fails closed. A verifier on the wrong checkout should not pass. Set that variable from the same commit the trusted job sealed.

Use a table when you want to rerun

Local signal	Note	Retry	Merge
exit 0, digest matches, record sealed	ignored	no	only through this required check
exit 75, budget left, first attempt	optional, after the seal	one local retry	no, until a new sealed pass exists
exit 75, budget already spent	optional	no	no
exit 1, unknown failure	optional, advisory	no	no
note says flake, exit is 1	present	no	no
explain job down or empty	absent	unchanged	unchanged
hash mismatch, or record still writable	any	no	no

The sixth row is the one to rehearse. Model downtime must not block a sealed pass. It also must not turn a fail into a pass. If your forge fails the whole commit whenever any job fails, the explain job is publishing a status it should not own. Narrow the token. Do not widen the retry rule to compensate.

The last row is the corruption case. A writable record, or a hash that does not match, means you stop. You do not ask the model whether the mismatch looks harmless. Harmless-looking mismatches are how a second writer learns it can edit the gate.

Limits, and who should skip this

This workflow does not rank models, and it does not show that a free server is faster or cheaper than a runner you already operate. It does not replace a fixture manifest check, a clean worktree assertion after tests, or a lock on replay artifacts. Those answer different questions. If you lack them, add them. Do not trade them for a clearer paragraph.

You should not use this approach when branch protection is off, when every green check is treated as equal, or when "flake" is a feeling instead of an exit code in the harness. You should not use the remote explain lane for release signing or for a production deploy. A diagnostic box is not a promotion environment. Skip the remote call entirely if you cannot deny status scope on that job's token, or if the log holds data you would refuse to paste into a third-party prompt.

Turn the explain job on only after the verifier already passes with no note file present. Until that rehearsal is done, a model in the pipeline is extra surface, not extra certainty. If the rehearsal holds and you still want a drafted note without another machine of your own, MonkeyCode's free model access and free server option are enough to try the explain job. Stop if that job can write the check your branch protection requires, or if the current docs no longer match the workflow you merged.