DEV Community: Emery Lin

A Green Pipeline That Ran Three of Five Shards Is Not Green

Emery Lin — Sat, 17 Oct 2026 06:25:04 +0000

Your merge queue trusts a color. It should trust a coverage claim.

A sharded CI run where shard 3 died during setup, shard 5 collected zero tests, and shard 1 passed on its third attempt produces exactly the same green check as a clean sweep of the whole matrix. Nothing in the check name tells you which one you got. The fix is not a smarter retry policy — it is a per-shard manifest plus a gate that refuses to merge anything the manifest cannot prove.

Four ways a shard quietly disappears

The matrix shrinks. --splits is computed from a test count that changed between the plan job and the test job. Fewer jobs get scheduled, needs is satisfied, the workflow is green.
A partition is empty. Shard 4 is assigned no tests. pytest exits with code 5, and someone already appended || true to keep the pipeline moving.
A step is allowed to fail. continue-on-error: true left over from a migration week means a crashed shard reports success at the job level.
A retry hides the first attempt. pytest-rerunfailures turns a flake into a pass, and the report contains no field that says "this took three tries."

All four are coverage bugs, not flake bugs. Treat them that way.

The contract: one manifest per shard

Stop reading the job summary. Have every shard write a small JSON file next to its report, and let the gate read only those files.

The manifest needs six fields and nothing more: index, commit, collected, node_ids, status, attempts. Add quarantined if you run a quarantine lane. If a field is missing, the gate blocks — absence is not consent.

Step 1 — Discover once, split deterministically

Collect node IDs in a single job, persist them, and split from that file. Never recompute the split inside the test job.

python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
wc -l < nodes.txt   # the number every later step must agree with

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      shards: ${{ steps.plan.outputs.shards }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - id: plan
        run: |
          python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
          n=$(wc -l < nodes.txt)
          echo 'shards='$(( (n + 199) / 200 )) >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with: { name: nodes, path: nodes.txt }

Step 2 — Emit the manifest from a hook

Do not parse the terminal output. Use pytest_sessionfinish and derive everything from session.items.

# conftest.py
import json, os, pathlib

def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path(os.environ.get('SHARD_DIR', 'shard-reports'))
    out.mkdir(parents=True, exist_ok=True)
    nodes = sorted({item.nodeid for item in session.items})
    payload = {
        'index': int(os.environ['SHARD_INDEX']),
        'commit': os.environ['GITHUB_SHA'],
        'collected': len(nodes),
        'node_ids': nodes,
        'status': 'passed' if exitstatus == 0 else 'failed',
        'attempts': int(os.environ.get('SHARD_ATTEMPT', '1')),
        'quarantined': os.environ.get('QUARANTINE') == '1',
    }
    (out / f'shard-{payload["index"]}.json').write_text(json.dumps(payload))

Expose SHARD_INDEX from the matrix, not from a counter inside the process. Counter drift is how two shards end up writing shard-0.json and the gate sees four files where five were expected.

Step 3 — Gate on the manifest, not the color

Stdlib only, no dependencies, no network. It exits non-zero on the first violation it can explain.

# gate_shards.py
import json, os, pathlib, sys

head = os.environ['GITHUB_SHA']
expected = int(os.environ['SHARD_COUNT'])
files = sorted(pathlib.Path(sys.argv[1]).glob('shard-*.json'))

def block(msg):
    print(f'::error::{msg}')
    sys.exit(1)

if len(files) != expected:
    block(f'found {len(files)} manifests, expected {expected}')

indexes, owner, retried = set(), {}, []
for path in files:
    m = json.loads(path.read_text())
    commit, index, status = m['commit'], m['index'], m['status']
    if commit != head:
        block(f'{path.name} tested {commit[:8]}, head is {head[:8]}')
    if index in indexes:
        block(f'shard {index} reported twice')
    indexes.add(index)
    if status != 'passed':
        block(f'shard {index} finished as {status}')
    if m['collected'] == 0:
        block(f'shard {index} collected zero tests')
    if m['attempts'] > 1 and not m['quarantined']:
        retried.append(index)
    for node in m['node_ids']:
        if node in owner:
            block(f'{node} ran in shards {owner[node]} and {index}')
        owner[node] = index

missing = sorted(set(range(expected)) - indexes)
if missing:
    block(f'no result for shard(s) {missing}')
if retried:
    block(f'green only after a retry in shard(s) {retried}')
print(f'OK: {len(owner)} node ids across {expected} shards on {head[:8]}')

Note what the duplicate check buys you: a split bug that sends the same 40 tests to two shards now fails the gate instead of inflating your pass count.

Step 4 — Wire it as its own required check

  gate:
    needs: [plan, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: shard-*
          path: shard-reports
          merge-multiple: true
      - env:
          SHARD_COUNT: ${{ needs.plan.outputs.shards }}
          GITHUB_SHA: ${{ github.event.pull_request.head.sha }}
        run: python3 gate_shards.py shard-reports

Two details earn their keep. if: always() stops the gate from being skipped when test fails, and a skipped gate reads like a passed one in most UIs. Overriding GITHUB_SHA with the PR head SHA means the manifest is compared against the code you reviewed, not the merge commit.

Step 5 — Make the retry ledger boring

The attempts > 1 rule is the flake policy, and it is deliberately blunt: a test that only goes green on the second attempt blocks the merge until someone either fixes it or moves it to the quarantine lane with an owner and an expiry date. Rerunning until green is not a decision, it is a coin flip with a paper trail.

Track the top five quarantined files weekly. If the list does not shrink, your quarantine is where tests go to be forgotten.

Where a free model actually fits

Not in the gate. The gate must stay deterministic, offline, and explainable in one line of output.

Triage is a different job: turning twenty scattered failure snippets into three probable root causes is exactly the kind of summarization that saves a reviewer ten minutes. MonkeyCode is an open-source project that offers free model access and a free server option, which makes it a low-friction place to run that advisory step before you decide to pay for anything. The project advertises a 10M-token allowance and a free server tier — treat both as vendor claims and check the current numbers in its console before you build a dependency on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# triage.py — advisory only; never writes status
import json, os, urllib.request, pathlib

fails = []
for path in pathlib.Path('shard-reports').glob('shard-*.json'):
    fails.extend(json.loads(path.read_text()).get('failures', []))
if not fails:
    raise SystemExit('nothing to triage')

prompt = 'Group by shared root cause. Markdown, max 200 words.\n\n' + '\n'.join(fails[:20])
body = json.dumps({'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0}).encode()
req = urllib.request.Request(
    os.environ['MONKEYCODE_BASE_URL'] + '/chat/completions',
    data=body,
    headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY'],
             'Content-Type': 'application/json'},
)
print(urllib.request.urlopen(req, timeout=60).read().decode())

Read the base URL and model ID from the project's own docs; endpoints move, and a hardcoded one in a tutorial rots the week after publication. Scrub secrets and customer identifiers from fails before it leaves the runner — failure logs are the most common accidental leak in this pattern.

Hard rule: triage output lands in a PR comment. It never sets status, never flips quarantined, never authorizes a merge.

Limits, and who should skip this

Small suites. Under roughly 200 tests, sharding overhead exceeds the wall-clock win, and one manifest beats five.
Non-hermetic runners. If your runner image digest can change mid-run, a passed shard is not reproducible and the manifest proves less than you think.
Regulated audit trails. Don't put a compliance gate behind a free tier whose quota and retention you have not verified in writing.
Anyone tempted to parse model prose. If you cannot keep the machine summary out of the decision path, don't add the summary.

The checker above is stdlib Python 3.12 and untested against your report format — adapt the status mapping to your runner before you make the gate required. Once it is required, the interesting metric stops being pass rate and becomes coverage: how many node IDs, from how many shards, on which commit.

If you want to try the triage step before wiring a billing account, the project's free model access and free server option are the shortest path to finding out whether it earns a slot in your pipeline.

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

Emery Lin — Fri, 16 Oct 2026 05:25:39 +0000

Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

A retry counts as evidence only when it replays from a cold cache and leaves a record a later job can verify. A warm rerun proves that your cache works. It does not prove that your branch builds.

Most advice about flaky CI ends at "just retry the job". That advice quietly hands your merge queue a pass that was never produced under the same conditions as a fresh clone, and it leaves nothing behind for the merge job to inspect. This post fixes the boring half of the problem: make one replay emit a machine-checkable admission record, then make the merge path refuse anything else.

The workflow below is a proposal you can adapt, not a transcript of a run I am handing you as a benchmark. Label your own numbers when you measure them.

The failure mode, stated precisely

A pull request fails on a cold runner because a dependency resolves differently without the warm module cache. Someone clicks retry. The job runs on the same runner, reuses the toolchain, and goes green.

The green check is real, but its subject is different. It tested the tree plus the warm cache, and your merge queue only knows about the tree. When main is built from scratch later, the original failure returns, now attributed to a merge that "passed CI".

So the rule is short: a replay resets the environment and writes down what the environment was. Everything else is a cache hit wearing a check mark.

Write the admission record first

Define the record before you write the workflow. You want fields that the merge job can compare against current reality, not a prose summary. Keep it to one small JSON artifact.

{
  "schema": "admission/v1",
  "tree_sha": "9f1c4e2",
  "base_sha": "3ab07d1",
  "runner_image": "ghcr.io/acme/ci-runner@sha256:91a0...",
  "hook_version": 7,
  "cache": "cold",
  "seed": 4211,
  "toolchain": { "node": "20.11.1", "python": "3.12.3" },
  "result": "pass",
  "started_at": "2026-10-16T09:12:04Z"
}

Four fields do the heavy lifting. cache must equal cold, or the record is void by definition. base_sha catches the case where main moved after your replay. runner_image catches a silent runner rotation. hook_version lets you invalidate every old record in one commit when you change the test harness itself.

If you have already learned not to let a cancelled run count as a pass, the same logic applies here: a cancelled or timed-out replay writes no record, and a missing record is a failure, never a neutral state.

Step 1: purge, then replay

The workflow has one job that runs on a clean runner and deliberately destroys every cache it can reach. Note that the retry path and the first attempt use the same job; there is no separate "fast" path for reruns.

name: replay
on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  checks: write

jobs:
  cold-replay:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0

      - name: Record the base commit
        run: echo BASE_SHA=${{ github.event.pull_request.base.sha }} >> $GITHUB_ENV

      - name: Record the tree commit
        run: echo TREE_SHA=$(git rev-parse HEAD^{tree}) >> $GITHUB_ENV

      - name: Purge every warm artifact
        run: rm -rf node_modules .venv ~/.npm ~/.cache/pip ~/.cargo/registry

      - name: Read the runner image digest
        run: echo IMAGE_DIGEST=$(jq -r .image /etc/runner-manifest.json) >> $GITHUB_ENV

      - name: Install pinned toolchain
        run: |
          corepack enable
          npm ci --no-audit --no-fund

      - name: Run the suite with a fixed seed
        run: npm test -- --seed=${{ github.run_number }}4211 --reporter=json > test-report.json

      - name: Emit the admission record
        if: always()
        run: python3 scripts/emit_record.py

      - uses: actions/upload-artifact@v4
        with:
          name: admission-record
          path: admission.json
          retention-days: 14

Two details matter more than the rest. The seed is derived from the run number so a retry is a genuinely different ordering, not a repeat of the same lucky interleaving. And the record upload uses if: always(), so a failing replay still produces an artifact that says "result: fail" — invisible failures are how teams end up re-running until green.

Step 2: gate the merge on the record

The merge job never runs the test suite. It downloads the record and checks it against the repository as it exists right now. If any input moved, the record is stale and the merge is refused.

#!/usr/bin/env python3
# scripts/admit.py
import json, subprocess, sys

REQUIRED = {
    'schema', 'tree_sha', 'base_sha', 'runner_image',
    'hook_version', 'cache', 'seed', 'result',
}

def git(*args):
    return subprocess.check_output(['git', *args], text=True).strip()

def admit(record, head_ref, base_sha, image_digest, hook_version):
    missing = REQUIRED - record.keys()
    if missing:
        return False, f'missing fields: {sorted(missing)}'
    if record['schema'] != 'admission/v1':
        return False, 'unknown record schema'
    if record['cache'] != 'cold':
        return False, 'record came from a warm cache'
    if record['result'] != 'pass':
        return False, 'record is not a pass'
    if record['tree_sha'] != git('rev-parse', head_ref + '^{tree}'):
        return False, 'tree changed after the replay'
    if record['base_sha'] != base_sha:
        return False, 'base moved; rebase and replay'
    if record['runner_image'] != image_digest:
        return False, 'runner image rotated; replay on the new image'
    if str(record['hook_version']) != str(hook_version):
        return False, 'hook version changed'
    return True, 'admitted'

if __name__ == '__main__':
    record = json.load(open(sys.argv[1]))
    ok, reason = admit(
        record,
        head_ref=sys.argv[2],
        base_sha=sys.argv[3],
        image_digest=sys.argv[4],
        hook_version=sys.argv[5],
    )
    print(reason)
    sys.exit(0 if ok else 1)

Keep this script in the default branch and call it from a required check that the pull request author cannot edit. A gate that lives in the same diff it approves is decoration.

Step 3: classify the flake before you spend a retry

Not every red run deserves a replay. Split failures into two buckets before you rerun anything, because a product failure replayed three times is three times the noise.

Infrastructure failures look like exit code 143, OOM kills, DNS resolution errors, or a registry 5xx in the log. These are the only ones eligible for a replay, and each replay voids the previous record. Product failures — an assertion, a snapshot diff, a type error — block the branch and go to a human.

Budget the replays per pull request, not per job. Two replays is a common ceiling; if a third is needed, the branch has a real ordering dependency and you should fix it instead of paying for it.

Step 4: the green-to-merge decision table

Situation	Action
Record passes; tree, base, image and hook version all match	Auto-merge
Any infrastructure replay happened	Require a fresh cold replay before merge
Failure is a product assertion	Block; human triage
Base branch moved after the replay	Rebase, then cold replay
Runner image rotated since the replay	Cold replay on the new image
No record (cancelled, timeout, OOM)	Treat as failing, never neutral
Hook version changed in this diff	Reject the diff; harness changes land separately

That last row is the one people skip, and it is the one that keeps the gate honest. If a pull request can change the harness and the code it tests at the same time, your record describes a moving target.

Where a free model and a free server fit

If your patches come from an agent, the same record shape still applies — the agent is just another producer of a tree hash. In that setup the operator states that MonkeyCode offers free model access plus a free server option, which is enough to try this loop before committing paid CI minutes to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the part you can control: generate the patch with the free model access, run the cold replay on the free server option, and keep the admission record identical to the one your paid runners emit. Do not hard-code quota or hardware numbers into your workflow — free-tier terms change, so read the current page before you depend on a specific allowance, and never treat a free runner as a compliance or isolation boundary.

Limitations, and who should skip this

Cold replay costs time. A suite that takes twelve minutes warm can take twenty-five or more cold, and monorepos with large native builds will feel it first. If your build already requires a licensed artifact you cannot download on a clean machine, this approach will block you until you solve caching deliberately — which is a different project.

Skip it if your team has no required check that pull request authors cannot edit, because the record will not be enforceable. Skip it if every failure is already a product failure and your flake rate is near zero. The overhead only pays for itself when a warm cache is a plausible explanation for a green run.

A five-line checklist for the next pull request

Does the failing job emit an admission record even when it fails?
Does the record include the tree, the base, the runner image, and the hook version?
Is the merge gate reading that record from a path the author cannot modify?
Is the replay budget per pull request, not per job?
When the base moves, does the old record become invalid automatically?

If you can answer yes to all five, a green check means something you can defend after the outage.

A Cancelled Review Run Is Not a Passing Review

Emery Lin — Thu, 15 Oct 2026 11:30:30 +0000

The rule, in one line. A cancelled run is missing evidence, and a missing verdict must never be read as a pass.

That sentence is the article. The rest is the machinery that enforces it in a pipeline where one non-deterministic lane — an AI review step, a fuzz run, an end-to-end suite — sits next to the checks that actually block a merge.

Why the cancelled lane is the expensive one

Most teams bolt on an AI review step because it is cheap and advisory. Then they do two reasonable things that combine badly.

They set concurrency: cancel-in-progress: true, so a fresh push cancels the review of the previous commit. That is the correct cost decision.
They either require the review check in branch protection, or they treat the bot's comment as the verdict. Both make a cancelled run load-bearing.

Put those together and you get the worst kind of bug: a lane that has never once emitted "no evidence", merged into a system where nobody has tested what "no evidence" means. A check whose failure path has never run is not a check. It is a habit.

The same shape appears in browser suites cancelled on the third retry, scanners that time out, and any job with a 10-minute timeout sitting next to a 6-minute model call.

Bind the verdict to fields, not to a comment

A comment is not evidence. Evidence is a small artifact with fields the gate can verify without trusting the producer. Four fields carry the weight:

head_sha — the commit the verdict is about. Without it, a review of commit A authorizes a merge of commit B.
input_digest — a hash of exactly what the lane read: tracked file modes and contents plus lockfiles. Without it, a review of last week's lockfile authorizes today's dependency bump.
run_attempt — which attempt produced this. A verdict from attempt 1 must not authorize attempt 3.
status: complete — set only on normal completion of the lane. A cancelled or killed job writes nothing.

That last point is the design: a cancelled lane produces no artifact. Absence is the signal. You never have to teach the gate to interpret cancelled, because there is nothing to interpret.

Step 1 — the advisory workflow

Reference implementation, not something I have run against your repository. Treat it as a starting point, and verify the cancellation behaviour on your own runner first, because step-level conditions during cancellation have changed between runner versions.

name: advisory-review
on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: advisory-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true

permissions:
  contents: read
  pull-requests: write

jobs:
  review:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}

      - name: Compute input digest
        id: digest
        run: |
          DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Run review lane
        id: lane
        env:
          MONKEYCODE_API_KEY: ${{ secrets.MONKEYCODE_API_KEY }}
        run: ./ci/run-review.sh > review.md

      - name: Write verdict
        if: success() || failure()
        run: |
          STATUS="fail"
          [ "${{ steps.lane.outcome }}" = "success" ] && STATUS="pass"
          jq -n \
            --arg sha "${{ github.event.pull_request.head.sha }}" \
            --arg digest "${{ steps.digest.outputs.digest }}" \
            --arg attempt "${{ github.run_attempt }}" \
            --arg status "$STATUS" \
            '{status: "complete", conclusion: $status, head_sha: $sha,
              input_digest: $digest, run_attempt: $attempt}' > verdict.json

      - name: Upload verdict
        if: success() || failure()
        uses: actions/upload-artifact@v4
        with:
          name: verdict-${{ github.event.pull_request.number }}-${{ github.run_attempt }}
          retention-days: 7
          path: verdict.json

Note the asymmetry: the artifact is uploaded on both success and failure of the lane, but not on cancellation. A red verdict is a verdict; a missing verdict is not green.

Step 2 — the required gate

Keep the gate in a separate workflow triggered by workflow_run, or as a job that downloads the artifact. It reads the repository and nothing else. No secrets, no tokens, no write access.

#!/usr/bin/env bash
# ci/gate.sh — required check. Fails closed on any missing or stale evidence.
set -euo pipefail

VERDICT="${1:-verdict.json}"
[[ -f "$VERDICT" ]] || { echo "No verdict artifact: fail closed." >&2; exit 1; }

HEAD_SHA="$(git rev-parse HEAD)"
DIGEST="$( { git ls-files -s | sort; sha256sum package-lock.json 2>/dev/null || true; } | sha256sum | cut -d' ' -f1 )"

jq -e --arg sha "$HEAD_SHA" --arg digest "$DIGEST" '
  .status            == "complete"
  and .conclusion    == "pass"
  and .head_sha      == $sha
  and .input_digest  == $digest
' "$VERDICT" >/dev/null \
  || { echo "Verdict rejected: stale, partial, or mismatched input." >&2; exit 1; }

echo "Evidence accepted for $HEAD_SHA"

The digest formula has to be identical in both places. Copy it from one file into the other, or export it from a shared script — the moment the two commands drift by one sort flag, every verdict is silently rejected, and someone "fixes" it by loosening the gate.

Step 3 — a pre-push hook that keeps the deterministic subset honest

Hooks do not replace CI. They keep the cheap, deterministic checks off the critical path, so the required lanes stay short enough that cancelling them is never tempting.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
BASE="${BASE_REF:-origin/main}"

CHANGED="$(git diff --name-only "$BASE"...HEAD || true)"

if echo "$CHANGED" | grep -qE '^(ci/|\.github/workflows/)'; then
  echo "Pipeline config changed — running the gate self-test locally." >&2
  ./ci/gate.sh --self-test
fi

./ci/test.sh --subset=hermetic --no-retry

Enable it with git config core.hooksPath .githooks, and add one CI job that asserts the hook path is still set — otherwise the hook is decoration on everyone's machine but yours.

Step 4 — quarantine flakes with an expiry date, not a retry count

A retry nobody records is a false green with extra steps. Keep a small ledger, and make the gate read it.

# ci/flakes.yaml
- id: "tests/test_upload.py::test_resume_after_timeout"
  first_seen: 2026-09-02
  expires: 2026-10-02
  owner: "@emery"
  reason: "S3 mock intermittently returns 500"

Two rules make the ledger work:

The gate fails if an entry is expired. Not warns. Fails. An expiry you can ignore is a permanent exemption with a nice date on it.
Any retry increments retry_count in the verdict, and a non-zero retry_count on a required lane opens a tracking issue automatically. The retry stays visible instead of dissolving into a green check.

Choosing what is allowed to block

Lane	Required?	Can write to the repo?	Verdict artifact?
Advisory AI review	No	No (comments only)	Yes, complete-only
Hermetic unit tests	Yes	No	No
Integration suite	Yes, after quarantine review	No	No
Gate / evidence check	Yes	No	Reads only
Deploy preview	No	Preview environment only	No

The rule behind the table: anything that can produce a verdict must not be able to produce the input it is judging, and anything that can write to the repository must not be required.

Limitations, and who should skip this

This costs roughly a day of setup and one more artifact to keep alive. If you merge a handful of PRs a month, or your tests are not hermetic yet, the quarantine ledger becomes a place where failures go to be forgotten. Fix flakiness first.

Three more cases where this is the wrong trade:

Repositories where the model cannot legally see the code. If your data policy forbids sending source to a hosted model, the advisory lane does not exist and neither does most of this article.
Teams without artifact retention. If artifacts expire in a day and PRs live for three, the gate fails on old PRs for the wrong reason.
Pipelines where the expensive lane is genuinely required. A required non-deterministic lane is a queue you cannot merge through. Make it deterministic or make it advisory; do not make it required and cancellable.

Where a free lane actually fits

The advisory review lane is the right place to experiment, because it is the one lane in this design that cannot authorize a merge, cannot write to the repository, and can be cancelled without losing evidence.

The operator states that the MonkeyCode open-source project offers free model access and a free server option. Those are the two claims I am passing along. I am deliberately not quoting a token figure, a model name, or a duration here, because free-tier quotas and terms are exactly the kind of data that goes stale between writing and reading. Check the project's current terms before you size a lane around them.

If you run the lane on a free server, treat the box as ephemeral: no durable caches, no long-lived secrets, and rotate anything you put there. The gate above assumes the advisory lane is untrusted, which is the correct assumption for a free host.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One way to use it: point the advisory lane at the free server on a scratch fork, then push a second commit while the first review is still running and watch what your gate does. If it accepts the cancelled verdict, you have found a real bug — and the bug was never in the model.

Do Not Let the Explain Job Authorize the Retry

Emery Lin — Tue, 13 Oct 2026 02:29:21 +0000

A model can explain a red job. It must not authorize the retry, and it must not mint the merge. You freeze a local decision record before any assistant runs, and you point branch protection at that record alone.

This is a practical split you can add beside checks you already trust. The scripts are a proposed example. They were not executed for this draft. Run them on a branch, read the diff, and only then consider requiring the new check.

Stop treating a summary as an exit code

Logs are easier to summarize than they were a year ago. Your merge rule did not get stricter by itself. A paragraph that says "this looks flaky" invites a rerun. The rerun sometimes goes green for a reason the paragraph never named, and the paragraph becomes the story the team remembers.

You break that habit by refusing to parse prose in the gate. The classifier sees an exit class, a fixture digest, and a remaining flake budget. The explain job sees a redacted log. Those readers do not share a pen.

Give each artifact one writer

You want three files, and you want three different write permissions.

ci/decision.json is the merge evidence. The trusted test job writes it once per attempt, then the file becomes read-only.

ci/notes/failure.md is a comment. An explain job may create it. No required check scores its wording.

ci/retry-ledger.jsonl is an append-only list of rerun reasons. The allowed reasons are a short local vocabulary. "The note sounded flaky" is not in that vocabulary.

If one token can write all three, you do not have a gate. You have a notebook with permission to go green. Split the tokens before you split the workflow file.

Seal the record before any network call

Run the classifier on the trusted runner, immediately after the suite, with no model call and no socket. Exit code 75 is a convention you define in the harness for a known flake signature. It is not a platform default. If that signature does not exist in code, set retry_allowed to false and leave it there.

GITHUB_SHA below is the GitHub Actions name for the commit under test. Map it to your forge's own SHA variable. The rule is the seal, not the vendor.

#!/usr/bin/env bash
set -euo pipefail
# Proposed post-test hook. Unexecuted. No network.
rec=ci/decision.json
digest="$(sha256sum fixtures/manifest.json | awk '{print $1}')"
budget="$(jq -r '.flake_budget' policy/ci.json)"
used="$(wc -l < ci/retry-ledger.jsonl || true)"
used="${used:-0}"
remain=$((budget - used))
class="fail"
retry=false
if [[ "${TEST_EXIT}" == "0" ]]; then
  class="pass"
elif [[ "${TEST_EXIT}" == "75" && "${remain}" -gt 0 ]]; then
  class="flake_candidate"
  retry=true
fi
umask 022
jq -n \
  --arg head "${GITHUB_SHA}" \
  --arg digest "${digest}" \
  --arg class "${class}" \
  --argjson remain "${remain}" \
  --argjson retry "${retry}" \
  '{schema:1,job:"trusted-test",head:$head,fixture_sha256:$digest,exit_class:$class,flake_budget_remaining:$remain,retry_allowed:$retry,note_status:"absent"}' \
  > "${rec}"
sha256sum "${rec}" > ci/decision.json.sha256
chmod a-w "${rec}" ci/decision.json.sha256

note_status starts as absent and is not a slot for the model to fill in later. A rewrite of exit_class or retry_allowed must fail the hash check. You are freezing a small tuple, not collecting a richer story.

Keep policy/ci.json in git, with flake_budget as a plain integer. Review a budget change the way you review a production flag. A budget edited in the same pull request that is currently red is a smell. It is not a fix.

Walk the red job in five steps

Run the suite on the trusted runner. Save the raw exit code. Do not hide a retry inside the test process, and do not let a wrapper convert a timeout into success.
Run the hook above. It writes and freezes ci/decision.json for this attempt. If the hook fails, the attempt fails. You do not continue to an explain job without a seal.
If retry_allowed is true, append one ledger line with reason=local_flake_signature, then rerun once under a new attempt id. Write a new sealed record. Do not edit the first file in place.
Start the explain job only after a terminal failure. It may draft ci/notes/failure.md. It must not call the status API, and it must not push to the branch.
Require a pinned check that runs the verifier in the next section. Do not require the explain job. A missing note leaves the merge decision unchanged.

The ledger line is supposed to look boring. One reason, one attempt, no prose:

printf '%s attempt=%s reason=local_flake_signature head=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${ATTEMPT}" "${GITHUB_SHA}" >> ci/retry-ledger.jsonl

A second automatic retry means you do not have a budget. You have a loop. If you want another attempt, a person changes the policy in git and reviews that change in the open.

Host the note where it cannot set status

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The explain job is the only step where a hosted assistant belongs. MonkeyCode's free model access and free server option can host that optional job when you want a drafted note and you do not want to stand up another build machine. Treat both as current product terms you must read, not as facts this article measured. No token quota, machine size, time limit, or permanence is claimed here. If the docs you can cite disagree with the button you clicked, skip the remote job until they match.

Put a read-only redacted log on that runner. Limit its write path to ci/notes/. Do not mount the decision record read-write. Do not inject a credential that can create check runs. A free server is still someone else's CPU. You would not paste production secrets into a chat box. Do not upload them in a log either.

Stamp the note before any prose:

---
role: advisory
writer: explain-job
may_affect_status: false
---

Redact first. Remove Authorization headers, echoed environment files, signed URLs, and customer identifiers. If you cannot redact with confidence, write a local note that says the log was withheld, and do not call the model. The gate does not need a guess. It already has an exit class.

When the explain call times out, returns empty text, or never starts, you leave the sealed record alone. A missing comment is not a missing test. People get this backwards when a red X from the note job sits next to a real failure. Fix the job so it cannot publish a commit status. Do not "fix" the X by teaching reviewers to ignore red.

Verify the seal and ignore the tone

This verifier is the required check. It is a proposed, unexecuted script. It does not grade sentences. It assumes a Unix mode bit, which matches the chmod line above.

#!/usr/bin/env python3
"""Merge gate. Reads the sealed record. Does not score note prose."""
import hashlib, json, os, sys
from pathlib import Path

rec = Path("ci/decision.json")
seal = Path("ci/decision.json.sha256").read_text().split()[0]
body = rec.read_bytes()
if hashlib.sha256(body).hexdigest() != seal:
    sys.exit("decision record hash mismatch")
if rec.stat().st_mode & 0o222:
    sys.exit("decision record still has a write bit")
data = json.loads(body)
if data.get("schema") != 1:
    sys.exit("unexpected schema")
expected = os.environ.get("EXPECTED_SHA", "")
if not expected or data.get("head") != expected:
    sys.exit("head mismatch")
if data.get("job") != "trusted-test":
    sys.exit("unexpected writer")
if data.get("exit_class") != "pass":
    sys.exit("not green: %s" % data.get("exit_class"))
if data.get("retry_allowed") is True:
    sys.exit("final record still allows retry")
note = Path("ci/notes/failure.md")
if note.exists():
    text = note.read_text(encoding="utf-8")
    if "may_affect_status: false" not in text or "role: advisory" not in text:
        sys.exit("note missing advisory stamp")
print("merge-gate ok")

Give this check a stable name and protect that name. Do not let the explain workflow reuse it. The success line is there so you can grep the log. The script does not print the note. Once the note is in the required log, someone will screenshot it into the merge thread and treat the screenshot as evidence.

An empty EXPECTED_SHA fails closed. A verifier on the wrong checkout should not pass. Set that variable from the same commit the trusted job sealed.

Use a table when you want to rerun

Local signal	Note	Retry	Merge
exit 0, digest matches, record sealed	ignored	no	only through this required check
exit 75, budget left, first attempt	optional, after the seal	one local retry	no, until a new sealed pass exists
exit 75, budget already spent	optional	no	no
exit 1, unknown failure	optional, advisory	no	no
note says flake, exit is 1	present	no	no
explain job down or empty	absent	unchanged	unchanged
hash mismatch, or record still writable	any	no	no

The sixth row is the one to rehearse. Model downtime must not block a sealed pass. It also must not turn a fail into a pass. If your forge fails the whole commit whenever any job fails, the explain job is publishing a status it should not own. Narrow the token. Do not widen the retry rule to compensate.

The last row is the corruption case. A writable record, or a hash that does not match, means you stop. You do not ask the model whether the mismatch looks harmless. Harmless-looking mismatches are how a second writer learns it can edit the gate.

Limits, and who should skip this

This workflow does not rank models, and it does not show that a free server is faster or cheaper than a runner you already operate. It does not replace a fixture manifest check, a clean worktree assertion after tests, or a lock on replay artifacts. Those answer different questions. If you lack them, add them. Do not trade them for a clearer paragraph.

You should not use this approach when branch protection is off, when every green check is treated as equal, or when "flake" is a feeling instead of an exit code in the harness. You should not use the remote explain lane for release signing or for a production deploy. A diagnostic box is not a promotion environment. Skip the remote call entirely if you cannot deny status scope on that job's token, or if the log holds data you would refuse to paste into a third-party prompt.

Turn the explain job on only after the verifier already passes with no note file present. Until that rehearsal is done, a model in the pipeline is extra surface, not extra certainty. If the rehearsal holds and you still want a drafted note without another machine of your own, MonkeyCode's free model access and free server option are enough to try the explain job. Stop if that job can write the check your branch protection requires, or if the current docs no longer match the workflow you merged.

Refuse the Rerun Until the Fixture Manifest Still Matches

Emery Lin — Sun, 11 Oct 2026 21:55:56 +0000

A rerun is only a rerun if the files under test did not move. If the same workspace can edit a fixture and then go green, the required check is grading a different contract than the one that failed.

You do not need a smarter retry button for that. You need a frozen manifest, a job that cannot write it, and a verdict small enough to reject.

What this path actually proves

This workflow separates three facts that teams often collapse into one green square.

The fixture bytes at failure time.
The advisory opinion about whether a retry is even allowed.
The required status check that a merge queue is permitted to trust.

The first and third stay on a runner with no model credential. The second may call a model. It may not push, commit, or rewrite the check name.

None of this measures flake rate. It only stops a class of false greens: a suggestion lane that repairs the evidence and then reports success.

1. Pin the fixture manifest in the required job

Start from an explicit list. Do not hash the whole repository. A lockfile bump or a changelog edit is not a fixture change, and treating it as one will train people to ignore the check.

# Proposal: run this inside the required job, before tests.
# It has not been executed against your repository.
mkdir -p .ci
find tests/fixtures -type f | sort > .ci/fixture-list.txt
sha256sum $(cat .ci/fixture-list.txt) > .ci/fixture-manifest.txt
cp .ci/fixture-manifest.txt "$RUNNER_TEMP/fixture-manifest.txt"
test -s .ci/fixture-list.txt

Keep the committed list under review. The generated manifest is an artifact, not a file the test job may commit back.

If tests/fixtures is empty, fail closed. A missing contract is not a passing contract. Add the directory to the job's checkout only. Do not generate replacement files in this step just to make the hash command succeed.

2. Upload the red log, then leave the workspace

On failure, publish the log and the manifest. Do not publish the worktree. The worktree is how a later step smuggles a rewrite back in.

# Illustrative fragment, not a live workflow from this account.
# Pin the action version your organization already trusts.
- name: Preserve failure evidence
  if: failure()
  uses: actions/upload-artifact
  with:
    name: red-evidence-${{ github.run_id }}
    path: |
      .ci/fixture-manifest.txt
      test-output.log
    if-no-files-found: error

The required job ends here when tests fail. It does not call a model. It does not apply a patch. It does not rerun itself.

That split matters more than the model you pick later. A credential on the required runner is a write path, even if today's prompt says only summarize.

3. Let the advisory job read, not repair

Use a second job. Give it contents: read and the downloaded artifact. Deny it pull-requests: write and any deploy key.

This is the one place a hosted model is useful. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied availability claim used here is limited: free model access, plus a free server option, can host this advisory step. This draft does not assert a token quota, a model name, a hardware shape, a time limit, or that the free tier will stay as it is.

The advisory job may return only this JSON shape:

{
  "schema": "retry-verdict/v1",
  "action": "stop",
  "failure_class": "assertion",
  "reason": "expected status drifted; fixture bytes unchanged"
}

Allowed action values are retry, quarantine, and stop. There is no patch field. If a field appears that the schema does not list, the verdict is invalid.

# Unexecuted proposal. Adapt paths before you rely on it.
import hashlib, json, sys
from pathlib import Path

ALLOWED = {"retry", "quarantine", "stop"}
REQUIRED = {"schema", "action", "failure_class", "reason"}

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def main() -> int:
    verdict = json.loads(Path(sys.argv[1]).read_text())
    if set(verdict) != REQUIRED or verdict["schema"] != "retry-verdict/v1":
        print("reject: schema")
        return 2
    if verdict["action"] not in ALLOWED:
        print("reject: action")
        return 2
    if "\n" in verdict["reason"] or len(verdict["reason"]) > 240:
        print("reject: reason")
        return 2
    pinned = Path(sys.argv[2]).read_text().splitlines()
    for line in pinned:
        digest, rel = line.split(None, 1)
        rel = rel.lstrip("*").strip()
        if sha256_file(Path(rel)) != digest:
            print("reject: fixture moved: " + rel)
            return 3
    print("accept")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Call it with the verdict file and the pinned manifest. A nonzero exit blocks the rerun. It does not open a pull request.

Feed the model the log excerpt and the manifest hash, not the repository remote. Ask it to fill the schema. Then throw the prose away and keep the JSON only if the checker accepts it.

4. Decide with a table, not with a vibe

Use the failure class only after the hash check passes. If the hash check fails, the table does not apply. The bytes already told you to stop.

Failure class	Fixture hash	Verdict action	What the required lane does
timeout	unchanged	retry	one rerun, same job name
assertion	unchanged	stop	stay red, no rerun
missing fixture	changed or missing	stop	stay red, no rerun
network	unchanged	quarantine	rerun only if a human label already allows it
unknown	any	stop	stay red

retry is not a merge. It is permission to schedule the same required job again, against the same manifest. The second green is still just that job. Your branch protection rule should name it, and it should not name the advisory job.

Quarantine is the narrow middle. Use it for failures you have already labeled as environmental, in a file a human reviews. Do not let the advisory job add labels to that file. Otherwise the model can grant itself the next retry.

Cap the retry at one. A second red is a stop, even if the model would say retry again. Record the attempt id in the job output so a later reader can see the budget was spent.

5. Re-hash immediately before the rerun

Download the original manifest into the new run. Recompute hashes before the test command. If they differ, exit 3 and skip the suite.

# Proposal: first step of the rerun job.
python3 scripts/check_verdict.py "$VERDICT_PATH" "$MANIFEST_PATH"
status=$?
if [ "$status" -ne 0 ]; then
  echo "rerun refused"
  exit "$status"
fi

Do not fix up a mismatch in this step. A mismatch is the finding. Fix the fixture in a normal commit, with review, and let the required job hash the new bytes on a clean run.

6. Leave the advisory name out of branch protection

Name the required check something stable, such as tests (manifest). Name the advisory job retry-verdict (advisory). Protect the branch with the first name only.

You will still see advisory failures. That is fine. They are not the merge signal. If you add the advisory name to the required list, you have glued the suggestion lane back onto the gate this workflow just removed.

Also skip the rerun when the verdict file is missing. Absence is not retry. A down advisory host should leave the original red check in place, not invent a pass.

Inspect a refused rerun

Three commands are enough. You are looking for which gate fired, not for a narrative from the model.

# Proposal commands. Replace the artifact path with the run you are debugging.
sha256sum -c .ci/fixture-manifest.txt
python3 scripts/check_verdict.py verdict.json .ci/fixture-manifest.txt; echo "exit=$?"
git status --short

sha256sum -c failing means the workspace is not the failed commit's fixture set. Exit 2 from the checker means the verdict itself is the problem. A dirty git status means something wrote the tree after checkout. Stop there. Do not rerun tests until the tree matches the manifest.

If all three are clean and the action is stop, read the test log. The workflow has done its job. The bug is in the change, not in the gate.

Where to hook this

Put the manifest step in pre-test, not in post-test. A post-test hash tells you the suite already ran against whatever the workspace had become.

If you use a local hook runner, call the same script from a pre-push hook only as a convenience. The server-side required job is the one that counts. A laptop hook can be skipped. Branch protection cannot, unless an admin bypasses it, which you should treat as an incident rather than a path.

Keep model credentials in the advisory job's secret store. Do not copy them into the required job to make debugging easier. That convenience recreates the write path you just removed.

Limitations

This catches byte-level fixture edits. It does not catch a test that reads an unlisted file, a clock, or a live service. Those need their own pins.

The verdict can be wrong and still be schema-valid. retry on a real assertion bug wastes a run. It does not, by itself, hide the bug, provided the rerun uses the same assertions.

Free model access and a free server option are availability claims, not a capacity plan. If the advisory host is down, the required job must still fail closed without it. Do not block merges on the advisory job, and do not let a missing verdict become an implicit retry.

The sample code is a proposal. It is not a benchmark, and it is not a result from this account's repository. Paths, action versions, and permission keys will need to match your forge before you trust them.

Who should skip this

Skip it if you have no required status check. The manifest then has nothing to protect.

Skip it if fixtures are generated inside the test job on purpose. Pin the generator and its inputs instead of hashing outputs that are supposed to change.

Skip it if you want a model to open the fixing pull request automatically. That is a different risk, and this path refuses it on purpose.

If your required check already cannot write fixtures, the remaining step is small. Put the summarizer on a read-only job, and keep its token off the merge runner. MonkeyCode's free model access and free server option are enough for that advisory role when the suggestion does not need to deploy, commit, or rename the check.

Keep the Draft Lane From Applying Its Own Patch

Emery Lin — Sat, 10 Oct 2026 18:34:08 +0000

A remote green is not merge authority. If a free runner can both write the change and certify it, the required check is theater. Freeze the proposal, apply it on a trusted job, and fail closed when the receipt is missing, cancelled, or quota-stopped.

That is the gate. The rest of this piece is how you wire it so a noisy draft lane cannot bless itself.

The failure you are actually blocking

The badge goes green and the log is long, so the pull request looks finished. You merge. Later the fixture no longer matches the assertion, because the same job edited both.

A second failure mode is quieter. The inner test process exits non-zero, the wrapper still exits zero, and branch protection records success. Cancellation and quota stops get laundered the same way when a skip is treated as a pass.

Generated workflow files make this easier to miss. The diff is large, the job name sounds official, and the first green check arrives before anyone reads the runner label. You do not fix that with a longer prompt. You fix it by splitting lanes.

Where a free draft lane fits

Use the free lane to draft. Do not use it as the merge gate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project with free model access and a free server option. Those two availability claims are the only product facts used here.

This article does not state a token quota, a hardware size, a time limit, a benchmark, or a promise that the free option stays fixed. Read the current project docs before you depend on either lane.

Point free model access at a patch draft. Point the free server option at a non-required smoke job. You get a cheap place to explore a change. The required job still has to live on a runner you control, with a name you pin in branch protection.

Decide the lanes before you write YAML

Put the rule in a table, then make the hook enforce the table. A comment in the workflow will not stop a later edit.

Lane	May write	May execute fixtures	May satisfy the required check
Draft: free model plus free server	`proposals/<id>/` only	short smoke, non-required	no
Trusted CI runner	apply the frozen diff	full fixture set, once	yes, only when the apply receipt matches
Laptop pre-push hook	nothing, unless you are the author	same fixture digest, local only	no; it only blocks your push

If one job can edit workflow files and also set the required check name, stop. You have collapsed the table into a single lane, and a single lane cannot police itself.

Receipt contract

The draft lane writes proposals/<id>/receipt.json next to change.diff. The trusted job writes artifacts/apply-receipt.json after it applies that diff and reruns fixtures. Your merge hook compares the overlapping fields. It does not compare log tone.

Require all of these:

proposal_digest: sha256: plus the digest of change.diff
fixture_digest: digest of the fixture tree recorded before apply
runner_class: draft on the proposal, trusted on the apply artifact
exit_class: passed, failed, cancelled, or quota_stopped
inner_exit: exit code of the test process, not the wrapper
draft_attempts: integer, used only to stop noisy draft reruns

A missing field is a failure. An absent file is a failure. Do not default either one to pass.

Do not skip the trusted fixture suite just because the draft receipt says passed. That field means the draft lane thinks it finished. It is not merge evidence.

Numbered path from draft to merge

Freeze fixtures first. Compute fixture_digest in the workflow environment, and do not let the draft patch rewrite that value.
Restrict the draft lane to proposals/<id>/. If the diff touches any other path, the guard exits non-zero and you do not apply.
Run smoke tests on the free server job. Leave that job off branch protection. continue-on-error is acceptable only because this job is not the merge gate.
On the trusted runner, recompute the patch digest. Refuse apply when it differs from proposal_digest.
Confirm the worktree fixture digest still equals the frozen value. Then apply the diff and run the fixture suite once.
Write the apply receipt with runner_class set to trusted. Set exit_class to passed only when inner_exit is 0.
Name the required check after that trusted job. Keep the draft job name off the protection rule, even if the draft name looks cleaner in the UI.
If exit_class is cancelled or quota_stopped, leave the proposal unapplied. Do not convert that status into a skipped success.

Guard script

The following script is an unexecuted example. Review it before you add it to a repository. It is not a measured benchmark, and it was not run against a live server for this article.

#!/usr/bin/env python3
"""Refuse a draft proposal that is incomplete or out of bounds."""
import hashlib
import json
import sys
from pathlib import Path

REQUIRED = (
    "proposal_digest",
    "fixture_digest",
    "runner_class",
    "exit_class",
    "inner_exit",
    "draft_attempts",
)
BLOCKED = {"cancelled", "quota_stopped", "failed"}

def added_paths(text):
    paths = []
    for line in text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:]
            if path != "/dev/null":
                paths.append(path)
    return paths

def main():
    proposal = Path(sys.argv[1])
    receipt = json.loads((proposal / "receipt.json").read_text())
    patch = (proposal / "change.diff").read_bytes()
    missing = [key for key in REQUIRED if key not in receipt]
    if missing:
        print("missing fields: " + ", ".join(missing), file=sys.stderr)
        return 2
    if receipt["runner_class"] != "draft":
        print("runner_class must be draft", file=sys.stderr)
        return 2
    if receipt["exit_class"] in BLOCKED:
        print("refuse apply: " + receipt["exit_class"], file=sys.stderr)
        return 3
    if int(receipt["draft_attempts"]) > 2:
        print("draft attempts exceeded; stop and review", file=sys.stderr)
        return 3
    digest = "sha256:" + hashlib.sha256(patch).hexdigest()
    if receipt["proposal_digest"] != digest:
        print("proposal_digest does not match change.diff", file=sys.stderr)
        return 4
    bad = [p for p in added_paths(patch.decode()) if not p.startswith("proposals/")]
    if bad:
        print("patch leaves proposals/: " + ", ".join(bad), file=sys.stderr)
        return 5
    print("draft proposal is eligible for trusted apply")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Call the guard from the draft job, then again at the start of the trusted job. These commands are part of the same unexecuted example.

python3 scripts/check_proposal.py "proposals/${PROPOSAL_ID}"
sha256sum "proposals/${PROPOSAL_ID}/change.diff"

After trusted apply, assert the artifact. set -euo pipefail matters here. A missing file must fail the job, not print a warning and continue.

set -euo pipefail
test "$(jq -r .runner_class artifacts/apply-receipt.json)" = "trusted"
test "$(jq -r .exit_class artifacts/apply-receipt.json)" = "passed"
test "$(jq -r .inner_exit artifacts/apply-receipt.json)" = "0"
test "$(jq -r .fixture_digest artifacts/apply-receipt.json)" = "${FIXTURE_DIGEST}"

Wire the jobs so the required name is the trusted job. if: always() is deliberate. You want the trusted job to run and fail when the draft receipt is blocked. A skipped required check is how a cancelled free-server run sneaks toward merge.

# Unexecuted example. Job names are illustrative.
jobs:
  draft_on_free_server:
    continue-on-error: true
    steps:
      - run: python3 scripts/check_proposal.py "proposals/${PROPOSAL_ID}"
  trusted_apply:
    needs: draft_on_free_server
    if: always()
    steps:
      - run: python3 scripts/trusted_apply.py "proposals/${PROPOSAL_ID}"
      - run: bash scripts/assert_apply_receipt.sh

Do not gate trusted_apply on needs.draft_on_free_server.result == 'success' alone. A wrapper can report success after a quota stop. The receipt check, plus a fresh fixture run, is the control. The draft result is only a hint.

Flake control that cannot borrow a draft green

You may rerun the draft lane when smoke is noisy. Record the count in draft_attempts. Above 2, stop and ask a person to look at the proposal.

Those reruns never satisfy the required check. The trusted job runs the fixture suite once per apply receipt. If you need another trusted run, write a new receipt with a new digest comparison. Do not reuse an older green artifact from the draft lane.

This is narrower than a repository-wide flake budget. You are blocking one specific wash: a red inner exit that later becomes a zero from a wrapper on a free server.

Debug a merge that should not have happened

When a pull request lands and the fixture looks wrong, reconstruct the lane before you rerun anything.

Read the required check name on the merge commit. If it equals the free-server job, the gate was never split.
Download artifacts/apply-receipt.json. If the artifact is absent, the job skipped or never started. Treat that as a failed gate, not as an empty pass.
Hash the diff that landed and compare it with proposal_digest. A mismatch means apply did not use the frozen proposal.
Recompute fixture_digest at the parent commit. Drift means something moved the fixture between freeze and apply.
Read inner_exit and any wrapper status separately. Keep inner_exit. Discard the wrapper.

Use a stable file order when you recompute the fixture digest. Directory iteration order is not a contract.

# Unexecuted illustration of a stable fixture digest.
find fixtures -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum

If step 1 already shows the draft job as the required check, stop digging through model logs. Fix the protection rule first. The rest of the receipt work will not help while the wrong job name can still go green.

Limitations

Matching digests does not prove the patch is good. It proves the trusted job applied the same bytes, against the fixture digest you froze, and recorded the inner exit you checked.

A compromised trusted runner can forge the apply receipt. Treat that runner like a release credential. Do not place it on the same account or the same machine as the free server lane.

The +++ b/ scan misses some rename and copy headers. Extend the parser before you treat this guard as a security boundary. Schema-check fixture files after apply as well. A draft model can still invent a field inside a path you allowed.

Free-server availability is not something this article can promise. Queue loss, image drift, and quota stops belong on the exit_class list. None of them is a pass, and none of them should delete the proposal you still need to inspect.

Who should not use this flow

Do not use it if you have only one runner and you plan to label it both draft and trusted. The split is the control. One machine wearing both labels removes the control.

Do not auto-apply patches that edit workflow files, branch protection, or signing configuration. Those edits need a human review path with a different required check.

Do not treat a proposal receipt as a release signature. It is a CI comparison. It is not a key ceremony, and it does not replace code review.

Try the split on one non-production branch

If a free model lane and a free server option are already available to you, aim them at proposals/ only. Confirm current access terms in the project docs, add the receipt asserts, and only then consider naming trusted_apply as the required check.

Pin the Merge Check Name, Then Keep the Noisy Lane Off It

Emery Lin — Fri, 09 Oct 2026 15:01:05 +0000

A merge rule should wait on one check name you pinned before the pull request opened. Other jobs may hunt noise, draft notes, or retry a shaky test. They do not get to publish that pinned name, rewrite the fixture behind it, or cancel the run the rule is watching.

That split is the whole gate. Rename the check, and the badge is a new label. Keep the name, and you still have to prove the job is read-only and bound to a fixture hash.

Why the badge string is the control

Required status checks match text. GitHub does not match your intention, and it does not match the story in the pull request body. Actions usually shows a check as a workflow name plus a job name, but you should copy the live string from a dry run into the ruleset instead of guessing. Confirm the current behavior in GitHub's workflow syntax docs and status-check docs, then trust the string your repository actually emits.

An agent that can edit .github/workflows can publish a friendly green under a new job name. If your ruleset still lists the old name, the new badge is decoration. If someone updates the ruleset to follow the new name without reviewing the job, the decoration becomes the gate. You close that gap by pinning the name in a policy file and failing the pull request when the workflow and the policy disagree.

This is not a claim about any model family. It is a claim about strings, permissions, and hashes. Coding agents make the gap easier to hit, because a generated workflow edit and a generated test can land in the same pull request. The ruleset still only sees names.

Set the policy before the workflow

Write the contract first. The table is the decision. The files only encode it.

Decision	Pinned merge check	Noisy lane
Check name	Exact string copied from a dry run	Any other name
Fixture directory	Read a hashed bundle	Scratch only, never uploaded as the bundle
`cancel-in-progress`	`false`	`true` is allowed
Token scope	`contents: read`	No write scope either
Model note	Ignored	Advisory artifact only
Listed in the ruleset	Yes	No

If the noisy lane must be green before merge, it is not a noisy lane. Move that test into the pinned check, or delete it. Do not keep a second required context "temporarily." Temporary required checks are how old names survive.

1. Store the pinned name outside the job

Create .github/merge-policy.json. Review it like production code. A pull request that changes it needs an owner review, even when the rest of the diff is a one-line test.

{
  "required_context": "gate/merge-fixture",
  "required_job_id": "merge-fixture",
  "quarantine_job_id": "flake-explore",
  "quarantine_context_prefix": "explore/flake-",
  "fixture_bundle": "fixtures/locked"
}

The context value above is an example string, not a name GitHub will invent for you. After a dry run on a non-default branch, compare it with the check name you actually see. If the live name includes the workflow title, put that full string in the ruleset, and keep the job name: field aligned with the portion you control.

2. Give the two jobs different lifetimes

This workflow is a proposal. It has not been run for this article. Pin action versions your organization has already approved before you adopt it.

name: merge-gate
on:
  pull_request:
permissions:
  contents: read
jobs:
  merge-fixture:
    name: gate/merge-fixture
    runs-on: ubuntu-latest
    permissions:
      contents: read
    concurrency:
      group: merge-fixture-${{ github.event.pull_request.number }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4
      - name: Check policy, permissions, and fixture hash
        run: python3 scripts/check_merge_gate.py --policy .github/merge-policy.json
      - name: Run tests bound to the locked bundle
        run: python3 -m unittest discover -s tests/locked -v
  flake-explore:
    name: explore/flake-note
    runs-on: ubuntu-latest
    permissions:
      contents: read
    concurrency:
      group: flake-note-${{ github.event.pull_request.number }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - name: Rerun the noisy suite without failing the pin
        run: python3 -m unittest discover -s tests/noisy -v
        continue-on-error: true
      - name: Write an advisory note
        if: always()
        run: python3 scripts/flake_note.py --log noisy.log --out flake-note.md

The pinned job cannot be cancelled by a newer push on that pull request. The noisy job can. That difference is intentional. A follow-up commit may abandon an exploration. It must not abandon the run your ruleset is waiting for.

Both jobs stay read-only. A required job that can push is a job that can repair the fixture it is about to trust. Keep contents: write off both lanes even if a later step "only" wants to commit a regenerated cassette.

3. Fail the pull request when the lanes collapse

Save the checker as scripts/check_merge_gate.py. It is a narrow proposal, not a full Actions language parser. It extracts two job blocks by indentation, then refuses the collisions that matter.

#!/usr/bin/env python3
"""Proposal checker. This article does not report an execution result."""

import argparse
import hashlib
import json
import pathlib
import sys

def sha256_tree(root: pathlib.Path) -> str:
    files = sorted(p for p in root.rglob("*") if p.is_file())
    if not files:
        sys.exit(f"empty fixture bundle: {root}")
    digest = hashlib.sha256()
    for path in files:
        rel = path.relative_to(root).as_posix().encode()
        digest.update(rel)
        digest.update(b"\0")
        digest.update(path.read_bytes())
        digest.update(b"\0")
    return digest.hexdigest()

def job_block(workflow: str, job_id: str) -> str:
    lines = workflow.splitlines()
    start = None
    for index, line in enumerate(lines):
        if line == f"  {job_id}:":
            start = index
            break
    if start is None:
        sys.exit(f"missing job id: {job_id}")
    block = [lines[start]]
    for line in lines[start + 1 :]:
        if line.startswith("  ") and not line.startswith("   ") and line.endswith(":"):
            break
        block.append(line)
    return "\n".join(block)

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--policy", required=True)
    parser.add_argument("--workflow", default=".github/workflows/merge-gate.yml")
    args = parser.parse_args()
    policy = json.loads(pathlib.Path(args.policy).read_text(encoding="utf-8"))
    workflow_path = pathlib.Path(args.workflow)
    workflow = workflow_path.read_text(encoding="utf-8")
    required = policy["required_context"]
    prefix = policy["quarantine_context_prefix"]
    if required.startswith(prefix):
        sys.exit("required context uses the quarantine prefix")
    required_job = job_block(workflow, policy["required_job_id"])
    quarantine_job = job_block(workflow, policy["quarantine_job_id"])
    if required_job.count(f"name: {required}") != 1:
        sys.exit("pinned job must publish the required context once")
    if required in quarantine_job:
        sys.exit("noisy lane reuses the pinned context")
    if "cancel-in-progress: false" not in required_job:
        sys.exit("pinned job must refuse cancel-in-progress")
    if "contents: read" not in required_job:
        sys.exit("pinned job must declare contents: read")
    if "contents: write" in required_job:
        sys.exit("pinned job must not request contents: write")
    digest = sha256_tree(pathlib.Path(policy["fixture_bundle"]))
    receipt = {
        "required_context": required,
        "fixture_sha256": digest,
        "workflow_file": workflow_path.as_posix(),
        "advisory_note": "not-a-status",
    }
    pathlib.Path("merge-receipt.json").write_text(
        json.dumps(receipt, indent=2) + "\n", encoding="utf-8"
    )
    print(digest)

if __name__ == "__main__":
    main()

Run it before you enable the ruleset entry:

mkdir -p fixtures/locked tests/locked tests/noisy
printf 'id: sample\n' > fixtures/locked/case.json
python3 scripts/check_merge_gate.py --policy .github/merge-policy.json
git status --short

You want a receipt whose fixture_sha256 matches the locked tree, and a worktree diff you can explain. If the checker rewrites merge-receipt.json on every run, commit that file only when the hash changed for a reason you reviewed. An unexpected hash change means the bundle moved. Do not shrug that into a green badge.

The indentation parser will miss a job written with a different layout. That is a reason to keep the workflow boring, not a reason to trust an unparsed file. If your organization already calls a reusable workflow from a locked repository, prefer that pin, and use this script only as a local tripwire.

4. Put model text on the noisy lane only

The advisory writer does not decide merge. It copies a tail of the noisy log so a reviewer can see why the exploration failed. The sample does not call a network.

#!/usr/bin/env python3
"""Advisory note writer. A failing suite must not change this exit code."""

import argparse
import pathlib

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--log", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    source = pathlib.Path(args.log)
    text = source.read_text(encoding="utf-8", errors="replace")[-4000:] if source.exists() else ""
    pathlib.Path(args.out).write_text(
        "Advisory only. Do not add this file's job to the ruleset.\n\n" + text,
        encoding="utf-8",
    )

if __name__ == "__main__":
    main()

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a shorter note than a raw log tail, MonkeyCode's free model access can sit behind this writer on the noisy lane. Keep that call out of merge-fixture. The free server option matters for the same boundary: run the draft there so a slow summary cannot occupy, cancel, or impersonate the pinned runner. This article does not name a model, a token quota, a machine size, or a period of free availability. Those details change, and they are not the gate. Read the current product documentation before you treat either option as present.

One use is enough. Try the free model access and the free server option for that advisory note if you want the draft off the required job, then leave the note out of the ruleset.

5. Turn the pin on in this order

Do not enable auto-merge in the same change that introduces the workflow.

Land the policy, the checker, and the workflow with the ruleset still pointing at your existing required checks.
Open a dry-run pull request and copy the live check name from the checks list. gh api repos/OWNER/REPO/commits/SHA/check-runs --jq '.check_runs[].name' is the command shape. Substitute your owner, repository, and commit.
If the live name differs from required_context, fix the policy in a reviewed change. Do not pick whichever badge is already green in the ruleset UI.
Add only that verified string as a required check. Do not add explore/flake-note.
On the next pull request, compare merge-receipt.json with a fresh hash of fixtures/locked/ at the merge commit.

A reviewer still reads the diff. The receipt is a breadcrumb, not an approval. If step 2 shows two names that both look official, stop and delete the extra job before you touch branch protection.

Who should skip this

Skip the pattern when you have no required checks at all. A local script cannot invent that platform feature. Skip it when one unreviewed credential can both edit workflows and approve merges. The pin is theater in that setup. Skip it when you need a model verdict to be the merge authority. This design refuses that role on purpose.

Also skip the in-repo tripwire if a locked reusable workflow already publishes the required check from a ref the pull request cannot move. Adding a second, weaker parser will only drift. Do not use the noisy lane to hide a product bug you should fix. Isolation is for tests you have already decided are non-blocking. It is not a pardon.

Limits of the artifact

The checker does not evaluate GitHub expressions. A job that concatenates the context name at runtime can slip past a string search. The receipt hash covers file bytes under fixtures/locked/, not the Actions cache, not container images, and not the runner image. continue-on-error: true on the noisy suite means a red exploration can still look quiet in the pull request conversation. That quiet is acceptable only because the job is absent from the ruleset.

Free model access and a free server option are availability claims supplied for this draft. They are not a capacity plan, an SLA, or a measured speedup. No pass rate is reported here because none was measured.

Pin the name you verified. Hash the bundle the pinned job reads. Let every other lane stay optional, including a model note written somewhere else.

Do Not Let the Required Job Write the Fixture It Trusts

Emery Lin — Thu, 08 Oct 2026 10:34:48 +0000

The required job should fail if it can still write a fixture. A green check from a job that shares that file with a draft run is not a merge signal. You want the merge job read-only on owned fixtures, and you want every model write confined to a scratch tree. Branch protection waits on the read-only job.

You do not need a headline to justify the split. If an agent can open a pull request, the job that unlocks merge needs a permission boundary. Color on the draft job is optional noise.

What a retry is hiding

You see this on a persistent runner, or in a test that repairs a missing JSON file. The draft rewrites fixtures/owned/paid.json until the new assertion matches. Pytest goes green. The next pull request still has the old bytes, so it fails in a way that looks like order.

Someone adds a retry. The retry passes because a later test put the file back. You did not fix a flake. You hid a shared writable fixture behind a second attempt.

Do not spend retries on that failure. Remove the write permission, and the second run stops being the control. The first run was never allowed to edit the file it claimed to trust.

Two trees

Keep trusted bytes under fixtures/owned/. Keep agent output under scratch/agent/. The merge job may read the first tree and must not write it. The draft job may write only the second tree.

Tests you intend to merge live in tests/merge/ and reference owned paths only. This is a directory contract plus a permission bit. You can rehearse it in a temp folder. Nothing in the gate calls a model.

Step 1: Abort the merge session if a fixture is writable

Drop this file beside the merge tests. It is a proposal until you run it. On pytest_sessionstart it refuses a missing catalog, an empty catalog, a writable path, or a scratch reference in a test module.

# tests/merge/conftest.py
import os
from pathlib import Path

OWNED = Path("fixtures/owned")

def pytest_sessionstart(session):
    if not OWNED.is_dir():
        raise SystemExit("missing fixtures/owned")
    files = [path for path in OWNED.rglob("*") if path.is_file()]
    if not files:
        raise SystemExit("fixtures/owned has no files")
    writable = [str(path) for path in files if os.access(path, os.W_OK)]
    if os.access(OWNED, os.W_OK):
        writable.append(str(OWNED))
    if writable:
        raise SystemExit("owned fixtures are writable:\n" + "\n".join(writable))
    leaked = []
    for test_file in Path("tests/merge").rglob("test_*.py"):
        text = test_file.read_text(encoding="utf-8")
        if "scratch/agent" in text or "scratch.agent" in text:
            leaked.append(str(test_file))
    if leaked:
        raise SystemExit("merge suite references scratch:\n" + "\n".join(leaked))

Scan test_*.py only. conftest.py has to name the forbidden path in order to reject it, so including that file makes the probe fail itself. If the probe can write, the suite stops. You do not get a skip, and you do not get a green collection.

Step 2: Clear the write bit in the required job

Git will not store this mode reliably. The executable bit is the permission Git tends to keep. The read-only bit is the job's responsibility, every run. Apply it after checkout and before pytest.

#!/bin/sh
# scripts/merge_suite.sh
set -eu
test -d fixtures/owned
find fixtures/owned -type f -exec chmod a-w {} +
find fixtures/owned -type d -exec chmod a-w {} +
find fixtures/owned -type d -exec chmod a+x {} +
python3 -m pytest tests/merge -q --maxfail=1

Do not point find at scratch/agent/. The draft job needs that tree writable. One combined root, and the split is gone.

A root checkout can change the bit back and then pass a probe you ran too early. Drop privileges before the probe, or mount fixtures/owned read-only. If you can do neither, do not mark the job required. A green root shell is not a lock.

Step 3: Leave the draft job off the required list

Sketch the two jobs in the CI system you already run. Pin actions to SHAs your org audits. The image name below is a placeholder, not a version claim.

jobs:
  agent-draft:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<pinned-sha>
      - run: |
          mkdir -p scratch/agent
          python3 scripts/draft_test.py --out scratch/agent/proposed_test.py
  merge-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<pinned-sha>
      - run: sh scripts/merge_suite.sh

Require merge-gate only. A red draft must not block merge. A green draft must not satisfy it. Put that sentence in the pull request template so the extra bubble stops looking like evidence.

draft_test.py can be a stub that writes a proposal under scratch. The merge suite does not import it. If you want the proposal kept, move a reviewed copy into tests/merge/ and add the fixture under fixtures/owned/ in a commit that does not also weaken the probe.

Step 4: Catch the scratch import before the pull request

Match the probe locally with a pre-push hook. It does not compute coverage, and it does not retry a failed job. It only blocks one path leak.

#!/bin/sh
set -eu
if git grep -n -E 'scratch/agent|scratch\.agent' -- tests/merge ':!tests/merge/conftest.py'; then
  echo "tests/merge must not reference scratch/agent"
  exit 1
fi

Use the hook only if the repo already runs one. A hook nobody installs is a note. The session probe remains the gate that branch protection can see.

Step 5: Rehearse a pass, a write failure, and a leak

Run this in a temp directory on a POSIX shell. Copy the conftest.py from step 1 into tests/merge/ first. You want three different endings. If the writable case still exits 0, the probe is not on pytest_sessionstart.

mkdir -p fixtures/owned scratch/agent tests/merge
printf '%s\n' '{"id":"paid-1","status":"paid"}' > fixtures/owned/paid.json
printf '%s\n' \
  'def test_paid_fixture_is_readable():' \
  '    text = open("fixtures/owned/paid.json", encoding="utf-8").read()' \
  '    assert text.startswith("{")' \
  > tests/merge/test_paid.py
chmod a-w fixtures/owned/paid.json
python3 -m pytest tests/merge -q

That run should pass if the file is read-only and the test only reads. Then force the failure you care about.

chmod u+w fixtures/owned/paid.json
python3 -m pytest tests/merge -q

Expect a non-zero exit and the line owned fixtures are writable. Restore the bit, then plant a leak.

chmod a-w fixtures/owned/paid.json
printf '%s\n' 'PATH = "scratch/agent/proposed_test.py"' > tests/merge/test_leak.py
python3 -m pytest tests/merge -q

The third run must name test_leak.py and stop. Delete that file when you are done. Do not commit a mode bit and call it the control. Commit the probe, and let CI apply chmod.

Where a model may run

Only agent-draft may call a model. The probe, the mode bits, and the merge pytest run stay offline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If your account includes MonkeyCode's free model access and free server option, bind both to the draft job and to nothing required. Check the current terms in the account before you depend on them. This article does not state a quota, a model name, a machine size, or a duration. When that free server is down, skip the draft job. Still run merge-gate.

A required check that needs a live model will go red for reasons outside the diff. Keep that call off the merge path. The draft job is the bench. It is not the signal.

How to read a noisy pull request

What you see	Merge?	Why
Draft green, merge job missing	No	The green job was allowed to write
Merge green, probe not started	No	You never proved the write bit was off
Probe lists a writable owned file	No	The suite could edit the answer
Merge test mentions `scratch/agent`	No	The two jobs share a tree again
Probe clean and merge pytest green	Only after review	The required job could not write fixtures

Review stays in the last row on purpose. The probe understands permission and path. It does not understand whether the JSON is the right scenario.

Limitations

os.access follows the current user. If that user is root, a cleared write bit is advisory, because root can set the bit again. The probe is a tripwire for normal CI users, not a sandbox against a hostile root step in the same job.

The scratch check is a literal substring. A test that concatenates path fragments will pass the scan. Review generated tests. Do not treat git grep as a type checker.

A read-only file can still be the wrong file. You can lock a bad payload and merge a matching bad assertion. Read fixture diffs. The gate will not do that for you.

POSIX chmod is part of the contract. On Windows runners, os.access and a-w do not mean the same thing. Use an ACL you have actually tested, or do not copy this script into that job and call it done.

Persistent self-hosted workspaces will fail the probe when a previous job left files writable. Clean the workspace, or stop using a dirty checkout for merge-gate. That failure is the point.

Who should not use this

Skip the split if the merge suite must regenerate fixtures and you have not extracted that step. People will delete the probe to get green, which is worse than no probe.

Skip it on a spike that cannot reach the default branch. The scratch rule is overhead until merge is real.

Skip any hosted model if policy forbids sending repo contents to a third party. A drafting bench does not rewrite that rule. Run the draft locally, or do not run it.

Do not treat a drafting bench as an availability plan, and do not treat the probe as a review replacement. One locks a path. The other can be skipped.

Close

Require the job that cannot write fixtures/owned/. Keep proposals in scratch/agent/. Ignore a green draft. You still read the assertion. You stop calling a shared writable file a passing test.

Assert a Clean Worktree After Tests, Then Require That Job

Emery Lin — Thu, 08 Oct 2026 08:15:31 +0000

A passing exit code is not a clean tree. If the suite writes files and the next job still looks green, you are merging a leftover, not a result. Fail the job when git status --porcelain is not empty after tests, unless every leftover path is on an allowlist you reviewed by hand.

Put that assertion on the merge path. A review comment will not stop an unprotected green check. The workflow that runs the suite should also run the status hook, and that job should be required on the branch you merge to.

The failure you are blocking

Reused workspaces fail quietly. A test drops a recording, rewrites a fixture, or leaves a dotenv file in the tree. The process exits zero. The following run sees those files and skips setup, so the badge stays green for a stale reason.

You do not fix that with a longer retry. You fix it with a post-suite hook that reads git status. The hook does not judge test quality. It only refuses success when the worktree changed outside a short allowlist.

Every snippet below is an unexecuted example. Copy it onto a branch and run it yourself before you protect the branch. Nothing here is a measured production result.

Step 1: Hash fixtures before the suite starts

Record a digest of the fixture directory before tests run. You want that line in the log when a later step reports a dirty tree. A one-line digest is enough. You do not need a database.

#!/usr/bin/env bash
# scripts/ci/hash-fixtures.sh
# Unexecuted example. Confirm the root before you depend on it.
set -euo pipefail

root="${1:-tests/fixtures}"
if [[ ! -d "$root" ]]; then
  echo "fixture root missing: $root" >&2
  exit 2
fi

find "$root" -type f -print0 \
  | sort -z \
  | xargs -0 sha256sum \
  | sha256sum \
  | awk '{print $1}'

Print it before the suite, then keep the root narrow. Hashing the whole repository will move for reasons that are not fixture drift, and the log line stops being useful.

echo "fixture_sha=$(bash scripts/ci/hash-fixtures.sh tests/fixtures)"

Step 2: Check out a wiped workspace

A runner label is not an isolation boundary. You still wipe the checkout, and you do not restore a cache into tracked paths on this job. Shared caches are a separate policy. Leave them out until you have one.

# .github/workflows/clean-tree.yml
# Unexecuted example. Action versions are placeholders, not a run report.
name: clean-tree
on:
  pull_request:
concurrency:
  group: clean-tree-${{ github.ref }}
  cancel-in-progress: true
jobs:
  tests-then-status:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          clean: true
          fetch-depth: 1
      - name: Hash fixtures before the suite
        run: echo "fixture_sha=$(bash scripts/ci/hash-fixtures.sh tests/fixtures)"
      - name: Run the suite
        run: bash scripts/ci/run-suite.sh
      - name: Assert a clean tree
        run: bash scripts/ci/assert-clean-tree.sh ci/clean-allowlist.txt

ubuntu-latest only names an image channel. It does not promise hardware, region, or how long the job may run. If your organization already pins runner images, keep that pin. The concurrency block cancels a superseded push on the same ref so two runs do not tell two workspace stories for one branch.

Step 3: Fail closed on porcelain

The allowlist is for reporter files you will never commit. It is not a pardon for new fixtures. Keep it short, and review every line as if it were a production ignore rule.

# ci/clean-allowlist.txt
coverage/cobertura.xml
test-results/junit.xml

#!/usr/bin/env bash
# scripts/ci/assert-clean-tree.sh
# Unexecuted example.
set -euo pipefail

allowlist="${1:-ci/clean-allowlist.txt}"
tmp="$(mktemp)"
trap 'rm -f "$tmp"' EXIT

git status --porcelain=v1 --untracked-files=all > "$tmp"

if [[ ! -s "$tmp" ]]; then
  echo "clean tree"
  exit 0
fi

fail=0
while IFS= read -r line; do
  path="${line:3}"
  if [[ "$path" == *" -> "* ]]; then
    path="${path##* -> }"
  fi
  if [[ -f "$allowlist" ]] && grep -Fxq -- "$path" "$allowlist"; then
    echo "allowed: $path"
    continue
  fi
  echo "dirty: $line" >&2
  fail=1
done < "$tmp"

if [[ "$fail" -ne 0 ]]; then
  echo "tree dirty after tests; refusing success" >&2
  exit 1
fi
echo "only allowlisted leftovers"

The suite must not be able to rewrite the gate and then pass. Hash the script and the allowlist before tests, and compare after. If the compare fails, the job stops before the status hook can trust a rewritten file.

before="$(sha256sum ci/clean-allowlist.txt scripts/ci/assert-clean-tree.sh | sha256sum | awk '{print $1}')"
bash scripts/ci/run-suite.sh
after="$(sha256sum ci/clean-allowlist.txt scripts/ci/assert-clean-tree.sh | sha256sum | awk '{print $1}')"
test "$before" = "$after"
bash scripts/ci/assert-clean-tree.sh ci/clean-allowlist.txt

That compare is part of this job, not a second system. A generated edit to the allowlist cannot bless itself.

Step 4: Classify leftovers and do not commit them

A dirty line needs a decision you can apply without debate in the workflow file. Do not add a step that commits leftovers back to the pull request. Upload the porcelain log if a human needs it later. That upload is not a commit, and it must not flip the job to success.

Leftover	Likely cause	What you do
Untracked recording or snapshot	The suite stored a new answer	Fail. Review the file in the pull request.
Modified file under `tests/fixtures`	A test mutated shared state	Fail. Restore the file and fix the test.
Dotenv file, token file, or key material	Setup wrote a secret into the tree	Fail. Scrub the log and rotate anything that printed.
Coverage or JUnit path on the allowlist	Reporter output	Keep it gitignored. Upload an artifact if you need it.
Edit to the gate script or the allowlist	The gate moved during the run	Fail. Those paths are not test output.

Step 5: Require the job, including when the gate itself changes

An optional workflow is a note to yourself. Mark tests-then-status required on the protected branch. A faster lint job is not a substitute, because lint does not see a mutated fixture.

Path filters are the easy way to skip this by accident. If you use them, still run the workflow when scripts/ci/** or ci/clean-allowlist.txt changes. A pull request that only edits the hook has to execute the hook. Otherwise the branch rule trusts a check that never ran.

Require the check on the commit you intend to merge. A green run of an older head is a different tree. The protection rule should look at the latest push, not at a stale badge on a previous one.

Retries are the wrong flake control here. A second attempt can pass because the first attempt left files behind. Wipe the checkout, cancel superseded runs, and fail on leftovers. Do not wrap the status step in a retry.

Drafting the hook is not the same as trusting it

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Free model access can draft the shell script and a first allowlist. The free server option can run the example job once if you do not already have a runner. That is the fit. You were not given quotas, model names, hardware, duration, or a promise that the free option stays available, so do not encode any of those in the workflow or in branch protection.

Read the draft before the check becomes required. An allowlist entry of tests/fixtures/ hides the mutation you are trying to see. After one example run, keep branch protection in your own repository.

Who should skip this pattern

Do not use it if the required job is meant to generate files and commit them under its own signed-commit rule. Do not use it if the CI checkout is not a git worktree. Do not use it yet if a reporter writes a new unlisted path on every honest run.

Widen the allowlist only for paths you would gitignore forever. Never set the step to continue on error just to go green. A skipped failure is how a dirty tree becomes the next false pass.

A clean tree does not mean the assertions were right. Clock-dependent tests, order-dependent tests, and tests that only write under /tmp can still be wrong. Keep the suite you already have. This hook sits beside it.

clean: true covers the workspace attached to the checkout. It does not list every cache your organization restores. Until a cache policy exists, do not restore caches into this job.

Prove the script on your machine first

You can verify the example in a temporary repository. The first run should pass. The second should fail because of one untracked file. If both calls exit zero, stop and fix the script before you touch branch protection.

dir="$(mktemp -d)"
cd "$dir"
git init -q
git config user.email "dev@example.com"
git config user.name "dev"
mkdir -p tests/fixtures scripts/ci ci
echo "seed" > tests/fixtures/sample.txt
printf '%s\n' "coverage/cobertura.xml" > ci/clean-allowlist.txt
# Copy assert-clean-tree.sh to scripts/ci/ before this commit.
git add .
git commit -qm "seed"
bash scripts/ci/assert-clean-tree.sh ci/clean-allowlist.txt
echo "leak" > tests/fixtures/leak.txt
bash scripts/ci/assert-clean-tree.sh ci/clean-allowlist.txt || echo "expected failure: $?"

Expect clean tree on the first call. Expect a dirty: line and a non-zero status on the second. If that pair does not behave this way, the allowlist is too wide or you ran a different file.

The local pair is the artifact. The workflow only wraps it. After you require the job, green means the suite exited zero and the worktree was empty enough to merge.

Write-Lock the Replay Before That Green Unlocks Merge

Emery Lin — Thu, 24 Sep 2026 21:26:41 +0000

A replay that can still rewrite fixtures is not a merge signal. It is a write that happened to exit zero.

Freeze the fixture digest first. Then replay the touched tests on a tree that cannot update those files. Unlock merge only when the digest before and after the replay is the same, and the flake window stays inside a budget you chose. The commands below are an unexecuted proposal. Adapt the paths. Do not treat them as a transcript from a live pipeline.

What a single green job hides

You already know a test can pass for the wrong reason. The annoying case is narrower than a flaky network. The job edits a fixture, the assertion matches the new bytes, and the check turns green.

That pattern shows up when an agent drafts tests, and it shows up when a person updates snapshots in a hurry. The exit code does not record which one happened.

Split the permissions. One step may classify the diff and record a digest. A later step may run tests. The later step must not be the one that updates fixtures. If those two steps share a writable worktree, you did not split anything.

Three artifacts, one rule

You only need three files to make the split reviewable.

A path list of fixture roots, maintained by people, not inferred from filenames.
A classify hook that fails closed when those paths change without an explicit label.
A replay script that checks the digest, drops write access as far as the runner allows, reruns the touched tests, and writes a receipt.

The rule is small. No receipt, no merge. A local green, an agent summary, or a log from a side machine does not replace the receipt on the SHA you are merging.

Decide before you add a check

Use a table so the hook stays dumb. Ambiguous diffs fail. They do not get a score.

Diff you see	Classify	Replay	Merge
App code only	Pass, no digest	Skip	Your existing required checks
Tests changed, fixture roots untouched	Pass, list test paths	Rerun those tests	Allow only inside the flake budget
Fixture root changed, no `fixture-update` label	Fail	Do not start	Block
Fixture root changed, label present, digest recorded	Pass	Rerun, then recheck digest	Allow only if digest is unchanged and budget holds
Gate workflow and fixtures in the same PR	Fail closed	Do not start	Split the PR

Read the last row as a review rule. A pull request that edits the gate and the evidence together is not something this receipt can vouch for.

Step 1: Name the roots

Put the roots in a file the hook can read with no extra parser.

# ci/fixture-roots.txt
tests/fixtures/
tests/http_fixtures/
src/components/__snapshots__/

You own every line. If a team stores fixtures beside source under another folder name, add that folder. Do not let a comment in the diff redefine the list.

Keep globs out until the hook has a real matcher. A prefix is easier to explain in review, and easier to fail closed when a path surprises you.

Step 2: Fail closed on unlabeled fixture diffs

The hook does not decide whether the new fixture is true. It decides whether a human allowed the write, then it records a digest.

#!/usr/bin/env bash
# proposal, unexecuted: ci/hooks/check-fixture-label.sh
set -euo pipefail

roots_file="${1:-ci/fixture-roots.txt}"
label="${FIXTURE_UPDATE_LABEL:-}"
base="${BASE_SHA:-origin/main}"

mapfile -t roots < <(grep -Ev '^\s*(#|$)' "$roots_file")
changed=$(git diff --name-only "${base}...HEAD")

hits=()
while IFS= read -r path; do
  [ -z "$path" ] && continue
  for root in "${roots[@]}"; do
    case "$path" in
      "$root"*) hits+=("$path") ;;
    esac
  done
done <<< "$changed"

mkdir -p ci/receipts

if [ "${#hits[@]}" -eq 0 ]; then
  echo "classify=no-fixture" | tee ci/receipts/classify.txt
  exit 0
fi

if [ "$label" != "fixture-update" ]; then
  printf 'unlabeled fixture path:\n' | tee ci/receipts/classify.txt
  printf '%s\n' "${hits[@]}" | tee -a ci/receipts/classify.txt
  exit 1
fi

# sha256sum splits on spaces; keep fixture paths space-free or switch to a NUL-safe digest
sha256sum "${hits[@]}" | sort > ci/receipts/fixture-digest.txt
echo "classify=frozen count=${#hits[@]}" | tee ci/receipts/classify.txt

Try the failure path before you try the success path:

FIXTURE_UPDATE_LABEL="" BASE_SHA=origin/main bash ci/hooks/check-fixture-label.sh

You want a non-zero exit when a listed root appears in the diff. Then repeat with FIXTURE_UPDATE_LABEL=fixture-update and confirm ci/receipts/fixture-digest.txt lists only those paths. If the digest lists a source file, your roots are too wide. Fix the list before you write the replay.

Step 3: Replay without a writable fixture tree

chmod a-w is not a boundary. The same user can turn the bit back on and write the file. Treat a mode change as a tripwire only.

Prefer a stronger lock when the runner allows it. Copy the fixture tree, bind-mount it read-only over the working paths, and run the tests as a user who cannot mount or change mode. If you cannot get a read-only mount, say so in the receipt and do not pretend the lock held.

#!/usr/bin/env bash
# proposal, unexecuted: ci/replay-locked.sh
set -euo pipefail

digest_file="${1:-ci/receipts/fixture-digest.txt}"
test_cmd="${2:-pytest -q tests/test_order_replay.py}"
budget="${FLAKE_BUDGET:-3}"
lock_mode="${LOCK_MODE:-unset}"

[ -f "$digest_file" ] || { echo "missing digest; classify did not freeze fixtures"; exit 1; }

sha256sum -c "$digest_file"
before_ok=$?
if [ "$before_ok" -ne 0 ]; then
  echo "digest mismatch before replay"
  exit 1
fi

if [ "$lock_mode" != "ro-mount" ]; then
  echo "lock_mode=${lock_mode}; refusing to call this a write-lock" | tee ci/receipts/replay.txt
  exit 1
fi

fails=0
i=1
while [ "$i" -le "$budget" ]; do
  if ! bash -lc "$test_cmd"; then
    fails=$((fails + 1))
  fi
  i=$((i + 1))
done

if ! sha256sum -c "$digest_file"; then
  echo "digest moved during replay" | tee ci/receipts/replay.txt
  exit 1
fi

if [ "$fails" -ne 0 ]; then
  echo "flake window ${fails}/${budget}" | tee ci/receipts/replay.txt
  exit 1
fi

printf 'replay_ok fails=%s budget=%s lock=%s\n' "$fails" "$budget" "$lock_mode" \
  | tee ci/receipts/replay.txt

The script exits 1 unless you deliberately pass LOCK_MODE=ro-mount after you have actually mounted the tree read-only. That default is the point. A green receipt must not appear just because the variable was forgotten.

A budget of three is a filter for tests that fail every other run. It will not catch a rare flake. Write that limit into the job summary so the number is not mistaken for a confidence interval.

Paths with spaces will break sha256sum field splitting. Ban spaces in fixture paths, or replace the digest with a NUL-safe tool you already trust. Do not discover that in the merge queue.

Step 4: Require both receipts on the same SHA

Wire classify and replay as separate jobs. The merge job only checks files. It does not rerun tests, and it does not regenerate fixtures.

# proposal, unexecuted — pin the action SHAs your org already reviews
name: fixture-replay
on:
  pull_request:
jobs:
  classify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout
        with:
          fetch-depth: 0
      - id: lab
        run: |
          val=""
          while read -r name; do
            if [ "$name" = "fixture-update" ]; then val="fixture-update"; fi
          done < <(jq -r '.pull_request.labels[]?.name' "$GITHUB_EVENT_PATH")
          echo "value=${val}" >> "$GITHUB_OUTPUT"
      - env:
          FIXTURE_UPDATE_LABEL: ${{ steps.lab.outputs.value }}
        run: bash ci/hooks/check-fixture-label.sh
      - uses: actions/upload-artifact
        with:
          name: fixture-receipts
          path: ci/receipts/

  replay:
    needs: classify
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout
      - uses: actions/download-artifact
        with:
          name: fixture-receipts
          path: ci/receipts/
      - name: Mount fixtures read-only before this step
        run: |
          echo "Replace this step with the ro mount your runner actually supports."
          echo "Do not export LOCK_MODE=ro-mount until that mount is in place."
          exit 1
      - run: LOCK_MODE=ro-mount FLAKE_BUDGET=3 bash ci/replay-locked.sh
      - uses: actions/upload-artifact
        with:
          name: replay-receipt
          path: ci/receipts/replay.txt

  merge-receipt:
    needs: replay
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact
        with:
          name: replay-receipt
          path: ci/receipts/
      - run: |
          grep -q '^replay_ok ' ci/receipts/replay.txt
          grep -q 'lock=ro-mount' ci/receipts/replay.txt

Two details are easy to get wrong. merge-receipt must download the replay artifact. A job that only tests a path it never fetched will fail, or worse, pass if a stale file was baked into the image. The action names above are placeholders. Pin the SHAs your org already reviews. Do not copy a floating tag into a required check.

Watch merge-receipt fail on an unlabeled fixture edit before you depend on it. A check you have never seen fail is a decoration, not a gate.

Draft the failing test somewhere cheap, then throw that green away

You still need a scratch loop for the first red test. That loop should not be the merge runner.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can draft the failing test and a first pass at fixture-roots.txt. The free server option is a fit for one rehearsal of the replay script before the change spends shared CI minutes. This article does not name a model, a token quota, a machine size, or a retention period. Those are product terms, and they move. Read the current terms before you plan around them.

Keep that session in draft territory. Copy the test command into the repo. Let the required job produce the receipt on the pull request SHA. A green log from the free server is a rehearsal. It is not the merge signal, even when the same script printed replay_ok.

If you want that rehearsal before the first push, use the free server to confirm the test fails for the reason you think, then land the hook in a separate pull request that does not touch fixtures.

Who should not use this

Skip the pattern when you cannot name fixture roots without guessing. A hook with a vague list becomes a bypass target in a week.

Skip it when fixtures live in object storage or a shared database. A read-only mount on the runner does not freeze those bytes, and a replay may write into a shared environment.

Skip it when the suite is not safe to run three times. Billing jobs, email jobs, and tests that mutate a shared tenant will turn the flake window into an incident.

Skip it when policy forbids a third-party drafting server from seeing fixture contents. Use an internal runner for the rehearsal. The merge rule does not require the free server. The free server is only a place to draft.

Also skip it if fixture-update is applied by a bot. A label a workflow can add is not a human allow. Remove that automation, or drop the label check and review fixture diffs by hand.

Limitations you should write next to the check

The digest proves the listed files did not change during the replay. It does not prove the fixture was correct, complete, or free of secrets.

The flake budget proves the command stayed green for N runs on one runner. It does not prove the test is stable on another architecture, at another hour, or under load.

jq on $GITHUB_EVENT_PATH assumes a pull_request event with a labels array. A workflow_dispatch run will not have that shape. Do not reuse this workflow file unchanged for every event.

The classify hook trusts BASE_SHA. If that SHA is attacker-controlled, the diff can be made empty. Set the base from your protected branch in the workflow, not from a pull request input.

What to land first

Land ci/fixture-roots.txt and the classify hook in a pull request that does not change fixtures. Watch it fail on a deliberate unlabeled edit. Add the replay job only after that failure is visible. Require merge-receipt last.

Until both receipts name the SHA under review, the path is not green. Do not merge on the rehearsal.

Pin the Cassette SHA Before the Agent Writes an HTTP Test

Emery Lin — Wed, 23 Sep 2026 17:13:21 +0000

Do not merge an agent-written API test until it replays a locked HTTP cassette. A green unit job is not a contract. If that test can still open the network, you shipped a flake machine.

Agents are good at emitting clients, fixtures, and “happy path” calls. They are bad at leaving the network alone. Your job is to make live HTTP a merge-blocking event, then give the agent a narrow lane: client code plus tests that replay a hash-pinned cassette.

This is not a model bake-off. It is a green-to-merge path you can keep in the repo.

The failure you are actually debugging

An agent adds tests/test_billing_client.py. CI is green on the author’s laptop. Overnight the job fails: DNS, 429, a staging field rename, a clock skew in Expires.

You did not lose coverage. You lost determinism. The test was a live probe wearing a pytest filename.

Treat every agent HTTP test as untrusted until three files exist together:

A cassette (recorded request/response).
A SHA256 pin of that cassette in CI.
A flake budget that forbids retries and forbids the network.

No pin, no merge.

What this gate checks

You are not asking “did the agent write tests?” You are asking four narrower questions:

Does the new test import a replay helper instead of requests/fetch against a live host?
Is every cassette path under fixtures/http/ and listed in cassettes.lock?
Does sha256sum of each cassette match the lockfile on main?
Is flake-budget.yml set to retries: 0 and allow_network: false for this job?

If any answer is no, fail the required check. Do not “warn.” Warnings do not block merge.

Decision table

Diff touches	Cassette lock	Network in test	Merge
Client only, tests replay existing cassette	hash unchanged	blocked	allow
New endpoint + new cassette	hash added in same PR	blocked	allow after human review of cassette
Tests only, cassette missing	missing	n/a	fail
Cassette bytes change, lockfile not updated	mismatch	blocked	fail
Any `https://` in `tests/` outside fixtures	n/a	open	fail
Agent edits `.github/workflows` to skip the check	n/a	n/a	fail (keep this job required and name-pinned)

Print this table in the PR template. Agents follow checklists more reliably than prose.

1. Record the cassette against a stub, not production

Stand up a local stub. Do not record against a shared staging host. Staging is not a fixture. It moves.

# labeled example: local OpenAPI stub, not a production capture
python -m http.server 4010 --directory ./stubs/billing &
curl -sS http://127.0.0.1:4010/invoices/inv_1 > /tmp/inv.json

Keep the stub tiny. One resource, one error shape, one pagination cursor. If the cassette contains timestamps that change, strip them in a normalizer before you hash.

# labeled example: normalize before lock
import json, hashlib, pathlib, re

DROP = {"request_id", "generated_at", "trace_id"}

def normalize(raw: bytes) -> bytes:
    data = json.loads(raw)
    if isinstance(data, dict):
        for k in list(data):
            if k in DROP:
                data.pop(k)
    text = json.dumps(data, sort_keys=True, separators=(",", ":"))
    text = re.sub(r"\d{4}-\d{2}-\d{2}T[^\"]+", "<ts>", text)
    return text.encode()

def pin(path: pathlib.Path) -> str:
    digest = hashlib.sha256(normalize(path.read_bytes())).hexdigest()
    return f"{digest}  {path.as_posix()}"

Hash the normalized bytes, not the raw capture. Otherwise the agent “updates” the cassette every run and the lockfile becomes theater.

2. Lock the cassette in the same PR as the client

cassettes.lock is a text file. One line per fixture. Commit it with the client change. Split the PR if the agent also rewrites unrelated snapshots.

# cassettes.lock
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  fixtures/http/get_invoice_inv_1.json

The lockfile is the contract. The YAML workflow is only the referee.

# .github/workflows/agent-http-replay.yml
name: agent-http-replay
on:
  pull_request:
    paths:
      - "src/**"
      - "tests/**"
      - "fixtures/http/**"
      - "cassettes.lock"
      - "flake-budget.yml"
jobs:
  replay:
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - name: pin-and-replay
        run: bash scripts/check_agent_http_tests.sh

Keep timeout-minutes low. Agent jobs that wander the network should die, not soak.

3. Put the checks in a script the hook and CI both run

Hooks that only live in GitHub will be skipped on a laptop. Duplicate the same script in pre-commit and in Actions.

#!/usr/bin/env bash
# scripts/check_agent_http_tests.sh
set -euo pipefail

fail() { echo "::error::$1"; exit 1; }

test -f cassettes.lock || fail "cassettes.lock missing"
test -f flake-budget.yml || fail "flake-budget.yml missing"

python - <<'PY'
import sys, yaml, pathlib
b = yaml.safe_load(pathlib.Path("flake-budget.yml").read_text())
job = b["jobs"]["agent-http-replay"]
assert job.get("retries") == 0, "retries must be 0"
assert job.get("allow_network") is False, "network must be false"
PY

# live hosts in tests/ are a hard fail
if grep -RInE 'https?://[^/]*' tests --include='*.py' --include='*.ts' | grep -v 'fixtures/http'; then
  fail "live URL in tests/; use fixtures/http cassettes"
fi

# every cassette on disk must be locked; every lock line must exist
while read -r hash path; do
  [[ -z "${hash:-}" || "$hash" =~ ^# ]] && continue
  test -f "$path" || fail "locked cassette missing: $path"
  actual=$(python -c "from scripts.normalize import pin; from pathlib import Path; print(pin(Path('$path')).split()[0])")
  test "$actual" = "$hash" || fail "cassette hash mismatch: $path"
done < cassettes.lock

# tests that name an HTTP client must import the replay helper
if git diff --name-only origin/main...HEAD | grep -E '^tests/.*client' >/dev/null; then
  grep -RIn 'replay_cassette\|vcr\.use_cassette\|nock' tests >/dev/null \
    || fail "HTTP client tests must call a replay helper"
fi

echo "agent-http-replay: ok"

Install it once:

chmod +x scripts/check_agent_http_tests.sh
cat > .git/hooks/pre-push <<'HOOK'
#!/usr/bin/env bash
set -euo pipefail
bash scripts/check_agent_http_tests.sh
HOOK
chmod +x .git/hooks/pre-push

If someone bypasses the hook with --no-verify, CI still fails. The hook is for speed. The required check is for truth.

4. Write the test so replay is the only I/O

Label this as a template, not a captured production run.

# tests/test_billing_client.py — labeled example
from pathlib import Path
from billing_client import BillingClient
from http_replay import replay_cassette

CASSETTE = Path("fixtures/http/get_invoice_inv_1.json")

def test_get_invoice_shape():
    with replay_cassette(CASSETTE, allow_network=False):
        inv = BillingClient("http://cassette.local").get_invoice("inv_1")
    assert inv["id"] == "inv_1"
    assert "amount_cents" in inv
    assert "request_id" not in inv  # stripped by normalizer; do not assert chatter

allow_network=False must be an argument the helper enforces, not a comment. If the helper cannot intercept the socket, fail the job in the helper, not in a linter after the fact.

5. Give the agent a write budget, then measure it

Agents pad tests. Cap what they may touch.

# flake-budget.yml
jobs:
  agent-http-replay:
    retries: 0
    allow_network: false
    max_new_cassettes: 3
    max_test_lines: 200
    timeout_seconds: 60

Count new cassette files and new test lines in the PR diff. If the agent dumps a hundred near-duplicate GET fixtures, fail. You wanted a contract, not a traffic dump.

new_cassettes=$(git diff --name-only --diff-filter=A origin/main...HEAD -- fixtures/http | wc -l)
test "$new_cassettes" -le 3 || { echo "too many new cassettes"; exit 1; }

Short cap. Hard fail. No “we will tidy later.”

Where a free local agent fits

You still need something to draft the client test. Keep paid cloud keys out of GitHub Secrets if you can. Run the agent next to the stub, on a machine you control, then let CI only replay.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that draft loop without stuffing a vendor key into Actions, MonkeyCode’s free model access and free server option are one way to generate the client test locally, then push only the cassette, lockfile, and replay test. CI should never need the model. CI needs the pin.

Do not ask the agent to “refresh cassettes” in GitHub-hosted runners. That reintroduces the network. Generation is a laptop (or free server) step. Merge is a replay step.

Green-to-merge path, in order

Human or stub records one cassette; normalizer strips chatter; SHA goes into cassettes.lock.
Pre-push runs scripts/check_agent_http_tests.sh.
Required check agent-http-replay runs the same script on the PR head.
Branch protection requires that exact check name. Renames fail closed.
Merge. Not before.

If you already split merge work from soak work, keep this job on the merge lane. Soak can probe staging later, off the critical path, with a different check name. Do not let soak results silently satisfy merge.

Limitations

This gate does not prove the backend is correct. It proves the client still understands a frozen conversation. If the real API changed and nobody updated the stub, you will merge a faithful client of a dead contract. Pair this with a scheduled contract job against a versioned schema, not against a live shared tenant.

Cassettes can leak secrets. Scan fixtures for tokens before you hash. If a cassette contains Authorization, fail. Redact, re-record, then lock.

Binary bodies, streaming endpoints, and websocket upgrades do not belong here. Use a different artifact for those.

Who should not use this

Skip this if your tests are already fully in-process with no HTTP. Skip it if you cannot keep a stub. Skip it if the agent is allowed to edit workflow files that define the required check — fix that permission first. Skip it if you need live canaries on every PR; those belong in soak, with an explicit budget, not in merge.

You want merge to be boring. Pin the cassette SHA. Then let the agent write the client test. Anything else is a live request with extra steps.

Fail the Job If the Agent Invents a Field

Emery Lin — Tue, 22 Sep 2026 15:26:28 +0000

Stop treating a green unit test as proof the agent spoke to your API correctly. If the model invents a query field, a required header, or a verb your service never shipped, the merge path should fail before any live call leaves CI.

That is the whole method. Schema-check the tool JSON. Keep live inference off the merge job. Replay recorded fixtures so a flaky upstream cannot buy you a green check.

You already know the failure mode. An agent “helps” by filling in limit, pretty, or async=true because those names showed up in training data. Your stub server shrugs. Production 400s after merge. The CI log still says success because nobody asserted the call shape.

What you are actually gating

You are not grading prose. You are grading a contract.

A tool call is an HTTP intent with a name, a method, a path, and a JSON body. If that payload does not validate against the schema you committed, the agent did not finish the work. Fail the job. Do not retry the model hoping the next sample is nicer.

Live model access belongs in a labeled soak job, not in pull_request. Free endpoints are still endpoints. They rate-limit. They change. They are the wrong merge dependency.

Decision table: merge vs soak vs skip

Signal in the PR	Merge job	Soak job (opt-in label)
Agent only edits application code	Replay fixtures, schema-check tool JSON	Off
Agent adds or renames a tool	Replay fixtures and fail if schema file was not updated in the same diff	Off until a human adds `soak-tools`
Fixture hash mismatch	Fail. Do not silently rerecord	Allowed to rerecord behind the label
HTTP 429 / timeout from a hosted model	Must not run	Retry with a hard cap, then skip (non-blocking)
Schema itself changed	Contract tests must change in the same commit	Optional live probe of the new fields

Print this table in the PR template if your reviewers skim. One row per signal. No extra narrative.

1. Commit the contract next to the tool

Put the schema where the agent can see it and where CI can hash it. Do not hide it in a wiki.

{
  "$id": "https://example.local/schemas/search-tickets.json",
  "title": "search_tickets",
  "type": "object",
  "additionalProperties": false,
  "required": ["query", "status"],
  "properties": {
    "query": { "type": "string", "minLength": 1, "maxLength": 200 },
    "status": { "enum": ["open", "pending", "closed"] },
    "page": { "type": "integer", "minimum": 1, "maximum": 50 }
  }
}

additionalProperties: false is the whole point. Invented fields must be a CI failure, not a warning the log swallows.

Pin a tiny allowlist of tool names in the same directory.

tools/
  search_tickets.schema.json
  create_ticket.schema.json
  ALLOWLIST.txt

ALLOWLIST.txt is one name per line. If the agent emits a tool that is not on that list, fail. New tools need a schema file and a human-edited allowlist in the same commit.

2. Record the tool JSON, not the essay

You need a fixture the merge job can replay without a network. Capture only the structured call.

# tests/support/record_tools.py
# Proposal: drop this next to your agent runner. Label it experimental until you wire it to your real tracer.
from __future__ import annotations

import json
from pathlib import Path
from typing import Any

FIXTURE_DIR = Path("tests/fixtures/tool_calls")


def dump_tool_call(run_id: str, tool_name: str, arguments: dict[str, Any]) -> Path:
    FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
    path = FIXTURE_DIR / f"{run_id}__{tool_name}.json"
    payload = {"name": tool_name, "arguments": arguments}
    path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    return path

Do not record chain-of-thought. Do not record API keys. If your tracer includes headers, strip Authorization before write.

Rerecord on purpose:

python -m tests.support.record_tools --run-id pr-842 --from-session ./tmp/agent-session.jsonl
git add tests/fixtures/tool_calls

If the agent changed behavior and nobody updated fixtures, merge CI must go red. That is the receipt.

3. Schema-check in pytest, on every PR

# tests/test_tool_contracts.py
from __future__ import annotations

import json
from pathlib import Path

import jsonschema
import pytest

SCHEMA_DIR = Path("tools")
FIXTURE_DIR = Path("tests/fixtures/tool_calls")
ALLOWLIST = {
    line.strip()
    for line in Path("tools/ALLOWLIST.txt").read_text(encoding="utf-8").splitlines()
    if line.strip() and not line.startswith("#")
}


def load_schema(name: str) -> dict:
    path = SCHEMA_DIR / f"{name}.schema.json"
    if not path.exists():
        pytest.fail(f"tool {name!r} has no schema at {path}")
    return json.loads(path.read_text(encoding="utf-8"))


@pytest.mark.parametrize("fixture_path", sorted(FIXTURE_DIR.glob("*.json")))
def test_recorded_tool_matches_schema(fixture_path: Path) -> None:
    payload = json.loads(fixture_path.read_text(encoding="utf-8"))
    name = payload["name"]
    if name not in ALLOWLIST:
        pytest.fail(f"{fixture_path.name}: tool {name!r} is not in ALLOWLIST.txt")
    jsonschema.validate(instance=payload["arguments"], schema=load_schema(name))

Run it locally the same way CI will.

python -m pip install jsonschema pytest
python -m pytest tests/test_tool_contracts.py -q

One assertion per fixture. No sleeps. No retries. If this file needs the network, you built the wrong test.

4. Keep the live model off the merge path

Merge CI should be boring. Checkout. Install. Replay fixtures. Validate schemas. Upload the junit file. Stop.

# .github/workflows/tool-contracts.yml
name: tool-contracts
on:
  pull_request:
  push:
    branches: [main]

jobs:
  replay:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install jsonschema pytest
      - run: pytest tests/test_tool_contracts.py --junitxml=junit-tools.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: tool-contract-junit
          path: junit-tools.xml

That job never calls a model. If an agent rewrites this workflow to inject a live key, you have a different problem: protect the workflow file with CODEOWNERS and a required review. Do not solve that inside the pytest file.

5. Optional soak: live calls behind a label

You still want a place to see what a hosted model does with the same prompt. Make it opt-in, non-blocking, and budgeted.

This is the only place a free model path and a free server option belong. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already use MonkeyCode, its free model access and free server option can host that soak runner so the merge job stays fixture-only. They are extra capacity, not a required check name.

# .github/workflows/tool-soak.yml
name: tool-soak
on:
  pull_request:
    types: [labeled, synchronize, opened]

jobs:
  soak:
    if: contains(github.event.pull_request.labels.*.name, 'soak-tools')
    runs-on: ubuntu-latest
    timeout-minutes: 15
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Probe live tool calls with a hard retry cap
        env:
          SOAK_BASE_URL: ${{ secrets.SOAK_BASE_URL }}
        run: python tests/support/soak_probe.py --max-attempts 3 --timeout-sec 20

The probe must refuse to invent retries. Three attempts. Then skip.

# tests/support/soak_probe.py
# Proposal: wire SOAK_BASE_URL to whatever host you actually control.
from __future__ import annotations

import os
import sys
import time
import urllib.error
import urllib.request


def main(max_attempts: int, timeout_sec: int) -> int:
    url = os.environ.get("SOAK_BASE_URL")
    if not url:
        print("SOAK_BASE_URL unset; skipping soak")
        return 0
    last_err = None
    for attempt in range(1, max_attempts + 1):
        try:
            with urllib.request.urlopen(url, timeout=timeout_sec) as resp:
                print(f"soak ok status={resp.status} attempt={attempt}")
                return 0
        except (urllib.error.URLError, TimeoutError) as err:
            last_err = err
            print(f"soak attempt {attempt} failed: {err}")
            time.sleep(min(2 ** attempt, 8))
    print(f"soak skipped after {max_attempts} attempts: {last_err}")
    return 0  # non-blocking


if __name__ == "__main__":
    raise SystemExit(main(max_attempts=3, timeout_sec=20))

Notice the exit code. Soak never blocks merge. If the free server is busy, you learn that in the log, not by wedging the queue.

6. Fail closed on three cheap signals

Add a short guard so the agent cannot “fix” CI by deleting tests.

# scripts/assert_contract_surface.sh
set -euo pipefail
test -f tools/ALLOWLIST.txt
test -f tests/test_tool_contracts.py
count=$(find tests/fixtures/tool_calls -name '*.json' | wc -l)
if [ "$count" -lt 1 ]; then
  echo "no tool-call fixtures; refusing to pass"
  exit 1
fi

Call it from the merge workflow before pytest. Empty fixture directories are not a pass. They are an untested agent.

You can also hash the schema directory and echo it into the job summary. When the hash changes without a matching fixture update, reviewers see it without reading YAML.

schema_hash=$(find tools -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum)
echo "schema_hash=$schema_hash" >> "$GITHUB_STEP_SUMMARY"

What this does not prove

A valid tool payload is not a valid product decision. The agent can still search the wrong tenant, page past your max, or call a legal tool in an illegal order. Schema checks do not replace authz tests, idempotency tests, or a human reading the diff.

This approach also assumes you own the schema. If the upstream API is a moving SaaS spec you do not vendor, freeze a copy. Do not fetch OpenAPI from the internet during pull_request. That is another flaky dependency wearing a documentation hat.

Hosted inference — free or not — can still return a well-typed wrong answer. Soak is for drift detection. It is not a substitute for the replay job.

Who should not use this

Skip this workflow if your “agent” never emits tool JSON. Skip it if every call is already a hand-written client with compile-time types and no LLM in the loop. Skip it if policy forbids sending prompts or customer text to a hosted model, free server included.

Do not use soak-as-merge if you cannot tolerate a 429. Do not store raw session traces that contain secrets. Do not let continue-on-error: true leak onto the replay job.

A 15-minute checkout you can run today

Add tools/search_tickets.schema.json and tools/ALLOWLIST.txt.
Drop one recorded fixture under tests/fixtures/tool_calls.
Copy tests/test_tool_contracts.py and run pytest until it fails on an invented field.
Wire .github/workflows/tool-contracts.yml as a required check.
Only then add tool-soak.yml behind the soak-tools label.

Break the fixture on purpose. Add "pretty": true to the arguments. Watch pytest fail. That red log is the feature.

If you want a soak host that is not your laptop, MonkeyCode’s free model access and free server option are enough to run the labeled job. Keep them off the required check list. The merge path should still be fixtures, schemas, and a boring green.