DEV Community: Emery Lin

If the Agent Touches .github, Fail the Job. Then Measure Coverage Delta.

Emery Lin — Mon, 21 Sep 2026 13:47:13 +0000

A green CI run is not a merge decision when the same pull request rewrote the job that produced it. You need a path-deny check and a coverage-delta check that the agent cannot edit in the same diff.

Agents are good at making tests pass. They are also good at moving the finish line. One deleted assertion. One quieter workflow. One coverage floor that used to be a hard fail and is now a skip. The log still says success.

You already know the smell. The PR looks helpful. The check is green. The change you actually wanted is buried under a chore: stabilize CI file.

This article is a concrete gate, not a prompt. You will add three things: a deny list for pipeline files, CODEOWNERS on those paths, and a coverage-delta job that compares the branch to the merge base. A model may summarize the failure. It does not get to define the threshold.

The failure you are actually debugging

The bug is not that the model is too eager. The bug is that your merge contract lives in files the agent can patch.

GitHub will run whatever workflow is on the head ref for pull_request. If the agent edits .github/workflows/ci.yml in that same PR, it can rename a required check, skip a job with an if:, or drop --cov-fail-under. Branch protection cannot save you if the protected check never runs.

You fix that by treating pipeline files as a different class of change. Product code can land through the normal jobs. Pipeline code cannot land in the same PR as product code, and it cannot land without a human owner.

What this gate decides

Keep the rule small enough to implement in an afternoon. Write it down before you touch YAML.

If the PR diff touches .github/, CODEOWNERS, coverage config, or the scripts that compute the delta, fail a dedicated job named pipeline-freeze.
If those paths are clean, run tests and publish a coverage total.
Fail coverage-delta when the branch total is lower than the merge-base total by more than your budget. Start that budget at 0.
Never let a model write the deny list, the budget, or the job name.

That is the whole policy. Everything else is wiring.

1. Freeze the files that define the gate

Put the deny list in-repo, next to the workflow, and keep it boring. Humans edit it. Agents do not.

# ci/pipeline-deny.txt
.github/
CODEOWNERS
.coveragerc
pyproject.toml
codecov.yml
scripts/ci/

pyproject.toml belongs on the list only if it holds tool.coverage or pytest addopts. If your coverage config lives elsewhere, deny that file instead. Do not deny the entire repository. You want the agent to change product code.

You will read this file in CI with git diff --name-only. Exact prefix match. No regex from the model.

2. Add a path-deny job

Create a workflow the agent is not allowed to edit in the same PR. That sounds circular until you split the controls. CODEOWNERS blocks the human review path. A required check name, set in the branch-protection UI, blocks the robot path.

Proposal you can adapt. Pin actions/checkout to a full-length SHA before you make the job required.

# .github/workflows/pipeline-freeze.yml
name: pipeline-freeze

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  pipeline-freeze:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Fail if the PR edits the merge contract
        env:
          BASE: ${{ github.event.pull_request.base.sha }}
          HEAD: ${{ github.event.pull_request.head.sha }}
        run: bash scripts/ci/deny_pipeline_paths.sh

#!/usr/bin/env bash
# scripts/ci/deny_pipeline_paths.sh
set -euo pipefail

: "${BASE:?BASE sha required}"
: "${HEAD:?HEAD sha required}"

deny_file="ci/pipeline-deny.txt"
mapfile -t denied < <(grep -v '^[[:space:]]*#' "$deny_file" | grep -v '^[[:space:]]*$')

changed=$(git diff --name-only "$BASE...$HEAD")
hits=()

while IFS= read -r path; do
  [ -z "$path" ] && continue
  for prefix in "${denied[@]}"; do
    prefix="${prefix%/}"
    case "$path" in
      "$prefix"|"$prefix"/*)
        hits+=("$path")
        break
        ;;
    esac
  done
done <<< "$changed"

if [ "${#hits[@]}" -gt 0 ]; then
  echo "pipeline-freeze: these paths are not mergeable with product changes:"
  printf '  %s\n' "${hits[@]}"
  echo "Open a separate PR that touches only pipeline files, and wait for CODEOWNERS."
  exit 1
fi

echo "pipeline-freeze: no merge-contract paths in the diff"

Mark the script executable in git. If this job is not a required status check, the agent can ignore it. Name the check pipeline-freeze and require that exact name in the branch rule. Do not let the workflow rename itself in the same PR. That is the point of the deny list.

3. Require human owners on those paths

Path deny is a robot. CODEOWNERS is the human backup when someone needs a legitimate CI change.

# CODEOWNERS
.github/          @your-org/maintainers
CODEOWNERS        @your-org/maintainers
ci/               @your-org/maintainers
scripts/ci/       @your-org/maintainers
.coveragerc       @your-org/maintainers

Turn on “Require review from Code Owners.” A second agent account is not a code owner. If your org lets bots satisfy review, exclude them from this file.

Legitimate pipeline work still happens. It happens in a PR that only contains pipeline files, reviewed by a person who can explain why the floor moved.

4. Compute coverage delta from the merge base

A frozen workflow can still ship a PR that deletes assertions and keeps a green job. You need a number that is not negotiated in the diff.

Proposal: two coverage JSON files, one for BASE, one for HEAD, compared by a script the deny list already protects.

  coverage-delta:
    needs: pipeline-freeze
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install and test HEAD
        run: |
          python -m pip install -e ".[dev]"
          pytest --cov --cov-report=json:coverage.head.json

      - name: Test merge base
        env:
          BASE: ${{ github.event.pull_request.base.sha }}
        run: |
          git stash --include-untracked || true
          git checkout "$BASE"
          python -m pip install -e ".[dev]"
          pytest --cov --cov-report=json:coverage.base.json
          git checkout -

      - name: Compare totals
        env:
          MAX_DROP: "0"
        run: python scripts/ci/coverage_delta.py coverage.base.json coverage.head.json

# scripts/ci/coverage_delta.py
"""Fail when HEAD coverage is below BASE by more than MAX_DROP percent."""
from __future__ import annotations

import json
import os
import sys
from pathlib import Path


def total_percent(path: Path) -> float:
    payload = json.loads(path.read_text())
    totals = payload["totals"]
    covered = float(totals["covered_lines"])
    num = float(totals["num_statements"])
    if num == 0:
        raise SystemExit(f"{path}: no statements in coverage totals")
    return 100.0 * covered / num


def main() -> None:
    if len(sys.argv) != 3:
        raise SystemExit("usage: coverage_delta.py BASE.json HEAD.json")
    base = total_percent(Path(sys.argv[1]))
    head = total_percent(Path(sys.argv[2]))
    max_drop = float(os.environ.get("MAX_DROP", "0"))
    drop = base - head
    print(
        f"coverage base={base:.2f} head={head:.2f} "
        f"drop={drop:.2f} max_drop={max_drop:.2f}"
    )
    if drop > max_drop + 1e-9:
        raise SystemExit(
            "coverage-delta: HEAD dropped below the merge-base budget"
        )
    print("coverage-delta: within budget")


if __name__ == "__main__":
    main()

Start MAX_DROP at 0. If a noisy generated file makes that unusable, raise it in a pipeline-only PR, not in the same diff that removes tests.

Checking out the merge base inside the job is slow. It is also honest. Caching the base total from main is a later optimization. Do not skip the comparison because an agent offered a cached number.

5. Keep the model off the decision

Run the same two scripts locally before you open the PR. That is the whole preflight.

export BASE="$(git merge-base HEAD origin/main)"
export HEAD="$(git rev-parse HEAD)"
bash scripts/ci/deny_pipeline_paths.sh
pytest --cov --cov-report=json:coverage.head.json
# use a worktree for BASE if you do not want to dirty the tree
python scripts/ci/coverage_delta.py coverage.base.json coverage.head.json

A coding agent can still draft the product diff. It should not draft ci/pipeline-deny.txt.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already use MonkeyCode, the useful part here is not a new merge rule from the model. Free model access and the free server option can run this same preflight before GitHub ever sees the branch: the scripts decide pass or fail, and the model only explains which denied path or which coverage total moved. That explanation is optional. The exit code is not.

Do not paste the deny list into a prompt and ask whether the PR is probably fine. Prefix match is cheaper than inference, and it does not invent a path.

Decision table for the PR template

Paste this into .github/PULL_REQUEST_TEMPLATE.md so reviewers do not re-litigate the policy on every agent branch.

Diff contains	`pipeline-freeze`	`coverage-delta`	Review
`src/` or tests only, coverage holds	pass	pass	normal review
`src/` plus deleted assertions, coverage drops	pass	fail	do not merge
any file under `.github/` with product code	fail	skipped	split the PR
pipeline files only	fail here; use a pipeline-only change	n/a	CODEOWNERS required
renamed workflow job that used to be required	fail	skipped	reject; restore the name
model-written rationale, no script change	n/a	n/a	ignore as evidence

If a row is not in this table, the answer is fail closed.

Limitations

This does not detect a test that still runs but no longer asserts anything useful. Coverage can stay flat while behavior rots. Pair the delta with at least one characterization test you actually read.

Checking out the merge base twice per PR costs minutes on large installs. Monorepos will need path filters for the test command itself, without letting those filters live in an editable workflow on the same PR.

git diff --name-only does not replace review on pipeline-only changes. It does see added workflows. You still want secret scanning and a human on those PRs.

GitHub required checks are configured in the hosting UI. If someone with admin rights removes pipeline-freeze from the list, this article becomes a blog post. Protect the branch rules the same way you protect CODEOWNERS.

The scripts above are a proposal. Run them against a throwaway branch before you make them required.

Who should not use this

Skip it if you do not accept agent PRs, or if a single maintainer already reviews every line and coverage is not a merge input.

Skip it if your tests are screenshots or manual QA with no numeric total. The delta job needs a denominator.

Skip it if pipeline files and product files must ship together because of a framework constraint you cannot split. In that case you need a different control plane, not a prefix match.

Do not use a model as the coverage accountant. Do not use this gate as permission to stop reading the product diff.

The merge question is simple after the jobs run. Did the agent edit the contract. Did coverage fall. If either answer is yes, the PR is not green, no matter how confident the summary sounds.

Lock the Golden Files. Split Agent Diffs That Touch the Answers.

Emery Lin — Sun, 20 Sep 2026 11:43:24 +0000

Treat golden files like a lockfile. If an agent can rewrite expected output in the same pull request as the code under test, the green check is circular.

You do not need a new platform to stop that. You need a committed hash list, a split-diff rule, and a required check that ignores whatever the model claimed about its own patch.

The failure you are blocking

Watch a typical agent PR. It changes src/pricing.py. It also refreshes tests/golden/invoice.json. Both files look related. CI runs. The new function matches the new JSON. Nothing failed, because the spec moved with the implementation.

That is not coverage. That is a tautology. The test did not catch the bug. It absorbed it.

Humans still refresh goldens on purpose. You want that path. You do not want it on the default agent lane, and you do not want CI to rewrite fixtures.lock for anyone.

The contract

Keep three rules. They should fit in a review comment.

Every golden file has a SHA-256 in fixtures.lock.
An agent-labeled PR may change production code or golden files, not both, unless a human applied fixtures:allow.
The merge job runs tests once. No retry loop. A scheduled job can retry. Merge cannot.

If rule 2 fires, the PR stays open. A person inspects the bytes, applies the label, updates the lock in a visible commit, and re-runs the check.

Decision table

Use this at review time. Do not invent extra states until this table holds for a few weeks.

Diff shape	`agent` label	`fixtures:allow`	Contract result
Source only	yes	n/a	Pass contract, run tests once
Goldens only	yes	no	Fail split-diff
Source and goldens	yes	no	Fail split-diff
Source and goldens	yes	yes	Pass split; lockfile must update in the same PR
Source and goldens	no (human PR)	n/a	Pass split; `CODEOWNERS` still reviews goldens
Lockfile drift, goldens unchanged	any	no	Fail hash check

The table is the spec. The scripts below only enforce it.

Step 1 — Inventory the answers

List the files that define expected output. JSON goldens, text fixtures, recorded HTTP payloads, snapshot files. Put the list in the repo so a check cannot quietly drop a path.

mkdir -p tests/golden tests/snapshots scripts

find tests/golden tests/snapshots -type f \
  \( -name '*.json' -o -name '*.snap' -o -name '*.txt' -o -name '*.csv' \) \
  | sort > fixtures.manifest

git add fixtures.manifest

Generate the lock from a clean main. Do this on a tree you already trust, not on the agent branch.

git checkout main
python scripts/write_fixture_lock.py
git add fixtures.lock
git commit -m "Lock golden file hashes"

Example writer. Label this as a starting script, not a benchmarked production service.

# scripts/write_fixture_lock.py
from __future__ import annotations

import hashlib
import pathlib
import sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
MANIFEST = ROOT / "fixtures.manifest"
LOCK = ROOT / "fixtures.lock"


def expand(pattern: str) -> list[pathlib.Path]:
    rel = pattern.strip()
    if not rel or rel.startswith("#"):
        return []
    return sorted(p for p in ROOT.glob(rel) if p.is_file())


def sha256(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def main() -> int:
    if not MANIFEST.exists():
        print("missing fixtures.manifest", file=sys.stderr)
        return 2
    rows: list[str] = []
    for line in MANIFEST.read_text().splitlines():
        for path in expand(line):
            rel = path.relative_to(ROOT).as_posix()
            rows.append(f"{sha256(path)}  {rel}")
    LOCK.write_text("\n".join(rows) + ("\n" if rows else ""))
    print(f"wrote {len(rows)} hashes to {LOCK.relative_to(ROOT)}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Boring is the point. If the writer gets clever, the gate gets negotiable.

Step 2 — Fail closed on a drifted hash

The merge job recomputes hashes from the tree it just checked out. If fixtures.lock does not match, and the PR is not allowed to edit goldens, fail. CI must never rewrite the lock. Updating the lock is a human commit you can git blame.

# scripts/check_fixture_lock.py
from __future__ import annotations

import hashlib
import pathlib
import sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
LOCK = ROOT / "fixtures.lock"


def sha256(path: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def main() -> int:
    if not LOCK.exists():
        print("missing fixtures.lock", file=sys.stderr)
        return 2
    failed = 0
    seen: set[str] = set()
    for raw in LOCK.read_text().splitlines():
        if not raw.strip():
            continue
        digest, rel = raw.split("  ", 1)
        seen.add(rel)
        path = ROOT / rel
        if not path.is_file():
            print(f"MISSING  {rel}")
            failed += 1
            continue
        actual = sha256(path)
        if actual != digest:
            print(f"DRIFT    {rel}")
            print(f"  lock   {digest}")
            print(f"  tree   {actual}")
            failed += 1
    if failed:
        print(f"fixture lock failed: {failed} path(s)")
        return 1
    print(f"fixture lock ok: {len(seen)} path(s)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it the same way CI will:

python scripts/check_fixture_lock.py

A mismatch is a red X. Three common meanings: the agent edited a golden, a human edited a golden and forgot the lock, or someone deleted a file still listed in the lock. All three are merge blockers until a person chooses.

Step 3 — Split the diff for agent PRs

Detect the lane with an agent label, or with a commit trailer Agent-Assisted: true. Then inspect the name-only diff against the base branch. Production paths and golden paths must not both appear unless fixtures:allow is present.

# scripts/split_agent_diff.py
from __future__ import annotations

import os
import subprocess
import sys

GOLDEN_PREFIXES = ("tests/golden/", "tests/snapshots/", "fixtures.lock")
SOURCE_PREFIXES = ("src/", "lib/", "app/")


def names(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def main() -> int:
    base = os.environ.get("BASE_SHA", "origin/main")
    labels = set(os.environ.get("PR_LABELS", "").split(","))
    labels = {item.strip() for item in labels if item.strip()}
    is_agent = "agent" in labels or os.environ.get("AGENT_ASSISTED") == "1"
    if not is_agent:
        print("human PR: split-diff skipped")
        return 0
    changed = names(base)
    gold = [p for p in changed if p.startswith(GOLDEN_PREFIXES)]
    src = [p for p in changed if p.startswith(SOURCE_PREFIXES)]
    if src and gold and "fixtures:allow" not in labels:
        print("agent PR changed source and golden files without fixtures:allow")
        print("source:")
        print("\n".join(f"  {p}" for p in src))
        print("goldens:")
        print("\n".join(f"  {p}" for p in gold))
        return 1
    print("agent split-diff ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

This is the rule that stops the tautology. The model can still propose a code fix. It can still open a second PR that only updates goldens. It cannot smuggle both through one green check.

Local dry run against main:

export BASE_SHA=origin/main
export PR_LABELS=agent
python scripts/split_agent_diff.py

If you use trailers instead of labels, set AGENT_ASSISTED=1 when git log -1 --pretty=%B contains Agent-Assisted: true.

Step 4 — Wire a required check

Example GitHub Actions workflow. Adapt branch names and prefixes. Do not paste this into a repo whose fixtures live somewhere else and assume it holds.

# .github/workflows/fixture-contract.yml
name: fixture-contract
on:
  pull_request:
    branches: [main]

jobs:
  fixture-contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Split agent diffs
        env:
          BASE_SHA: ${{ github.event.pull_request.base.sha }}
          PR_LABELS: ${{ join(github.event.pull_request.labels.*.name, ',') }}
        run: python scripts/split_agent_diff.py
      - name: Verify fixture lock
        run: python scripts/check_fixture_lock.py
      - name: Tests once, no retry
        run: |
          if [ -f pyproject.toml ] || [ -f pytest.ini ]; then
            python -m pytest -q --maxfail=1
          else
            echo "no pytest config; skip test step in this example"
          fi

Protect main. Require the fixture-contract check. Put CODEOWNERS on the answers so a green job cannot bypass a human on spec files:

# .github/CODEOWNERS
/tests/golden/    @your-org/maintainers
/tests/snapshots/ @your-org/maintainers
/fixtures.lock    @your-org/maintainers
/fixtures.manifest @your-org/maintainers

Write access is not the same as permission to change the spec. Owners review goldens. The check only proves the hashes and the split.

Step 5 — Keep retries off the merge job

Flakes exist. They do not get a vote on merge.

python -m pytest -q --maxfail=1

If a test is unstable, you will see it. That is the point. A retry loop on the PR check is how a weak assertion becomes "green enough." Put soak retries on a schedule after merge if you need them. Do not hide them inside the required job.

When fixtures:allow is present, require the lock update in the same PR:

python scripts/write_fixture_lock.py
git diff --exit-code fixtures.lock

If that diff is non-empty after a golden edit, the author forgot to refresh the lock. Fail. Do not refresh it in CI and push back. The lock is a reviewed artifact, not a build cache.

A local pass is optional

You can run the same three commands on a laptop before git push. A coding assistant that can read the diff will often catch a golden rewrite earlier than you will. That preview is useful. It is not a receipt.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want that walkthrough on a box you do not have to size yourself, MonkeyCode's free model access and free server option are enough to run check_fixture_lock.py and split_agent_diff.py and list the files that would fail the check. Treat the output as a preview. The required GitHub job is still the only merge authority.

What this does not catch

The hash gate does not stop assert True. It does not stop deleting a test. It does not stop moving a golden file out of the manifest so the lock no longer sees it.

Add a cheap assertion-count diff if that is your next hole. Fail if net assert / self.assert* count drops without tests:allow. That is a different check. Do not overload the lockfile job until you have watched this one hold.

Also watch empty goldens. A file that becomes {} still hashes. The lock will match. The spec died anyway. If that shows up, add a minimum-size rule for selected paths, or review those files by owner every time.

Who should not use this

Skip this if you have no golden files. Skip it if snapshot files are the product, as in some visual-regression suites where every intentional UI tweak must update pixels. Skip it on throwaway spikes that never merge to main.

Also skip it if your agent label is world-writable and you cannot restrict who applies fixtures:allow. A gate that anyone can disable is documentation, not a gate. Restrict label changes, or require the allow label plus a CODEOWNER approval on fixtures.lock.

Close

Commit the lockfile. Split the agent diff. Run tests once. When a human really must refresh expected output, they say so with a label and they own the review.

The model can still write code. It does not get to rewrite the answers in the same move.

Put the Green-to-Merge Path in the Repo, Then Verify the Receipt

Emery Lin — Sat, 19 Sep 2026 09:47:37 +0000

A green check is not merge permission. Permission is a receipt that matches a versioned path file in the repo. If the path is only in someone’s head, agent patches will skip a gate and still look done.

You already know the failure mode. A model rewrites a handler, updates a fixture, and your laptop reports green. CI later retries a flaky test, the check turns green again, and the PR merges. The sequence that was supposed to protect main never ran as a sequence. It ran as a pile of jobs.

Encode the path. Dry-run it locally. Let CI sign a receipt. Merge only when those three agree.

What the path file is for

Keep the merge contract next to the code. Not in a wiki. Not in a branch-protection screenshot from last quarter.

The file names the gates, their order, and how many flakes each gate may spend. CI becomes a verifier. You stop arguing about which check “counts.”

# .ci/merge-path.yml
version: 1
fixture_lock: tests/fixtures.lock
allow_merge_on: merge-receipt
required_sequence:
  - id: fixture-lock
    check_name: fixture-lock
    flake_budget: 0
  - id: unit
    check_name: unit
    flake_budget: 0
  - id: contract
    check_name: contract
    flake_budget: 1
  - id: merge-receipt
    check_name: merge-receipt
    flake_budget: 0

Short file. Strict meaning. If a job is not in this list, it cannot unlock merge. If a job is in the list and silent, merge stays closed.

Freeze fixtures before the patch can travel

Agent diffs love to “fix” snapshots. That is how a broken contract becomes a new golden file. You want that change visible as a lock update, not as silent drift.

#!/usr/bin/env bash
# scripts/hash-fixtures.sh
set -euo pipefail
root="$(git rev-parse --show-toplevel)"
lock="$root/tests/fixtures.lock"
tmp="$(mktemp)"

find "$root/tests/fixtures" -type f | sort | while read -r f; do
  rel="${f#$root/}"
  printf '%s  %s\n' "$(git hash-object "$f")" "$rel"
done > "$tmp"

if [[ "${1:-}" == "--write" ]]; then
  mv "$tmp" "$lock"
  echo "wrote $lock"
  exit 0
fi

diff -u "$lock" "$tmp"
rm -f "$tmp"

Run it two ways. --write is a deliberate lock bump. A plain run is a comparison. The hook below refuses a push when the comparison fails.

#!/usr/bin/env bash
# .githooks/pre-push
set -euo pipefail
repo="$(git rev-parse --show-toplevel)"
"$repo/scripts/hash-fixtures.sh"

Install it once:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push scripts/hash-fixtures.sh

You now have a local fail-closed gate. The fixture tree cannot move with the patch unless the lock moves too. That is the first step of the merge path, and it happens before CI spends a minute.

Dry-run the same sequence you will ask CI to sign

Do not wait for GitHub to tell you the order was wrong. Replay the path on the commit you intend to push.

#!/usr/bin/env bash
# scripts/merge-dry-run.sh
set -euo pipefail
repo="$(git rev-parse --show-toplevel)"
cd "$repo"

echo "== fixture-lock =="
./scripts/hash-fixtures.sh

echo "== unit =="
python -m pytest tests/unit -q --maxfail=1

echo "== contract =="
python -m pytest tests/contract -q --maxfail=1

echo "dry-run ok on $(git rev-parse HEAD)"

Call it before every agent PR:

chmod +x scripts/merge-dry-run.sh
./scripts/merge-dry-run.sh

If this script fails, you do not open the PR. Green later in CI will not repair a path you never ran. The dry-run is not a substitute for CI. It is a filter so CI only sees patches that already survived the sequence.

Spend flakes as a budget, then write them down

Retries hide instability when they are infinite. They are usable when they are counted, named, and stored on the receipt.

# scripts/run_with_budget.py
from __future__ import annotations

import json, subprocess, sys, time
from pathlib import Path

def run(cmd: list[str]) -> int:
    p = subprocess.run(cmd)
    return p.returncode

def main() -> int:
    gate, budget = sys.argv[1], int(sys.argv[2])
    cmd = sys.argv[3:]
    spends = 0
    last = 1
    attempts = budget + 1
    for i in range(attempts):
        last = run(cmd)
        if last == 0:
            break
        spends += 1
        if spends > budget:
            break
        time.sleep(min(2 ** i, 8))
    Path("receipt.partial.json").write_text(json.dumps({
        "gate": gate,
        "flake_budget": budget,
        "flake_spent": spends if last == 0 else spends,
        "ok": last == 0 and spends <= budget,
        "command": cmd,
    }, indent=2))
    return last if last != 0 else (0 if spends <= budget else 1)

if __name__ == "__main__":
    raise SystemExit(main())

Unit tests get a budget of zero. Contract tests may spend one retry. The receipt records the spend. A green job that burned three retries against a budget of one is not green for merge. It is a failed gate with a polite color.

CI writes one receipt. Merge reads only that receipt.

The last required check is not “all jobs finished.” It is a document. Head SHA, fixture lock digest, gate order, and flake spends have to match .ci/merge-path.yml.

# scripts/write_receipt.py
from __future__ import annotations

import hashlib, json, os, subprocess, sys
from pathlib import Path

import yaml

def sha() -> str:
    return subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()

def digest(path: str) -> str:
    h = hashlib.sha256()
    h.update(Path(path).read_bytes())
    return h.hexdigest()

def main() -> int:
    path = yaml.safe_load(Path(".ci/merge-path.yml").read_text())
    partials = [json.loads(p.read_text()) for p in sorted(Path(".ci/partials").glob("*.json"))]
    expected = [g["id"] for g in path["required_sequence"] if g["id"] != "merge-receipt"]
    seen = [p["gate"] for p in partials]
    if seen != expected:
        print(f"sequence mismatch: {seen} != {expected}", file=sys.stderr)
        return 1
    for gate, partial in zip(path["required_sequence"], partials):
        if not partial["ok"] or partial["flake_spent"] > gate["flake_budget"]:
            print(f"gate failed: {gate['id']}", file=sys.stderr)
            return 1
    receipt = {
        "head_sha": sha(),
        "fixture_lock_sha256": digest(path["fixture_lock"]),
        "path_version": path["version"],
        "gates": partials,
        "allow_merge": True,
    }
    out = Path(os.environ.get("RECEIPT_OUT", "merge-receipt.json"))
    out.write_text(json.dumps(receipt, indent=2) + "\n")
    print(out)
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire the jobs so order is real, not decorative.

# .github/workflows/merge-path.yml
name: merge-path
on:
  pull_request:
  push:
    branches: [main]

jobs:
  fixture-lock:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/hash-fixtures.sh
      - run: mkdir -p .ci/partials && python -c "import json,pathlib; pathlib.Path('.ci/partials/01-fixture-lock.json').write_text(json.dumps({'gate':'fixture-lock','flake_budget':0,'flake_spent':0,'ok':True}))"
      - uses: actions/upload-artifact@v4
        with:
          name: partials
          path: .ci/partials

  unit:
    needs: [fixture-lock]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/run_with_budget.py unit 0 python -m pytest tests/unit -q --maxfail=1
      - run: mkdir -p .ci/partials && mv receipt.partial.json .ci/partials/02-unit.json
      - uses: actions/upload-artifact@v4
        with:
          name: partials
          path: .ci/partials

  contract:
    needs: [unit]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/run_with_budget.py contract 1 python -m pytest tests/contract -q --maxfail=1
      - run: mkdir -p .ci/partials && mv receipt.partial.json .ci/partials/03-contract.json
      - uses: actions/upload-artifact@v4
        with:
          name: partials
          path: .ci/partials

  merge-receipt:
    needs: [contract]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: partials
          path: .ci/partials
      - run: pip install pyyaml
      - run: python scripts/write_receipt.py
      - uses: actions/upload-artifact@v4
        with:
          name: merge-receipt
          path: merge-receipt.json

Set branch protection to one required check: merge-receipt. Leave the other jobs visible. They feed the receipt. They do not, by themselves, open main.

Decision table you can paste into the PR template

What you observe	Merge?	Why
Unit is green, no receipt artifact	No	Sequence never closed
Receipt `head_sha` ≠ PR HEAD	No	Stale or replayed document
Fixture files changed, lock did not	No	Contract drift
Gate spent more flakes than budget	No	Instability was painted green
Jobs ran, but not in `required_sequence`	No	Extra color is not a path
Receipt matches path, SHA, lock, budgets	Yes	Permission is documented

Print the table in the PR body. Reviewers stop asking “is it green?” They ask “does the receipt match?”

Where generation happens does not change the path

You can write the patch by hand. You can also draft it in a workspace that offers free model access and a free server option. The merge path does not care which editor produced the diff. It cares whether fixtures stayed locked, whether the dry-run passed, and whether CI signed a matching receipt.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you use MonkeyCode for that draft step, treat it as a source of a patch, not as a substitute for merge-dry-run.sh or the receipt job. Generate, then hash fixtures, then run the path. Skip that order and you are back to a green dot with no permission document.

Limitations

This pattern assumes you can name a short, ordered sequence. A matrix of thirty OS and language jobs does not belong in the receipt as thirty gates. Fold the matrix behind one named gate, or the path file becomes noise.

The hook only protects people who push through Git. A force-push from the GitHub UI, a bot with admin, or a workflow that commits using GITHUB_TOKEN can bypass pre-push. Defend those routes separately, or the lock is theater.

The receipt is only as strong as branch protection. If merge-receipt is not required, the JSON is a log line. If required checks can be skipped by administrators every afternoon, you still have a social process, not a gate.

Flake budgets do not fix flakes. They bound how much instability you will paper over on the way to merge. A contract gate that spends its one retry every day needs a bug, not a larger budget.

Who should not use this

Do not install this on a repo with no fixtures and one smoke test. You will maintain YAML for no gain.

Do not use it as a way to let generated tests rewrite themselves. If the agent changes tests/fixtures and src in the same commit, the lock bump should be a human review item. Auto-writing the lock in CI defeats the hook.

Do not use it when your “CI” is a single vendor check you cannot split. The receipt needs partials. One opaque green square cannot be sequenced.

A working order you can run today

Add .ci/merge-path.yml with four gates and budgets of 0/0/1/0.
Hash current fixtures into tests/fixtures.lock with ./scripts/hash-fixtures.sh --write.
Point core.hooksPath at .githooks and push a no-op branch to prove the hook fires.
Run ./scripts/merge-dry-run.sh on a known-good commit, then on a commit that edits a fixture without updating the lock. The second run must fail.
Land the workflow. Make merge-receipt the only required check. Open a PR that skips unit and confirm merge stays blocked.
Only then send an agent-authored patch through the same path.

The work that remains after a model drafts code is still engineering work. You name the path. You freeze the fixtures. You count the retries. You merge on a receipt that matches the file in the repo. That is the green-to-merge path. Everything else is a color.

Pin the Check Name, Attest the SHA, Then Allow Merge

Emery Lin — Fri, 18 Sep 2026 08:09:43 +0000

A green check is not a merge decision. If an agent can add a similarly named workflow, or if you merge on any success, you will ship a SHA that never passed the gate you think you have. Pin the required check to a workflow digest, make CI write a merge attestation for that exact SHA, and refuse merge until both match.

You do not need a new platform to do this. You need a pin file, a hook that fails closed, and one required check whose name cannot drift.

Why agent diffs make name-only gates fail

Agent-written patches arrive faster than review capacity. That is useful. It is also how a yellow workflow becomes a fake green.

GitHub branch protection matches check names. Names are strings. Strings get copied. If protection says tests and an agent opens .github/workflows/tests-agent.yml with a job named tests, you can get a green mark that is not your harness.

Keep the merge rule boring. One check. One workflow file. One digest. Everything else is noise.

The contract

Three objects, one SHA:

A pin file that names the required check and identifies the workflow that produces it.
A CI job that runs fixtures, records flake retries, and writes merge-attestation.json.
A verifier that blocks push and merge when the attestation is missing, stale, or unbound to HEAD.

The agent owns the diff. CI owns green. The merge path only reads the attestation.

The scripts below are examples you can adapt. They are not a production audit of your GitHub org. Do not paste secrets into the attestation.

1. Pin the required check

Put the pin next to the workflow so a path change is a pin change.

.ci/merge-pin.json:

{
  "required_check_name": "merge-gate",
  "workflow_path": ".github/workflows/merge-gate.yml",
  "max_flake_retries": 1,
  "fixture_glob": "tests/fixtures/**",
  "attestation_path": ".ci/merge-attestation.json"
}

On GitHub, the required check string is not always the name: you put on the workflow. After the first run, copy the exact check name from the branch protection UI into required_check_name. If the UI shows merge-gate / merge-gate, pin that full string. Do not guess.

Hash the workflow in CI and in the hook with the same command. A stored digest that nobody updates is a false sense of safety. Compute it from the file on that SHA:

python3 - <<'PY'
import hashlib, pathlib
p = pathlib.Path(".github/workflows/merge-gate.yml")
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY

If that file moves, the pin is wrong. Fail closed.

2. Fail the push if the pin cannot be proven locally

Install a pre-push hook that refuses the push when HEAD cannot prove the pin. You are not running the full suite on every laptop. You are proving the pin still points at a real file, the check name is stable, and the fixture glob still matches something.

.githooks/pre-push (example):

#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import glob, json, pathlib, sys
pin = json.loads(pathlib.Path(".ci/merge-pin.json").read_text())
wf = pathlib.Path(pin["workflow_path"])
if not wf.is_file():
    print("merge pin points at missing workflow:", wf, file=sys.stderr)
    sys.exit(1)
if pin["required_check_name"] not in {"merge-gate", "merge-gate / merge-gate"}:
    print("required check name drifted:", pin["required_check_name"], file=sys.stderr)
    sys.exit(1)
files = glob.glob(pin["fixture_glob"], recursive=True)
if not files:
    print("fixture glob matched nothing; fail closed", file=sys.stderr)
    sys.exit(1)
print("fixtures pinned:", len(files))
PY

Enable it once:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push

If a contributor bypasses hooks, CI still has to refuse. The hook is a fast local filter, not the merge authority.

3. Make CI write the attestation

The required workflow is the only job allowed to declare green. Give the job the check name you pinned. Run fixtures. Retry at most once if the pin allows it. Then write an attestation bound to GITHUB_SHA.

.github/workflows/merge-gate.yml (example; pin third-party actions by commit SHA in production):

name: merge-gate
on:
  pull_request:
  push:
    branches: [main]
jobs:
  merge-gate:
    name: merge-gate
    runs-on: ubuntu-latest
    permissions:
      contents: read
    env:
      GIT_SHA: ${{ github.sha }}
    steps:
      - uses: actions/checkout@v4
      - name: Run fixture harness with one recorded retry
        run: |
          set -euo pipefail
          glob=$(python3 -c 'import json; print(json.load(open(".ci/merge-pin.json"))["fixture_glob"])')
          FLAKE_RETRIES_USED=0
          if ! python3 scripts/run_fixtures.py --glob "$glob"; then
            FLAKE_RETRIES_USED=1
            python3 scripts/run_fixtures.py --glob "$glob"
          fi
          echo "FLAKE_RETRIES_USED=$FLAKE_RETRIES_USED" >> "$GITHUB_ENV"
      - name: Write merge attestation
        if: success()
        run: python3 scripts/write_attestation.py
      - name: Verify attestation on this SHA
        if: success()
        run: python3 scripts/verify_attestation.py
      - name: Upload attestation
        if: success()
        uses: actions/upload-artifact@v4
        with:
          name: merge-attestation-${{ github.sha }}
          path: .ci/merge-attestation.json
          if-no-files-found: error

scripts/write_attestation.py (example):

#!/usr/bin/env python3
"""Example only. Writes a SHA-bound merge attestation."""
import glob, hashlib, json, os, pathlib, time

pin = json.loads(pathlib.Path(".ci/merge-pin.json").read_text())
wf = pathlib.Path(pin["workflow_path"]).read_bytes()
fixtures = sorted(glob.glob(pin["fixture_glob"], recursive=True))
digest = hashlib.sha256()
for path in fixtures:
    digest.update(pathlib.Path(path).read_bytes())

att = {
    "sha": os.environ["GIT_SHA"],
    "required_check_name": pin["required_check_name"],
    "workflow_path": pin["workflow_path"],
    "workflow_sha256": hashlib.sha256(wf).hexdigest(),
    "fixture_sha256": digest.hexdigest(),
    "fixture_count": len(fixtures),
    "flake_retries_used": int(os.environ.get("FLAKE_RETRIES_USED", "0")),
    "max_flake_retries": pin["max_flake_retries"],
    "written_at_unix": int(time.time()),
}
if att["flake_retries_used"] > att["max_flake_retries"]:
    raise SystemExit("flake retries exceeded pin")
if att["fixture_count"] == 0:
    raise SystemExit("no fixtures hashed; fail closed")
pathlib.Path(pin["attestation_path"]).write_text(json.dumps(att, indent=2) + "\n")
print("wrote", pin["attestation_path"])

Keep scripts/run_fixtures.py deterministic. If a fixture needs network, it does not belong in the merge gate. Record flake retries in the environment the writer reads. A silent rerun that never lands in the attestation is how you launder flakes into greens. If the second run fails, the job fails. No attestation. No merge.

4. Verify before merge

Branch protection should require exactly the pinned check. Then a merge helper downloads the artifact for that SHA and checks the fields.

scripts/verify_attestation.py (example):

#!/usr/bin/env python3
import hashlib, json, os, pathlib, sys

pin = json.loads(pathlib.Path(".ci/merge-pin.json").read_text())
att = json.loads(pathlib.Path(pin["attestation_path"]).read_text())
want_sha = os.environ.get("GIT_SHA", "")
errors = []
if not want_sha:
    errors.append("GIT_SHA is required")
if att.get("sha") != want_sha:
    errors.append(f"attestation sha {att.get('sha')} != GIT_SHA")
if att.get("required_check_name") != pin["required_check_name"]:
    errors.append("check name does not match pin")
wf_hash = hashlib.sha256(pathlib.Path(pin["workflow_path"]).read_bytes()).hexdigest()
if att.get("workflow_sha256") != wf_hash:
    errors.append("workflow digest does not match the file on this SHA")
if int(att.get("flake_retries_used", 99)) > pin["max_flake_retries"]:
    errors.append("flake retries exceed pin")
if errors:
    print("\n".join(errors), file=sys.stderr)
    sys.exit(1)
print("merge attestation ok for", att["sha"])

Call the verifier as the last step of merge-gate as well. If the writer and the verifier disagree, you want that failure on the PR, not at 17:00 on a Friday.

A maintainer helper can stay small:

GIT_SHA=$(git rev-parse HEAD)
# Download the merge-attestation-$GIT_SHA artifact from the merge-gate run, then:
GIT_SHA="$GIT_SHA" python3 scripts/verify_attestation.py && gh pr merge --merge

Do not gh pr merge on a green UI check alone.

Green-to-merge path

Run it in this order every time:

Human or agent opens a diff. The pin files stay out of that diff unless a human is reviewing the gate itself.
Pre-push hook confirms the workflow path, check name, and fixture glob.
merge-gate runs fixtures, records retries, writes the attestation, verifies it, and uploads it.
Branch protection requires only that check.
Merge helper re-reads the attestation for git rev-parse HEAD.
If any field drifts, you do not merge. You fix the pin or the fixtures.

That is the whole path. Chat transcripts are not an input.

Decision table

Signal	Merge?	Why
`merge-gate` green, attestation SHA matches HEAD, workflow digest matches pin	Yes	Gate and evidence agree
Some other check named `tests` is green	No	Name is not the pin
Attestation missing on the SHA	No	Green without evidence
Workflow file changed, digest mismatch	No	Pin broken; re-review the gate
`flake_retries_used` greater than the pin	No	Flakes were laundered
Attestation SHA is the parent commit	No	Evidence is for a different tree
Hook bypassed, CI still red	No	Hook is not authority
Pin and workflow edited in the same PR as product code	No	The gate changed itself

Paste the table into the PR template if people keep asking why merge is blocked.

Where a coding agent belongs on this path

The agent may draft the diff. It may even draft a new fixture. It does not choose the required check name, and it does not write the attestation. If you let the model edit .github/workflows/merge-gate.yml and .ci/merge-pin.json in the same PR, you no longer have a pin. Split those paths onto a review rule, or fail the gate when both change together.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already use a coding assistant to generate the patch, run the same scripts/run_fixtures.py command the gate will run. MonkeyCode's free model access and free server option can host that harness so the agent, your laptop, and CI are not inventing three different test commands. That is the only role it has here: a shared place to execute the fixture command you already pinned. It does not replace branch protection, and it does not get a vote on merge.

Limitations

This does not sign the runner image. A compromised hosted runner, an unpinned third-party action, or a pull_request_target workflow with write permissions can still lie. Pin actions by commit SHA before you treat the attestation as strong evidence.

The attestation is not SLSA provenance. It is a JSON file your own job wrote. That is enough to stop check-name impersonation and unbound greens. It is not enough for a regulated supply-chain audit.

Hooks are optional on the client. Fork PRs may not upload artifacts the way you expect. Secrets must never enter the attestation. Timestamp skew is not a security control.

If your suite is non-deterministic, a retry of one will not save you. Fix the fixtures. The pin's max_flake_retries is a fuse, not a reliability strategy.

Who should skip this

Skip it if you merge to main from a laptop with no branch protection. Skip it if one required check already is a merge queue with a SHA-locked policy you trust. Skip it if the repo has no fixtures worth hashing. Skip it if you wanted a product tour instead of a gate.

You wanted a green-to-merge path. Pin the check, attest the SHA, and let everything else wait.

Split CI Into a Merge Lane and a Soak Lane

Emery Lin — Thu, 17 Sep 2026 06:30:06 +0000

A single green check is a weak merge signal. Soak jobs, retries, and agent-drafted required steps should not share it. Split CI into two lanes. Put fixtures in a merge lane with zero retries. Park flakes and long soaks in an advisory lane. An agent may draft jobs. It does not rewrite the lane map.

That split is the whole contract. Everything below is how you enforce it.

What a mixed green check hides

You already know the failure mode. A pull request edits application code, a workflow file, and a test that used to flake. CI retries the flake. A soak job fails and is marked optional. The required job is green because the retry landed on a quiet runner. The merge button lights up.

None of that proved the SHA. It proved that noise and signal shared a status check. Agent-authored diffs make the mix worse. They add jobs quickly. They copy retry blocks from the last workflow that "worked." They promote a soak job to required because the prompt said "make CI pass."

You do not fix that with a longer test matrix. You fix it by making lane membership a reviewed file, not a side effect of YAML edits.

The two-lane contract

Keep two lanes, and only two.

Merge lane. Deterministic fixtures. No retries. Fail closed. A red job here blocks merge.
Soak lane. Load, timing, known flakes, long suites. Failures are visible. They do not vote on merge.

Unmapped jobs fail closed. If a workflow job is not in the lane map, treat it as merge-lane and fail the PR that introduced it. Promotion is a human edit to the map. Drafting a job is not promotion.

Label this as a proposed contract, not a claim about your current org. You can run it on a throwaway repo today.

Step 1: Write a lane map the workflows cannot ignore

Create ci/lanes.yml. Keep it boring. Boring files get reviewed.

# ci/lanes.yml
version: 1
merge:
  - unit-fixtures
  - contract-diff
  - lane-guard
soak:
  - nightly-load
  - flake-watch
rules:
  merge_retries: 0
  unmapped: fail-closed
  agent_may_edit_lanes: false

Job names in this file must match jobs.<id> in your workflows. If they drift, the guard fails. That is the point.

Step 2: Forbid retries on the merge lane

Retries belong to soak. Merge-lane retries convert flakes into greens. You want the opposite: a flake in the merge lane is a fixture bug or a product bug. Fix the fixture. Do not roll the dice.

# .github/workflows/merge-lane.yml
name: merge-lane
on:
  pull_request:
  push:
    branches: [main]

jobs:
  lane-guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python ci/check_lanes.py --strict

  unit-fixtures:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/fixtures -q

  contract-diff:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python ci/check_lanes.py --diff origin/${{ github.base_ref || 'main' }}

Soak stays separate. continue-on-error: true is allowed there. It is not allowed in merge-lane YAML. The guard parses both files.

# .github/workflows/soak-lane.yml
name: soak-lane
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  flake-watch:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/soak -q --maxfail=1

Step 3: Put the rules in a script you can run locally

CI should not be the first place the contract runs. Run the same file on the laptop and in the workflow. The script below is a complete, labeled example. It is not a production security scanner. It is a lane contract.

# ci/check_lanes.py
from __future__ import annotations

import argparse, subprocess, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("pip install pyyaml", file=sys.stderr)
    raise

ROOT = Path(__file__).resolve().parents[1]
LANES = ROOT / "ci" / "lanes.yml"
WORKFLOWS = ROOT / ".github" / "workflows"
AGENT_MARKERS = ("Generated-By:", "Co-authored-by: agent", "Agent-Commit: true")


def load_lanes():
    data = yaml.safe_load(LANES.read_text())
    merge = set(data.get("merge") or [])
    soak = set(data.get("soak") or [])
    overlap = merge & soak
    if overlap:
        raise SystemExit(f"job in both lanes: {sorted(overlap)}")
    return data, merge, soak


def workflow_jobs():
    jobs = {}
    for path in sorted(WORKFLOWS.glob("*.yml")):
        doc = yaml.safe_load(path.read_text()) or {}
        for name, body in (doc.get("jobs") or {}).items():
            jobs[name] = {"file": path.name, "body": body or {}}
    return jobs


def changed_files(base: str) -> set[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return {line.strip() for line in out.splitlines() if line.strip()}


def commit_message() -> str:
    return subprocess.check_output(["git", "log", "-1", "--pretty=%B"], text=True)


def has_retry(body: dict) -> bool:
    text = yaml.safe_dump(body)
    return ("max-attempts" in text) or ("retry" in text and "retry-on" in text)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--strict", action="store_true")
    parser.add_argument("--diff", default="")
    args = parser.parse_args()

    data, merge, soak = load_lanes()
    jobs = workflow_jobs()
    errors = []

    mapped = merge | soak
    extra = set(jobs) - mapped
    missing = mapped - set(jobs)
    if extra:
        errors.append(f"unmapped jobs fail closed: {sorted(extra)}")
    if missing:
        errors.append(f"lane map names missing jobs: {sorted(missing)}")

    for name in merge:
        body = (jobs.get(name) or {}).get("body") or {}
        if body.get("continue-on-error") is True:
            errors.append(f"merge job {name} cannot continue-on-error")
        if has_retry(body):
            errors.append(f"merge job {name} cannot retry")
        if int(data.get("rules", {}).get("merge_retries", 0)) != 0:
            errors.append("rules.merge_retries must be 0")

    if args.diff:
        changed = changed_files(args.diff)
        lanes_touched = "ci/lanes.yml" in changed
        msg = commit_message()
        agent = any(m.lower() in msg.lower() for m in AGENT_MARKERS)
        if lanes_touched and agent and not data["rules"].get("agent_may_edit_lanes"):
            errors.append("agent commit cannot edit ci/lanes.yml")

    if errors:
        print("lane contract failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"lane contract ok: {len(merge)} merge, {len(soak)} soak")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Install PyYAML in the job and locally. Then run the same command in both places.

pip install pyyaml
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

Step 4: Add a hook so the laptop fails before the PR exists

A hook is cheaper than a round trip through GitHub. Keep it thin. Call the same script.

# .git/hooks/pre-push  (or a tracked script that installs it)
#!/bin/sh
set -e
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

chmod +x .git/hooks/pre-push

If your team hates client hooks, run lane-guard as a required check and refuse to merge without it. The hook is for speed. The required check is for truth.

Step 5: Classify every new job before you write assertions

When a PR wants a new job, fill this table in the description. If the row is blank, the lane guard should already fail. The table is for humans. The YAML is for the script.

Job id	Lane	Retry allowed	Blocks merge	Who may edit the lane row
unit-fixtures	merge	no	yes	human
contract-diff	merge	no	yes	human
lane-guard	merge	no	yes	human
flake-watch	soak	yes	no	human
nightly-load	soak	yes	no	human

Promotion is a row change in ci/lanes.yml, not a comment in the PR. If the agent drafted a soak job that looks stable, a person moves the name from soak to merge after the fixture is deterministic. Not before.

Where a free coding agent belongs

Drafting fixtures is useful. Promoting jobs is not. Keep the agent in the directories that feed the merge lane: tests/fixtures, golden files, and small contract tests. Keep it out of ci/lanes.yml.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option. That pairing matters here for one reason. You can generate fixture drafts on the free server, pull them into a worktree, and run python ci/check_lanes.py --strict before GitHub ever sees the branch. The server does not become CI. CI still rebuilds the merge lane. The agent still does not vote.

Point the agent at a failing fixture, not at the workflow that marks the job required. If it rewrites .github/workflows and ci/lanes.yml in one commit, the guard should fail. That is a successful contract, not a rude model.

If you already have a local agent session, use it to propose soak tests. Leave lane promotion for review. One place to start is the project's own repo when you want that draft loop without standing up extra hardware.

A 20-minute local drill

Do this on a throwaway clone. Do not run it against production secrets.

Copy ci/lanes.yml, ci/check_lanes.py, and the two workflow files into an empty repo.
Add a passing tests/fixtures/test_ok.py and a soak test that fails half the time.
Run python ci/check_lanes.py --strict. It should print lane contract ok.
Add retry YAML to unit-fixtures. Run the script again. It should fail.
Commit with a body line Generated-By: demo-agent and edit ci/lanes.yml to move flake-watch into merge. Run --diff origin/main. It should fail.
Revert the lane edit. Keep the soak failure. Open a PR. Merge lane stays green. Soak stays visible and non-blocking.

If step 4 or 5 stays green, the contract is not installed. Fix the script path before you trust the badge.

Limitations, and who should skip this

This does not detect every retry mechanism. GitHub Actions has more retry styles than one string scan. Extend the parser if you use a reusable workflow or a third-party retry action. Until you do, assume the guard is incomplete.

It also does not prove tests are deterministic. Zero retries only expose flakes. You still need fixtures that do not sleep on the wall clock. Soak-lane continue-on-error can hide a broken product if nobody reads the advisory job. Name an owner for soak failures or they become wallpaper.

Skip this approach if you ship a single workflow with one job, or if every suite is already short and deterministic. Skip it if your merge queue requires one check name and you cannot split required versus advisory. Do not use it as a substitute for secret scanning or workflow review. Lane maps do not make untrusted YAML safe.

The merge button should mean the merge lane passed once, on this SHA, with no retries. Anything else is a soak result. Keep the agent in the fixture directory. Keep the lane file in human hands.

Split CI Into a Merge Lane and a Soak Lane

Emery Lin — Thu, 17 Sep 2026 06:24:31 +0000

That split is the whole contract. Everything below is how you enforce it.

What a mixed green check hides

You do not fix that with a longer test matrix. You fix it by making lane membership a reviewed file, not a side effect of YAML edits.

The two-lane contract

Keep two lanes, and only two.

Merge lane. Deterministic fixtures. No retries. Fail closed. A red job here blocks merge.
Soak lane. Load, timing, known flakes, long suites. Failures are visible. They do not vote on merge.

Unmapped jobs fail closed. If a workflow job is not in the lane map, treat it as merge-lane and fail the PR that introduced it. Promotion is a human edit to the map. Drafting a job is not promotion.

Label this as a proposed contract, not a claim about your current org. You can run it on a throwaway repo today.

Step 1: Write a lane map the workflows cannot ignore

Create ci/lanes.yml. Keep it boring. Boring files get reviewed.

# ci/lanes.yml
version: 1
merge:
  - unit-fixtures
  - contract-diff
  - lane-guard
soak:
  - nightly-load
  - flake-watch
rules:
  merge_retries: 0
  unmapped: fail-closed
  agent_may_edit_lanes: false

Job names in this file must match jobs.<id> in your workflows. If they drift, the guard fails. That is the point.

Step 2: Forbid retries on the merge lane

Retries belong to soak. Merge-lane retries convert flakes into greens. You want the opposite: a flake in the merge lane is a fixture bug or a product bug. Fix the fixture. Do not roll the dice.

# .github/workflows/merge-lane.yml
name: merge-lane
on:
  pull_request:
  push:
    branches: [main]

jobs:
  lane-guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python ci/check_lanes.py --strict

  unit-fixtures:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/fixtures -q

  contract-diff:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python ci/check_lanes.py --diff origin/${{ github.base_ref || 'main' }}

Soak stays separate. continue-on-error: true is allowed there. It is not allowed in merge-lane YAML. The guard parses both files.

# .github/workflows/soak-lane.yml
name: soak-lane
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  flake-watch:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/soak -q --maxfail=1

Step 3: Put the rules in a script you can run locally

# ci/check_lanes.py
from __future__ import annotations

import argparse, subprocess, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("pip install pyyaml", file=sys.stderr)
    raise

ROOT = Path(__file__).resolve().parents[1]
LANES = ROOT / "ci" / "lanes.yml"
WORKFLOWS = ROOT / ".github" / "workflows"
AGENT_MARKERS = ("Generated-By:", "Co-authored-by: agent", "Agent-Commit: true")


def load_lanes():
    data = yaml.safe_load(LANES.read_text())
    merge = set(data.get("merge") or [])
    soak = set(data.get("soak") or [])
    overlap = merge & soak
    if overlap:
        raise SystemExit(f"job in both lanes: {sorted(overlap)}")
    return data, merge, soak


def workflow_jobs():
    jobs = {}
    for path in sorted(WORKFLOWS.glob("*.yml")):
        doc = yaml.safe_load(path.read_text()) or {}
        for name, body in (doc.get("jobs") or {}).items():
            jobs[name] = {"file": path.name, "body": body or {}}
    return jobs


def changed_files(base: str) -> set[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return {line.strip() for line in out.splitlines() if line.strip()}


def commit_message() -> str:
    return subprocess.check_output(["git", "log", "-1", "--pretty=%B"], text=True)


def has_retry(body: dict) -> bool:
    text = yaml.safe_dump(body)
    return ("max-attempts" in text) or ("retry" in text and "retry-on" in text)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--strict", action="store_true")
    parser.add_argument("--diff", default="")
    args = parser.parse_args()

    data, merge, soak = load_lanes()
    jobs = workflow_jobs()
    errors = []

    mapped = merge | soak
    extra = set(jobs) - mapped
    missing = mapped - set(jobs)
    if extra:
        errors.append(f"unmapped jobs fail closed: {sorted(extra)}")
    if missing:
        errors.append(f"lane map names missing jobs: {sorted(missing)}")

    for name in merge:
        body = (jobs.get(name) or {}).get("body") or {}
        if body.get("continue-on-error") is True:
            errors.append(f"merge job {name} cannot continue-on-error")
        if has_retry(body):
            errors.append(f"merge job {name} cannot retry")
        if int(data.get("rules", {}).get("merge_retries", 0)) != 0:
            errors.append("rules.merge_retries must be 0")

    if args.diff:
        changed = changed_files(args.diff)
        lanes_touched = "ci/lanes.yml" in changed
        msg = commit_message()
        agent = any(m.lower() in msg.lower() for m in AGENT_MARKERS)
        if lanes_touched and agent and not data["rules"].get("agent_may_edit_lanes"):
            errors.append("agent commit cannot edit ci/lanes.yml")

    if errors:
        print("lane contract failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"lane contract ok: {len(merge)} merge, {len(soak)} soak")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Install PyYAML in the job and locally. Then run the same command in both places.

pip install pyyaml
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

Step 4: Add a hook so the laptop fails before the PR exists

A hook is cheaper than a round trip through GitHub. Keep it thin. Call the same script.

# .git/hooks/pre-push  (or a tracked script that installs it)
#!/bin/sh
set -e
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

chmod +x .git/hooks/pre-push

If your team hates client hooks, run lane-guard as a required check and refuse to merge without it. The hook is for speed. The required check is for truth.

Step 5: Classify every new job before you write assertions

When a PR wants a new job, fill this table in the description. If the row is blank, the lane guard should already fail. The table is for humans. The YAML is for the script.

Job id	Lane	Retry allowed	Blocks merge	Who may edit the lane row
unit-fixtures	merge	no	yes	human
contract-diff	merge	no	yes	human
lane-guard	merge	no	yes	human
flake-watch	soak	yes	no	human
nightly-load	soak	yes	no	human

Where a free coding agent belongs

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A 20-minute local drill

Do this on a throwaway clone. Do not run it against production secrets.

Copy ci/lanes.yml, ci/check_lanes.py, and the two workflow files into an empty repo.
Add a passing tests/fixtures/test_ok.py and a soak test that fails half the time.
Run python ci/check_lanes.py --strict. It should print lane contract ok.
Add retry YAML to unit-fixtures. Run the script again. It should fail.
Commit with a body line Generated-By: demo-agent and edit ci/lanes.yml to move flake-watch into merge. Run --diff origin/main. It should fail.
Revert the lane edit. Keep the soak failure. Open a PR. Merge lane stays green. Soak stays visible and non-blocking.

If step 4 or 5 stays green, the contract is not installed. Fix the script path before you trust the badge.

Limitations, and who should skip this

The merge button should mean the merge lane passed once, on this SHA, with no retries. Anything else is a soak result. Keep the agent in the fixture directory. Keep the lane file in human hands.

CI Rebuilds the Test Matrix. The Agent Does Not Get a Vote.

Emery Lin — Wed, 16 Sep 2026 04:22:40 +0000

You should not merge an agent patch because the agent said the tests passed. The merge queue has to rebuild the fixture set from the paths that actually changed. Local green is a hint. It is not a ticket.

This matters more now that generated diffs show up in ordinary review queues. A model can emit a plausible Tested-with: trailer, a tidy fixture list, and a green local run. None of that is evidence CI can reuse. Treat the agent's test list as untrusted input, the same way you treat a user-supplied filename.

What goes wrong if you import the list

An agent optimizes for the run it just did. It will drop fixtures that look expensive. It will keep the one test that already passed. It will retry a flaky file until the LED turns green, then hand you that retry count as if it were policy.

You cannot see that from a check mark. You only see a matrix that was chosen by the patch author. When the author is a model, that matrix is part of the patch. It is not a review of the patch.

So split the jobs. Local hooks record paths. Humans own the path-to-fixture map. CI recomputes the matrix at the merge SHA. Flake retries live in a ledger CI writes, not in the PR body.

The selector split

Use two sources of truth, and keep them apart.

Path record (untrusted author, trusted git). git diff --name-only against the merge base. This is the only agent-adjacent signal you keep.
Fixture map (human-owned). A YAML file on the default branch. Agents may read it. They may not edit it in the same commit that they change product code.
Recomputed matrix (CI-owned). A script CI runs on the candidate SHA. Its stdout is the only matrix the merge job is allowed to consume.
Flake ledger (CI-owned). Retry counts keyed by fixture id and SHA. Local retries do not transfer.

If a job reads fixtures from a git trailer, a PR label, or a comment, that job is misconfigured. Fail the workflow for that reason alone.

Artifact: recompute from paths, never from trailers

The following is a proposed workflow you can copy. It is not a production benchmark. Label it as a contract you run on a throwaway branch first.

Create ci/fixture-map.yml. Keep it small and boring.

# ci/fixture-map.yml — humans edit this; agents do not in product commits
version: 1
rules:
  - paths: ["src/billing/**"]
    fixtures: ["billing-unit", "billing-contract"]
  - paths: ["src/checkout/**"]
    fixtures: ["checkout-unit", "checkout-http"]
  - paths: ["src/shared/**", "pkg/**"]
    fixtures: ["shared-unit"]
  - paths: [".github/workflows/**", "ci/**"]
    fixtures: ["workflow-lint", "fixture-map-lint"]
require_map_hit: true
max_fixtures: 12

Then add ci/recompute_fixtures.py. It accepts a changed-file list. It refuses to read claimed fixtures.

#!/usr/bin/env python3
"""Recompute CI fixtures from changed paths. Proposed helper, not a product."""
from __future__ import annotations

import argparse
import fnmatch
import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("install pyyaml before running this helper", file=sys.stderr)
    sys.exit  = 2  # type: ignore
    raise

FORBIDDEN_FLAGS = ("--claimed", "--agent-list", "--pr-body")


def load_map(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict) or "rules" not in data:
        raise SystemExit("fixture map missing rules")
    return data


def match_fixtures(changed: list[str], rules: list[dict]) -> tuple[list[str], list[str]]:
    selected: list[str] = []
    unmatched: list[str] = []
    for rel in changed:
        hits = []
        for rule in rules:
            if any(fnmatch.fnmatch(rel, pat) for pat in rule.get("paths", [])):
                hits.extend(rule.get("fixtures", []))
        if hits:
            selected.extend(hits)
        else:
            unmatched.append(rel)
    # stable unique
    seen = set()
    ordered = []
    for name in selected:
        if name not in seen:
            seen.add(name)
            ordered.append(name)
    return ordered, unmatched


def main() -> int:
    for flag in sys.argv:
        if flag.startswith(FORBIDDEN_FLAGS):
            print(f"refusing untrusted fixture source: {flag}", file=sys.stderr)
            return 2

    parser = argparse.ArgumentParser()
    parser.add_argument("--map", required=True)
    parser.add_argument("--changed-file", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()

    changed = [line.strip() for line in Path(args.changed_file).read_text().splitlines() if line.strip()]
    data = load_map(Path(args.map))
    fixtures, unmatched = match_fixtures(changed, data["rules"])

    if data.get("require_map_hit") and unmatched:
        print("unmapped paths (informational; do not import an agent list to cover them):", file=sys.stderr)
        for rel in unmatched:
            print(f"  {rel}", file=sys.stderr)
        # Fail because the map is incomplete, not because the agent omitted a test.
        return 1

    limit = int(data.get("max_fixtures", 99))
    if len(fixtures) > limit:
        print(f"recomputed matrix too large: {len(fixtures)} > {limit}", file=sys.stderr)
        return 1

    payload = {
        "selector": "ci-recompute",
        "fixtures": fixtures,
        "changed": changed,
        "unmatched": unmatched,
    }
    Path(args.out).write_text(json.dumps(payload, indent=2) + "\n")
    print(json.dumps(payload["fixtures"]))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Fix the accidental sys.exit = 2 line if you paste carelessly. The import failure should be sys.exit(2) followed by raise. The point of the script is the refusal list: there is no flag that accepts the agent's fixtures.

Numbered path from clone to merge

Run this as a drill on a feature branch. Do not point it at main until the map covers your tree.

Classify the patch. Require a trailer Patch-Class: agent or Patch-Class: human on every commit that touches src/. Missing class fails the hook. The class does not choose fixtures. It only chooses how loud the logs are.
Record paths only. In a pre-push hook, write git diff --name-only origin/main...HEAD to .ci/changed.txt. Strip any Claimed-Fixtures, Tested-with, or Local-Green trailers before the push proceeds.
Block map edits beside product edits. If ci/fixture-map.yml and src/ both appear in the same diff, fail. Map changes need their own review, because they change what CI is allowed to skip.
CI recomputes. The first job checks out the candidate SHA, runs recompute_fixtures.py, and uploads matrix.json as an artifact. Later jobs consume that artifact. They do not parse the PR body.
CI owns flake. Each fixture may retry at most N times, recorded as fixture_id + sha + attempt. A local retry count in the commit message is ignored. If the ledger would exceed N, the fixture fails. It does not get a silent extra attempt because a model already burned retries on a laptop.
Promote only the recomputed green. Merge is allowed when the recomputed matrix is green at that SHA. A green local run with a different matrix is a comment, not a gate.

A minimal hook sketch:

#!/usr/bin/env bash
set -euo pipefail
# proposed pre-push: paths in, claimed fixtures out
range="origin/main...HEAD"
git diff --name-only "$range" > .ci/changed.txt

if git log --format=%B "$range" | grep -Eiq '^(Claimed-Fixtures|Tested-with|Local-Green):'; then
  echo "strip claimed fixture trailers; CI will recompute" >&2
  exit 1
fi

if git diff --name-only "$range" | grep -q '^src/' ; then
  if ! git log --format=%B "$range" | grep -q '^Patch-Class: '; then
    echo "add Patch-Class: agent  or  Patch-Class: human" >&2
    exit 1
  fi
fi

map_changed=$(git diff --name-only "$range" | grep -c '^ci/fixture-map.yml$' || true)
src_changed=$(git diff --name-only "$range" | grep -c '^src/' || true)
if [[ "$map_changed" -gt 0 && "$src_changed" -gt 0 ]]; then
  echo "do not edit the fixture map in the same commit as product code" >&2
  exit 1
fi

Proposed CI job (GitHub Actions). Notice the matrix file is an artifact, not a parameter from the PR.

name: recompute-matrix
on:
  pull_request:
jobs:
  select:
    runs-on: ubuntu-latest
    outputs:
      fixtures: ${{ steps.pack.outputs.fixtures }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: list changed paths from git, not from the PR body
        run: |
          mkdir -p .ci
          git diff --name-only origin/${{ github.base_ref }}...HEAD > .ci/changed.txt
      - name: refuse imported lists
        run: |
          if git log --format=%B origin/${{ github.base_ref }}...HEAD | grep -Eiq '^(Claimed-Fixtures|Tested-with):'; then
            echo "untrusted fixture list present; strip it and rerun" >&2
            exit 1
          fi
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - id: pack
        run: |
          python ci/recompute_fixtures.py \
            --map ci/fixture-map.yml \
            --changed-file .ci/changed.txt \
            --out .ci/matrix.json
          echo "fixtures=$(python -c 'import json; print(json.dumps(json.load(open(".ci/matrix.json"))["fixtures"]))')" >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with:
          name: recomputed-matrix
          path: .ci/matrix.json

  run-fixtures:
    needs: select
    if: needs.select.outputs.fixtures != '[]'
    strategy:
      fail-fast: false
      matrix:
        fixture: ${{ fromJSON(needs.select.outputs.fixtures) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: run one CI-selected fixture with a CI-owned retry ceiling
        env:
          FLAKE_MAX: "1"
        run: ./ci/run-fixture.sh "${{ matrix.fixture }}"

./ci/run-fixture.sh is yours. Keep FLAKE_MAX in CI. Do not read it from the commit.

Decision table: what may gate merge

Source	Show on the PR?	Gate merge?	Why
`git diff` paths at candidate SHA	Yes	Indirectly, via recompute	Paths are observed, not claimed
`ci/fixture-map.yml` at that SHA	Yes	Yes	Human-owned selector
Agent / local fixture list	At most as a comment	No	Author-controlled
PR body, labels, review comments	No	No	Trivial to spoof
Local flake retries	No	No	Burns budget off-ledger
Prior pipeline on another SHA	No	No	Wrong tree
Recomputed matrix job on this SHA	Yes	Yes	Only selector the queue trusts

Print this table in the workflow summary if you want reviewers to stop arguing with the agent log.

Where a free coding environment fits — and where it does not

You can generate the patch on a laptop, in a container, or on a hosted box. The merge rule does not change. The environment that wrote the diff does not select fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to draft a candidate patch and run a local smoke command. That is a writing environment. It is not your merge queue. Do not point the recompute job at that server. Do not import its logs as Claimed-Fixtures. If you try it, keep the output on a branch and let CI rebuild the matrix from ci/fixture-map.yml at the SHA you intend to merge.

No model names, quotas, or hardware claims are required for this workflow. The workflow only needs git, a map, and a job that refuses extra flags.

Flake control without inheriting the agent's budget

Agents retry. That is their loop. If you copy their retry count into CI, you launder flake into policy.

Cap CI at a small integer. One extra attempt is a reasonable default for known-noisy fixtures. Zero extra attempts is the default for everything else. Write each attempt as JSONL:

{"sha":"abc123","fixture":"checkout-http","attempt":1,"result":"fail","reason":"timeout"}
{"sha":"abc123","fixture":"checkout-http","attempt":2,"result":"pass","reason":"replay"}

If attempt 2 would exceed FLAKE_MAX, stop. Do not look at the commit message to see whether the agent already "proved" it was flaky. Proof of flake is a ticket against the fixture, opened by a human, not a trailer on the patch that needs the fixture to go green.

Limitations

Path maps lie when behavior changes without a path change. A shared library tweak can break billing without touching src/billing/**. If that is your architecture, the map must send pkg/** and src/shared/** through the heavier fixtures, or you run a scheduled full suite that is not part of the merge gate.

Dynamic test discovery (collect whole trees at runtime, then shard) does not need this script. Recompute becomes a no-op if your only matrix is "run everything." In that case the untrusted-list problem shrinks, and you should spend the effort on SHA pinning and flake ledgers instead.

The hook that forbids map edits beside product edits will annoy legitimate refactors. Split those into two PRs. If your team will not do that, the map will rot and CI will fail closed on unmatched paths.

This article does not claim the recompute step catches semantic bugs the fixtures miss. It claims something narrower: the agent does not get to shrink the gate.

Who should not use this

Do not install this if you have no agent traffic and every PR already runs the full suite. You will add YAML without changing risk.

Do not use it as a substitute for code review of .github/workflows/**. A workflow that still reads the PR body can bypass the script.

Do not use it on a repo where ci/fixture-map.yml is generated by the same model that writes src/. That collapses the selector split.

Do not treat a free remote coding box as a self-hosted runner for the merge queue. Different trust domain. Different secrets. Different failure mode.

What to do on the next agent PR

Take one open patch. Write .ci/changed.txt from git diff --name-only. Run the recompute script. Compare the printed fixtures with whatever the agent listed. If the lists differ, keep CI's list. If they match, you still run CI's list, because matching is not the same as importing.

That is the whole rule. The agent can write code. It does not choose the exam.

Park Agent Diffs in a Replay Worktree Before CI Starts

Emery Lin — Sun, 13 Sep 2026 20:35:54 +0000

An agent-authored patch should not be allowed to start your paid CI. Local green is not a ticket. A hermetic replay against frozen fixtures, written to a report your workflow can reject, is the ticket.

You already know the failure. A model emits a plausible diff. You run two tests on your laptop. You open the pull request. GitHub Actions then burns minutes on a change that never would have survived a frozen fixture pack. The patch is not evil. It is just unreplayed.

This article gives you a small gate for that gap. You park the diff in a throwaway worktree, run only the fixtures the diff claims to touch, and emit replay_report.json. CI reads the report before it schedules anything expensive. If the report is missing, stale, or hashed against the wrong tree, the workflow stops. No merge-queue lore. No second lane. Just a report that must exist before CI starts.

The failure you are gating

Agent patches fail in boring ways. They edit the test instead of the code. They miss the path the fixture encodes. They rename a helper and leave the caller on the old symbol. None of that needs a GPU. It needs a replay you can hash.

Paid CI is the wrong place to discover those misses. Runners are shared. Caches lie. Secrets are in scope. You want the cheap, secret-free check to happen first, on a worktree that dies when the script exits.

Treat the model as an untrusted patch source. You would not merge a zip file from the internet because a unit test passed on someone's laptop. Do not treat an agent diff as stronger evidence than that zip file.

What the report must contain

Keep the contract tiny. If a field does not help you reject a run, drop it.

{
  "schema": "replay-report/v1",
  "git_head": "4f2c91a0e1b7d3aa",
  "patch_sha256": "b7c1…",
  "worktree": "/tmp/replay-4f2c91a0",
  "fixtures": [
    {"id": "cart.total.v3", "path": "fixtures/cart/total.json", "sha256": "aa19…"},
    {"id": "cart.tax.v1", "path": "fixtures/cart/tax.json", "sha256": "0c44…"}
  ],
  "commands": [
    {"cmd": "python -m pytest tests/test_cart.py -q", "exit": 0}
  ],
  "result": "pass",
  "created_at": "2026-09-14T12:04:11Z"
}

git_head pins the base. patch_sha256 pins the bytes you applied. Each fixture hash pins the input the test actually saw. CI will recompute those three and compare. If any drift, the job fails closed.

Do not put model prompts, API keys, or hostnames in this file. The report is an admission ticket, not a chat log.

1. Freeze a fixture manifest

Start with the files your tests already read. You do not need a new framework. You need hashes.

# tools/hash_fixtures.sh
set -euo pipefail
manifest="fixtures/manifest.json"
python - <<'PY'
import hashlib, json, pathlib, sys
rows = []
root = pathlib.Path("fixtures")
for path in sorted(root.rglob("*.json")):
    if path.name == "manifest.json":
        continue
    digest = hashlib.sha256(path.read_bytes()).hexdigest()
    rows.append({"id": path.stem, "path": str(path), "sha256": digest})
pathlib.Path("fixtures/manifest.json").write_text(json.dumps({"fixtures": rows}, indent=2) + "\n")
print(f"wrote {len(rows)} fixture hashes", file=sys.stderr)
PY

Commit fixtures/manifest.json. When someone edits a fixture, the manifest changes, and every old replay report becomes invalid. That is the point. Replay is not a vibe. It is a hash match.

If a test reads a path that is not in the manifest, fail the replay. Unmapped inputs are how silent skips sneak in. You do not need a long policy document. One missing hash is enough to stop the run.

2. Apply the patch in a disposable worktree

Never replay on your dirty checkout. You will mix agent bytes with leftover edits and then trust the result. Use a worktree, apply the patch, run the listed tests, write the report, then remove the worktree.

# tools/replay_agent_patch.sh
set -euo pipefail
patch_file="${1:?usage: replay_agent_patch.sh <patch>}"
head_sha="$(git rev-parse --short=16 HEAD)"
work="/tmp/replay-${head_sha}-$$"
report="replay_report.json"

git worktree add --detach "$work" HEAD
cleanup() { git worktree remove --force "$work" 2>/dev/null || rm -rf "$work"; }
trap cleanup EXIT

patch_sha="$(python - <<PY
import hashlib, pathlib
print(hashlib.sha256(pathlib.Path("$patch_file").read_bytes()).hexdigest())
PY
)"

(cd "$work" && git apply --check "$patch_file" && git apply "$patch_file")

# Fail closed if the patch touches files with no fixture mapping.
python tools/check_diff_to_fixtures.py "$work" "$patch_file" fixtures/manifest.json

set +e
(cd "$work" && python -m pytest tests/test_cart.py -q)
exit_code=$?
set -e

python tools/write_replay_report.py \
  --head "$head_sha" \
  --patch-sha "$patch_sha" \
  --worktree "$work" \
  --manifest fixtures/manifest.json \
  --cmd "python -m pytest tests/test_cart.py -q" \
  --exit "$exit_code" \
  --out "$report"

test "$exit_code" -eq 0

The script is boring on purpose. Boring scripts get copied. Clever wrappers get skipped when someone is in a hurry.

Label the mapping checker as a proposal if your repo does not yet have path rules. A minimal version is enough to start:

# tools/check_diff_to_fixtures.py
"""Fail if the patch touches src/cart but no fixtures/cart/* hash exists."""
import pathlib, json, subprocess, sys

work, patch, manifest_path = sys.argv[1], sys.argv[2], sys.argv[3]
manifest = json.loads(pathlib.Path(manifest_path).read_text())
mapped_dirs = {pathlib.Path(row["path"]).parent.as_posix() for row in manifest["fixtures"]}

diff = subprocess.check_output(["git", "apply", "--numstat", patch], text=True)
changed = []
for line in diff.splitlines():
    parts = line.split()
    if len(parts) >= 3:
        changed.append(parts[-1])

for path in changed:
    if not path.startswith("src/"):
        continue
    expected = "fixtures/" + "/".join(pathlib.Path(path).parts[1:-1])
    hits = [d for d in mapped_dirs if d.startswith(expected) or expected.startswith(d)]
    if not hits:
        raise SystemExit(f"unmapped path {path}; no fixture under {expected}")

You can replace the src/ prefix with whatever layout you use. The rule is the same: a changed production path must name a hashed fixture directory, or the replay never writes result: pass.

3. Make CI refuse work without the report

The workflow does one thing first. It verifies the report. Only then does it run the suite you actually pay for.

# .github/workflows/require-replay.yml
name: require-replay
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  replay-ticket:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 1
      - name: Require replay_report.json
        run: |
          test -f replay_report.json
          python tools/verify_replay_report.py replay_report.json
      - name: Paid tests (only after replay)
        run: python -m pytest -q

verify_replay_report.py should recompute, not trust:

# tools/verify_replay_report.py
import hashlib, json, pathlib, subprocess, sys

report = json.loads(pathlib.Path(sys.argv[1]).read_text())
if report.get("schema") != "replay-report/v1":
    raise SystemExit("unknown replay schema")
if report.get("result") != "pass":
    raise SystemExit("replay did not pass")

head = subprocess.check_output(["git", "rev-parse", "--short=16", "HEAD"], text=True).strip()
if report["git_head"] != head:
    raise SystemExit(f"report head {report['git_head']} != {head}")

manifest = json.loads(pathlib.Path("fixtures/manifest.json").read_text())
wanted = {row["path"]: row["sha256"] for row in manifest["fixtures"]}
for row in report["fixtures"]:
    on_disk = hashlib.sha256(pathlib.Path(row["path"]).read_bytes()).hexdigest()
    if wanted.get(row["path"]) != row["sha256"] or on_disk != row["sha256"]:
        raise SystemExit(f"fixture drift: {row['path']}")
print("replay report ok")

Commit the report with the pull request, or upload it as a workflow artifact from a pre-job you control. Either way, the paid job must not start on a missing file. A skipped step is not a pass.

4. Decide what belongs in the replay

Not every test earns a place in the worktree. Use a table, not a meeting.

Kind of check	In the replay?	Why
Pure unit test with a JSON fixture	Yes	Hashable, secret-free, fast
Contract test against a recorded HTTP cassette	Yes, if the cassette is in the manifest	Replay needs the same bytes
Live staging call	No	Not hermetic; do not hash the network
Job that needs production secrets	No	The worktree should not see those secrets
UI screenshot diff	Only if the baseline image is hashed	Otherwise the report cannot fail closed
Flaky integration suite	No	A replay that needs retries is not a ticket

If a check cannot be hashed, it does not belong in replay_report.json. Put it behind the report, not inside it.

Where a spare server actually helps

The harness above runs on a laptop. Many agent patches still waste the next hop: you regenerate the diff, forget to replay, and push anyway. A small always-on box that only runs replay_agent_patch.sh removes that skip. You drop a patch on the box. You get a report or a rejection. You do not get a GitHub Actions bill for the misses.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a place to host that loop without standing up your own runner, MonkeyCode's free model access and free server option can hold the replay harness and regenerate a patch when the report comes back result: fail. Keep your schema. Do not outsource the hash check. The product is a spare environment, not a replacement for verify_replay_report.py.

Use the model only after a failed replay, with the report as the prompt context: which fixture hash failed, which command exited nonzero, which path was unmapped. A regenerated patch that cannot produce a new report still does not start CI. That is the whole policy.

Limitations

This gate does not review design. A replay can pass on a patch that deletes a feature you still need. Humans still read the diff.

It does not replace branch protection. If someone can force-push and rewrite replay_report.json without running the script, you have a process hole, not a tooling hole. Protect the workflow file. Require the verify job.

It does not make non-deterministic tests honest. If pytest talks to the clock, the network, or a shared database, the report will flap and you will start ignoring it. Ignore the report once, and the gate is gone. Keep replay on hashed inputs only.

It also does not prove the patch is the patch GitHub will merge. Rebase, squash, and late review comments all change bytes. After those events, run the script again. An old report on a new head is a failed verify, which is the correct outcome.

Who should not use this

Skip this if your pull requests never include model-authored hunks and your laptop tests already match CI. You would be hashing ceremony.

Skip it if your only tests are live end-to-end runs against shared staging. You cannot freeze those inputs, so you cannot write a report worth verifying. Fix the test shape first.

Skip it if you cannot keep secrets out of the worktree. A replay box that mounts production credentials is a new incident, not a cheaper CI.

If you do use it, start with one package and two fixtures. Expand the manifest when a missed path burns you, not before. The merge path does not care which laptop produced the patch. It cares that the replay hash still matches.

Give the Merge Queue a SHA-Bound Ticket

Emery Lin — Sat, 12 Sep 2026 18:37:22 +0000

A green job is a boolean. Merge eligibility is a contract. If an agent-authored diff can reach main because unit went green, you skipped fixture ownership, the flake cap, and the required-context list. Write a SHA-bound merge ticket locally, refuse a missing or stale ticket in CI, and only then enqueue.

You already know the failure mode. The PR is green. The queue is quiet. Then a fixture nobody owned moves, a flaky test is rerun twice, and a workflow file changes under a check that never reads YAML. The ticket exists to make those facts visible before merge, not after rollback.

What the ticket is for

The ticket is not another status-check name. It is a small JSON document bound to the commit SHA you intend to merge. CI recomputes the same fields from the diff. If they diverge, the job fails closed.

You keep promotion boring. Local hooks mint the ticket. CI verifies it. The merge queue reads the verified artifact, not a recycled success bit from a renamed job.

Decision table: mergeable or not

Use this table as the contract. Adapt the paths to your repo. Do not treat a blank cell as allow.

Signal	Human PR	Agent-authored PR	Fail closed when
Ticket missing or SHA mismatch	block	block	always
Diff touches `.github/workflows/`	require CODEOWNERS	block	agent + workflow path
Diff touches unowned fixtures	require review	block unless `fixtures-ok`	unmapped fixture path
Flake reruns on this SHA	cap 1	cap 0	reruns exceed cap
Required contexts	exact name set	exact name set plus `ticket-verify`	any missing or renamed
Retry / rebase storms	concurrency group	concurrency group, max 1 retry	second running job for SHA

That table is the policy. The code below only encodes it. If a row does not match your risk, change the table first, then the script.

1. Define a tiny ticket document

Keep it greppable in logs. Do not commit it. A committed ticket goes stale on the next amend.

{
  "sha": "REPLACE_WITH_GIT_SHA",
  "agent_authored": true,
  "allowed_path_prefixes": ["src/", "tests/unit/"],
  "touched_paths": ["src/billing.py", "tests/unit/test_billing.py"],
  "unowned_fixtures": [],
  "workflow_touched": false,
  "flake_reruns": 0,
  "flake_cap": 0,
  "required_contexts": ["lint", "unit", "ticket-verify"],
  "retry_count": 0,
  "max_retries": 1
}

Store it as .git/merge-ticket.json on the laptop and as a CI artifact named merge-ticket.json. The queue should demand the artifact SHA, not the conversation.

2. Map fixtures so unmapped paths fail closed

You need an ownership file you can grep. Start smaller than you think.

# fixtures.owners
# prefix<space>team
tests/fixtures/billing/  billing-team
tests/fixtures/auth/     auth-team
tests/fixtures/shared/   platform

Walk one example. tests/fixtures/billing/invoice.json is owned. tests/fixtures/tmp/dump.json is not. An agent diff that adds the dump is a merge block, not a review comment.

Unmapped fixture paths are not probably fine. They are a closed gate. Agent diffs should not expand fixtures.owners without a human review of the map itself.

3. Mint the ticket in a pre-push hook

The hook is the first gate. If the ticket cannot be built, you do not push.

#!/usr/bin/env bash
# .git/hooks/pre-push — example template, not a production dump
set -euo pipefail
SHA="$(git rev-parse HEAD)"
AGENT=0
if git log -1 --format=%B | grep -qiE '(^Co-authored-by:.*agent)|\[agent\]'; then
  AGENT=1
fi
python3 scripts/merge_ticket.py mint \
  --sha "$SHA" \
  --base origin/main \
  --owners fixtures.owners \
  --agent-authored "$AGENT" \
  --out .git/merge-ticket.json
python3 scripts/merge_ticket.py verify --ticket .git/merge-ticket.json

Install it explicitly. A hook that is not executable is not a control.

chmod +x scripts/merge_ticket.py .git/hooks/pre-push
git fetch origin main

Fetch before you mint. origin/main...HEAD is a lie if you never updated origin/main.

4. Recompute the ticket in CI

CI must recompute, not trust an uploaded JSON. Recompute from git diff --name-only origin/main...HEAD. Compare. Fail closed on drift.

Pin the job name. If someone renames ticket-verify to ticket_verify to dodge branch protection, the ticket's required_contexts list no longer matches. That is the point.

# .github/workflows/merge-ticket.yml — example template
name: merge-ticket
on:
  pull_request:
  merge_group:

concurrency:
  group: ticket-${{ github.event.pull_request.head.sha || github.sha }}
  cancel-in-progress: true

jobs:
  ticket-verify:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Recompute and verify ticket
        env:
          AGENT: ${{ contains(join(github.event.pull_request.labels.*.name, ','), 'agent') }}
          FLAKE_RERUNS: ${{ vars.FLAKE_RERUNS || '0' }}
        run: |
          git fetch origin main --depth=1 || git fetch origin main
          python3 scripts/merge_ticket.py mint \
            --sha "${GITHUB_SHA}" \
            --base origin/main \
            --owners fixtures.owners \
            --agent-authored "$AGENT" \
            --flake-reruns "${FLAKE_RERUNS}" \
            --out merge-ticket.json
          python3 scripts/merge_ticket.py verify --ticket merge-ticket.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: merge-ticket
          path: merge-ticket.json

Required contexts stay exact: lint, unit, ticket-verify. The merge queue admits the SHA only when those names exist and the artifact SHA equals HEAD.

5. Put the flake cap on the SHA, not the PR

Reruns attach to the commit, not the conversation. A new push gets a new SHA and a new budget. An agent retry of the same SHA gets none.

flake_cap(human) = 1
flake_cap(agent) = 0

If unit is flaky, you spend the human budget or you fix the test. You do not let an agent click re-run until the queue is full. Treat a missing rerun sidecar as zero. Missing must not mean unlimited.

Worked example: mint and verify

The script below is a compact example you can drop into scripts/merge_ticket.py and extend. It is a proposal. It is not a claim that it ran on your default branch.

#!/usr/bin/env python3
"""SHA-bound merge ticket. Example only; unexecuted here."""
from __future__ import annotations

import argparse
import json
import subprocess
import sys
from pathlib import Path

DEFAULT_ALLOWED = ("src/", "tests/unit/")
WORKFLOW_PREFIX = ".github/workflows/"
FIXTURE_HINTS = ("tests/fixtures/", "testdata/")


def git_output(args: list[str]) -> str:
    return subprocess.check_output(args, text=True).strip()


def diff_names(base: str) -> list[str]:
    out = git_output(["git", "diff", "--name-only", f"{base}...HEAD"])
    return [line for line in out.splitlines() if line]


def load_owners(path: Path) -> list[tuple[str, str]]:
    rows: list[tuple[str, str]] = []
    if not path.exists():
        return rows
    for raw in path.read_text().splitlines():
        line = raw.split("#", 1)[0].strip()
        if not line:
            continue
        prefix, team = line.split()
        rows.append((prefix, team))
    return rows


def unowned_fixtures(paths: list[str], owners: list[tuple[str, str]]) -> list[str]:
    bad: list[str] = []
    for p in paths:
        if not p.startswith(FIXTURE_HINTS):
            continue
        if not any(p.startswith(prefix) for prefix, _ in owners):
            bad.append(p)
    return bad


def truthy(value: str | bool) -> bool:
    if isinstance(value, bool):
        return value
    return str(value).strip().lower() in {"1", "true", "yes", "on"}


def build_ticket(args: argparse.Namespace) -> dict:
    touched = diff_names(args.base)
    owners = load_owners(Path(args.owners))
    agent = truthy(args.agent_authored)
    return {
        "sha": args.sha or git_output(["git", "rev-parse", "HEAD"]),
        "agent_authored": agent,
        "allowed_path_prefixes": list(DEFAULT_ALLOWED),
        "touched_paths": touched,
        "unowned_fixtures": unowned_fixtures(touched, owners),
        "workflow_touched": any(p.startswith(WORKFLOW_PREFIX) for p in touched),
        "flake_reruns": int(args.flake_reruns),
        "flake_cap": 0 if agent else 1,
        "required_contexts": ["lint", "unit", "ticket-verify"],
        "retry_count": int(args.retry_count),
        "max_retries": 1,
    }


def violations(ticket: dict, current_sha: str) -> list[str]:
    problems: list[str] = []
    if ticket["sha"] != current_sha:
        problems.append("sha-mismatch")
    if ticket["unowned_fixtures"]:
        problems.append("unowned-fixtures:" + ",".join(ticket["unowned_fixtures"]))
    if ticket["agent_authored"] and ticket["workflow_touched"]:
        problems.append("agent-touched-workflow")
    if int(ticket["flake_reruns"]) > int(ticket["flake_cap"]):
        problems.append("flake-budget-exceeded")
    if int(ticket["retry_count"]) > int(ticket["max_retries"]):
        problems.append("retry-storm")
    skip_prefixes = FIXTURE_HINTS + (WORKFLOW_PREFIX,)
    for p in ticket["touched_paths"]:
        if not ticket["agent_authored"]:
            break
        if p.startswith(skip_prefixes):
            continue
        allowed = ticket["allowed_path_prefixes"]
        if not any(p.startswith(a) for a in allowed):
            problems.append(f"path-not-allowed:{p}")
    return problems


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--sha", default="")
    parser.add_argument("--base", default="origin/main")
    parser.add_argument("--owners", default="fixtures.owners")
    parser.add_argument("--agent-authored", default="0")
    parser.add_argument("--flake-reruns", default="0")
    parser.add_argument("--retry-count", default="0")
    parser.add_argument("--out", default=".git/merge-ticket.json")
    parser.add_argument("--ticket", default=".git/merge-ticket.json")
    sub = parser.add_subparsers(dest="cmd", required=True)
    sub.add_parser("mint")
    sub.add_parser("verify")
    args = parser.parse_args()
    current_sha = args.sha or git_output(["git", "rev-parse", "HEAD"])
    if args.cmd == "mint":
        ticket = build_ticket(args)
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps(ticket, indent=2) + "\n")
        print(args.out)
        return 0
    ticket = json.loads(Path(args.ticket).read_text())
    problems = violations(ticket, current_sha)
    if problems:
        print("ticket failed:", file=sys.stderr)
        for item in problems:
            print(f"  - {item}", file=sys.stderr)
        return 1
    print("ticket ok", ticket["sha"])
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Reproducible test plan

Run these as ordinary unit tests. Mock ticket dicts. Do not hit the network.

SHA mismatch: ticket sha is aaa, current SHA is bbb. Expect sha-mismatch and exit 1.
Owned fixture: touched_paths includes tests/fixtures/billing/invoice.json and fixtures.owners maps that prefix. Expect no unowned-fixtures.
Unmapped fixture: tests/fixtures/tmp/dump.json with an agent ticket. Expect a closed gate.
Workflow path: agent ticket with .github/workflows/ci.yml. Expect agent-touched-workflow.
Flake cap: human ticket with flake_reruns=2, flake_cap=1. Expect flake-budget-exceeded. Agent ticket with flake_reruns=1 must fail even faster.
Path allowlist: agent ticket touching infra/terraform/main.tf. Expect path-not-allowed unless you changed the table.

A minimal pytest sketch for step 3:

from merge_ticket import violations

def test_unmapped_fixture_fails_closed():
    ticket = {
        "sha": "abc",
        "agent_authored": True,
        "allowed_path_prefixes": ["src/", "tests/unit/"],
        "touched_paths": ["tests/fixtures/tmp/dump.json"],
        "unowned_fixtures": ["tests/fixtures/tmp/dump.json"],
        "workflow_touched": False,
        "flake_reruns": 0,
        "flake_cap": 0,
        "required_contexts": ["lint", "unit", "ticket-verify"],
        "retry_count": 0,
        "max_retries": 1,
    }
    problems = violations(ticket, "abc")
    assert any(p.startswith("unowned-fixtures:") for p in problems)

If you cannot run that assertion locally, do not enable the merge queue path. The ticket is only useful when the verifier is in the required set.

Green-to-merge path

A human or an agent produces a commit.
pre-push mints .git/merge-ticket.json from the diff and fixtures.owners.
Push is refused if the ticket already fails locally.
CI checks out the SHA, recomputes the ticket, and publishes the artifact.
lint and unit may run in parallel, but the queue still requires ticket-verify.
The merge queue admits the SHA only when required_contexts match exactly and the artifact SHA equals HEAD.
A flake rerun is a new ticket field, not a silent click. Agent SHAs get a zero cap.

Skip a step and you are back to merging a boolean.

Where a model belongs — and where it does not

Some teams want a natural-language check: does this PR description match the touched paths? That is not merge authority. It is a linter on the ticket's story.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you run that classifier at all, run it on a redacted path list with no secrets, no GITHUB_TOKEN, and no deploy keys. MonkeyCode's free model access and free server option are enough for that isolated step: the model returns yes or no on a path/intent mismatch, and ticket-verify still owns the fail-closed decision. Do not let the model rename required contexts or increment the flake cap.

Keep the prompt boring. Feed the touched paths, the allow prefixes, and the PR title. Ask for a single token: MATCH or DRIFT. Parse only those two strings. Anything else is a failed classifier, not a merge.

Limitations

The ticket is only as strong as CI's ability to recompute it. If ticket-verify can be skipped, renamed, or run on a fork workflow you do not control, the file is theater.

A model classifier can be wrong. Treat it as an extra signal, never as the owner map. Inference on a free server does not see your private fixtures unless you paste them. Do not paste them.

git diff origin/main...HEAD is wrong if main moved and you did not fetch. Shallow clones will lie about names. The workflow uses fetch-depth: 0 for a reason.

This does not replace code review, CODEOWNERS, or secret scanning. It serializes merge eligibility. That is all.

Branch protection that matches .*verify.* will accept a renamed job. Pin the exact context string. If your queue cannot require an exact check name, fix the queue before you add JSON.

Who should not use this

Do not add a merge ticket if you are a solo repo with no agents, no fixtures, and one required check. The ceremony will cost more than the risk.

Do not use it to auto-merge agent PRs. The point is to block, not to accelerate.

Do not point a model at the full tree, .env, or Actions secrets to explain the diff. The classifier belongs on a redacted path list.

If you cannot list fixture owners, stop. An empty fixtures.owners plus fail-closed means every fixture change blocks. That is safer than a silent allow, and it will also halt the team until you fill the map.

Start with the map, then pin the name

Start with fixtures.owners and a verifier that fails closed. Then pin ticket-verify. Only after that should you care whether a model helped write the PR body or classify the path list.

If you run the classifier on a free model and a free server, keep it read-only on the diff and leave merge authority in CI.

Unmapped Paths Fail Closed: A Diff-to-Fixture Contract for the Merge Queue

Emery Lin — Fri, 11 Sep 2026 16:09:51 +0000

A green check that never executed tests for the files in your diff is not a merge signal. It is a hole with a badge. Fail closed when git diff contains a path that no fixture pack owns.

That rule is the article. The rest is a contract you can enforce locally, in CI, and at queue admission.

Whole-suite green hides the wrong skip

Most pipelines still run a default job list. You change internal/billing/, and CI spends its budget on docs snapshots and a flaky browser pack. The billing fixtures never ran. The check is still green.

Agent-authored diffs make the hole larger. They often add a helper, a config stub, and a file you did not ask for. Your suite maps none of those extras. You merge a path that no named test owns.

You do not need a new coverage percentage. You need a path contract.

The contract: every changed path names a pack

Keep a matrix in the repo. Each row is a glob, a fixture directory, a timeout, and a flake policy. CI loads the matrix, intersects it with the merge-base diff, and runs only the matching packs. An unmatched path is a failed job, not a skip.

Here is a starting file. Treat it as a template, not as measured production data.

# ci/path-matrix.yml
version: 1
unmapped: fail
packs:
  - id: billing-unit
    glob: ["internal/billing/**", "pkg/money/**"]
    fixtures: tests/fixtures/billing
    command: go test ./internal/billing ./pkg/money -count=1
    timeout_sec: 180
    flake:
      max_retries: 0
      quarantine_file: ci/quarantine.yml
  - id: http-contract
    glob: ["internal/api/**", "openapi/*.yaml"]
    fixtures: tests/fixtures/http
    command: go test ./internal/api -count=1
    timeout_sec: 240
    flake:
      max_retries: 1
      quarantine_file: ci/quarantine.yml
  - id: ci-self
    glob: [".github/workflows/**", "ci/**", "scripts/ci/**"]
    fixtures: tests/fixtures/ci
    command: python scripts/ci/check_matrix.py --self-test
    timeout_sec: 60
    flake:
      max_retries: 0
      quarantine_file: ci/quarantine.yml

Keep unmapped: fail as the default. If docs should be allowed to skip unit packs, add an explicit docs/** pack. Do not weaken the default so noisy diffs can pass.

Step 1 — Classify the diff from the merge base

Run this from the merge base, not from HEAD~1. Stacked commits lie.

BASE="${GITHUB_BASE_REF:-main}"
git fetch origin "$BASE" --depth=50
git diff --name-only "origin/${BASE}...HEAD" | sort -u > /tmp/changed.txt
cat /tmp/changed.txt

That list is the only input the contract should see. Job names, previous greens, and chat summaries are noise.

Step 2 — Resolve packs, then refuse leftovers

The checker should be boring. Boring is reviewable.

# scripts/ci/check_matrix.py
from __future__ import annotations

import argparse
import fnmatch
import json
import pathlib
import sys

import yaml


def load_matrix(path: pathlib.Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if data.get("version") != 1:
        raise SystemExit("unsupported matrix version")
    return data


def resolve(changed: list[str], matrix: dict) -> tuple[set[str], list[str]]:
    claimed: dict[str, str] = {}
    unknown: list[str] = []
    for path in changed:
        hits = [
            pack["id"]
            for pack in matrix["packs"]
            if any(fnmatch.fnmatch(path, glob) for glob in pack["glob"])
        ]
        if not hits:
            unknown.append(path)
            continue
        claimed[path] = hits[-1]
    return set(claimed.values()), unknown


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--matrix", default="ci/path-matrix.yml")
    parser.add_argument("--changed", default="/tmp/changed.txt")
    parser.add_argument("--out", default="/tmp/promotion.json")
    parser.add_argument("--self-test", action="store_true")
    args = parser.parse_args()

    matrix = load_matrix(pathlib.Path(args.matrix))
    if args.self_test:
        packs, unknown = resolve(["ci/path-matrix.yml"], matrix)
        if "ci-self" not in packs or unknown:
            raise SystemExit("self-test failed")
        print("self-test ok")
        return 0

    changed = [
        line.strip()
        for line in pathlib.Path(args.changed).read_text().splitlines()
        if line.strip()
    ]
    packs, unknown = resolve(changed, matrix)
    record = {
        "changed": changed,
        "packs": sorted(packs),
        "unknown": unknown,
        "unmapped_policy": matrix.get("unmapped", "fail"),
        "matrix_path": args.matrix,
    }
    pathlib.Path(args.out).write_text(json.dumps(record, indent=2) + "\n")
    print(json.dumps(record, indent=2))
    if unknown and record["unmapped_policy"] == "fail":
        print("unmapped paths:", *unknown, sep="\n  ", file=sys.stderr)
        return 2
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

This is a worked example. Port the language later if you need to. The behavior you want is exit code 2 on leftovers, plus a JSON record the queue can fetch.

Step 3 — Use the same script in a pre-push hook

CI-only enforcement is how unmapped files reach the queue. Put the same check in pre-push.

# .githooks/pre-push
#!/usr/bin/env bash
set -euo pipefail
BASE="${UPSTREAM_BASE:-origin/main}"
git diff --name-only "${BASE}...HEAD" | sort -u > /tmp/changed.txt
python scripts/ci/check_matrix.py --changed /tmp/changed.txt --out /tmp/promotion.json

Point Git at that directory once:

git config core.hooksPath .githooks
chmod +x .githooks/pre-push

You now fail before the push. That is cheaper than occupying a red queue slot.

Step 4 — Run only the resolved packs

Do not keep a second, hand-maintained job list. Read promotion.json and execute those commands.

# scripts/ci/run_packs.py
from __future__ import annotations

import argparse
import json
import pathlib
import subprocess
import sys

import yaml


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--record", required=True)
    parser.add_argument("--matrix", default="ci/path-matrix.yml")
    args = parser.parse_args()

    record = json.loads(pathlib.Path(args.record).read_text())
    matrix = yaml.safe_load(pathlib.Path(args.matrix).read_text())
    wanted = set(record["packs"])
    by_id = {pack["id"]: pack for pack in matrix["packs"]}
    missing = sorted(wanted - set(by_id))
    if missing:
        print("unknown pack ids:", missing, file=sys.stderr)
        return 2

    for pack_id in sorted(wanted):
        pack = by_id[pack_id]
        retries = int(pack.get("flake", {}).get("max_retries", 0))
        timeout = int(pack["timeout_sec"])
        attempt = 0
        while True:
            attempt += 1
            print(f"==> {pack_id} attempt {attempt}")
            try:
                subprocess.run(
                    pack["command"],
                    shell=True,
                    check=True,
                    timeout=timeout,
                )
                break
            except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as exc:
                if attempt > retries:
                    print(f"pack {pack_id} failed: {exc}", file=sys.stderr)
                    return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Keep flake retries pack-scoped. A billing unit test that needs a retry is a different incident from a browser pack that always needs one. A repo-wide retry counter hides the pack that is rotting.

Wire both scripts in one workflow. Example GitHub Actions job:

# .github/workflows/path-contract.yml
name: path-contract
on:
  pull_request:
jobs:
  admit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: List changed paths
        run: |
          git diff --name-only "origin/${{ github.base_ref }}...HEAD" | sort -u > /tmp/changed.txt
      - name: Enforce path matrix
        run: python scripts/ci/check_matrix.py --changed /tmp/changed.txt --out /tmp/promotion.json
      - name: Upload promotion record
        uses: actions/upload-artifact@v4
        with:
          name: promotion-record
          path: /tmp/promotion.json
      - name: Run resolved packs
        run: python scripts/ci/run_packs.py --record /tmp/promotion.json --matrix ci/path-matrix.yml

If the matrix job cannot produce the artifact, later admission must not treat a skipped upload as success.

Step 5 — Quarantine by pack, and date the skip

A global flake list becomes folklore. Quarantine rows should name a pack id, a test id, and an expiry.

# ci/quarantine.yml
- pack: http-contract
  test: TestRateLimitWindow
  reason: "clock skew on shared runner; tracked in issue 1841"
  expires: "2026-10-01"

Reject expired rows in run_packs.py before you honor max_retries. An open-ended skip is a merge-policy hole. Date it or delete it.

Step 6 — Admit the queue with the record

A required check named path-contract is still a boolean at the branch-protection layer. Download promotion.json and assert the fields you actually care about.

test "$(jq '.unknown | length' /tmp/promotion.json)" -eq 0
test "$(jq -r '.unmapped_policy' /tmp/promotion.json)" = "fail"
test "$(jq '.packs | length' /tmp/promotion.json)" -gt 0

If those asserts pass, you may enqueue. If they fail, you do not. A green job that cannot produce the record does not count.

Drafting new rows without trusting the draft

When the checker prints an unmapped path, you need a glob and a fixture directory. That edit is mechanical. It is also easy to over-match.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already use MonkeyCode, its free model access and free server option are enough to draft a candidate row from git diff --name-only plus a listing of tests/fixtures/. You still read the glob. You still type the commit. The assistant does not own unmapped: fail.

Keep the prompt small:

Given these unmapped paths:
internal/ledger/entry.go
internal/ledger/entry_test.go
Propose one path-matrix.yml pack row.
Constraints: glob must not include vendor/** or .github/**
fixtures dir must already exist or be named tests/fixtures/ledger
max_retries must be 0
Return YAML only.

Review the glob before you merge it. internal/** is not a pack. It is a surrender.

Who should not use this

This contract does not prove fixtures are meaningful. A pack can run and still assert nothing about the lines you changed. Pair it with tests you already trust.

It does not replace review. An unmapped-path check will not catch a mapped file with a logic error.

Skip it for generated trees, vendored blobs, or monorepos that cannot list globs without a platform team. If a typical diff contains thousands of paths, you need package ownership first.

Do not use this if CI cannot fetch the merge base. HEAD~1 classifies the wrong snapshot.

Do not switch unmapped to warn so agent PRs stop bouncing. Fix the matrix. That bounce is the signal.

Admission checklist

Copy this into the PR template if you want the queue to stay honest:

[ ] python scripts/ci/check_matrix.py exits 0 on this branch
[ ] promotion.json lists every changed path under a pack id
[ ] New globs do not include vendor/**, dist/**, or secret paths
[ ] Quarantine rows have an expires date
[ ] core.hooksPath is .githooks and pre-push is executable

You are done when an extra file blocks merge. You are not done when the default suite is green.

Treat Merge as Promotion: Four Stages Between Local Green and main

Emery Lin — Thu, 10 Sep 2026 04:26:50 +0000

A green check is a local fact. Merge is a promotion.

If you let an agent-written diff jump from “tests passed on my laptop” to main, you skipped three decisions you still own: what was frozen, what CI re-ran, and what you will revert if the next hour is worse than the last. Copy the path below. Do not copy a feeling.

The conclusion you should keep

You should not merge because a required check is green. You should merge because the patch survived a promotion sequence you can replay.

That sequence has four stages. Each stage writes evidence. The next stage refuses to start without it.

Local fail-closed, with a receipt that freezes fixtures and paths.
Pull-request checks that re-run tests and recompute the freeze.
A merge queue that isolates one flake retry from a real failure.
A post-merge soak window with a documented revert command.

This is a tutorial. You leave with a receipt schema, a pre-push hook, a GitHub Actions workflow, and a decision table. Treat the scripts as a proposed workflow you can run in a throwaway branch. They are not a claim about your production fleet.

Why one green check is a skipped promotion

CI answers a narrow question: did this SHA fail the jobs you paid to run? Merge answers a wider one: is this SHA allowed to become everyone else’s default?

Those questions diverge when the author is an agent. The agent can rewrite the fixture that made the test pass. It can expand the diff into lockfiles, snapshots, or golden files you never intended to touch. A boolean check will still go green. Your history will not.

So you split the path. Local work is allowed to be optimistic. main is not.

Stage 1 — Fail closed before the network

Run the cheap proofs on the machine that wrote the diff. Do it before git push.

You are not trying to replace CI. You are trying to refuse a class of diffs that CI will rubber-stamp: fixture edits, allowlist escapes, and “I deleted the failing test” patches.

Receipt schema

Keep the receipt small. It is a freeze, not a novel.

{
  "schema": "promote-receipt/v1",
  "git_sha": "REPLACE_WITH_HEAD_SHA",
  "tree_sha": "REPLACE_WITH_TREE_SHA",
  "fixture_lock": "sha256:REPLACE",
  "paths_changed": ["src/billing/quote.py", "tests/test_quote.py"],
  "allowlist_ok": true,
  "fixtures_updated": false,
  "commands": [
    {"name": "unit", "exit": 0},
    {"name": "lint", "exit": 0}
  ]
}

Proposed generation script:

#!/usr/bin/env bash
# scripts/write_receipt.sh — proposed local helper
set -euo pipefail

root="$(git rev-parse --show-toplevel)"
cd "$root"

allow_re='^(src/|tests/test_)'
forbid_re='^(tests/fixtures/|package-lock\.json|uv.lock|poetry.lock)$'

git_sha="$(git rev-parse HEAD)"
tree_sha="$(git rev-parse HEAD^{tree})"
mapfile -t paths < <(git diff --name-only origin/main...HEAD)

allowlist_ok=true
fixtures_updated=false
for p in "${paths[@]}"; do
  if [[ "$p" =~ $forbid_re ]]; then
    fixtures_updated=true
  fi
  if [[ ! "$p" =~ $allow_re ]]; then
    allowlist_ok=false
  fi
done

fixture_lock="sha256:$(find tests/fixtures -type f -print0 2>/dev/null | sort -z | xargs -0 sha256sum | sha256sum | awk '{print $1}')"

python -m pytest -q
python -m ruff check src tests

mkdir -p .ci
python - <<'PY'
import json, os
receipt = {
  "schema": "promote-receipt/v1",
  "git_sha": os.environ["GIT_SHA"],
  "tree_sha": os.environ["TREE_SHA"],
  "fixture_lock": os.environ["FIXTURE_LOCK"],
  "paths_changed": os.environ["PATHS"].split("\n") if os.environ["PATHS"] else [],
  "allowlist_ok": os.environ["ALLOWLIST_OK"] == "true",
  "fixtures_updated": os.environ["FIXTURES_UPDATED"] == "true",
  "commands": [{"name": "unit", "exit": 0}, {"name": "lint", "exit": 0}],
}
open(".ci/receipt.json", "w", encoding="utf-8").write(json.dumps(receipt, indent=2) + "\n")
PY

Export the env vars before the Python block, or inline them. The point is the freeze, not the shell dialect.

Pre-push hook

#!/usr/bin/env bash
# .git/hooks/pre-push — proposed
set -euo pipefail
./scripts/write_receipt.sh

python - <<'PY'
import json, subprocess, sys
r = json.load(open(".ci/receipt.json", encoding="utf-8"))
msg = subprocess.check_output(["git", "log", "-1", "--format=%B"], text=True)
if r["fixtures_updated"] and "Fixtures-Update: true" not in msg:
    sys.stderr.write("fixture or lockfile changed without Fixtures-Update: true\n")
    sys.exit(1)
if not r["allowlist_ok"]:
    sys.stderr.write("path allowlist failed; split the unrelated files out\n")
    sys.exit(1)
PY

You now have a local gate. Push is no longer “the agent said it was fine.”

Drafting the patch can still be cheap. If you sketch the diff in MonkeyCode, use it as a typewriter, not as a merge oracle.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option can host that draft loop. They do not sign the receipt, re-run CI, or own the revert.

Stage 2 — Re-run, do not replay

CI must recompute the freeze. A committed receipt is a claim. It is not evidence that the tests passed on GitHub-hosted runners.

Proposed workflow:

# .github/workflows/promote.yml
name: promote
on:
  pull_request:
    types: [opened, synchronize, reopened]
  merge_group:

jobs:
  recompute:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install
        run: pip install pytest ruff
      - name: Verify receipt against this SHA
        run: python scripts/verify_receipt.py
      - name: Re-run tests
        run: |
          python -m ruff check src tests
          python -m pytest -q --junitxml=junit.xml
      - name: Upload junit
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: junit-${{ github.sha }}
          path: junit.xml

Proposed verifier:

# scripts/verify_receipt.py — proposed
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ALLOW = ("src/", "tests/test_")
FORBID = ("tests/fixtures/", "package-lock.json", "uv.lock", "poetry.lock")


def sha_tree(root: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(root.rglob("*")):
        if p.is_file():
            h.update(p.as_posix().encode())
            h.update(p.read_bytes())
    return "sha256:" + h.hexdigest()


def main() -> int:
    receipt_path = Path(".ci/receipt.json")
    if not receipt_path.exists():
        print("missing .ci/receipt.json; run scripts/write_receipt.sh locally", file=sys.stderr)
        return 2

    r = json.loads(receipt_path.read_text(encoding="utf-8"))
    head = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
    if r.get("git_sha") != head:
        print(f"receipt sha {r.get('git_sha')} != HEAD {head}", file=sys.stderr)
        return 1

    base = os.environ.get("GITHUB_BASE_REF", "origin/main")
    names = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"], text=True
    ).splitlines()

    fixtures_updated = any(n.startswith(FORBID) or n in FORBID for n in names)
    allowlist_ok = all(n.startswith(ALLOW) or n == ".ci/receipt.json" for n in names)

    msg = subprocess.check_output(["git", "log", "-1", "--format=%B"], text=True)
    if fixtures_updated and "Fixtures-Update: true" not in msg:
        print("CI freeze: fixtures changed without trailer", file=sys.stderr)
        return 1
    if not allowlist_ok:
        print("CI freeze: path allowlist failed", file=sys.stderr)
        return 1

    lock = sha_tree(Path("tests/fixtures")) if Path("tests/fixtures").exists() else "sha256:empty"
    if r.get("fixture_lock") != lock:
        print("fixture lock drifted between laptop and runner", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Notice what this job does not do. It does not trust the local exit codes. It trusts the freeze, then it runs the tests again.

Stage 3 — Queue with one retry, not infinite hope

A merge queue is where flakes pretend to be product bugs. You want one retry for jobs you have already labeled flaky. You do not want a while-loop until green.

Protect main. Require promote / recompute. Turn on a merge queue if your host supports it. Then add a thin classifier so a timeout is not treated like an assertion failure.

# scripts/classify_junit.py — proposed
import sys
import xml.etree.ElementTree as ET

FLAKY_MARKERS = ("socket.timeout", "Temporary failure in name resolution")


def classify(path: str) -> str:
    root = ET.parse(path).getroot()
    failures = []
    for case in root.iter("testcase"):
        bad = case.find("failure") or case.find("error")
        if bad is None:
            continue
        text = (bad.get("message") or "") + "\n" + (bad.text or "")
        failures.append(text)
    if not failures:
        return "pass"
    if all(any(m in t for m in FLAKY_MARKERS) for t in failures):
        return "flake"
    return "fail"


if __name__ == "__main__":
    print(classify(sys.argv[1]))

Wire it as a second job that only runs on merge_group:

  queue-retry:
    if: github.event_name == 'merge_group'
    needs: recompute
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest
      - id: first
        continue-on-error: true
        run: python -m pytest -q --junitxml=junit.xml
      - id: class
        if: always()
        run: echo "kind=$(python scripts/classify_junit.py junit.xml)" >> "$GITHUB_OUTPUT"
      - name: One flake retry
        if: steps.class.outputs.kind == 'flake'
        run: python -m pytest -q
      - name: Real failure blocks the queue
        if: steps.class.outputs.kind == 'fail'
        run: exit 1

One retry. Then stop. If the second run fails, the SHA does not promote.

Stage 4 — Soak, then keep or revert

Promotion is not finished at merge. You need a window where main can still lose.

Tag the merge. Watch a short, named soak. Keep the revert command in the same runbook as the receipt.

# proposed soak notes, not a production SLO claim
git tag "soak-$(git rev-parse --short HEAD)"
git push origin "soak-$(git rev-parse --short HEAD)"

# if the next deploy or nightly is worse than the parent
git revert --no-edit HEAD
git push origin main

If you lack nightlies, soak is still a decision: “we will look at error rate for this SHA before the next human leaves.” Write that down. A tag without a watcher is jewelry.

Decision table

Use this at review time. Do not negotiate it in the merge comment thread.

Evidence	Local	PR job	Merge queue	Merge?
Unit + lint green, allowlist ok, fixture lock matches	pass	pass	pass	Promote
Tests green, fixtures changed, no `Fixtures-Update: true`	fail-closed	fail	n/a	Split the PR
Tests green, lockfile or snapshot slipped into the diff	fail-closed	fail	n/a	Split the PR
Assertion failure on runner	n/a	fail	fail	Fix the code
Known network timeout, first queue run only	n/a	pass	one retry	Retry once
Same timeout after retry	n/a	pass	fail	Do not merge
Receipt SHA ≠ `GITHUB_SHA`	n/a	fail	fail	Regenerate receipt

The table is the merge policy. The workflow is just enforcement.

Limitations, and who should not use this

This path assumes you can name an allowlist. Monorepos that ship lockfiles, generated clients, and screenshot goldens on every change will drown in Fixtures-Update trailers. If that is you, replace the allowlist with CODEOWNERS on generated paths instead of copying this script.

Do not use a laptop receipt as a substitute for CI. Runners differ. Clock skew, case-sensitive filesystems, and missing system libraries will lie to you in the other direction.

Do not use the flake retry as a general “run until green” knob. If you cannot name the timeout string, it is not a flake policy. It is denial.

Tiny personal repos with one job and no agents do not need four stages. A required check and a human look is enough. Teams that already run a vendor merge queue with required re-runs can keep that queue and only steal the fixture freeze.

The draft environment is optional. The promotion path is not. If you already sketch patches on a free server, point write_receipt.sh at that working tree before the first push, then let CI disagree with you in public.

Status Checks Don't Review the Workflow Files They Run

Emery Lin — Wed, 09 Sep 2026 16:24:32 +0000

A green check means the workflows already on the branch executed. It does not mean the YAML you just changed is safe to become the next default-branch runner. Treat .github/workflows/** as production code: dry-run it, attach fixtures for every new job, then attach a merge receipt before you click merge.

Status checks are a report about a run. Workflow files are the machine that will run on every later push. Those are different objects. If an agent rewrites a job, retargets a secret, or swaps pull_request for pull_request_target, the old green check is the wrong signal.

This article is a local-first merge path for CI YAML. You keep the remote check. You stop treating it as a review of the workflow files themselves.

The gap you actually have

GitHub (and most CI products) evaluate the workflows that exist on the head SHA. They do not certify that those files are well-formed, secret-safe, or equivalent to what you tested last week. A passing unit job can sit next to a broken deploy.yml you never executed.

Agent-written YAML makes the gap wider. The model will invent a runner label, a context, or an environment name that looks plausible. Your app tests still pass. The next default-branch push is when the fiction becomes a privileged job.

You need a gate that lives outside the files under edit. If the only required check is defined in the YAML being rewritten, a broken edit can delete the gate.

What “good” means before merge

Use four artifacts, not another emoji reaction on the PR:

A path lock: .github/workflows/** is CODEOWNERS-protected.
A dry-run: lint plus a local execution attempt for every changed workflow.
Fixtures: a matrix file that names the jobs you added and the inputs they expect.
A merge receipt: a committed YAML document the remote check verifies, signed by the local guard.

The receipt is the merge ticket. The green check is only evidence that the receipt’s commands ran.

Step 1: Freeze the protected surface

Put ownership on the directory that can mint secrets and runners. One reviewer is enough if that reviewer is not the author of the YAML.

# .github/CODEOWNERS
.github/workflows/    @your-org/ci-maintainers
.github/merge-receipt.schema.json  @your-org/ci-maintainers
scripts/workflow_merge_guard.py    @your-org/ci-maintainers

Keep the guard script and the schema off the path an agent casually rewrites. If a PR touches both a workflow and the guard, stop. That combination is a policy change, not a feature branch.

Add a local hook so the rule fires before the remote ever sees the YAML.

# .git/hooks/pre-push  (install via core.hooksPath)
#!/usr/bin/env bash
set -euo pipefail
python3 scripts/workflow_merge_guard.py --mode pre-push

Remote enforcement without a local hook is late. Local enforcement without a remote check is optional. You want both.

Step 2: Dry-run the YAML, not the app

Install two tools that do not need your cloud minutes. actionlint catches syntax, undefined actions, and impossible if: expressions. nektos/act attempts a local run. Neither is GitHub-hosted Actions. Both beat merging on hope.

# Install once. Pin versions in your team doc.
# actionlint: https://github.com/rhysd/actionlint
# act:        https://github.com/nektos/act

changed=$(git diff --name-only origin/main...HEAD -- '.github/workflows')

if [ -z "$changed" ]; then
  echo "no workflow files in the diff"
  exit 0
fi

actionlint $changed

for f in $changed; do
  # List jobs; fail closed if the file does not parse.
  act --list --workflows "$f"
done

If act cannot pull an image, that is a finding, not a skip. Record it on the receipt as dry_run: blocked and do not merge. A workflow that only runs on a label you do not own is untested YAML.

Step 3: Require fixtures for every new job

A new job without a named fixture is an untested production entrypoint. You do not need a full end-to-end deploy. You need a documented input set and a command that exercises the job graph.

Keep the contract next to the workflow, not in a chat log.

# .github/workflow-fixtures/ci-unit.yaml
workflow: ci.yml
job: unit
event: pull_request
inputs:
  python_version: "3.12"
secrets_expected: []
local_command: |
  act pull_request -W .github/workflows/ci.yml -j unit \
    --input python_version=3.12

Rules you can enforce in code:

Every job id in a changed workflow must appear in .github/workflow-fixtures/.
secrets_expected must be a subset of secrets referenced by that job.
local_command must be the command the guard actually ran.
A job that lists secrets cannot use pull_request_target without an explicit privilege: elevated flag and a human owner.

That last rule is the one agents miss. pull_request_target plus a checkout of untrusted code is how a green PR becomes a secret leak. The fixture file makes the privilege visible.

Step 4: Write a merge receipt the remote can verify

The receipt is small on purpose. If it needs a novel, your change is too large.

# .github/merge-receipt.yml
schema: merge-receipt/v1
pr_head: "${GIT_SHA}"
protected_paths:
  - .github/workflows/
changed_workflows:
  - path: .github/workflows/ci.yml
    jobs: [unit, lint]
    privilege: unprivileged
dry_run:
  actionlint: pass
  act: pass
  notes: "act used the pull_request event; no secrets injected"
fixtures:
  - .github/workflow-fixtures/ci-unit.yaml
  - .github/workflow-fixtures/ci-lint.yaml
flake_ledger: .github/flake-ledger.json
generated_by: human   # or "agent" — see below
owner: "@ci-maintainers"

generated_by: agent is not a slur. It is a risk label. Agent YAML gets the same dry-run plus a required human owner. It does not get a faster merge.

Here is a guard you can run locally and in CI. It is a proposal: read it, pin the schema, then wire it to your hooks.

#!/usr/bin/env python3
"""workflow_merge_guard.py — fail closed on workflow diffs without a receipt."""
from __future__ import annotations

import argparse, json, subprocess, sys
from pathlib import Path

import yaml  # PyYAML; pin it in requirements-dev.txt

ROOT = Path(__file__).resolve().parents[1]
RECEIPT = ROOT / ".github" / "merge-receipt.yml"
FIXTURE_DIR = ROOT / ".github" / "workflow-fixtures"
WF_DIR = ".github/workflows"


def git_changed(against: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{against}...HEAD", "--", WF_DIR],
        text=True,
    )
    return [line for line in out.splitlines() if line]


def load_receipt() -> dict:
    if not RECEIPT.exists():
        sys.exit("missing .github/merge-receipt.yml")
    data = yaml.safe_load(RECEIPT.read_text()) or {}
    if data.get("schema") != "merge-receipt/v1":
        sys.exit("unsupported merge-receipt schema")
    return data


def job_ids(path: Path) -> set[str]:
    doc = yaml.safe_load(path.read_text()) or {}
    jobs = doc.get("jobs") or {}
    if not isinstance(jobs, dict):
        sys.exit(f"{path}: jobs must be a mapping")
    return set(jobs)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--mode", choices=["pre-push", "ci"], required=True)
    parser.add_argument("--against", default="origin/main")
    args = parser.parse_args()

    changed = git_changed(args.against)
    if not changed:
        print("workflow_merge_guard: no workflow diff")
        return 0

    receipt = load_receipt()
    listed = {row["path"] for row in receipt.get("changed_workflows", [])}
    missing = set(changed) - listed
    if missing:
        sys.exit(f"receipt missing workflows: {sorted(missing)}")

    for row in receipt["changed_workflows"]:
        path = ROOT / row["path"]
        expected = set(row.get("jobs") or [])
        actual = job_ids(path)
        if expected != actual:
            sys.exit(f"{path}: receipt jobs {expected} != yaml jobs {actual}")
        if row.get("privilege") == "elevated" and receipt.get("generated_by") == "agent":
            sys.exit("agent-generated elevated workflows are blocked")

    fixture_files = {ROOT / p for p in receipt.get("fixtures", [])}
    if not fixture_files:
        sys.exit("receipt lists no fixtures")
    for f in fixture_files:
        if not f.exists():
            sys.exit(f"missing fixture {f}")

    dry = receipt.get("dry_run") or {}
    if dry.get("actionlint") != "pass" or dry.get("act") != "pass":
        sys.exit("dry_run is not pass/pass; refuse merge")

    print(f"workflow_merge_guard: {args.mode} ok for {changed}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it the same way in both places. Drift between local and CI is how people start ignoring the hook.

python3 scripts/workflow_merge_guard.py --mode pre-push --against origin/main
python3 scripts/workflow_merge_guard.py --mode ci --against origin/main

Step 5: Enforce the receipt from a file you rarely touch

Put the remote job in a workflow that is not the one being rewritten this week. Give it a stable name. Mark that name required on the default branch. See GitHub’s branch protection docs for the exact UI; the important part is the check name, not the file name.

# .github/workflows/merge-receipt.yml
name: merge-receipt
on:
  pull_request:
    paths:
      - ".github/workflows/**"
      - ".github/merge-receipt.yml"
      - ".github/workflow-fixtures/**"
      - "scripts/workflow_merge_guard.py"

jobs:
  verify-receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python3 scripts/workflow_merge_guard.py --mode ci --against origin/${{ github.base_ref }}

If a PR deletes this file, CODEOWNERS plus a required review should stop it. If your org supports required workflows at the organization level, prefer that. A check defined only in a deletable file is not a control.

Where a coding assistant is allowed to help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The assistant is a stub generator, not a merge authority. You can point a free-tier coding model at a failing actionlint log and ask for a fixture file that names the new jobs. You still run the guard. You still own generated_by.

MonkeyCode is an open-source coding assistant with free model access and a free server option. That combination is useful when you want a scratch box to draft .github/workflow-fixtures/*.yaml or to summarize why act --list rejected a file. It is not useful as a replacement for actionlint, act, CODEOWNERS, or the receipt.

If the model proposes pull_request_target, a new environment:, or a secret that is not in secrets_expected, discard the patch. The receipt’s generated_by: agent field exists so the next reviewer does not have to guess.

Decision table

Diff contains	Dry-run	Fixtures	generated_by	Merge?
App code only	n/a	n/a	any	Existing app CI decides
Workflow YAML, unprivileged jobs	actionlint + act pass	One file per new job	human or agent	Yes, with receipt
Workflow YAML, secrets or `environment:`	pass, and secrets listed	Required	human	Yes, owner required
Workflow YAML, `pull_request_target` or elevated	pass	Required	human only	Yes only with `privilege: elevated`
Workflow YAML + edits to the guard/schema	anything	anything	anything	No — split the PR
Dry-run blocked (missing runner/image)	fail	present	any	No

Print this table in the PR template. Reviewers merge faster when the exceptions are written down.

Limitations

actionlint does not execute jobs. act is not GitHub-hosted Actions: container images, service names, and GITHUB_TOKEN permissions differ. A local pass is necessary and still incomplete.

The receipt is a file. Anyone who can commit to the branch can write dry_run: pass. The hook and the required check exist to make that lie expensive, not impossible. If you skip core.hooksPath, you skipped the control.

This path does not classify flaky tests and it does not parse JUnit. It only answers one question: may these workflow files land?

Free model access and a free server do not give you a replica of GitHub’s hosted runners. Do not send workflows that embed production secrets to any hosted assistant. Redact. Then dry-run on a machine you control.

Who should not use this

Skip the receipt if you have no default-branch protection and no one will read CODEOWNERS. A YAML file nobody enforces is documentation.

Skip agent-assisted fixture generation if your workflows are the secret boundary and your policy forbids leaving the building. Generate stubs on paper. Run act on an offline runner.

Skip act as a merge oracle if your jobs are macOS-only, billed runners, or OIDC-to-cloud deploys that cannot be faked locally. Keep actionlint and the privilege table. Replace the act field with a documented staging workflow that is not the default branch.

The merge button is not a linter. Put the dry-run and the receipt in front of .github/workflows, and let the green check mean what it actually means: the current machine ran. Not that the next one is safe.