Refuse the Rerun Until the Fixture Manifest Still Matches

A rerun is only a rerun if the files under test did not move. If the same workspace can edit a fixture and then go green, the required check is grading a different contract than the one that failed.

You do not need a smarter retry button for that. You need a frozen manifest, a job that cannot write it, and a verdict small enough to reject.

What this path actually proves

This workflow separates three facts that teams often collapse into one green square.

1. The fixture bytes at failure time.
2. The advisory opinion about whether a retry is even allowed.
3. The required status check that a merge queue is permitted to trust.

The first and third stay on a runner with no model credential. The second may call a model. It may not push, commit, or rewrite the check name.

None of this measures flake rate. It only stops a class of false greens: a suggestion lane that repairs the evidence and then reports success.

1. Pin the fixture manifest in the required job

Start from an explicit list. Do not hash the whole repository. A lockfile bump or a changelog edit is not a fixture change, and treating it as one will train people to ignore the check.

# Proposal: run this inside the required job, before tests.
# It has not been executed against your repository.
mkdir -p .ci
find tests/fixtures -type f | sort > .ci/fixture-list.txt
sha256sum $(cat .ci/fixture-list.txt) > .ci/fixture-manifest.txt
cp .ci/fixture-manifest.txt "$RUNNER_TEMP/fixture-manifest.txt"
test -s .ci/fixture-list.txt

Keep the committed list under review. The generated manifest is an artifact, not a file the test job may commit back.

If tests/fixtures is empty, fail closed. A missing contract is not a passing contract. Add the directory to the job's checkout only. Do not generate replacement files in this step just to make the hash command succeed.

2. Upload the red log, then leave the workspace

On failure, publish the log and the manifest. Do not publish the worktree. The worktree is how a later step smuggles a rewrite back in.

# Illustrative fragment, not a live workflow from this account.
# Pin the action version your organization already trusts.
- name: Preserve failure evidence
  if: failure()
  uses: actions/upload-artifact
  with:
    name: red-evidence-${{ github.run_id }}
    path: |
      .ci/fixture-manifest.txt
      test-output.log
    if-no-files-found: error

The required job ends here when tests fail. It does not call a model. It does not apply a patch. It does not rerun itself.

That split matters more than the model you pick later. A credential on the required runner is a write path, even if today's prompt says only summarize.

3. Let the advisory job read, not repair

Use a second job. Give it contents: read and the downloaded artifact. Deny it pull-requests: write and any deploy key.

This is the one place a hosted model is useful. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied availability claim used here is limited: free model access, plus a free server option, can host this advisory step. This draft does not assert a token quota, a model name, a hardware shape, a time limit, or that the free tier will stay as it is.

The advisory job may return only this JSON shape:

{
  "schema": "retry-verdict/v1",
  "action": "stop",
  "failure_class": "assertion",
  "reason": "expected status drifted; fixture bytes unchanged"
}

Allowed action values are retry, quarantine, and stop. There is no patch field. If a field appears that the schema does not list, the verdict is invalid.

# Unexecuted proposal. Adapt paths before you rely on it.
import hashlib, json, sys
from pathlib import Path

ALLOWED = {"retry", "quarantine", "stop"}
REQUIRED = {"schema", "action", "failure_class", "reason"}

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def main() -> int:
    verdict = json.loads(Path(sys.argv[1]).read_text())
    if set(verdict) != REQUIRED or verdict["schema"] != "retry-verdict/v1":
        print("reject: schema")
        return 2
    if verdict["action"] not in ALLOWED:
        print("reject: action")
        return 2
    if "\n" in verdict["reason"] or len(verdict["reason"]) > 240:
        print("reject: reason")
        return 2
    pinned = Path(sys.argv[2]).read_text().splitlines()
    for line in pinned:
        digest, rel = line.split(None, 1)
        rel = rel.lstrip("*").strip()
        if sha256_file(Path(rel)) != digest:
            print("reject: fixture moved: " + rel)
            return 3
    print("accept")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Call it with the verdict file and the pinned manifest. A nonzero exit blocks the rerun. It does not open a pull request.

Feed the model the log excerpt and the manifest hash, not the repository remote. Ask it to fill the schema. Then throw the prose away and keep the JSON only if the checker accepts it.

4. Decide with a table, not with a vibe

Use the failure class only after the hash check passes. If the hash check fails, the table does not apply. The bytes already told you to stop.

Failure class	Fixture hash	Verdict action	What the required lane does
timeout	unchanged	retry	one rerun, same job name
assertion	unchanged	stop	stay red, no rerun
missing fixture	changed or missing	stop	stay red, no rerun
network	unchanged	quarantine	rerun only if a human label already allows it
unknown	any	stop	stay red

retry is not a merge. It is permission to schedule the same required job again, against the same manifest. The second green is still just that job. Your branch protection rule should name it, and it should not name the advisory job.

Quarantine is the narrow middle. Use it for failures you have already labeled as environmental, in a file a human reviews. Do not let the advisory job add labels to that file. Otherwise the model can grant itself the next retry.

Cap the retry at one. A second red is a stop, even if the model would say retry again. Record the attempt id in the job output so a later reader can see the budget was spent.

5. Re-hash immediately before the rerun

Download the original manifest into the new run. Recompute hashes before the test command. If they differ, exit 3 and skip the suite.

# Proposal: first step of the rerun job.
python3 scripts/check_verdict.py "$VERDICT_PATH" "$MANIFEST_PATH"
status=$?
if [ "$status" -ne 0 ]; then
  echo "rerun refused"
  exit "$status"
fi

Do not fix up a mismatch in this step. A mismatch is the finding. Fix the fixture in a normal commit, with review, and let the required job hash the new bytes on a clean run.

6. Leave the advisory name out of branch protection

Name the required check something stable, such as tests (manifest). Name the advisory job retry-verdict (advisory). Protect the branch with the first name only.

You will still see advisory failures. That is fine. They are not the merge signal. If you add the advisory name to the required list, you have glued the suggestion lane back onto the gate this workflow just removed.

Also skip the rerun when the verdict file is missing. Absence is not retry. A down advisory host should leave the original red check in place, not invent a pass.

Inspect a refused rerun

Three commands are enough. You are looking for which gate fired, not for a narrative from the model.

# Proposal commands. Replace the artifact path with the run you are debugging.
sha256sum -c .ci/fixture-manifest.txt
python3 scripts/check_verdict.py verdict.json .ci/fixture-manifest.txt; echo "exit=$?"
git status --short

sha256sum -c failing means the workspace is not the failed commit's fixture set. Exit 2 from the checker means the verdict itself is the problem. A dirty git status means something wrote the tree after checkout. Stop there. Do not rerun tests until the tree matches the manifest.

If all three are clean and the action is stop, read the test log. The workflow has done its job. The bug is in the change, not in the gate.

Where to hook this

Put the manifest step in pre-test, not in post-test. A post-test hash tells you the suite already ran against whatever the workspace had become.

If you use a local hook runner, call the same script from a pre-push hook only as a convenience. The server-side required job is the one that counts. A laptop hook can be skipped. Branch protection cannot, unless an admin bypasses it, which you should treat as an incident rather than a path.

Keep model credentials in the advisory job's secret store. Do not copy them into the required job to make debugging easier. That convenience recreates the write path you just removed.

Limitations

This catches byte-level fixture edits. It does not catch a test that reads an unlisted file, a clock, or a live service. Those need their own pins.

The verdict can be wrong and still be schema-valid. retry on a real assertion bug wastes a run. It does not, by itself, hide the bug, provided the rerun uses the same assertions.

Free model access and a free server option are availability claims, not a capacity plan. If the advisory host is down, the required job must still fail closed without it. Do not block merges on the advisory job, and do not let a missing verdict become an implicit retry.

The sample code is a proposal. It is not a benchmark, and it is not a result from this account's repository. Paths, action versions, and permission keys will need to match your forge before you trust them.

Who should skip this

Skip it if you have no required status check. The manifest then has nothing to protect.

Skip it if fixtures are generated inside the test job on purpose. Pin the generator and its inputs instead of hashing outputs that are supposed to change.

Skip it if you want a model to open the fixing pull request automatically. That is a different risk, and this path refuses it on purpose.

If your required check already cannot write fixtures, the remaining step is small. Put the summarizer on a read-only job, and keep its token off the merge runner. MonkeyCode's free model access and free server option are enough for that advisory role when the suggestion does not need to deploy, commit, or rename the check.