A Red Model Check Is Not a Rerun Signal: Triage, Repair Once, Then Merge

Rerunning a red model check until it turns green is how a merge gate becomes decoration. The fix is not a better prompt. It is a policy that decides what kind of red you have before anything is allowed to run a second time.

Below is the gate I would wire into an existing pipeline: a JSONL ledger, a five-class triage table, one bounded repair attempt, and a quarantine rule that expires. Total new code is about 120 lines. No agent framework required.

The three reds you keep mixing together

A model-backed check can fail for three unrelated reasons, and a rerun answers none of them. Your diff genuinely broke something, your fixture or prompt moved underneath the check, or the endpoint answered differently for identical input. Only one of those is a code problem.

When you hit Rerun failed jobs without evidence, you convert the second and third cases into a green check. That is how a team ends up with a gate that has never caught anything in six months — and no one can prove it either way.

The rule: a rerun is only legal on an unchanged commit, and it produces a record, not just a new status badge.

What every attempt must record

You need five fingerprints before a verdict means anything. The prompt hash and the fixture hash are separate on purpose — collapsing them is what makes "the test started failing" impossible to diagnose.

Field	Why it exists
diff_fp	Hash of the patch, not the commit SHA. Survives rebases.
prompt_sha	Which review instructions produced this verdict.
fixture_sha	Which cases the check was evaluated against.
verdict	pass / fail, parsed, not eyeballed.
attempt	How many times this exact diff has been judged.

Step 1 — Fingerprint the diff, not the commit

Commit SHAs lie to you. An empty commit, a rebase, or a merge from main changes the SHA and changes nothing about the code under review. Hash the patch instead.

# scripts/model_gate.py (part 1)
import hashlib, json, pathlib, subprocess

ROOT = pathlib.Path(__file__).resolve().parents[1]

def sh(*args: str) -> str:
    return subprocess.run(args, cwd=ROOT, capture_output=True,
                          text=True, check=True).stdout

def sha(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

def diff_fingerprint(base: str) -> tuple[str, list[str]]:
    patch = sh("git", "diff", "--binary", f"{base}...HEAD")
    files = sh("git", "diff", "--name-only", f"{base}...HEAD").split()
    return hashlib.sha256(patch.encode()).hexdigest()[:16], files

Step 2 — Classify the red before touching it

The classifier compares the new record against the last record for the same check. That is the whole trick: you are not comparing against yesterday, you are comparing against the last judgment of this check.

# scripts/model_gate.py (part 2)
def triage(rec: dict, prev: dict | None) -> str:
    if prev is None:
        return "first-run"
    if rec["diff_fp"] != prev["diff_fp"]:
        return "diff-changed"          # your code moved: rerun is legitimate
    if rec["fixture_sha"] != prev["fixture_sha"]:
        return "fixture-drift"         # the check moved, not the code
    if rec["verdict"] != prev["verdict"]:
        return "nondeterministic"      # identical inputs, different answer
    return "reproducible-red"          # identical inputs, same failure

Read the branches carefully. nondeterministic can only be reached when every fingerprint matches — meaning you re-ran the same commit with no edits in between. If you edit anything between runs, you land in diff-changed and you have learned nothing about flakiness. You have learned about your edit.

Class	Trigger	Action	Blocks merge
first-run	No earlier record	Report, don't repair	Yes
diff-changed	diff_fp differs	One rerun allowed	Yes
fixture-drift	fixture_sha differs, diff same	Stop, fix inputs	Yes
nondeterministic	All fingerprints equal, verdict flipped	Quarantine + ledger note	No, but time-boxed
reproducible-red	All fingerprints equal, stable fail	Exactly one repair attempt	Yes

Implement the decision as data, not as if chains scattered through YAML:

def decide(cls: str, verdict: str) -> tuple[str, int]:
    if cls == "fixture-drift":
        return "block:fixture", 1
    if cls == "nondeterministic":
        return "quarantine", 0        # advisory, and it expires
    if verdict == "pass":
        return "merge", 0
    if cls == "reproducible-red":
        return "repair", 1            # budget: one attempt
    return "report", 1

Step 3 — Bound the repair loop to one attempt

An unbounded agent that keeps patching until the check passes is a machine for producing unrelated diffs. Give the repair step a written contract and enforce it with a shell guard, not with trust.

# scripts/repair_guard.sh — a repair may not leave the original diff
BASE="${BASE:-origin/main}"
allowed=$(git diff --name-only "$BASE...HEAD" | sort -u)
touched=$(git diff --name-only HEAD | sort -u)
extra=$(comm -13 <(echo "$allowed") <(echo "$touched"))
if [ -n "$extra" ]; then
  echo "repair escaped the diff: $extra" >&2
  exit 1
fi

The contract, as a comment at the top of your repair prompt, is three lines and no more: only edit files already in the diff; do not touch fixtures, prompts, or CI config; if the failure cannot be fixed inside that boundary, output ESCALATE and stop. Anything else and the agent will "fix" the gate instead of the code — which passes review, because the diff looks confident.

Step 4 — Wire the hooks so the policy is not optional

Two entry points, one script. Local runs replay the fixture lane and never touch the network, so git push stays fast.

#!/usr/bin/env bash
# .git/hooks/pre-push
set -euo pipefail
python scripts/model_gate.py --mode local

# .github/workflows/gate.yml (trimmed)
- name: Fixture lane
  run: python scripts/model_gate.py --mode local
- name: Model lane
  run: |
    python scripts/model_gate.py --mode ci --base "origin/${{ github.base_ref }}" \
      | tee .ci/last-run.txt

The fixture lane is the part that keeps your local hook honest: recorded verdicts plus expected results, checked in, replayed in milliseconds. When someone edits a fixture case without updating the expectation, fixture-drift fires in CI and the local hook stays green — which is exactly the asymmetry you want.

Step 5 — Make quarantine expire

nondeterministic is the class most likely to become a permanent hiding place. A check that flips is a check you cannot use, so give it a deadline measured in merges, not vibes.

Write the entry to .ci/quarantine.txt with the merge count at entry time, then fail a nightly job when the entry is older than your agreed window. Two rules keep this honest: a quarantined check must be labelled on the PR by a human, and removing the label requires a new ledger record with matching fingerprints. Without the second rule, quarantine is just a quieter rerun button.

Where a free endpoint actually helps

The gate above needs two things teams stall on: an endpoint to call and somewhere to run the job. MonkeyCode documents free model access and a free server option, which maps onto exactly those two blockers — the model lane gets an endpoint, and the fixture lane plus ledger can run on the provided server instead of waiting on a runner procurement ticket.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat the free allowance as a way to start measuring, not as a capacity plan. Check the project's current terms for what is included, because token allowances and server specs change and a gate built on a stale number is worse than no gate. If you want the second lane running this week without a billing conversation, that is the shortest path I can describe honestly.

Limitations, and who should skip this

• Temperature 0 is not determinism. Providers change routing and quantisation behind a stable endpoint name. The ledger detects this; it does not prevent it.
• You are sending code to an endpoint. If your repository cannot leave your network, run --mode local only and accept that the model lane stays advisory.
• Large diffs degrade the check. Truncate at a stated byte limit and record the truncation in the ledger, or your reproducible-red is really input-too-big.
• The ledger grows forever. Rotate monthly; keep the last record per prompt_sha plus everything in quarantine.
• Skip this entirely if your team cannot commit to reading the ledger. A gate nobody reads is a rerun button with extra steps.

Merge checklist

1. diff_fp is computed from the patch, not the commit SHA.
2. Prompt and fixture hashes are recorded separately.
3. Reruns are only permitted on unchanged fingerprints.
4. Repair is bounded to files already in the diff, enforced by repair_guard.sh.
5. Quarantine entries carry an entry timestamp and expire on a merge count.
6. Local hook replays fixtures and touches no network.
7. The ledger is rotated, not deleted, and the quarantine slice survives rotation.

If you only adopt one item, adopt item three. Everything else is bookkeeping around the single decision that keeps a model-backed gate worth having: you may rerun, but never to hide what you just learned.