The Gate Said No. Now What? A Triage Procedure for Rejected Agent Patches

A gate that rejects a patch is only half a policy. The other half is what happens after the rejection. In most pipelines, a failing agent patch produces one of three outcomes: a human stares at the log, the patch is rebuilt blindly, or the test is deleted. All three are wrong in different ways.

This article is a decision procedure instead. It classifies every rejected patch into one of three failure classes, assigns one action to each class, and keeps a quarantine ledger with an expiry date. The reproducible artifact is triage_gate.py: a small script that re-runs the failing test, compares fixture hashes, and freezes only the flakes.

The three failure classes

An agent patch fails CI for a reason. The reason is rarely "the code is bad" in the abstract sense. Every failure I have triaged lands in one of three buckets:

1. Class A — deterministic regression. The test fails on the first run and on every re-run. The assertion fails with the same input each time. This is the only class where the agent's code is the primary suspect.
2. Class B — fixture drift. The test passes locally and fails on the server. A fixture ID, snapshot, or generated seed changed outside the patch. The agent patch may be innocent.
3. Class C — flake. The test fails intermittently. Re-runs flip between red and green. Timing, ordering, or shared state is the suspect, not the patch.

The trap is that all three look identical in the first CI log line. The only way to separate them is a controlled re-run.

The triage procedure

Run these steps in order. Each one narrows the failure space by a single dimension.

1. Re-run the exact failing test three times with the same seed and the same command. All green → Class C candidate. Mixed results → Class C. All red → continue.
2. Hash the fixtures the test loads. Compare the hash with the value recorded at the last green commit. Mismatch → Class B. Regenerate the fixture, re-run, and leave the agent patch untouched.
3. Everything else is Class A. Fix the code — or, better, promote the failing input into the property test corpus so the same bug becomes a property violation instead of a one-off assertion.
4. Freeze Class C properly. Append the test to quarantine.json with a first-seen timestamp, a reason, and an expiry date. CI skips quarantined tests. The expiry is the guard: a freeze without a deadline is just deletion with extra steps.
5. Log every verdict to the same ledger. The ledger becomes the report.

The script

triage_gate.py is the minimal version of that procedure. It takes a test id and a command, runs the command three times, and writes a verdict.

#!/usr/bin/env python3
"""triage_gate.py — classify one failed test and update the quarantine ledger."""
import hashlib
import json
import subprocess
import sys
from datetime import datetime, timedelta
from pathlib import Path

LEDGER = Path("quarantine.json")
RETRIES = 3
QUARANTINE_DAYS = 7


def load() -> dict:
    if not LEDGER.exists():
        return {"fixtures": {}, "quarantine": {}}
    return json.loads(LEDGER.read_text())


def save(ledger: dict) -> None:
    LEDGER.write_text(json.dumps(ledger, indent=2))


def run(cmd: list[str]) -> int:
    return subprocess.run(cmd, capture_output=True).returncode


def fixture_hash(test_id: str) -> str:
    # In a real setup, hash the fixture files for this test.
    return hashlib.sha256(test_id.encode()).hexdigest()[:12]


def classify(test_id: str, cmd: list[str], ledger: dict) -> dict:
    results = [run(cmd) for _ in range(RETRIES)]
    passes = sum(1 for r in results if r == 0)

    if passes == RETRIES:
        verdict, reason = "C", "transient: green on all re-runs"
    elif 0 < passes < RETRIES:
        verdict, reason = "C", "flaky: inconsistent re-runs"
    else:
        current = fixture_hash(test_id)
        recorded = ledger["fixtures"].get(test_id, current)
        if current != recorded:
            verdict, reason = "B", f"fixture drift: {recorded} -> {current}"
        else:
            verdict, reason = "A", "deterministic regression"

    if verdict == "C":
        ledger["quarantine"][test_id] = {
            "first_seen": datetime.utcnow().isoformat(),
            "expires": (datetime.utcnow() + timedelta(days=QUARANTINE_DAYS)).isoformat(),
            "reason": reason,
        }
    else:
        ledger["fixtures"][test_id] = fixture_hash(test_id)
    return {"test": test_id, "verdict": verdict, "reason": reason}


if __name__ == "__main__":
    args = sys.argv[1:]
    test_id, cmd = args[0], args[1:]
    ledger = load()
    print(json.dumps(classify(test_id, cmd, ledger)))
    save(ledger)

Usage:

python triage_gate.py tests/test_parser.py::test_roundtrip \
  pytest -q tests/test_parser.py::test_roundtrip

The verdict is the contract. A, B, or C, plus a reason. Everything downstream — merge decision, reassignment to a human, or quarantine — keys off that verdict.

Note what this script is not. It is not a general flake detector: three runs is a sample, not a proof. It is a triage instrument, and its job is to make the decision explicit and repeatable, not perfect.

A ledger entry looks like this:

{
  "fixtures": {
    "tests/test_parser.py::test_roundtrip": "a1b2c3d4e5f6"
  },
  "quarantine": {
    "tests/test_io.py::test_write_then_read": {
      "first_seen": "2026-08-29T09:12:00",
      "expires": "2026-09-05T09:12:00",
      "reason": "flaky: inconsistent re-runs"
    }
  }
}

Decision table

Class	Re-run pattern	Fixture hash	Verdict	Action
A	fails every run	unchanged	deterministic regression	fix the patch, or promote the failing input into the property corpus
B	fails every run	changed	fixture drift	regenerate the fixture; the agent patch is not the suspect
C	mixed or all green	unchanged	flake	quarantine with expiry; CI skips until 5 consecutive clean runs

The Class A promotion deserves emphasis. Adding the failing input to the property test turns a single assertion into a generative check. The next time an agent patch breaks the same edge case, the gate does not just reject — it names the violated property.

Where a free tier fits — and where it does not

The triage loop has two boring jobs. The first is re-running one test three times. The second is maintaining the ledger so the verdict survives restarts. Both are tiny workloads that are idle most of the time.

This is where MonkeyCode's free model access and free server option fit the workflow. The free server option can host the ledger and the re-run loop without standing up paid infrastructure; the free model access can draft the one-paragraph failure summary attached to each ledger entry. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use that summary as context for the next step, not as the verdict. The verdict stays with the deterministic checks above. The model's note exists for the human who opens the ledger an hour later.

Do not stretch the free tier where it does not belong. A free server is a poor place for a long integration suite, a load test, or any job that needs fixed capacity. Check the current terms before building a workflow on any availability claim; quotas and policies change.

Limitations and who should not use this

The procedure assumes your failures are re-runnable. If tests depend on real network calls, external services, or wall-clock time, three re-runs classify noise, not signal. Fix that first.

The quarantine ledger has a failure mode of its own. Expiry is enforced by the script; review is enforced by you. A test that keeps re-entering quarantine for the same reason is a test that needs deletion or a rewrite, not another freeze.

Skip this approach if your suite runs in under five minutes and you have no CI log retention. The triage overhead outweighs the benefit. Skip it also if you already run retry budgets and snapshot testing at the framework level; you would be solving a problem you already solved.

The ledger is the deliverable

The gate says no. The ledger says why, and when to look again. Run the script against your last ten failed patches and count the A/B/C split. That split is the factual basis for the next decision: fix the agent, fix the fixtures, or fix the flakiness budget.