A Zero-Token Triage for Agent-Patch Failures

#ai #ci #llm #testing

An agent opens a pull request at 2 AM. The summary says small refactor, all tests pass; nobody ran the tests. The reviewer clicks approve anyway.

This is the new review loop. AI turned every developer into a reviewer. The review tooling never arrived. The fix is not a better prompt. The fix is a cheaper gate.

Most agent-patch failures have three causes. The model changed behavior. The server was unavailable. The test itself was flaky. A pull request cannot tell these apart. A triage pipeline can. It runs before any LLM review. It spends zero tokens.

Run fixtures first

Every accepted patch leaves a trace. Save its inputs and outputs as JSON files under tests/fixtures. A new patch must reproduce those outputs exactly. The check runs in seconds. Most regressions die here. No model call happens.

The fixture corpus is your project memory. Capture new fixtures automatically on merge. Delete or edit them only with a human decision.

Run properties second

Fixtures cover known cases. Properties cover unknown ones. State invariants about the changed functions. A parser should satisfy parse(format(x)) == x. A time parser should reject impossible dates. A deduplicator should preserve order.

Hypothesis generates the edge cases for you. The checks are deterministic and fast. They cost nothing. They catch what fixtures miss.

Freeze flaky tests third

A random failure destroys trust in the pipeline. Give a suspect test three attempts. Three failures in a row mean quarantine. A quarantined test never blocks a review.

Review the quarantine weekly. Fix the test or delete it. Freezing is not forgetting. It moves noise off the critical path.

Review only survivors

The patch passed fixtures, properties, and reruns. Now a model may look. A free model endpoint writes the summary. The summary joins the classified report. No flaky noise reaches the model. The human reads a clean map and makes the final call.

The artifact

The script below implements that order. It classifies failures as passed, failed, or flaky. It prints a JSON report you can attach to the pull request. This is a sketch, not a package. Adapt it to your repository.

#!/usr/bin/env python3
"""Zero-token triage for agent patches. A sketch, not a package."""
import json
import subprocess
import sys
from pathlib import Path

FIXTURE_DIR = Path("tests/fixtures")
RERUNS = 3

def run(cmd, timeout=300):
    try:
        result = subprocess.run(
            cmd, shell=True, capture_output=True, text=True, timeout=timeout
        )
        return result.returncode, (result.stdout + result.stderr)[-400:]
    except subprocess.TimeoutExpired:
        return 124, "timeout"

def triage(patch_dir):
    report = {"passed": [], "failed": [], "flaky": []}

    for fixture in sorted(FIXTURE_DIR.glob("*.json")):
        code, out = run(f"python fixture_runner.py {fixture} {patch_dir}")
        entry = {"fixture": fixture.name, "exit": code, "out": out}
        target = report["passed"] if code == 0 else report["failed"]
        target.append(entry)

    code, out = run("pytest tests/properties -q", timeout=600)
    entry = {"suite": "properties", "exit": code, "out": out}
    target = report["passed"] if code == 0 else report["failed"]
    target.append(entry)

    code = 0
    for attempt in range(RERUNS):
        code, _ = run("pytest tests/flaky_candidates -q", timeout=900)
        if code == 0:
            break
    if code != 0:
        report["flaky"].append({"quarantined": True, "attempts": RERUNS})

    return report

if __name__ == "__main__":
    print(json.dumps(triage(sys.argv[1]), indent=2))

The sketch assumes two pytest directories exist. tests/properties holds the invariant checks. tests/flaky_candidates holds the repeat offenders. Both need a placeholder test. Pytest exits nonzero on an empty suite. The fixture_runner.py script is your responsibility. Load the fixture, apply the patch, compare outputs.

A sample report looks like this:

{
  "passed": [{"fixture": "accept_01.json", "exit": 0}],
  "failed": [{"suite": "properties", "exit": 1}],
  "flaky": [{"quarantined": true, "attempts": 3}]
}

Anything in failed blocks the review. Anything in flaky enters quarantine quietly. Everything in passed moves to the model summary. That split is the point. You learn what to distrust before you pay for a verdict.

Why a free tier fits

This loop was designed for small budgets. Free servers sleep and throttle. Free model access has rate limits. None of that matters. The triage rarely calls a model. The expensive step sees only survivors.

MonkeyCode is an open-source project for agent workflows. It fits at exactly this point. Its free model access can generate the summary. Its free server option can host the triage run. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

As of late August 2026, the free tier includes a ten-million-token grant and a free server option. Free-tier terms change often. Confirm the current limits in the MonkeyCode repository before relying on them.

Limits

This triage cannot catch semantic drift. Fixture replay checks recorded behavior only. Property checks need an invariant worth stating. A quarantine without weekly review hides real defects. Free tiers can throttle at the worst moment. Use paid infrastructure for strict SLAs. Use this triage for deterministic code and a tiny budget.

Do not treat the model summary as approval. Treat it as a map for a human. Run this script before your next review. If the budget fits, try MonkeyCode's free tier as a starting point. Confirm today's numbers in the repository. Run it once. Read the report. Then decide if the patch ships.

DEV Community