A Patch Triage Matrix: Which Tests Actually Matter for an Agent's Diff?

Running the full suite on every agent patch is the most expensive way to get almost no signal. A five-line diff that renames a variable inside a pure function can trigger a flaky integration test and mark the patch red. The failure is noise, but it eats minutes and attention.

The solution is not a bigger suite. It is a triage step before the suite runs: map the patch to a category, then execute only the checks that can catch real regressions in that category. This article shows that matrix, a Python script that implements it, and where a free model endpoint and a free server fit into the workflow.

The cost of "run everything"

A broad regression run feels safe. It also multiplies three problems:

1. Runtime cost. Full integration suites on every patch consume CI minutes that small teams and free tiers often do not have.
2. Signal dilution. Most tests are unrelated to the changed lines; their failures teach you little about the patch.
3. Flaky amplification. The more tests you run, the higher the chance that one unrelated flaky test fails and blocks the merge.

A triage matrix moves the decision from "what passed?" to "what could this patch have broken?"

The triage matrix

Change category	Checks to run	Where to execute
Pure function logic added or modified	Property tests + unit regression	Local worker or main runner
Signature or return type changed	Compile/type check + call-site scan	Fast local step
Error path moved or introduced	Error-contract tests	Worker
I/O boundary touched (DB, HTTP, filesystem)	Fixture-contract tests + fixture drift check	Server with fixed fixture snapshot
Dependency or config changed	Smoke test + environment diff	Small VM or cron runner
Concurrency changed	Short soak test / race detector	Dedicated worker

The categories are intentionally coarse. Their job is not perfect classification; it is to prevent the default "run all 2,000 tests" response.

Encoding the matrix as a script

Here is a runnable Python classifier. It starts with a rule-based heuristic, so it works even without a model. If a model is available, it can refine the category before the checks are chosen.

# classify_patch.py
import re

CHECKS = {
    "pure_logic": ["property", "unit"],
    "signature": ["compile", "call_scan"],
    "error_path": ["error_contract"],
    "io_boundary": ["fixture_contract", "fixture_drift"],
    "config_change": ["smoke", "env_diff"],
    "concurrency": ["soak_short"],
}

def heuristic_category(diff_text: str) -> str:
    if any(k in diff_text for k in ("asyncio", "Lock(", "threading", "Queue(")):
        return "concurrency"
    if any(k in diff_text for k in ("try:", "except ", "raise ")):
        return "error_path"
    if "def " in diff_text and re.search(r"->\s*\w+", diff_text):
        return "signature"
    if any(k in diff_text for k in ("session.", "client.", "open(", "connect(", "requests.")):
        return "io_boundary"
    if any(k in diff_text for k in ("os.environ", ".env", "config")):
        return "config_change"
    return "pure_logic"

def make_plan(diff_text: str, with_model: bool = False) -> dict:
    category = heuristic_category(diff_text)
    if with_model:
        # Optional refinement step. Hook this to an OpenAI-compatible endpoint.
        refined = ask_model_for_category(diff_text)
        category = refined or category
    return {"category": category, "checks": CHECKS[category]}

The ask_model_for_category function is intentionally left as an interface. You can point it at the free model access included in MonkeyCode's product offering; any compatible model or a local LLM works, because the model only refines a heuristic that already has a reasonable default.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Executing the plan on a free server

The second half of the workflow is execution. The plan runs on a disposable server rather than a laptop. The server does three things:

1. Picks the check names from the plan.
2. Masks or overlays the fixture snapshot for fixture_contract runs.
3. Freezes any failing test for a bounded time if the failure is classified as flaky.

The free server option fits here: a small VM is enough for this, and it keeps the review pipeline isolated from local state. A cron entry can handle the nightly batch of patches:

0 2 * * * cd /repo && python classify_patch.py --plan "$(git diff HEAD~1)" && ./run_checks

This is not a replacement for full CI. It is a cheap pre-filter that gives a reviewer a focused red/green signal on the parts of the patch that can actually fail.

Limitations and who should not use this

The triage matrix has real trade-offs.

• The heuristic categories are syntax-based, so they can misclassify a patch that touches both I/O and error handling.
• Model refinement is useful, but it should never be trusted without human review; a low-quality suggestion can silently skip an important check.
• Fixture snapshots add maintenance overhead. If your data model changes daily, the snapshot can become the source of flaky failures.
• A bounded flaky freeze can hide a real bug if the expiration period is too long. Keep it short (a 48-hour expiry is a reasonable default) and log every freeze.

Do not use this approach for safety-critical systems, for regulatory audits, or in projects where a missed test can directly cause data loss. Those contexts need a deterministic, exhaustive pipeline, not a triage heuristic.

The matrix will never be the final word on agent patches. It is a decision aid that forces a question many reviewers skip: which failure modes are actually plausible for this diff? That question is worth asking before you spend ten minutes waiting for a suite that can't tell you anything.

What does your matrix look like? I am curious which categories you would add before approving an agent-generated diff.