Scope Creep Is the Bug: Measure an Agent Patch's Blast Radius Before You Trust Its Tests

A patch that picks which tests it needs is a patch that decides how much it gets checked. That is the second half of agent-patch review, and it is the half most teams skip. Oracle independence tells you whether the check could have failed; scope tells you whether the check was pointed at the code the diff actually touched.

This is a walkthrough of a small planner that answers the second question from the diff itself, then hands the merge decision back to a human with an explicit promote-or-block verdict.

The failure mode, stated concretely

An agent edits app/pricing/tax.py, adds tests/test_tax_case_7.py, and reports "12 passed" on the file it wrote. The suite for that file is green. Nothing in the report mentions that app/pricing/tax.py is imported by app/checkout/session.py, whose tests never ran.

That is not a flaky test, and it is not a weak assertion. It is a run that was scoped by the author of the change. The example is illustrative, not a reproduction of a specific incident, but the shape is common enough that it deserves a mechanical check rather than a reviewer's memory.

One prerequisite: you need a pinned baseline revision and a pinned pre-fix seed for any property band you run. If the baseline moves under you, every number below is noise.

Step 1 — Freeze the revision and read the diff

The planner starts from a commit range, never from a working tree. A dirty tree means the file list and the executed code can disagree, and that disagreement is invisible in the output.

git rev-parse HEAD > run-manifest/revision.txt
git diff --name-only --diff-filter=ACMR "$BASE...HEAD"

--diff-filter=ACMR keeps added, copied, modified, and renamed paths. Deletions still matter for imports, so if your toolchain can break on a removed module, widen the filter and handle it explicitly.

Step 2 — Build the import closure from the diff outward

The question is not "which files changed" but "which modules can reach what changed." A reverse import closure answers that, and Python's standard library is enough to build it for a first pass.

#!/usr/bin/env python3
"""impact_set.py — test-scope planner for agent patches.

Reference implementation. Adapt the resolver to your build system.
Exit codes: 0 = scoped run is sufficient, 2 = promote to full suite,
            3 = scope escape.
"""
from __future__ import annotations

import ast
import subprocess
from pathlib import Path

PKG = "app"
TESTS = "tests"
PROTECTED = ("app/billing", "app/auth", "app/migrations")


def git(*args: str) -> str:
    return subprocess.run(["git", *args], check=True,
                          capture_output=True, text=True).stdout


def changed_files(base: str) -> list[str]:
    out = git("diff", "--name-only", "--diff-filter=ACMR", f"{base}...HEAD")
    return [p for p in out.splitlines() if p.strip()]


def module_of(path: Path) -> str | None:
    if path.suffix != ".py":
        return None
    parts = [p for p in path.with_suffix("").parts if p != "__init__"]
    return ".".join(parts) if parts else None


def imports_of(path: Path) -> set[str]:
    try:
        tree = ast.parse(path.read_text(encoding="utf-8"))
    except SyntaxError:
        return set()
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names |= {a.name for a in node.names}
        elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module:
            names.add(node.module)
    return {n for n in names if n == PKG or n.startswith(PKG + ".")}


def build_graph(roots: tuple[str, ...]) -> dict[str, set[str]]:
    graph: dict[str, set[str]] = {}
    for root in roots:
        for f in Path(root).rglob("*.py"):
            mod = module_of(f)
            if mod:
                graph[mod] = imports_of(f)
    return graph


def reverse_closure(seeds: set[str], graph: dict[str, set[str]]) -> set[str]:
    dependents: dict[str, set[str]] = {}
    for mod, deps in graph.items():
        for dep in deps:
            dependents.setdefault(dep, set()).add(mod)
    seen, stack = set(seeds), list(seeds)
    while stack:
        for up in dependents.get(stack.pop(), ()):
            if up not in seen:
                seen.add(up)
                stack.append(up)
    return seen

Two properties matter here. The closure is computed from the repository, not from the patch description, and it only ever grows the run. A planner that can shrink the suite is a planner that can hide a regression.

Step 3 — Resolve the closure onto test files

Test files enter the impact set in two ways: they changed, or they import something reachable from the change. Everything else is out of scope for the fast band.

def impact_tests(changed: set[str], reachable: set[str]) -> list[str]:
    selected = {t for t in changed if t.startswith(TESTS + "/")}
    for t in Path(TESTS).rglob("test_*.py"):
        if imports_of(t) & reachable:
            selected.add(str(t))
    return sorted(selected)

Run the planner and read the verdict rather than the list:

python tools/impact_set.py --base "$(cat baseline.txt)"
echo "exit=$?"

Step 4 — Turn the verdict into a merge rule

Exit codes are only useful if something acts on them. The table below is the part I would copy first, because it encodes the decisions reviewers usually make by feel.

Signal from the planner	Required action	Merge status
Changed files inside declared scope, impact set inside scope	Run impact tests plus the property band	Allowed if green
Changed file outside declared scope (exit 3)	Full suite; update the scope file first	Blocked until scoped
Impact set touches a protected module (exit 2)	Full suite regardless of declared scope	Allowed only on full-suite green
Flake ledger has an entry with no expiry	Full suite; ledger is not fail-closed	Blocked
Property band errors or times out	No retry on a new seed; keep the failing seed	Blocked

The declared scope should live in the repository as a plain text list of path prefixes, edited by a human in the same pull request. If the agent can edit the scope file and the code in one commit, you have moved the trust boundary rather than established one.

Step 5 — Hash the fixtures the run depends on

Fixtures drift quietly. A test that reads tests/fixtures/pricing_seed.json can pass for months against a snapshot nobody re-generated.

find tests/fixtures -type f -print0 | sort -z \
  | xargs -0 sha256sum > run-manifest/fixtures.sha256
sha256sum -c run-manifest/fixtures.sha256

Record the manifest alongside the revision. If a fixture hash changes while no test file changed, treat that as a scope escape and re-run the full suite. The manifest is a record, not a lock; its value comes from being compared against the previous run, not from existing.

Step 6 — Keep the flake ledger fail-closed

A quarantine list without an expiry becomes a permanent exemption, and a permanent exemption is an unchecked test wearing a status badge. Every entry needs an owner and a date, the check needs to fail when either is missing, and it needs to fail closed rather than skipping the affected test silently.

I covered the freeze rule for that in an earlier post; the only thing this planner adds is a precondition. If the ledger check exits non-zero, the impact-set verdict does not matter, because the fast band you are about to run contains tests you have already agreed not to trust.

Where to run the planner

The planner must run on the patched revision in a clean checkout. Agent-authored changes often arrive with local artifacts, uncommitted edits, or a virtualenv that differs from CI, and none of that should influence which tests get selected.

That is a good fit for a disposable remote box. MonkeyCode provides free model access and a free server option, and I have used both as the environment for this kind of harness: the server gives you a clean checkout to run the planner against, and the free model access is useful for drafting candidate property checks that a human then classifies. Treat any quota, hardware, or lifetime claim as unverified until you check it against the live product, and do not put secrets in a shared run environment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations, and who should not use this

Static import analysis does not see dynamic imports, plugin registries, dependency-injection containers, string-based module loading, generated code, or template-rendered call paths. In those repositories the closure under-reports, and an under-reported closure is more dangerous than no closure, because it looks precise. Use coverage instrumentation from a known-good run to supplement the graph, and keep the full suite in the merge path for protected areas regardless.

This approach is also a poor fit if you cannot pin a baseline commit, if your tests depend on shared mutable state across files, or if your team has no owner for the flake ledger. In those conditions, the planner will produce confident numbers that nothing downstream can act on.

Start with the scope file and the exit codes, run the planner in report-only mode for two weeks, and compare its impact sets against what your full suite actually catches. If the fast band and the full suite disagree even once, the disagreement is the finding.

If you want to try it without touching your own CI, the free server option in MonkeyCode is a reasonable place to run the first version.