Put Property Checks on the Impact Cone, Not on the Edited Function

An agent patch is local. Failures are not. A green pytest job on the files a model touched does not prove that callers still hold their invariants. Classify the patch, map the reverse-dependency cone, and attach property checks plus hashed fixtures to those public entry points. Quarantine flakes by failure signature, not by test name.

Green on the edited file is the wrong unit

Agent diffs cluster. One function is rewritten. Tests that live beside it are the cheapest to update, and they often land in the same commit. That pairing is not independent evidence. It is a restatement of the new code.

Callers keep old assumptions about types, ordering, error codes, and partial failure. Those callers are usually outside the path the agent executed. The suite can be green. The cone can be broken.

Grep is not a cone. Registries, plugin hooks, and string-based dispatch will not match the function identifier. You still want a conservative static cone: importers of the changed module, name-based callees, and every public function in the same package that imports the changed module. Promote the set when the static picture is incomplete. Extra tests are cheaper than a silent behavior change at a public entry.

Classify the patch before you choose tests

One CI job for every agent commit treats a comment-only change and a control-flow rewrite as the same event. They are not. Classify first. Promote the class when the signal is mixed. A false cosmetic label is a merge incident. Extra CI minutes are not.

Class	Diff signal	Required evidence
Cosmetic	Comments, whitespace, import order only	AST dump equal after stripping location fields
Local	One pure function, public signature unchanged	Properties on that function and cone smoke
Contract	Signature, types, or serialized shape changed	Contract checks at every public cone entry
Cross-module	Several modules or a control-flow rewrite	Characterization pins on the full cone

The class is a CI input, not a label in the pull request description. If the agent writes "refactor, no behavior change," ignore it. The trees decide.

1. Extract a structural diff

Text hunks lie. Formatters and comment rewrites look large. Parse both sides and compare dumps, not lines. The snippet below is a proposed harness. Run it against two worktrees or two git blobs. It does not execute the patch.

# cone_classify.py — proposed harness
from __future__ import annotations

import ast
import hashlib
from dataclasses import dataclass


@dataclass(frozen=True)
class FileClass:
    path: str
    kind: str  # unchanged | cosmetic | local | contract | cross
    before_hash: str
    after_hash: str


def _zero_locs(tree: ast.AST) -> ast.AST:
    for node in ast.walk(tree):
        for field in ("lineno", "col_offset", "end_lineno", "end_col_offset"):
            if hasattr(node, field):
                setattr(node, field, 0)
    return tree


def module_dump(src: str) -> str:
    return ast.dump(_zero_locs(ast.parse(src)), include_attributes=False)


def top_signatures(src: str) -> dict[str, str]:
    tree = ast.parse(src)
    out: dict[str, str] = {}
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            args = ast.dump(node.args, include_attributes=False)
            returns = ast.dump(node.returns, include_attributes=False) if node.returns else ""
            out[node.name] = f"{args}->{returns}"
    return out


def classify_source(path: str, before: str, after: str) -> FileClass:
    bh = hashlib.sha256(before.encode()).hexdigest()[:12]
    ah = hashlib.sha256(after.encode()).hexdigest()[:12]
    if before == after:
        return FileClass(path, "unchanged", bh, ah)
    if module_dump(before) == module_dump(after):
        return FileClass(path, "cosmetic", bh, ah)
    sig_b, sig_a = top_signatures(before), top_signatures(after)
    if sig_b != sig_a:
        return FileClass(path, "contract", bh, ah)
    changed_fns = [
        name for name, dump in _fn_dumps(before).items()
        if _fn_dumps(after).get(name) != dump
    ]
    kind = "local" if len(changed_fns) <= 1 else "cross"
    return FileClass(path, kind, bh, ah)


def _fn_dumps(src: str) -> dict[str, str]:
    tree = _zero_locs(ast.parse(src))
    return {
        n.name: ast.dump(n, include_attributes=False)
        for n in tree.body
        if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))
    }

Package-level class is the maximum of file classes, with cross > contract > local > cosmetic. Unchanged files drop out. If any file is contract or cross, the cone must run. Cosmetic-only diffs can stop after the AST equality check.

2. Build the impact cone from imports and names

Walk the package once. Record which modules import the changed module. Record which functions call the changed names. Union those sets. That union is the cone. It is conservative on purpose. Dynamic dispatch will still leak; the next section compensates with public-entry pins, not with a promise of completeness.

# impact_cone.py — proposed static walk
from __future__ import annotations

import ast
from pathlib import Path


def iter_py(root: Path):
    yield from root.rglob("*.py")


def imports_of(src: str) -> set[str]:
    tree = ast.parse(src)
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            names.update(a.name.split(".")[0] for a in node.names)
        elif isinstance(node, ast.ImportFrom) and node.module:
            names.add(node.module.split(".")[0])
    return names


def call_names(src: str) -> set[str]:
    tree = ast.parse(src)
    out: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Call):
            f = node.func
            if isinstance(f, ast.Name):
                out.add(f.id)
            elif isinstance(f, ast.Attribute):
                out.add(f.attr)
    return out


def public_functions(src: str) -> list[str]:
    tree = ast.parse(src)
    return [
        n.name for n in tree.body
        if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef))
        and not n.name.startswith("_")
    ]


def cone(root: Path, changed_modules: set[str], changed_names: set[str]) -> dict:
    entries: list[str] = []
    for path in iter_py(root):
        src = path.read_text(encoding="utf-8")
        mod = path.stem
        touched = bool(imports_of(src) & changed_modules)
        touched = touched or bool(call_names(src) & changed_names)
        touched = touched or mod in changed_modules
        if not touched:
            continue
        for fn in public_functions(src):
            entries.append(f"{mod}.{fn}")
    return {"public_entries": sorted(set(entries))}

Feed changed_modules from the git path list, not from the model summary. Feed changed_names from the structural diff. Persist the JSON. The next CI step fails if a public entry in that JSON has no pin.

3. Require a characterization pin per public entry

Each public function in the cone needs one fixture on disk: canonical inputs, canonical outputs, and a lock hash. Agent patches may not rewrite the lockfile in the same commit as the code. If both change together, the pin certified the new behavior and proved nothing about the old one.

# pins.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

LOCK = Path("tests/cone_locks.json")


def digest(obj) -> str:
    blob = json.dumps(obj, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()


def load_lock() -> dict:
    return json.loads(LOCK.read_text(encoding="utf-8")) if LOCK.exists() else {}


def assert_pin(entry: str, output) -> None:
    lock = load_lock()
    if entry not in lock:
        raise AssertionError(f"missing cone pin for {entry}")
    got = digest(output)
    if got != lock[entry]["sha256"]:
        raise AssertionError(
            f"cone pin drift for {entry}: {got} != {lock[entry]['sha256']}"
        )

Pins are inherited evidence. They live on main before the agent runs. A separate, reviewed commit may rotate a pin after a human names the behavior change. Same-commit rotation is a class error: treat it as cross with no inherited oracle.

4. Attach properties to cone entries, not to the edited function

Local property tests on the rewritten function catch local bugs. They miss caller invariants. Put the properties on the public entries the cone named. The edited function is an implementation detail of those entries.

# tests/test_cone_properties.py — example properties, adapt to your entries
import json
from pathlib import Path

import pytest

from pins import assert_pin
from yourpkg.api import parse_record, list_records  # replace with real entries

FIXTURES = Path("tests/fixtures/cone")


@pytest.mark.parametrize("name", sorted(p.stem for p in FIXTURES.glob("*.json")))
def test_parse_record_roundtrip_on_cone(name):
    payload = json.loads((FIXTURES / f"{name}.json").read_text(encoding="utf-8"))
    parsed = parse_record(payload)
    assert parsed["id"] == payload["id"]
    assert parsed["id"]  # non-empty
    dumped = json.loads(json.dumps(parsed))
    assert dumped["id"] == parsed["id"]
    assert_pin("api.parse_record", dumped)


def test_list_records_preserves_order_and_ids():
    rows = list_records(limit=50)
    ids = [r["id"] for r in rows]
    assert ids == sorted(ids) or ids == list(dict.fromkeys(ids))
    assert len(ids) == len(set(ids))
    assert_pin("api.list_records", ids)

The properties are boring on purpose. Identity, order, uniqueness, and round-trip are the checks callers actually rely on. Domain-specific properties belong here too, but they still hang off the cone entry, not off the helper the agent rewrote.

5. Run inherited tests only on the first gate

Split the suite. Inherited tests are files that exist on main and are not modified in the agent commit. Candidate tests are files the agent added or edited. The first gate runs inherited tests plus cone pins. Candidate tests are a second, reviewed gate. They may become inherited after a human accepts them. They may not greenlight the patch that introduced them.

# proposed CI shape — labels only, not a measured runtime
git fetch origin main
INHERITED=$(git diff --name-only origin/main...HEAD -- tests | sed 's/^/^/' )
pytest tests --ignore-glob='tests/agent_*' -q
python cone_classify.py
python impact_cone.py
pytest tests/test_cone_properties.py -q

If you generate candidate patches through MonkeyCode's free model access and run this harness on the free server option, treat the model as a patch source only. Classification and cone pins still run on the checkout. They do not run on the model's summary of what it changed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

6. Quarantine flakes by failure signature, not by test name

Name-based skips hide the next bug in the same test. A frozen test_list_records will not fire when the assertion changes from order to uniqueness. Key the quarantine on the test node id plus a normalized traceback shape. Drop line numbers and assertion values. Keep the assertion type and the last application frame. If the signature changes, the freeze does not apply.

# tests/conftest.py — proposed signature freeze
from __future__ import annotations

import hashlib
import json
import re
from pathlib import Path

FREEZE = Path("tests/flake_signatures.json")
APP_ROOT = "yourpkg/"


def _shape(longrepr: str) -> str:
    text = str(longrepr)
    text = re.sub(r":\d+", ":N", text)
    text = re.sub(r"0x[0-9a-fA-F]+", "0xADDR", text)
    text = re.sub(r"\d+", "D", text)
    frames = [ln.strip() for ln in text.splitlines() if APP_ROOT in ln or "AssertionError" in ln]
    return "|".join(frames[-6:])


def signature(nodeid: str, longrepr: str) -> str:
    raw = nodeid + "\n" + _shape(longrepr)
    return hashlib.sha256(raw.encode()).hexdigest()[:16]


def pytest_runtest_makereport(item, call):
    if call.when != "call" or call.excinfo is None:
        return
    frozen = json.loads(FREEZE.read_text()) if FREEZE.exists() else {}
    sig = signature(item.nodeid, call.excinfo.exconly() + "\n" + str(item.location))
    rec = frozen.get(item.nodeid)
    if rec and rec.get("sig") == sig:
        item.add_marker("xfail(strict=False, reason='signature-frozen flake')")

Write freeze records in a reviewed commit, never in the agent commit. A new signature on a frozen node id is a failure. That is the point. The same test can still catch a different bug.

What this strategy does not claim

An AST cone misses getattr, decorator-renamed functions, and runtime registries. Characterization pins freeze production bugs if main is already wrong. Coarse signatures can hide a new intermittent failure under an old key if you normalize too much. None of these checks replace review on auth, crypto, or money paths. They also do not score assertion wording. They score whether public entries still behave like main.

The harness above is a proposed workflow. It is not a benchmark. It does not assert a pass rate, a latency, or a model ranking. If classification disagrees with a human, promote the class and keep the cone.

Who should not use this

Skip this approach on a greenfield repo with no inherited suite. There is no cone to pin. Skip it if the agent is allowed to edit lockfiles, freeze records, and tests in the same commit: the cone will certify itself. Skip it for one-off scripts with a single entry point; a local property test is enough. Skip it when the patch is supposed to change behavior and nobody has written the new pins yet. In that case the merge artifact is the pin rotation, not a green inherited job.

If you already emit agent patches into a repo with a locked main, keep the cone lockfile next to the tests. Refuse merges that rewrite the lock and the code together. The edited function is not the unit under test. The public entries that depend on it are.