Map Callers Before You Trust a Green Agent Patch

A green suite after an agent patch is not evidence the patch is safe. The missing artifact is a caller map: which symbols changed, which tests actually reach those symbols, and which callers have no coverage at all. Fixture locks and property checks only matter on that blast radius. Flaky tests that steal budget from untested callers should be quarantined with an expiry, not silently skipped forever.

This article proposes a closed-gate workflow for Python services. It is a method, not a production case study. Examples below are labeled as such. Do not treat sample hashes, file names, or command output as measured results from a live fleet.

Why whole-suite green is the wrong signal

Agent patches rarely land only in the function named in the prompt. A helper used by four call sites can change. Three of those sites may have tests. The fourth may not. The suite still passes.

Coverage percentages hide the same hole. A line can be executed by an unrelated test and still miss the semantic contract the helper now violates. Selection by file path is also too coarse. Two functions in one module can have disjoint callers.

The gate therefore answers three questions, in order:

1. Which symbols did the patch touch?
2. Which callers of those symbols exist in production code?
3. Which of those callers are untested, fixture-unlocked, or only reached by a frozen flake?

If question 3 is non-empty, fail closed. Do not widen the suite to compensate.

Decision table: what the gate should do

Blast-radius finding	Fixture action	Property action	Flake action	Gate
Changed symbol has direct tests	Hash-lock inputs/outputs of those tests	Run invariants on the symbol	Keep flakes out of this set	Pass only if all three hold
Changed symbol has transitive tests only	Promote one characterization fixture on the public caller	Add one metamorphic pair on that caller	Do not count flake passes	Pass only after promotion
Changed symbol has an untested production caller	Block merge; do not generate fixtures from the agent	Do not invent properties from the patch	Irrelevant	Fail closed
Only tests that hit the symbol are frozen flakes	Unfreeze or replace before merge	Properties still required	TTL must be in the future and owned	Fail until a non-flake test exists
Diff touches tests but not production symbols	Re-hash fixtures; reject oracle edits without review	Unchanged	New skips need a freeze record	Fail on silent oracle rewrite

The table is the policy. The scripts below are one way to implement it locally.

1. Extract changed symbols from the diff

Parse a unified diff, then parse the new file with ast. Name-level granularity is enough for a first gate. Line-level coverage is a later refinement, not a substitute.

# proposed_harness/changed_symbols.py
from __future__ import annotations

import ast
import re
from pathlib import Path

HUNK_FILE = re.compile(r"^\+\+\+ b/(.+)$", re.M)

def files_in_diff(diff: str) -> list[str]:
    return HUNK_FILE.findall(diff)

def top_level_defs(source: str) -> dict[str, ast.AST]:
    tree = ast.parse(source)
    found: dict[str, ast.AST] = {}
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            found[node.name] = node
    return found

def changed_symbol_names(old: str, new: str) -> set[str]:
    """Return names whose AST dump changed, plus names only on one side."""
    old_defs = top_level_defs(old)
    new_defs = top_level_defs(new)
    names = set(old_defs) | set(new_defs)
    changed = set()
    for name in names:
        if name not in old_defs or name not in new_defs:
            changed.add(name)
            continue
        if ast.dump(old_defs[name], include_attributes=False) != ast.dump(
            new_defs[name], include_attributes=False
        ):
            changed.add(name)
    return changed

Run it on the working tree, not on the model's narration of the patch. Models omit files. Diffs do not.

git diff --unified=0 HEAD -- '*.py' > /tmp/agent.patch
python -c "from pathlib import Path; print(Path('/tmp/agent.patch').read_text()[:200])"

2. Build a reverse caller index

Walk production modules with ast. Record every Call whose function name matches a changed symbol. This is incomplete for getattr, decorators that rewrite names, and imports with aliases. Incomplete is acceptable if the gate fails open only on unresolved aliases, not on missing tests.

# proposed_harness/callers.py
from __future__ import annotations

import ast
from collections import defaultdict
from pathlib import Path
from typing import Iterable

class CallIndex(ast.NodeVisitor):
    def __init__(self, module: str) -> None:
        self.module = module
        self.current: str | None = None
        self.calls: dict[str, set[str]] = defaultdict(set)
        self.aliases: dict[str, str] = {}

    def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
        for alias in node.names:
            local = alias.asname or alias.name
            self.aliases[local] = alias.name
        self.generic_visit(node)

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        prev, self.current = self.current, f"{self.module}:{node.name}"
        self.generic_visit(node)
        self.current = prev

    visit_AsyncFunctionDef = visit_FunctionDef

    def visit_Call(self, node: ast.Call) -> None:
        name = None
        if isinstance(node.func, ast.Name):
            name = self.aliases.get(node.func.id, node.func.id)
        elif isinstance(node.func, ast.Attribute):
            name = node.func.attr
        if name and self.current:
            self.calls[name].add(self.current)
        self.generic_visit(node)

def index_tree(roots: Iterable[Path]) -> dict[str, set[str]]:
    merged: dict[str, set[str]] = defaultdict(set)
    for root in roots:
        for path in root.rglob("*.py"):
            if "test" in path.parts:
                continue
            mod = ".".join(path.with_suffix("").parts)
            visitor = CallIndex(mod)
            visitor.visit(ast.parse(path.read_text()))
            for symbol, callers in visitor.calls.items():
                merged[symbol].update(callers)
    return merged

Keep tests out of this index. A test that calls a helper is evidence of a test, not evidence of a production caller. Mixing the two lists is how untested production paths disappear.

3. Classify tests that actually name the symbol

A second walk over tests/ records which test functions reference the changed names. String matching on source is a start. AST Name and Attribute nodes are better. Either way, publish the classification, not a single boolean.

# proposed_harness/classify.py
from __future__ import annotations

from dataclasses import dataclass

@dataclass(frozen=True)
class Radius:
    symbol: str
    production_callers: frozenset[str]
    tests: frozenset[str]
    untested_callers: frozenset[str]

def classify(symbol: str, callers: set[str], tests_by_symbol: dict[str, set[str]]) -> Radius:
    tests = tests_by_symbol.get(symbol, set())
    # proposed rule: a caller is "tested" only if a test names that caller or the symbol.
    tested_callers = {c for c in callers if c in tests or symbol in {t.split(":")[-1] for t in tests}}
    return Radius(
        symbol=symbol,
        production_callers=frozenset(callers),
        tests=frozenset(tests),
        untested_callers=frozenset(callers - tested_callers),
    )

Print one block per changed symbol. Humans review untested callers. Agents do not get to mark them wontfix.

# proposed sample output — not a measured run
symbol: normalize_range
production_callers:
  billing.quote:apply_discount
  billing.quote:prorate
tests:
  tests/test_quote.py::test_apply_discount_zero_items
untested_callers:
  billing.quote:prorate
GATE: fail-closed

4. Lock fixtures on the affected surface only

Do not snapshot the entire repository. Hash the fixtures that feed tests in the radius. If the agent rewrites a fixture to match a new helper, the hash changes and the gate fails. That failure is the point. Oracle edits need a human.

# proposed_harness/fixture_lock.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

def file_sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def lock_paths(paths: list[Path], lockfile: Path) -> None:
    payload = {str(p): file_sha256(p) for p in sorted(paths)}
    lockfile.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")

def verify_lock(lockfile: Path) -> list[str]:
    expected = json.loads(lockfile.read_text())
    broken = []
    for rel, digest in expected.items():
        path = Path(rel)
        if not path.is_file() or file_sha256(path) != digest:
            broken.append(rel)
    return broken

Pair the lock with a short allow-list of fixture directories the agent may not write:

git diff --name-only HEAD -- tests/fixtures | tee /tmp/fixture_writes.txt
test ! -s /tmp/fixture_writes.txt
python -c "from proposed_harness.fixture_lock import verify_lock; from pathlib import Path; err = verify_lock(Path('tests/fixtures.lock.json')); raise SystemExit(1 if err else 0)"

5. Property checks on the changed symbol, not the prompt

Characterization tests record what the code did. Property checks state what it must still do after a rewrite. Put properties next to the symbol the caller map named. One relation per changed helper is enough to start.

# proposed example — unexecuted
# tests/properties/test_normalize_range.py
import pytest

from billing.normalize import normalize_range

@pytest.mark.property
def test_normalize_range_is_idempotent():
    samples = [(0, 0), (1, 1), (2, 9), (9, 2), (-3, 4)]
    for lo, hi in samples:
        once = normalize_range(lo, hi)
        assert normalize_range(*once) == once

@pytest.mark.property
def test_normalize_range_span_non_negative():
    samples = [(0, 5), (5, 0), (3, 3)]
    for lo, hi in samples:
        a, b = normalize_range(lo, hi)
        assert b - a >= 0

Metamorphic pairs beat exact snapshots when the helper's byte-for-byte output is allowed to change. Idempotence, ordering, and conservation are typical. If you cannot name one relation, the symbol is not ready for an agent rewrite.

6. Freeze flakes as capacity theft, with a TTL

A flaky test that sometimes hits the changed symbol is not a pass. It is a hole that looks like coverage. Quarantine it in a file the gate reads. Require an owner and an expiry. After expiry, the test returns as a failure, not as a skip.

{
  "tests/test_quote.py::test_prorate_dst_boundary": {
    "reason": "timezone database drift on CI image",
    "owner": "billing-oncall",
    "expires": "2026-09-20",
    "blocks_symbols": ["prorate"]
  }
}

# proposed_harness/flake_freeze.py
from __future__ import annotations

import json
from datetime import date
from pathlib import Path

def load_freeze(path: Path, today: date) -> dict[str, dict]:
    data = json.loads(path.read_text())
    active = {}
    expired = {}
    for nodeid, meta in data.items():
        expires = date.fromisoformat(meta["expires"])
        if expires < today:
            expired[nodeid] = meta
        else:
            active[nodeid] = meta
    if expired:
        raise RuntimeError(f"expired flake freezes: {sorted(expired)}")
    return active

def radius_blocked_by_flakes(radius_tests: set[str], freeze: dict[str, dict]) -> set[str]:
    frozen = set(freeze) & radius_tests
    return frozen

If every test in the radius is frozen, the gate fails even though pytest -q is quiet. Re-admit requires a non-flake test, not a later expiry.

7. Wire the gate in front of the model loop

Keep the model off the critical path of the gate. Generate a patch, apply it on a branch, run the caller map, then run only the tests the map named plus the property module. The free-server option is useful here as a disposable worker that never holds production credentials.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can propose the patch. The free server option can run the gate. Neither replaces the caller map, the fixture lock, or the flake TTL. This article does not claim specific model names, quotas, hardware, or benchmark numbers.

A minimal operator loop looks like this:

# proposed operator loop — review before use on a shared repo
set -euo pipefail
git checkout -B agent/radius-gate
# apply the candidate patch from the working tree, then:
python -m proposed_harness.gate \
  --diff /tmp/agent.patch \
  --src billing \
  --tests tests \
  --fixture-lock tests/fixtures.lock.json \
  --flake-freeze tests/flake_freeze.json \
  --today 2026-09-06
pytest -q $(cat /tmp/required_tests.txt) tests/properties

The gate process should return non-zero when any changed symbol has an untested production caller, a broken fixture hash, an expired freeze, or a radius that is only reachable through active freezes.

Limitations

The AST caller index misses dynamic dispatch, string imports, frameworks that resolve views by name, and C extensions. Aliased imports are only handled one level deep. Cross-language repos get no signal.

Property checks written by the same model that wrote the patch are correlated failures. A human should name the relation. Fixture locks do not detect semantic oracle drift when the agent adds a new fixture file that is not yet in the lock. The lock step must scan for untracked files under tests/fixtures.

Flake freezes become a junk drawer if expiry is optional. Make expiry mandatory. Make ownership mandatory. Do not let the model edit flake_freeze.json.

This workflow does not claim mutation scores, wall-clock savings, or production incident reductions. Those require measurement on your suite.

Who should not use this

Do not use a fail-closed caller map as the only control on safety-critical code. Add human review, typed contracts, and, where the risk warrants it, mutation testing with a known tool you already run.

Skip this approach if the repository is not Python, if tests are generated at runtime without stable nodeids, or if production call graphs live only in a service mesh. Skip it if a mature test-impact system already selects by coverage traces and you will not maintain a second index.

Skip it if the team will treat freeze files as a way to ship. A TTL that is always extended is not a freeze. It is an untested caller with extra JSON.

What to implement first

Start with changed-symbol extraction and untested-caller failure. Add fixture hashes on the radius next. Add one property module for the hottest helper. Add flake TTL last. That order keeps the gate honest: untested callers cannot hide behind a skip.

If you already run patch loops against a free model on a free server, put this gate between apply and merge. The map is the review surface. The model is not.