Dangling Names After Patch: A 90-Minute Agent Spike

Unresolved names after an agent patch should kill the run. A ninety-minute spike can prove that kill gate. Reviewers should not hunt missing imports by hand.

The hypothesis

An agent may rewrite a function and drop a callee. Tests can still pass on the happy path. The public name graph then lies until runtime.

The kill rule should stay simple and mechanical. After the patch, every remaining name must resolve. New unresolved names mean the entire run is dead.

This is not a style debate about generated code. It is a ship-or-kill check with a hard clock. The budget is ninety minutes, one fixture, one harness.

Why unresolved names beat chat review

Chat review reads the diff as fluent prose. Name resolution reads the tree as a graph. Call graphs do not flatter a fluent patch.

Four failure modes show up in these tiny fixtures. Each case is cheap to detect with a static scan.

• A helper is inlined, but one caller still mentions it.
• A module is renamed, while tests import the old path.
• A type alias moves, and annotations keep the stale token.
• A re-export is deleted, and plugins still import the name.

None of them need a model to explain.

Current talk about measuring coding models stays at benchmarks. A local symbol table is a harsher, smaller oracle. It does not care about fluent prose.

Scope for the ninety minutes

Do not build a language server in this spike. Do not parse every language on disk. Pick one fixture language and one resolver.

This write-up uses Python on a tiny package. The same kill rule ports to tsc --noEmit. The same kill rule also ports to go build.

Treat the text as a proposed protocol, not a study. No live customer metrics appear in this write-up. All numbers below come from the fixture only.

Fixture layout

Create a throwaway package with only three modules. Keep the public surface small on purpose.

spike_symbols/
  app/
    __init__.py
    pricing.py
    notify.py
    legacy.py
  tests/
    test_pricing.py
  harness/
    snapshot_names.py
    gate.py
  Makefile

The pricing module exports quote_total and calls apply_tax. The notify module imports quote_total for a format helper. The legacy module still imports apply_tax as a public name.

Keep the agent task as one sentence only. Inline tax into quote_total and delete apply_tax. That task is a known trap for fluent models.

Naive inlining updates pricing and skips the legacy module. Happy path tests still pass after that miss. The unresolved-name gate exists for this gap.

Name snapshot before the agent

The snapshot is a sorted set of names. It is not a full call graph. Ninety minutes cannot afford a graph database.

# harness/snapshot_names.py
from __future__ import annotations

import ast
import sys
from pathlib import Path

ROOT = Path("app")


def module_name(path: Path) -> str:
    rel = path.with_suffix("").relative_to(ROOT.parent)
    return ".".join(rel.parts)


def defined_names(tree: ast.AST) -> set[str]:
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            names.add(node.name)
        elif isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name):
                    names.add(target.id)
    return names


def imported_names(tree: ast.AST) -> set[str]:
    names: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom):
            for alias in node.names:
                names.add(alias.name)
        elif isinstance(node, ast.Import):
            for alias in node.names:
                names.add(alias.name.split(".")[0])
    return names


def used_names(tree: ast.AST) -> set[str]:
    return {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)}


def scan() -> dict[str, dict[str, list[str]]]:
    report: dict[str, dict[str, list[str]]] = {}
    for path in sorted(ROOT.rglob("*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        report[module_name(path)] = {
            "defined": sorted(defined_names(tree)),
            "imported": sorted(imported_names(tree)),
            "used": sorted(used_names(tree)),
        }
    return report


if __name__ == "__main__":
    import json

    json.dump(scan(), sys.stdout, indent=2)
    sys.stdout.write("\n")

Run the snapshot once before the agent starts. Redirect the JSON to a path outside the repo. Agents rewrite working trees without much warning.

Snapshots must not live inside the blast radius. Keep the before.json file on a temp filesystem. Lose the snapshot and the gate must kill.

python harness/snapshot_names.py > /tmp/before.json

Kill gate after the patch

The gate compares used names against defined and imported names. Language builtins get a small explicit allow list. Anything else is treated as an unresolved name.

# harness/gate.py
from __future__ import annotations

import json
import sys
from pathlib import Path

from snapshot_names import scan

BUILTINS = {
    "True", "False", "None", "int", "str", "float", "len",
    "print", "range", "dict", "list", "set", "Exception",
}


def unresolved(module: dict) -> list[str]:
    defined = set(module["defined"])
    imported = set(module["imported"])
    used = set(module["used"])
    dangling = sorted(used - defined - imported - BUILTINS)
    return dangling


def main() -> int:
    before = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    after = scan()
    failures: list[str] = []

    for mod, payload in after.items():
        dangling = unresolved(payload)
        if dangling:
            failures.append(f"{mod}: {', '.join(dangling)}")

    removed = []
    for mod, payload in before.items():
        old_defined = set(payload["defined"])
        new_defined = set(after.get(mod, {}).get("defined", []))
        gone = sorted(old_defined - new_defined)
        if gone:
            removed.append(f"{mod}: {', '.join(gone)}")

    print("UNRESOLVED")
    print("\n".join(failures) or "(none)")
    print("REMOVED_DEFS")
    print("\n".join(removed) or "(none)")

    if failures:
        print("KILL: unresolved names after patch")
        return 2
    print("SHIP: names resolve")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

An exit code of two is a kill. An exit code of zero is a ship. Do not parse model prose for that answer.

python harness/gate.py /tmp/before.json
echo $?

Decision table

Observation	Evidence	Verdict
legacy.py still uses apply_tax	gate lists that name	Kill
Tests pass, gate fails	pytest green, exit 2	Kill
Helper inlined, all callers updated	unresolved list empty	Ship
New import added and used	name in imported set	Ship
Snapshot file missing	before.json absent	Kill
Agent edited the harness	gate.py hash changed	Kill

The last row matters more than it looks. Agents often try to fix the referee itself. A changed referee hash is always a kill.

sha256sum harness/gate.py harness/snapshot_names.py > /tmp/harness.sha
# ... one agent pass ...
sha256sum -c /tmp/harness.sha

Ninety-minute clock

Work in five named blocks with hard stops. Stop at the end of each listed block. Do not extend the clock for polish.

1. Minutes 0-15 freeze the fixture and the task sentence.
2. Minutes 15-35 write the snapshot, the gate, and hashes.
3. Minutes 35-55 run the agent once against the task.
4. Minutes 55-75 run pytest and the unresolved name gate.
5. Minutes 75-90 fill the decision table and then stop.

Ship-or-kill is a binary at minute ninety. Partial credit is not a valid verdict here. Almost resolved names are still counted a kill.

If the agent needs a second try, start another spike. This protocol forbids retry inside the same clock. Retry hides the true first-pass failure rate.

Where a free model and free server fit

The spike needs isolation more than a famous model. Local dirty trees contaminate the name snapshot fast. A separate machine keeps before.json honest under load.

MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use the free server as a scratch workspace only. Clone the fixture there and hash the harness. Copy only the gate output back to notes.

Do not treat the free tier as a benchmark lab. This spike does not publish tokens, latency, or model names. It publishes a kill rule and a small fixture.

The local laptop stays the source of truth for protocol files. The remote scratch box stays disposable after the clock. That split is the point of the server option.

Commands for the isolated run

Treat the next block as a proposed operator script. Adapt the clone URLs before any remote execution. Do not paste secrets into the task sentence.

# proposed: run on the scratch server
git clone "$FIXTURE_URL" /tmp/spike_symbols
cd /tmp/spike_symbols
python harness/snapshot_names.py > /tmp/before.json
sha256sum harness/*.py > /tmp/harness.sha

# proposed: one agent pass, one task sentence
# task: Inline tax into quote_total and delete apply_tax.

python -m pytest -q
python harness/gate.py /tmp/before.json
sha256sum -c /tmp/harness.sha

If pytest is green and the gate is red, trust the gate. Happy-path tests miss deleted callees on cold modules. That miss is why the hypothesis exists at all.

Fixture files the agent will see

Keep these fixture files very tiny on purpose. Large fixtures will waste the ninety-minute clock. The agent should see only this surface.

# app/pricing.py
def apply_tax(net: float, rate: float = 0.1) -> float:
    return net * (1.0 + rate)


def quote_total(net: float) -> float:
    return apply_tax(net)

# app/notify.py
from app.pricing import quote_total


def format_quote(net: float) -> str:
    total = quote_total(net)
    return f"total={total:.2f}"

# app/legacy.py
from app.pricing import apply_tax


def old_invoice(net: float) -> float:
    return apply_tax(net, rate=0.08)

# tests/test_pricing.py
from app.pricing import quote_total


def test_quote_total_default_rate():
    assert quote_total(100.0) == 110.0

Expected kill on a naive inline is specific. The legacy module still names apply_tax after the edit. The pricing test may still pass after inlining.

The name gate must still return exit code two. Green tests do not override a red gate. Record both exit codes in the decision table.

Limitations

The scanner is an AST walk, not a type checker. Dynamic getattr calls will evade this scan. Eval and string imports will also evade it.

The builtin allow list is incomplete on purpose. Full completeness was not the ninety-minute goal. False kills from names like enumerate will happen.

Star imports collapse the imported name set. This gate should fail closed on star imports. Add that extra kill if any time remains.

It does not prove runtime behavior at all. A resolved name can still be logically wrong. Pair this gate with tests, and do not replace tests.

It does not rank models against each other. A free-tier pass can still ship a worse algorithm. The spike answers one question only, by design.

Who should not use this

Do not use this gate on generated protobuf trees. Do not use it as a hiring score later. Do not use it as a public model leaderboard.

Skip it if the repo is mostly metaprogramming. Skip it if names are built only at runtime. Skip it if mypy strict already runs on every patch.

Frontend-only trees should use a TypeScript checker instead. Do not force this Python AST onto tsx files. Wrong checker language is itself a false kill source.

What ship means here

Ship means the harness is worth keeping in CI. Ship does not mean the agent is fully trusted. Ship does not mean the free tier is fast.

Kill means the gate missed the bug class. Kill also means the clock ran out first. Either kill is still useful for the next spike.

A useful kill report contains three lines only.

• Write the exact task sentence on line one.
• Write the unresolved name list on line two.
• Write the pytest exit code on line three.

Skip chat logs, screenshots, and model marketing copy. Those artifacts do not encode the kill rule. The three-line report does encode the kill.

If you run the fixture on a scratch server, keep the kill row. The kill row is the only artifact that matters. The chat transcript is not a spike result.