48-Hour Field Notes: Ghost Routes vs the Real Router

The ticket was one sentence: add a readiness client so the worker can wait for the API. I pointed an agent at a tiny FastAPI fixture and let it work. Twenty minutes later the diff looked finished. A ReadyClient class, three retries, GET /readyz, header X-Probe-Token.

The fixture has never served /readyz. The live router exposes /health. There is no probe token anywhere in the tree. Tests still went green because the agent wrote a mock that agreed with the fiction. That is the failure this note measures.

Agents do not only invent default arguments. They invent surface. A ghost route is a path, method, header, or environment name that appears in the patch but not in the application's declared HTTP inventory. Once a ghost lands, every later session treats it as heritage. The analogy is a city map that keeps drawing streets the pavement crew never poured. Navigation still "works" until a truck actually has to drive there.

I spent 48 hours on a fixture app, not a production network. Counts below are from that fixture. They are a method, not a benchmark of any hosted model.

Hour 0–4: freeze the real surface

Before the agent session, I generated an inventory file from source, not from memory. Memory is how ghosts survive. The fixture is a single app.py with three routes and one health check. The inventory script walks FastAPI's route table and writes method path lines.

# inventory_routes.py — labeled example, run against the fixture
from fastapi.routing import APIRoute

def freeze(app, dest="route_inventory.txt"):
    lines = []
    for route in app.routes:
        if isinstance(route, APIRoute):
            for method in sorted(route.methods - {"HEAD", "OPTIONS"}):
                lines.append(f"{method} {route.path}")
    text = "\n".join(sorted(set(lines))) + "\n"
    with open(dest, "w", encoding="utf-8") as handle:
        handle.write(text)
    return text

I also dumped environment names the app actually reads. String-searching .env.example is not the same job. Example files collect wishes. The AST pass below only records os.environ and os.getenv in Python the process would import.

# inventory_env.py
import ast
import pathlib
import sys

NAMES = set()

class Visitor(ast.NodeVisitor):
    def visit_Subscript(self, node):
        if isinstance(node.value, ast.Attribute) and node.value.attr == "environ":
            if isinstance(node.slice, ast.Constant) and isinstance(node.slice.value, str):
                NAMES.add(node.slice.value)
        self.generic_visit(node)

    def visit_Call(self, node):
        func = node.func
        if isinstance(func, ast.Attribute) and func.attr == "getenv":
            if node.args and isinstance(node.args[0], ast.Constant):
                NAMES.add(node.args[0].value)
        self.generic_visit(node)

root = pathlib.Path(sys.argv[1])
for path in root.rglob("*.py"):
    tree = ast.parse(path.read_text(encoding="utf-8"))
    Visitor().visit(tree)
print("\n".join(sorted(NAMES)))

Commands that actually ran against the fixture:

python -c "from app import app; from inventory_routes import freeze; freeze(app)"
python inventory_env.py . > env_inventory.txt
cp route_inventory.txt route_inventory.frozen.txt

The freeze file is the control. If you skip this step, you will grade the agent against a moving target, including its own previous ghosts. Think of it as photographing the street grid before anyone hands the model a pencil.

Hour 4–24: three tasks, one rule

I gave the agent three prompts on the same fixture, one at a time, resetting the tree between runs. Prompt A asked for the readiness client. Prompt B asked for a webhook caller that posts job results. Prompt C asked for a retry wrapper around "the status endpoint." None of the prompts named a path. That was the point. A named path is a spec. An unnamed path is an invitation to sketch.

After each run I saved git diff -- app.py tests/ and ran a ghost counter. The counter is deliberately dumb. It does not try to understand REST. It extracts string literals that look like paths or headers and set-differences them against the freeze file.

# ghost_surface.py — reproducible against a unified diff
import re
import sys

PATH_RE = re.compile(r"(?<![A-Za-z0-9])(/[a-zA-Z0-9._/-]{1,80})")
HEADER_RE = re.compile(r"['\"](X-[A-Za-z0-9-]+)['\"]")
ENV_RE = re.compile(
    r"os\.environ\[['\"]([A-Z0-9_]+)['\"]\]|os\.getenv\(['\"]([A-Z0-9_]+)['\"]\)"
)

def load_inventory(path):
    known = set()
    with open(path, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if not line:
                continue
            parts = line.split(" ", 1)
            known.add(parts[-1] if parts[0] in {"GET", "POST", "PUT", "PATCH", "DELETE"} else line)
    return known

def ghosts(diff_text, known_paths, known_env):
    paths = set(PATH_RE.findall(diff_text))
    headers = set(HEADER_RE.findall(diff_text))
    env = set(a or b for a, b in ENV_RE.findall(diff_text))
    return {
        "ghost_paths": sorted(
            p for p in paths if p not in known_paths and not p.startswith("/static")
        ),
        "headers": sorted(headers),
        "ghost_env": sorted(e for e in env if e not in known_env),
    }

if __name__ == "__main__":
    diff = sys.stdin.read()
    known_paths = load_inventory("route_inventory.frozen.txt")
    known_env = {
        line.strip()
        for line in open("env_inventory.txt", encoding="utf-8")
        if line.strip()
    }
    result = ghosts(diff, known_paths, known_env)
    print(result)
    if result["ghost_paths"] or result["ghost_env"]:
        raise SystemExit(2)

On the fixture, prompt A produced /readyz and PROBE_TOKEN. Prompt B produced /webhooks/jobs and X-Signature. Prompt C produced /status/v2 even though /status existed. I am not reporting a model ranking. I am reporting that three ordinary prompts, with no path named, were enough to grow a shadow API.

A pytest that pins the gate sits next to the fixture diff, not in the agent transcript.

# test_ghost_surface.py
from pathlib import Path
from ghost_surface import ghosts, load_inventory

def test_no_ghost_paths_in_sample_diff():
    diff = Path("fixtures/prompt_a.diff").read_text(encoding="utf-8")
    known = load_inventory("route_inventory.frozen.txt")
    known_env = set(Path("env_inventory.txt").read_text(encoding="utf-8").split())
    result = ghosts(diff, known, known_env)
    assert result["ghost_paths"] == []
    assert result["ghost_env"] == []

Wire it so a human does not have to remember the ritual:

git diff -- app.py tests/ | python ghost_surface.py
pytest test_ghost_surface.py -q

The test is the artifact you keep. The session transcript is not. Exit code 2 is louder than a paragraph of model self-confidence.

Hour 24–48: a second reader that cannot write

The regex gate is cheap and rude. It flags /health/live even when you meant to add it. It misses concatenated paths like prefix + "/readyz". I wanted a second pass that only classifies, never patches.

That is where a free remote model is actually useful. I uploaded the freeze file, the diff, and a closed instruction: label each extracted path as known, new-but-declared-in-the-ticket, or ghost. The model was not given write tools. If it suggested a rewrite, I discarded the suggestion. Classification is a different job from authorship. Mixing them is how the mock that hid /readyz got written in the first place.

I ran that classifier on MonkeyCode's free model access, on the free server option, so the overnight batch did not sit on the same laptop that was capturing diffs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product is not the meter. The meter is the freeze file plus ghost_surface.py. Remove the hosted pass and the gate still fails a build.

The prompt I used is a proposal, not a tuned eval.

You are a classifier. You may not propose code.
Inventory (known paths):
{freeze}

Diff:
{diff}

For each path or env name in the diff, reply with CSV:
name,kind,label
kind is path|header|env
label is known|ticketed|ghost
If the ticket text does not name the path, you may not use ticketed.

I treated ticketed as a human decision. The model does not get to promote a ghost by claiming the ticket implied it. Implication is how /readyz arrives wearing a lab coat.

What broke

The route freeze missed a WebSocket path because APIRoute is not WebSocketRoute. The agent then "discovered" /ws/ready and the gate stayed green. That is a hole in the inventory, not a clever model. I extended the freezer to walk app.routes without the APIRoute type check and regenerated the control file. The ghost appeared on the next run.

Header detection over-fired on X-Request-ID in a comment. I changed the parser to ignore diff lines whose payload, after the + or - prefix, starts with #. Comments are not surface. Agents still hide real calls in comments, so this filter is a trade. I would rather miss a commented-out ghost than fail the build on a lecture.

The OpenAPI file in the fixture was stale on purpose. Generating inventory from OpenAPI alone would have blessed /readyz if a previous agent had written it into the spec. Source of truth has to be the router module the process actually loads. A spec is a letter of intent. The route table is the pavement.

Concatenated paths still slip through. f"{base}/readyz" never matches PATH_RE on a single literal. I would not ship this gate as a security scanner. It is a session brake. Treat a clean run as "no obvious ghosts in literals," not "the agent told the truth."

What I would repeat

Freeze inventory before the first prompt, in CI, from the running route table. Fail the job on any ghost path or ghost env. Keep the second model read-only. Reset the tree between unrelated tickets so ghosts cannot become ancestry. I would not repeat letting the author model write mocks for endpoints it invented. If a test needs a transport, point it at the real /health or skip the test.

The useful number after 48 hours was not a token total. Prompt A left two ghosts, prompt B left two, prompt C left one path that looked like a version bump of a real route. Those five strings are enough to justify a gate. They are not enough to rank vendors.

Who should not use this

If you have no router, no OpenAPI, and no gateway config, you have no freeze file. Do not grade ghosts against a README. Teams that generate clients from a living spec already have a stronger check: compile the client. This meter is for brownfield services where the spec is a rumor.

Do not paste diffs that contain secrets into any hosted classifier, free or not. Strip values. Keep names. The approach also wastes time on greenfield work where inventing /readyz is the spec. Ghosts are a brownfield problem. On a blank repo, the same script will fail every honest first route.

Forty-eight hours did not produce a league table of models. It produced a brake. If you want the classifier off your laptop, the free server path is how I queued the overnight labels. The gate that matters still runs in pytest.