Case Study: Fail the Agent Plan When a Claim Has No File Citation

You should fail any agent plan that asserts a repo fact without citing a real file. Invented scripts, ports, and module paths often look confident while they quietly poison the first patch. This case study builds a small cite-or-fail gate around one fixture repository, not a platform. You can run the same commands on a laptop or on a free server without changing the contract.

Agentic coding tools now emit multi-step plans before they rewrite your tree, and those plans fill gaps with guesses. You do not need a fresh model announcement to see the failure mode: the agent states npm run lint when your repo only declares npm run check. A longer system prompt will not catch that class of error on its own. A machine-checked citation on every claim will.

This walkthrough is a worked example with fixtures. Treat the pass and fail outcomes below as outputs of those fixtures, not as production metrics from a live team.

Background: the failure this project exists to catch

You have probably watched an agent invent a fact that sounded local to your codebase. It names a Compose service that never existed, or it quotes an environment variable from a README you deleted last quarter. The patch then compiles just enough to waste a review cycle. The damage is not theatrical. It is a false map of the repo, written in fluent English.

The project under test here is a tiny billing CLI fixture, not a product you should copy into production. The tree only needs three files so you can see the gate fail for the right reason. You will keep the same schema when the real repo is larger, but you should not start there.

fixture-billing-cli/
  package.json
  src/invoice.py
  Makefile
  agent_plan.json          # emitted by the model, validated by you

The agent is allowed to propose edits. It is not allowed to narrate the tree as if memory were a checkout. That distinction is the whole case study.

Goal of the small project

You want a gate that reads one JSON plan and exits non-zero when a claim cannot be cited. You want that gate to run in a few seconds with the Python standard library, so CI and a free server stay interchangeable. You also want a reviewable artifact: a schema, two fixtures, and a decision table a teammate can argue with.

Success for this case study is narrow on purpose. The checker does not grade prose quality, security, or test coverage. It only answers whether each factual claim points at a file that still exists in the snapshot you handed the agent. If the citation is missing, stale, or the wrong kind of evidence, the plan fails before git apply.

Artifact: a cite-or-fail plan schema

Keep the plan boring. You should require an array of claims, and you should refuse free-form “notes” that smuggle extra facts. The schema below is the contract; treat it as the source of truth for this walkthrough.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "AgentPlan",
  "type": "object",
  "required": ["goal", "claims", "proposed_paths"],
  "additionalProperties": false,
  "properties": {
    "goal": { "type": "string", "minLength": 8 },
    "proposed_paths": {
      "type": "array",
      "items": { "type": "string", "minLength": 1 }
    },
    "claims": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["id", "kind", "text", "citation"],
        "additionalProperties": false,
        "properties": {
          "id": { "type": "string" },
          "kind": {
            "enum": ["command", "path", "symbol", "config"]
          },
          "text": { "type": "string", "minLength": 4 },
          "citation": {
            "type": "object",
            "required": ["path"],
            "properties": {
              "path": { "type": "string" },
              "start_line": { "type": "integer", "minimum": 1 },
              "end_line": { "type": "integer", "minimum": 1 },
              "excerpt": { "type": "string" }
            }
          }
        }
      }
    }
  }
}

You map kind to evidence as follows:

• command must appear in Makefile or in package.json scripts.
• path must name a file that already exists, not a file the patch hopes to create.
• symbol must appear as a substring on the cited line range.
• config must appear in a config or lockfile you listed, never in the agent’s memory.

If you already generate plans from a coding assistant with hosted free-model access, send the schema in the same turn as the repo snapshot. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option you can use to emit the JSON and run the checker; the gate itself stays in your repository and does not depend on any vendor remaining free.

Implementation: checker, fixtures, and commands

Label the next block as a complete worked example. You should paste it into cite_or_fail.py at the fixture root. It uses only the standard library so a free server image with Python 3 is enough.

#!/usr/bin/env python3
"""Fail an agent plan when a claim has no file citation."""
from __future__ import annotations

import json
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
KINDS = {"command", "path", "symbol", "config"}


def load_plan(path: Path) -> dict:
    plan = json.loads(path.read_text(encoding="utf-8"))
    for key in ("goal", "claims", "proposed_paths"):
        if key not in plan:
            raise SystemExit(f"missing key: {key}")
    return plan


def read_excerpt(path: Path, start: int | None, end: int | None) -> str:
    lines = path.read_text(encoding="utf-8").splitlines()
    if start is None:
        return "\n".join(lines)
    end = end or start
    if start < 1 or end > len(lines) or end < start:
        raise ValueError(f"{path}: bad range {start}-{end}")
    return "\n".join(lines[start - 1 : end])


def evidence_blob(root: Path) -> str:
    parts = []
    pkg = root / "package.json"
    if pkg.exists():
        parts.append(pkg.read_text(encoding="utf-8"))
    mk = root / "Makefile"
    if mk.exists():
        parts.append(mk.read_text(encoding="utf-8"))
    return "\n".join(parts)


def check_claim(root: Path, claim: dict, blob: str) -> str | None:
    kind = claim.get("kind")
    if kind not in KINDS:
        return f"{claim.get('id')}: unknown kind {kind!r}"
    citation = claim.get("citation") or {}
    rel = citation.get("path")
    if not rel:
        return f"{claim.get('id')}: citation.path is required"
    target = (root / rel).resolve()
    try:
        target.relative_to(root.resolve())
    except ValueError:
        return f"{claim.get('id')}: path escapes repo"
    if not target.is_file():
        return f"{claim.get('id')}: {rel} does not exist"
    start = citation.get("start_line")
    end = citation.get("end_line")
    try:
        excerpt = read_excerpt(target, start, end)
    except ValueError as exc:
        return f"{claim.get('id')}: {exc}"
    quoted = citation.get("excerpt")
    if quoted and quoted not in excerpt:
        return f"{claim.get('id')}: excerpt not found on cited lines"
    text = claim.get("text", "")
    if kind == "command" and text not in blob:
        return f"{claim.get('id')}: command {text!r} not in Makefile or package.json"
    if kind in {"symbol", "config", "path"} and rel not in str(claim.get("citation")):
        return f"{claim.get('id')}: path kind missing citation.path"
    if kind == "symbol" and text not in excerpt:
        return f"{claim.get('id')}: symbol {text!r} not on cited lines"
    if kind == "config" and text not in excerpt:
        return f"{claim.get('id')}: config token {text!r} not on cited lines"
    return None


def main() -> int:
    plan_path = ROOT / "agent_plan.json"
    if len(sys.argv) > 1:
        plan_path = Path(sys.argv[1])
    plan = load_plan(plan_path)
    blob = evidence_blob(ROOT)
    errors = []
    for claim in plan["claims"]:
        err = check_claim(ROOT, claim, blob)
        if err:
            errors.append(err)
    for rel in plan["proposed_paths"]:
        # Creating a file is allowed; inventing a source file as evidence is not.
        if rel.startswith(".."):
            errors.append(f"proposed_paths: {rel} escapes repo")
    if errors:
        print("cite-or-fail: REJECT")
        for item in errors:
            print(f"- {item}")
        return 1
    print("cite-or-fail: ACCEPT")
    print(f"claims_checked={len(plan['claims'])}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Commands you actually run

Create the fixture files first. You should keep them small enough that a reviewer can read every line in one sitting.

mkdir -p fixture-billing-cli/src
cd fixture-billing-cli

cat > package.json <<'EOF'
{
  "name": "fixture-billing-cli",
  "private": true,
  "scripts": {
    "check": "python -m compileall src",
    "test": "python -m unittest discover -s tests -q"
  }
}
EOF

cat > Makefile <<'EOF'
.PHONY: check test
check:
    python -m compileall src
test:
    python -m unittest discover -s tests -q
EOF

cat > src/invoice.py <<'EOF'
TAX_RATE = 0.07

def total(cents: int) -> int:
    return int(cents * (1 + TAX_RATE))
EOF

Then save two plans beside the checker: one that cites real evidence, and one that invents a script. You should run both so the reject path is not theoretical.

python3 cite_or_fail.py plans/pass.json
python3 cite_or_fail.py plans/fail-invented-script.json
echo "exit pass=$?"  # run them separately; expect 0 then 1

If you prefer a one-shot loop on a free server, wrap the same commands in a shell script and keep the fixture tree in git. Do not add model-specific flags here; the gate must stay vendor-neutral.

Sample fixtures

Passing plan, labeled as a fixture rather than a captured production trace:

{
  "goal": "Document how totals include TAX_RATE without changing behavior.",
  "proposed_paths": ["docs/totals.md"],
  "claims": [
    {
      "id": "c1",
      "kind": "symbol",
      "text": "TAX_RATE",
      "citation": {
        "path": "src/invoice.py",
        "start_line": 1,
        "end_line": 1,
        "excerpt": "TAX_RATE = 0.07"
      }
    },
    {
      "id": "c2",
      "kind": "command",
      "text": "python -m compileall src",
      "citation": {
        "path": "Makefile",
        "start_line": 3,
        "end_line": 4,
        "excerpt": "python -m compileall src"
      }
    }
  ]
}

Failing plan: the agent invents npm run lint and points at package.json anyway.

{
  "goal": "Add lint to the local check path.",
  "proposed_paths": ["package.json"],
  "claims": [
    {
      "id": "c1",
      "kind": "command",
      "text": "npm run lint",
      "citation": {
        "path": "package.json",
        "start_line": 1,
        "end_line": 10
      }
    }
  ]
}

Results from the fixture run

When you run the passing fixture, the checker should print cite-or-fail: ACCEPT and claims_checked=2. When you run the invented-script fixture, the checker should print cite-or-fail: REJECT and mention that npm run lint is not in Makefile or package.json. Those two lines are the entire result of this case study. They are not a benchmark and they are not a claim about model quality.

You should also try a third fixture that cites a path outside the tree, such as ../secrets.env. The resolve-and-relative check exists to stop that escape. If that case does not fail on your machine, you should stop and fix the path math before you wire the script into CI.

Expected shape of a local run, still a fixture outcome:

$ python3 cite_or_fail.py plans/pass.json
cite-or-fail: ACCEPT
claims_checked=2

$ python3 cite_or_fail.py plans/fail-invented-script.json
cite-or-fail: REJECT
- c1: command 'npm run lint' not in Makefile or package.json

Decision table: when the gate should fail

Use this table during review. If a row says fail, you should not negotiate the claim in the pull request thread.

Situation	Claim kind	Evidence required	Gate
Agent names a npm script	command	Exact string in package.json or Makefile	Fail if absent
Agent describes an existing module	path	File exists in the snapshot	Fail if missing
Agent quotes a constant	symbol	Token on cited line range	Fail if not on those lines
Agent states a config key	config	Key in the cited config file	Fail if only in prose
Agent wants to add a new file	n/a	Listed in proposed_paths only	Do not use it as a citation
Agent cites ../ or an absolute path	any	Must stay under repo root	Fail
Agent leaves claims empty	n/a	At least one claim	Fail

The last row matters more than it looks. An empty claim list is how a model hides guesses in the goal string. You should refuse that shape even when the proposed paths look harmless.

Limitations and who should skip this

This gate does not understand language. A cited line can still be the wrong line, and a matching substring can still be a coincidence. You should not treat ACCEPT as a merge decision. You should treat it as “the plan is no longer making unauditable claims.”

Skip this approach when you are in true greenfield work and the repo has no files worth citing. Skip it when the task is deliberately speculative, such as a spike that is allowed to invent APIs. Skip it when the source of truth is an external system the snapshot cannot see, such as a cloud console. The checker will only harass you in those settings.

Also skip it if you cannot pin the snapshot the model saw. A citation against main after you generated the plan against a dirty worktree is a false sense of safety. You should record the git sha next to agent_plan.json before you call the gate meaningful.

Lessons learned

You get more leverage from shrinking what the model is allowed to assert than from asking it to be careful. The schema did more work than any extra sentence in the prompt. The failing fixture was more useful than the passing one, because it named the exact invented command.

You should keep proposed new files out of the citation set. Creation is a patch act; citation is a claim about the present. Mixing those two is how an agent launders a guess through a file it plans to write later.

You should run the gate close to the files, not close to the chat transcript. A free server is enough if it can see the same tree the model saw. If you want a hosted generate-then-gate loop, MonkeyCode’s free model access and free server option are sufficient to replay these fixtures; keep the checker in your own repo so the contract survives the chat window.

The durable output of this case study is not a bot. It is a JSON file you can diff, a Python exit code you can put in CI, and a table your reviewer can point at when the plan starts guessing again.