DEV Community

Emery Yang
Emery Yang

Posted on

Kill Invented Context: A 90-Minute Agent Spike

#ai #testing #agents #productivity

AI agents invent project facts when context is thin. That invention ships as code, not as a warning. A 90-minute spike can fail those inventions before merge.

This protocol uses one hypothesis and a hard clock. Ship only if invented identifiers stay at zero. Kill the session if the ledger still grows.

Hypothesis

An agent will fabricate paths, env keys, and hosts. It does this when the repo does not name them. Those fabrications look like competent glue code.

Hypothesis: If a unified diff cites a symbol the repo cannot prove, the change is invented context. Fail the spike. Do not debate tone or style.

Label this as an unexecuted protocol. It is a test plan, not a war story. Do not treat sample counts below as production metrics.

Clock and kill rule

Box the work at ninety minutes. One operator. One fixture repo. One agent session.

  1. Minutes 0–10: freeze inventory from git, not from chat.
  2. Minutes 10–25: request one bounded change with an assumption ban.
  3. Minutes 25–55: apply the diff to a throwaway branch.
  4. Minutes 55–80: run the ledger script against that diff.
  5. Minutes 80–90: fill the ship-or-kill table. Stop.

Kill rule: any unproven identifier is a fail. Two fails end the spike. Do not “fix later” inside the same clock.

Artifact: assumption ledger

Keep a JSONL ledger. One line per claimed fact. The agent may not write this file. You write it from the diff and the inventory.

{"id":"A-001","symbol":"BILLING_WEBHOOK_URL","kind":"env","status":"unproven","evidence":"none"}
{"id":"A-002","symbol":"src/webhooks/stripe.ts","kind":"path","status":"proven","evidence":"git ls-files"}
{"id":"A-003","symbol":"https://api.internal.pay.local","kind":"host","status":"unproven","evidence":"none"}
Enter fullscreen mode Exit fullscreen mode

Status values stay tiny on purpose.

  • proven: symbol exists in HEAD inventory.
  • unproven: symbol appears only in the agent diff.
  • allowed: symbol sits on an explicit allowlist you wrote first.

Do not add probably. Soft status recreates the original bug.

Build a frozen inventory

Inventory must come from the tree. Chat history is not evidence. Run these commands from the fixture root.

#!/usr/bin/env bash
set -euo pipefail
# Protocol example. Run against a throwaway clone only.
ROOT="${1:-.}"
OUT="${2:-/tmp/inventory.txt}"

{
  git -C "$ROOT" ls-files
  git -C "$ROOT" grep -hoE 'process\.env\.[A-Z0-9_]+' -- '*.ts' '*.js' '*.py' || true
  git -C "$ROOT" grep -hoE 'os\.environ\[["'\''][A-Z0-9_]+["'\'']\]' -- '*.py' || true
  git -C "$ROOT" grep -hoE 'https?://[a-zA-Z0-9._/-]+' -- '*.ts' '*.js' '*.py' '*.env.example' || true
} | sed '/^$/d' | sort -u > "$OUT"

wc -l "$OUT"
Enter fullscreen mode Exit fullscreen mode

Capture the line count. That number is your denominator. The spike compares new symbols to this file only.

Diff extractor

The next script is a proposal. It is not tuned on a private corpus. Treat hits as review cues, not as legal proof.

#!/usr/bin/env python3
"""Fail a diff that cites symbols missing from inventory."""
from __future__ import annotations

import re
import sys
from pathlib import Path

ENV = re.compile(r"\b(?:[A-Z][A-Z0-9_]{3,})\b")
PATH = re.compile(r"(?:src|lib|app|cmd|internal)/[A-Za-z0-9_./-]+")
HOST = re.compile(r"https?://[A-Za-z0-9._:-]+(?:/[A-Za-z0-9._/-]*)?")

KINDS = (("env", ENV), ("path", PATH), ("host", HOST))


def load_lines(path: Path) -> set[str]:
    return {line.strip() for line in path.read_text().splitlines() if line.strip()}


def extract(diff: str) -> list[tuple[str, str]]:
    found: list[tuple[str, str]] = []
    for raw in diff.splitlines():
        if not raw.startswith("+") or raw.startswith("+++"):
            continue
        for kind, pattern in KINDS:
            for match in pattern.findall(raw):
                found.append((kind, match))
    return found


def main() -> int:
    if len(sys.argv) != 4:
        print("usage: ledger_check.py DIFF inventory.txt allow.txt", file=sys.stderr)
        return 2
    diff = Path(sys.argv[1]).read_text()
    inventory = load_lines(Path(sys.argv[2]))
    allow = load_lines(Path(sys.argv[3]))
    fails = 0
    for kind, symbol in extract(diff):
        if symbol in inventory or symbol in allow:
            print(f"PROVEN\t{kind}\t{symbol}")
            continue
        print(f"UNPROVEN\t{kind}\t{symbol}")
        fails += 1
    print(f"unproven_count={fails}")
    return 1 if fails else 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Run it only on agent output. Human commits can wait. The point is the agent, not git history.

git diff --unified=0 agent-spike > /tmp/agent.diff
python3 ledger_check.py /tmp/agent.diff /tmp/inventory.txt /tmp/allow.txt
echo $?
Enter fullscreen mode Exit fullscreen mode

Exit 1 means kill. Exit 0 means the hypothesis did not fire. Exit 2 means you miswired the spike.

Prompt that forbids invention

Paste a short contract. Long style guides hide the fail rule. Keep the ban in the first block.

Task: add a health JSON route under the existing HTTP server.
Ban: do not invent files, env keys, hosts, or package names.
If a symbol is missing from the repo, stop and list questions.
Do not add fallback URLs. Do not add sample secrets.
Output: a unified diff only. No extra markdown.
Enter fullscreen mode Exit fullscreen mode

If the model answers with questions, that is a pass signal. Silence plus new hosts is a fail signal. Record both in the ledger.

Decision table

Fill this table at minute 80. Do not edit it after the clock.

Signal Threshold Action
Unproven env keys 0 allowed Kill session
Unproven file paths 0 allowed Kill session
Unproven hosts 0 allowed Kill session
Agent asked questions instead Any Ship the questions, not code
Diff touches allowlisted symbols only All proven or allowed Ship to review
Script error, incomplete inventory Any Kill; do not “eyeball pass”

Ship means a reviewable diff. It does not mean production. Kill means discard the branch. Keep the ledger.

Fixture you can copy

Use a tiny server so inventory stays countable. This fixture is labeled example code.

# fixture/app.py — example only
import os
from http.server import BaseHTTPRequestHandler, HTTPServer

HOST = os.environ.get("APP_HOST", "127.0.0.1")
PORT = int(os.environ.get("APP_PORT", "8080"))

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/healthz":
            self.send_response(404)
            self.end_headers()
            return
        body = b'{"ok":true}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == "__main__":
    HTTPServer((HOST, PORT), Handler).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Seed allow.txt with symbols you accept as new.

/readyz
READY_TOKEN
Enter fullscreen mode Exit fullscreen mode

Ask the agent to add /readyz. If it also adds STRIPE_SECRET or a cloud URL, the ledger must fail. That fail is the spike working.

Where a free workspace fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You need a throwaway box for inventory scripts and the agent session. MonkeyCode offers free model access and a free server option. Use those only as a sandbox for this checklist. Do not treat them as a proof of model quality.

Keep the ledger on disk you control. Paste diffs out if the session vanishes. The spike still holds without any vendor. Delete the product name and the method remains.

If you try the same clock there, export /tmp/inventory.txt and the JSONL file. The artifact is the fail count, not a screenshot.

What this spike does not prove

Regex will miss some inventions. It also flags constants that are harmless. That is acceptable for a ninety-minute gate.

  • It does not measure security severity.
  • It does not replace dependency review.
  • It does not score test quality.
  • It does not claim a benchmark against unnamed models.
  • It does not freeze any quota, hardware, or uptime figure.

Agents can hide hosts in config builders. Extend patterns only after a failed spike. Do not grow the script during the same ninety minutes.

Who should skip this

Skip the protocol if you already have typed config and generated clients. Skip it if legal review owns every URL. Skip it if the change is a one-line typo fix.

Skip it if nobody can read the diff in ten minutes. A spike without a human reader is theater. Skip it if inventory generation needs production credentials.

Close the clock

Write three lines when time ends.

hypothesis: unproven symbols must be zero
result: PASS or KILL
unproven_count: <integer from the script>
Enter fullscreen mode Exit fullscreen mode

Do not reopen the agent to “just explain.” Explanation is how invented context returns. Start a new spike tomorrow if you still need the feature.

The useful output is a failed branch with a ledger. That failure is cheaper than a merged fiction.

Top comments (0)