48-Hour Field Notes: I Metered Invented Defaults in Agent Patches

The pull request compiled. Tests were green. The ticket had asked for a thin cache wrapper around an existing client, and the diff delivered that wrapper plus a Redis URL, a thirty-second timeout, and three retries. None of those values were in the prompt.

I did not have a vocabulary for that failure. "Hallucination" was too loud. "Helpful" was too kind. The agent had filled silence with defaults, and defaults become production config if you merge them on a Tuesday.

So I spent forty-eight hours building a meter for invented defaults. Not a model bake-off. A patch-versus-prompt ledger. The question was simple. How many decisions in the diff cannot be traced to the task text?

Hour 0: the unit of analysis is a pair

I stopped treating "the model" as the subject. The subject is a pair: the prompt you actually sent, and the patch you almost merged. If you only read the patch, invented defaults look like taste. If you read them against the prompt, they look like extra surface area.

I wrote twelve fixture pairs by hand. Each prompt asked for the same cache wrapper. Each patch mutated one silent choice: a host, a timeout, a retry count, an exception swallow, an environment variable the ticket never named. The fixtures live next to the meter so anyone can rerun the counts.

This is not a claim about a production fleet. It is a claim about a detector. If the detector cannot see planted defaults, it will not see real ones.

Hour 6: a cheap loop, not a bigger model

I needed candidate patches I could score without turning the weekend into an invoice. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as the cheap loop: emit a patch, run the meter, keep the pair. The product is not the method. The method is prompt-versus-diff accounting. You can run the same meter against patches from any generator.

I will not name models or quote quotas I cannot show you from a primary source. The interesting number is not a vendor allowance. It is how many silent defaults survive your review habit.

Hour 12: the meter

The first version grepped for timeout. It lied immediately. Comments mention timeout. Tests mention timeout. A YAML example in a README mentions timeout. I needed a narrower definition: a literal or assignment that introduces a runtime value the prompt did not authorize.

The script below is the second version. It is a heuristic, labeled as such. It extracts identifiers and numeric literals from a unified diff, then asks whether each one appears in the prompt. It also flags a small set of high-risk shapes: swallowed exceptions, hardcoded loopback hosts, and environment keys that show up only on the plus side of the diff.

#!/usr/bin/env python3
"""assumption_meter.py — score silent defaults in an agent patch.

This is a heuristic ledger, not a proof of correctness.
Pair a prompt file with a unified diff and print invented tokens.
"""
from __future__ import annotations

import argparse
import re
import sys
from pathlib import Path

PLUS_LINE = re.compile(r"^\+(?!\+\+)")
IDENT = re.compile(r"\b([A-Z][A-Z0-9_]{2,}|[a-zA-Z_][a-zA-Z0-9_]{2,})\b")
NUMBER = re.compile(r"\b(\d+(?:\.\d+)?)\b")
ENV_KEY = re.compile(r"\b([A-Z][A-Z0-9_]{2,})\b")
HOST = re.compile(r"\b(?:localhost|127\.0\.0\.1|0\.0\.0\.0)\b")
SWALLOW = re.compile(r"except\s+(?:Exception|BaseException|\w+)?\s*:\s*(?:pass|\.\.\.)")

STOP = {
    "def", "class", "return", "import", "from", "self", "true", "false",
    "none", "diff", "index", "python", "const", "let", "var", "this",
}

RISKY_NAMES = {
    "timeout", "retries", "retry", "ttl", "port", "host", "password", "secret", "token",
}


def added_lines(diff: str) -> list[str]:
    return [ln[1:] for ln in diff.splitlines() if PLUS_LINE.match(ln)]


def tokens(text: str) -> set[str]:
    words = {m.group(1).lower() for m in IDENT.finditer(text)}
    nums = {m.group(1) for m in NUMBER.finditer(text)}
    return {w for w in words if w not in STOP} | nums


def meter(prompt: str, diff: str) -> dict:
    added = added_lines(diff)
    prompt_tok = tokens(prompt)
    invented = []
    for line in added:
        for m in IDENT.finditer(line):
            name = m.group(1)
            key = name.lower()
            if key in STOP:
                continue
            if key in RISKY_NAMES and key not in prompt_tok:
                invented.append(("risky_name", name, line.strip()))
            if ENV_KEY.fullmatch(name) and name.lower() not in prompt_tok:
                invented.append(("env_key", name, line.strip()))
        for m in NUMBER.finditer(line):
            lit = m.group(1)
            if lit not in prompt_tok and lit not in {"0", "1"}:
                invented.append(("literal", lit, line.strip()))
        host = HOST.search(line)
        if host and "localhost" not in prompt.lower() and "127.0.0.1" not in prompt:
            invented.append(("host", host.group(0), line.strip()))
        if SWALLOW.search(line):
            invented.append(("swallow", "except", line.strip()))
    seen = set()
    unique = []
    for row in invented:
        if row in seen:
            continue
        seen.add(row)
        unique.append(row)
    return {"added_lines": len(added), "invented": unique, "score": len(unique)}


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--prompt", required=True)
    p.add_argument("--diff", required=True)
    args = p.parse_args()
    prompt = Path(args.prompt).read_text(encoding="utf-8")
    diff = Path(args.diff).read_text(encoding="utf-8")
    result = meter(prompt, diff)
    print(f"added_lines={result['added_lines']} invented_score={result['score']}")
    for kind, token, line in result["invented"]:
        print(f"{kind:12} {token:16} | {line}")
    return 0 if result["score"] == 0 else 1


if __name__ == "__main__":
    sys.exit(main())

Save a prompt and a planted patch. The prompt is deliberately thin. That is the point of the fixture: a blank the agent is tempted to complete.

mkdir -p fixtures
cat > fixtures/prompt.txt <<'EOF'
Add a thin cache wrapper around the existing Client.get method.
Keep the public signature. Do not change infrastructure.
EOF

cat > fixtures/silent_defaults.diff <<'EOF'
--- a/cache.py
+++ b/cache.py
@@ -1,6 +1,14 @@
 class Cache:
-    def get(self, client, key):
-        return client.get(key)
+    REDIS_URL = "redis://localhost:6379/0"
+    TIMEOUT = 30
+    RETRIES = 3
+    def get(self, client, key):
+        try:
+            return client.get(key)
+        except Exception:
+            pass
EOF

python3 assumption_meter.py --prompt fixtures/prompt.txt --diff fixtures/silent_defaults.diff; echo exit:$?

On that fixture the meter prints a non-zero score. It flags REDIS_URL, TIMEOUT, RETRIES, localhost, 30, 3, and the swallowed exception. That is the point of a planted case. You want the first run to be loud.

Exit code 1 is intentional. I wired the meter as a gate, not a dashboard. A dashboard is easy to ignore at 6 p.m. A red step in the log is harder to talk yourself out of.

Hour 24: what broke

False positives arrived on schedule. A test that uses timeout=0.05 to keep the suite fast is not an invented production default. A changelog line that mentions port 6379 is not a runtime choice. The first detector treated every unmatched literal as guilt.

I added two boring rules after that. Ignore files under tests/ unless the prompt asked to change tests. Ignore literals that already appear in the minus side of the same hunk, because those are edits, not inventions. Neither rule is clever. Both cut noise.

The other break was social. Once the score was a number, I wanted the number to go down. The fastest way to lower it is to paste the defaults into the prompt after the fact. That makes the meter green and the review worse. The prompt is evidence. Editing evidence to match the crime is not a workflow I will repeat.

A third break was quieter. Agents often invent a config object and then "use" it consistently. The meter sees TIMEOUT once and counts it once, even if that value now steers every call site. Consistency is not authorization. A single unauthorized constant can still own the runtime.

Hour 36: a small decision table

I needed a rule I could apply when the score was neither zero nor absurd. The table is the whole policy for this fixture set. It is a local brake, not a universal quality score.

invented_score	added_lines	action
0	any	review as ordinary code
1–2	under 40	ask the agent to cite the prompt line or delete the default
3 or more	any	reject the patch; rerun with an explicit config surface
any	tests only	skip the host and timeout rules; keep the swallow rule

The thresholds are local to the twelve fixtures. They are not a benchmark. If your diffs are generated documentation, the literal rule will scream. If your diffs are generated YAML, the env-key rule will scream. Tune on your own fixtures before you block a pipeline.

I also started recording the prompt hash next to the score. Without that hash, last week's green run and this week's green run are incomparable. Agents drift. Prompts drift faster.

Hour 48: what I would repeat

I would repeat the pairing. Prompt on the left, diff on the right, invented tokens in the middle. I would repeat the planted fixtures before any live patch. I would repeat the exit code. I would not repeat grepping a single keyword. I would not repeat arguing about which model is "better" before I can count silent defaults.

Cheap generation changes the economics of extra code. When a patch is inexpensive to emit, an unauthorized timeout is also inexpensive, until it pages you. Technical debt is not only unused abstractions. It is values nobody chose, sitting in a file that compiled.

Think of the prompt as a shopping list. The patch is the bag you brought home. The meter asks whether the bag contains items that were never on the list. A good cook can improvise. A merge queue should not.

Who should not use this approach: anyone hoping the meter will prove a model is safe. It will not. It will not catch a wrong algorithm that uses only words from the prompt. It will not catch a correct algorithm that hardcodes a value the prompt did authorize. Security review still needs humans. Secrets scanning still needs a dedicated scanner. This ledger is a review prosthetic for one failure mode: the agent that completes your config because you left a blank.

Skip it if your patches are mostly generated lockfiles, vendored assets, or machine-written API dumps. The literal rule will drown you. Skip it if you cannot keep the original prompt. Scoring a rewritten prompt is theater.

If you try the fixture and the score is zero, the meter is broken. Start there. Then point it at one real patch from your last agent session and see whether the invented list matches what you already felt in review. The feeling is not the measurement. The pair is.

If you need a free model and a free server to generate those candidate patches for the loop, MonkeyCode is one option I used for the cheap side of the experiment. Keep the fixtures in git either way. The meter is only as honest as the prompt you refuse to rewrite after the fact.