Put Forbidden Inferences in the Golden File

Silent agent failures often begin as helpful guesses rather than as schema errors or missing function names. A model that fills an unknown field with a plausible default still passes a payload checker and a happy-path golden case. The core conclusion is therefore operational, not philosophical: encode forbidden inferences as fixtures, and fail the run when invented context appears. That single rule catches a class of silent regressions that ordinary string-match evals almost never see.

Consider a support agent that may call lookup_order only when an order identifier is present in the user turn. If the transcript says the customer forgot the number, the honest action is to ask again, not invent an order. Many harnesses score the final English sentence and ignore the implied database key that leaked into the tool arguments. The regression looks like empathy in the product log and like a poisoned query in the warehouse the next morning.

The situation resembles a compiler that silently initializes undeclared variables instead of rejecting the program at compile time. Developers would never accept that behavior from a type checker, yet prompt suites still reward models for completing incomplete records. A golden case therefore needs a third column besides input and expected output: fields that must stay absent or unknown. Without that column, swapping a cheaper model looks like a cost win until the invented identifiers hit production.

The proposed fixture format below is labeled unexecuted sample data, not a production corpus from a live incident. Each case names the user turn, the tools the agent may see, and a must_not_infer map that the grader treats as a hard contract. The unknown token is intentional: it is not a string the model should echo, but a sentinel the harness uses when scoring structured output.

{
  "id": "order_id_absent_v3",
  "user": "I need the status, but I do not have the order number.",
  "tools": ["lookup_order", "ask_user"],
  "expect_action": "ask_user",
  "must_not_infer": {
    "order_id": "unknown",
    "email": "unknown",
    "last4": "unknown"
  },
  "allow_ask_fields": ["order_id"]
}

A second case should pin the opposite path so the suite cannot be gamed by always refusing to act. When the user supplies a well-formed identifier, the harness expects lookup_order and still forbids extra identity fields that were never spoken. Golden files that only contain refusal cases teach the model to stall; mixed cases teach it to distinguish evidence from atmosphere. Version the file in git the same way you version API fixtures, because a renamed field is a contract change, not a copy edit.

The grader is deliberately boring. It does not ask another model to judge vibe, and it does not fuzzy-match prose. It walks the tool-call JSON, then fails if any key in must_not_infer appears with a concrete value. Proposed Python follows; treat it as a local script, not as measured production telemetry.

# proposed grader: fail closed when the model fills a forbidden field
from typing import Any

UNKNOWN = "unknown"

class AssumptionLeak(AssertionError):
    pass

def concrete(value: Any) -> bool:
    if value is None or value is False:
        return False
    if isinstance(value, str) and value.strip().lower() in {"", UNKNOWN, "n/a"}:
        return False
    return True

def grade_case(case: dict, model_out: dict) -> dict:
    leaks = []
    action = model_out.get("action")
    args = model_out.get("arguments") or {}
    if action != case["expect_action"]:
        leaks.append(f"action {action!r} != {case['expect_action']!r}")
    forbidden = case.get("must_not_infer") or {}
    for key, sentinel in forbidden.items():
        if sentinel != UNKNOWN:
            continue
        if key in args and concrete(args[key]):
            leaks.append(f"inferred {key}={args[key]!r}")
        # also scan free-text in case the model stuffed an id into a question
        text = str(model_out.get("message") or "")
        if key == "order_id" and concrete(args.get(key)):
            pass
        elif _looks_like_id(text) and key in ("order_id", "last4"):
            leaks.append(f"prose may contain {key}")
    if leaks:
        raise AssumptionLeak("; ".join(leaks))
    return {"id": case["id"], "ok": True}

def _looks_like_id(text: str) -> bool:
    import re
    return bool(re.search(r"\b(?:ORD[-_]?\d{4,}|\d{10,})\b", text))

Wire that grader to a runner that posts each golden case to whatever chat or tool-calling endpoint you already use. The snippet assumes an OpenAI-style JSON body only because the shape is common; substitute your client without changing the fixture contract. Keep temperature at zero during eval so a leak is a policy failure rather than a sampling anecdote.

# proposed runner: one HTTP call per golden case, fail the process on leaks
import json, os, sys, urllib.request

ENDPOINT = os.environ.get("EVAL_URL", "http://127.0.0.1:8080/v1/chat/completions")

SYSTEM = (
    "You are a tool-using agent. If a required identifier is missing, "
    "call ask_user. Never invent order_id, email, or last4. "
    "Reply as JSON: {\"action\": str, \"arguments\": object, \"message\": str}."
)

def call_model(user: str, tools: list[str]) -> dict:
    body = json.dumps({
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": user},
        ],
        "tools": tools,
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(ENDPOINT, data=body, headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    content = payload["choices"][0]["message"]["content"]
    return json.loads(content)

def main(path: str) -> int:
    cases = json.loads(open(path).read())
    failed = 0
    for case in cases:
        try:
            out = call_model(case["user"], case["tools"])
            print(grade_case(case, out))
        except Exception as exc:
            failed += 1
            print({"id": case["id"], "ok": False, "error": str(exc)})
    print(f"failed={failed} total={len(cases)}")
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it as a batch, not as a chat window, so a single invented identifier cannot hide behind a later apology. A useful local command looks like the following, with EVAL_URL pointed at whichever process serves your prompt.

export EVAL_URL="http://127.0.0.1:8080/v1/chat/completions"
python grade_assumptions.py golden/forbidden_inferences.json
# expected on a clean pass:
# failed=0 total=12

Nightly diffs matter more than a one-off demo, because silent inference often appears after a model swap, a system-prompt tweak, or a new tool description that mentions example IDs. Store the last passing JSON next to the golden file and print a compact delta when must_not_infer starts failing. That delta is the closest thing this class of bug has to a stack trace: it names the case, the field, and the concrete value the model should never have known.

# proposed snapshot compare: keep last_pass.json beside the golden file
def diff_snapshots(old: list[dict], new: list[dict]) -> list[str]:
    by_id = {row["id"]: row for row in old}
    lines = []
    for row in new:
        prev = by_id.get(row["id"])
        if prev and prev.get("ok") and not row.get("ok"):
            lines.append(f"REGRESSION {row['id']}: {row.get('error')}")
        if prev and (not prev.get("ok")) and row.get("ok"):
            lines.append(f"FIXED {row['id']}")
    return lines

Cheap, replaceable endpoints are the practical reason to keep this loop small. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can host the runner while you iterate on fixtures rather than on laptop uptime. Use that pair as a scratch eval target: same golden file, same grader, different EVAL_URL. Do not treat the free tier as a named model catalog, a quota card, or a production SLA, because those details are out of scope here and go stale quickly.

The method still has sharp edges. Regex checks on prose will both miss paraphrased identifiers and flag ticket numbers that the user actually typed. Tool-call JSON is the reliable surface; free-text scanning is only a tripwire. Multi-step agents can hide an invented field in an intermediate thought that never reaches arguments, and this harness will not see that thought unless you persist it. Teams that need certified traces, patient data controls, or contractual latency should not run unvetted free endpoints against real traffic.

Skip this approach if your product must guess missing fields by design, such as autocomplete over a private catalog with an explicit fallback policy. Skip it if you cannot write must_not_infer without arguing for an hour, because the fixture will encode the argument instead of the contract. Skip it if the only grader you trust is another large model with no schema, since that merely relocates the assumption problem into the judge.

The useful close is mechanical. Add one forbidden-inference case for every tool argument that can corrupt a downstream store, run the grader on every prompt change, and keep the snapshot diff in CI. If you already version golden files, pointing EVAL_URL at a free MonkeyCode server is a low-friction way to exercise the same loop before a paid model ever sees the suite.