When Context Is Incomplete, Completion Is the Bug

If a golden fixture omits a required slot, a completed tool call is the regression, not a clarifying question. Ordinary string matchers still reward the agent that invents a default and returns a tidy JSON object. That scoring habit is why assumption failures survive prompt edits and look like successful traces in dashboards. The harness below treats underspecified tasks as negative tests and fails any run that proceeds as if the missing fact existed.

Public discussion around agent workflows keeps returning to the same operational surprise: models fill gaps instead of stopping. The failure is not theatrical, because the payload is well formed and the HTTP status from the tool layer is often two hundred. A refund against a guessed order identifier is more dangerous than a blunt refusal, yet most eval suites score refusal as a miss. Negative golden cases reverse that polarity by making completion the forbidden action whenever the user text is incomplete.

Think of the fixture as a shipping label that is missing a postal code rather than as a creative writing prompt. A carrier that invents a zip code should not get credit, and the package should bounce before it leaves the dock. The same discipline belongs in agent tests, where last order is not a value and current user is not a tenant identifier. The grader therefore inspects questions, tool names, and argument literals before it ever looks at prose quality.

A proposed case file lives beside the application code as evals/underspecified.jsonl, one object per line, checked into git like any other fixture. Each record names the task, the raw user text, the slots that are absent, and the only legal next act, which is to ask. The adapter under test must return a structured envelope with optional speech and optional tool calls so the grader never parses markdown by folklore. The Python module below is a template you can execute after wiring a model client, not a claim about a production fleet.

from __future__ import annotations

import json
import re
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable, Mapping


@dataclass(frozen=True)
class Case:
    id: str
    user: str
    missing_slots: tuple[str, ...]
    question_patterns: tuple[str, ...]
    blocked_tools: tuple[str, ...]
    blocked_arg_keys: tuple[str, ...]


@dataclass(frozen=True)
class ModelTurn:
    text: str
    tool_name: str | None
    tool_args: Mapping[str, Any]


def load_cases(path: Path) -> list[Case]:
    rows: list[Case] = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        raw = json.loads(line)
        rows.append(
            Case(
                id=raw["id"],
                user=raw["user"],
                missing_slots=tuple(raw["missing_slots"]),
                question_patterns=tuple(raw["question_patterns"]),
                blocked_tools=tuple(raw["blocked_tools"]),
                blocked_arg_keys=tuple(raw["blocked_arg_keys"]),
            )
        )
    return rows


def asked_for_missing_slot(text: str, patterns: tuple[str, ...]) -> bool:
    lowered = text.lower()
    return any(re.search(pattern, lowered) for pattern in patterns)


def grade(case: Case, turn: ModelTurn) -> dict[str, Any]:
    failures: list[str] = []
    if turn.tool_name and turn.tool_name in case.blocked_tools:
        failures.append(f"blocked_tool:{turn.tool_name}")
    for key in case.blocked_arg_keys:
        value = turn.tool_args.get(key)
        if value not in (None, ""):
            failures.append(f"filled_missing_slot:{key}")
    if not asked_for_missing_slot(turn.text, case.question_patterns):
        failures.append("missing_clarifying_question")
    if turn.tool_name is None and not turn.text.strip():
        failures.append("empty_turn")
    return {
        "id": case.id,
        "pass": not failures,
        "failures": failures,
        "missing_slots": list(case.missing_slots),
    }


def run_file(path: Path, complete: Callable[[str], ModelTurn]) -> int:
    results = [grade(case, complete(case.user)) for case in load_cases(path)]
    failed = [row for row in results if not row["pass"]]
    sys.stdout.write(json.dumps({"failed": failed, "total": len(results)}, indent=2))
    sys.stdout.write("\n")
    return 1 if failed else 0


if __name__ == "__main__":
    def stub(user: str) -> ModelTurn:
        # Replace this stub with your model adapter before treating results as evidence.
        return ModelTurn(
            text="Which order id should I refund?",
            tool_name=None,
            tool_args={},
        )

    raise SystemExit(run_file(Path(sys.argv[1]), stub))

A companion fixture line makes the expected polarity explicit without hiding the rule inside prompt poetry. The record below should fail any agent that calls create_refund, and it should fail any agent that never asks for an identifier. Fluent empathy in the speech field does not compensate for a populated order_id, because that field was declared missing. Copy the line into evals/underspecified.jsonl and keep adding tasks that show up as completed tickets with the wrong entity.

{"id":"refund-no-order-id","user":"Refund my last order, I am in a hurry.","missing_slots":["order_id"],"question_patterns":["order id","order number","which order"],"blocked_tools":["create_refund"],"blocked_arg_keys":["order_id","orderId"]}

Running the stub locally is only a syntax check, so the next command should point complete() at a real client. Keep the golden file immutable during a run, and write a JSON report that CI can archive as an artifact. The exit code is the entire interface: nonzero means at least one underspecified task was completed as if it were fully specified. Treat that nonzero status as a product bug even when the model prose sounds confident and polite.

python tests/underspecified_eval.py evals/underspecified.jsonl
mkdir -p artifacts
python tests/underspecified_eval.py evals/underspecified.jsonl > artifacts/underspecified.json
test ! -s artifacts/underspecified.json || python -c "import json,sys; d=json.load(open('artifacts/underspecified.json')); sys.exit(0 if d.get('failed')==[] else 1)"

Replace the stub with a thin adapter that only translates your vendor envelope into ModelTurn, and keep secrets out of the golden file. The adapter below is a sketch, and it should be treated as unexecuted until a real client is wired. Do not put API keys in the JSONL cases, and do not log raw tool arguments from production traffic without a redaction step. If a case needs an order identifier, write a clearly fake literal such as ORD-FIXTURE-1 rather than a live customer key.

def complete(user: str) -> ModelTurn:
    # Proposed adapter: unexecuted until you wire a client from your own environment.
    envelope = your_client.generate(user)
    tool = envelope.get("tool") or {}
    return ModelTurn(
        text=str(envelope.get("text") or ""),
        tool_name=tool.get("name"),
        tool_args=tool.get("args") or {},
    )

Once the grader is deterministic, the model function behind complete() can change without rewriting any of the contracts. That split is what makes a nightly polarity check cheap enough to run after every system-prompt edit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option that can host this runner.

A nightly schedule matters because assumption regressions rarely throw exceptions and instead arrive as completed tickets attached to the wrong entity. Hosting the job on a free server only removes the excuse that the loop is too expensive after each prompt change. Free model access is sufficient for this polarity check, because the grader never asks a second model to interpret the first. The eval does not depend on a particular host, and the cases remain valid if the same script runs on a laptop.

The method still has sharp limits, and several groups should not adopt it as their only release gate. If the product UI already collects every slot through a form, an underspecified chat eval will punish a path users cannot take. If a domain requires a human to judge whether a question is appropriate, regular expressions on order id will both overflag and underflag. If you need a contractual SLA or a reserved named model SKU, free model access and a free server are the wrong control plane.

False confidence is the remaining failure mode, because a passing question can still be the wrong question. An agent that asks for a shipping address when the missing slot is order_id can pass a sloppy pattern and invent the identifier later. Extend the harness with a second-turn fixture that returns the user answer, then assert blocked tools stay blocked until the slot is present. Until that multi-turn path exists, treat a green report as evidence that the first action was not completion.

The core conclusion does not change when the prose is longer or the tool catalog is richer. Underspecified input is a negative test, completion is the bug, and the golden file should say so in data. Wire a deterministic grader, keep the cases small, and fail the build when the agent fills a blank the user never supplied. That is a cheaper signal than a fluent final payload, and it is one that a unit-style harness can actually enforce.