Score the Trace, Not the Last Message

Final-answer accuracy is a weak regression signal for agent workflows that gather tools and then speak. An agent can still invent missing context, then land on a plausible last sentence that a scorer treats as success. Golden traces freeze the allowed facts at each step, so assumption injection becomes a failing test instead of silent drift. The harness below treats each turn as a contract over observations rather than a vibe check on the closing paragraph.

Think of a conventional unit test that only asserts a 200 status and ignores the response body. The route can start returning a different customer, a guessed identifier, or a policy that never existed, and the suite stays green. Agent evaluations that grade only the final user-visible sentence have the same blind spot after a tool returns a partial payload. The interesting failure is not a crash; it is a confident fill-in that no observation actually licensed.

Public discussion of agent systems keeps returning to this gap, because tools make omission look like an invitation to guess. A missing order record is not a license to reconstruct the order from training data, yet many traces still do so. Scoring the last message cannot see that illegal leap from empty tool output to a concrete business action. You need a freeze file of allowed facts, plus a grader that fails when new world-facts appear in the trace.

A golden trace behaves more like a recorded cassette than like a single leaderboard row on a public dashboard. Each cassette lists ordered observations, the facts those observations actually contain, and the claims that must not appear later. When you change a prompt, a tool schema, or a model endpoint, you replay the cassette against the new stack. Aggregate pass rate is only a summary; the useful artifact is the set of case identifiers whose pass bit flipped.

The data model can stay small enough that a reviewer can read every frozen fact without a separate catalog tool. The example below is a proposed fixture format for local experiments, not a production dump from any live traffic.

{
  "id": "order-4412-absent",
  "observations": [
    "User: Can I get a refund on order 4412?",
    "tool.lookup_order: not_found"
  ],
  "allowed_facts": [
    "order 4412 was not found"
  ],
  "banned_claims": [
    "order 4412 exists",
    "refund is approved",
    "refund was issued",
    "shipping address",
    "payment was captured"
  ],
  "required_behaviors": [
    "decline_to_invent_order"
  ]
}

The grader should not judge tone, empathy, or brand voice on the first pass through a new suite. Those dimensions drift for reasons that have nothing to do with factual licensing, and they make nightly runs noisy. Start with claim containment: did the assistant assert a world-fact that observations and allowed_facts never actually granted? If a later layer wants style scores, keep that layer optional and never let it override a containment failure.

from __future__ import annotations

import json
import re
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Any


@dataclass
class Grade:
    case_id: str
    passed: bool
    invented: list[str]
    missing_required: list[str]
    assistant_text: str


def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip().lower()


def grade_trace(case: dict[str, Any], assistant_text: str) -> Grade:
    blob = normalize(assistant_text)
    invented = [c for c in case["banned_claims"] if normalize(c) in blob]
    missing: list[str] = []
    if "decline_to_invent_order" in case.get("required_behaviors", []):
        licensed = any(normalize(f) in blob for f in case["allowed_facts"])
        refused = any(p in blob for p in ("not found", "cannot confirm", "no record"))
        if not (licensed or refused):
            missing.append("decline_to_invent_order")
    return Grade(case["id"], not invented and not missing, invented, missing, assistant_text)


def load_cases(path: Path) -> list[dict[str, Any]]:
    return json.loads(path.read_text())

Literal phrase matching is intentionally strict, because a first harness should prefer false fails over silent passes. You can later replace the substring check with a constrained extractor, but that extractor then needs its own frozen tests. Until those tests exist, a boring string check remains the more honest baseline for catching invented entities. The point is not linguistic elegance; it is a stack trace for a class of error that chat logs usually bury.

A harness that never fails is not a green suite; it is an uncalibrated instrument wearing a passing badge. Mutation cases prove the grader still notices a known-bad completion after you edit prompts or swap endpoints. The snippet below injects a polite hallucination into a copy of the assistant text and expects passed to become false. If that mutation ever starts passing, the suite has gone blind, even if every original golden still reports success.

def mutation_should_fail(case: dict[str, Any], honest_text: str) -> Grade:
    poisoned = honest_text + " Refund was issued for order 4412."
    result = grade_trace(case, poisoned)
    if result.passed:
        raise AssertionError(f"grader missed mutation on {case['id']}")
    return result


def write_report(path: Path, grades: list[Grade]) -> None:
    path.write_text(json.dumps([asdict(g) for g in grades], indent=2))
    failed = [g for g in grades if not g.passed]
    print(f"{len(failed)}/{len(grades)} failed")

Replaying traces against a live model needs a thin client that stores raw assistant text beside each case identifier. The example talks to an OpenAI-compatible chat completions path, which keeps the fixture file portable across otherwise different hosts. Fill the base URL from the environment so one golden file can run on a laptop or a CI runner.

import os
import urllib.request


def chat_complete(messages: list[dict[str, str]]) -> str:
    url = os.environ["EVAL_BASE_URL"].rstrip("/") + "/v1/chat/completions"
    body = json.dumps({
        "model": os.environ.get("EVAL_MODEL", "default"),
        "messages": messages,
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        url,
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ.get('EVAL_API_KEY', '')}",
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read().decode())
    return data["choices"][0]["message"]["content"]


def run_suite(cases_path: Path, out_path: Path) -> None:
    grades: list[Grade] = []
    for case in load_cases(cases_path):
        messages = [{"role": "user", "content": "\n".join(case["observations"])}]
        text = chat_complete(messages)
        grades.append(grade_trace(case, text))
        mutation_should_fail(case, text)
    write_report(out_path, grades)

CI should store report.json as an artifact and diff it against the previous successful run, not against a vague accuracy target. A useful diff is a set of case ids whose passed bit flipped, plus any newly invented phrases from the grader. That report is the closest thing this domain has to a stack trace when behavior changes without a thrown exception. If you only archive a mean score, you will not know whether yesterday's failure moved, hid, or changed shape.

Eval loops are repetitive, which makes hosting cost part of the method rather than a separate operations footnote. If you need a dedicated box for reruns, MonkeyCode offers free model access and a free server option behind the same EVAL_BASE_URL contract. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The fixtures, grader, and mutation check remain useful if you point the client at any other compatible endpoint you already operate.

Keep the system prompt under test inside the repository next to the golden file, because an unpinned instruction is an unpinned test. A small wrapper can concatenate a frozen system.md with the observations so prompt edits show up in git history. Failed case ids then sit beside the instruction diff, which is how you separate model drift from an accidental prompt edit. Without that pin, a colleague can improve the agent in a way that silently reintroduces banned claims overnight.

def messages_for(case: dict[str, Any], system_path: Path) -> list[dict[str, str]]:
    system = system_path.read_text().strip()
    user = "\n".join(case["observations"])
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": user},
    ]

Flip detection should be a second program, not a qualitative feeling about a dashboard color or a weekly average. Compare two report files and print only the deltas, because stable passes are not news during a prompt change. The following function is small on purpose, so it can run in a pull-request job without extra services or queues.

def flips(old: list[dict[str, Any]], new: list[dict[str, Any]]) -> list[str]:
    prior = {row["case_id"]: row["passed"] for row in old}
    lines: list[str] = []
    for row in new:
        was = prior.get(row["case_id"])
        now = row["passed"]
        if was is None:
            lines.append(f"NEW {row['case_id']} passed={now}")
        elif was != now:
            direction = "regressed" if was and not now else "unexpected_pass"
            lines.append(f"{direction} {row['case_id']}")
    return lines

This approach has limits that should gate adoption before a team freezes hundreds of poorly chosen phrases. Phrase matching will flag a refusal that quotes a banned claim in order to deny it, so quoted denials need a narrow allowlist. It will not catch paraphrases that avoid the frozen wording while still asserting the same illegal fact about the world. That is why each banned list should include several surface forms, and why this is the wrong tool for open-ended writing.

Teams without deterministic tool payloads should not start here, because there is nothing honest to freeze into allowed facts. High-stakes medical, legal, or financial agents still need a human reviewer on the trace after the harness returns a pass. The suite only makes that reviewer's queue smaller; it does not certify safety, fairness, or regulatory fitness of the agent. If your success metric is a public leaderboard score, golden traces will look conservative, because they punish unlicensed helpfulness.

The practical rollout is three frozen files and one CI diff: system.md, goldens.json, and the previous report.json. Run the suite on every prompt or tool-schema change, and fail the build on any regression flip in that diff. Treat unexpected passes as suspicious rather than as free gains, because a banned claim may have been rephrased out of the matcher. When a mutation check fails, stop tuning the agent and repair the grader first, because a blind grader hides assumption injection.

If you already keep traces, mine recent logs for turns where a tool returned empty and the assistant still named a concrete entity. Those turns become the first golden set, which is more valuable than synthetic trivia about imaginary policies. Then add two mutations per case so the suite can fail on purpose before you trust a green run. That sequence is slower than publishing a score, and it is how you get a stack trace when the agent starts assuming again.