Score the Trace, Not the Final Payload

Final-answer checks miss the costliest agent failure mode: a schema-valid payload produced by a tool path that has already drifted. A versioned golden-trace harness grades those intermediate steps, so a still-correct JSON object cannot hide a skipped confirmation or a merged lookup. The pattern below is a small Python runner with fixtures, a deterministic grader, and a command you can hang off CI. It stays useful even if the model behind the agent is swapped for a free endpoint tomorrow.

Answer-only evals treat the agent like a calculator that either emits the right number or does not. An agent is closer to a checkout clerk who can still bag the right items after skipping the ID check, substituting a nearby SKU, or charging the card twice. The customer walks away with a valid receipt, and the regression has no stack trace because the schema never broke. Teams then ship a prompt tweak that “still passes” while latency, tool spend, and side effects quietly change shape.

A golden trace records the path, not the punchline. Each fixture stores the user turn, the ordered tool names, the argument keys those tools must receive, and a hash of values that are allowed to be compared. Optional fields capture whether a clarifying question was required before any write. When a later model produces the same booking record through a different sequence, the harness fails on the path even though a JSON Schema validator would stay green. That is the whole point of scoring the trace.

The fixture format is intentionally boring so git diffs stay readable. The example below is a proposed on-disk contract, not a measured production suite, and it should be treated as a starting template rather than a benchmark.

{
  "id": "refund-requires-order-lookup",
  "input": {
    "user": "Refund order 1842 to the original card."
  },
  "expect": {
    "tool_names": ["get_order", "refund_payment"],
    "required_arg_keys": {
      "get_order": ["order_id"],
      "refund_payment": ["order_id", "method"]
    },
    "forbidden_tools": ["create_order", "charge_card"],
    "must_ask_before_write": true,
    "final_schema": {
      "type": "object",
      "required": ["status", "refund_id"],
      "properties": {
        "status": {"enum": ["ok", "needs_human"]},
        "refund_id": {"type": ["string", "null"]}
      }
    }
  }
}

A deterministic grader is the counterpart to that fixture. Large-language-model judges are useful for prose, and they are a poor fit here because the judge can drift on the same week the candidate model drifts. Sequence equality, key-set inclusion, and a forbidden-tool scan do not need another model in the loop. The following module is labeled as unexecuted example code; copy it, then point load_trace at whatever recorder your agent already has.

# trace_eval.py — proposed harness, not a published benchmark
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any


def load_json(path: Path) -> dict[str, Any]:
    return json.loads(path.read_text(encoding="utf-8"))


def tool_names(trace: dict[str, Any]) -> list[str]:
    return [step["name"] for step in trace.get("tools", [])]


def arg_keys(trace: dict[str, Any]) -> dict[str, list[str]]:
    out: dict[str, list[str]] = {}
    for step in trace.get("tools", []):
        keys = sorted((step.get("arguments") or {}).keys())
        out[step["name"]] = keys
    return out


def stable_hash(value: Any) -> str:
    blob = json.dumps(value, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def grade(fixture: dict[str, Any], trace: dict[str, Any]) -> dict[str, Any]:
    expect = fixture["expect"]
    names = tool_names(trace)
    keys = arg_keys(trace)
    failures: list[str] = []

    if names != expect["tool_names"]:
        failures.append(f"tool path {names!r} != {expect['tool_names']!r}")

    for tool, required in expect["required_arg_keys"].items():
        got = keys.get(tool, [])
        missing = [k for k in required if k not in got]
        if missing:
            failures.append(f"{tool} missing keys {missing}")

    used = set(names)
    banned = [t for t in expect.get("forbidden_tools", []) if t in used]
    if banned:
        failures.append(f"forbidden tools used: {banned}")

    asked = bool(trace.get("asked_clarification"))
    if expect.get("must_ask_before_write") and not asked:
        writes = [n for n in names if n.startswith(("refund_", "create_", "charge_"))]
        if writes:
            failures.append("write tools ran before a clarifying question")

    final = trace.get("final") or {}
    required_fields = expect["final_schema"].get("required", [])
    for field in required_fields:
        if field not in final:
            failures.append(f"final payload missing {field}")

    return {
        "id": fixture["id"],
        "pass": not failures,
        "failures": failures,
        "path_hash": stable_hash(names),
    }


def run_suite(fixture_dir: Path, traces_dir: Path) -> int:
    failed = 0
    for path in sorted(fixture_dir.glob("*.json")):
        fixture = load_json(path)
        trace_path = traces_dir / f"{fixture['id']}.json"
        if not trace_path.exists():
            print(f"MISSING trace for {fixture['id']}")
            failed += 1
            continue
        result = grade(fixture, load_json(trace_path))
        mark = "PASS" if result["pass"] else "FAIL"
        print(f"{mark} {result['id']} {result['path_hash']}")
        for item in result["failures"]:
            print(f"  - {item}")
        failed += int(not result["pass"])
    return failed


if __name__ == "__main__":
    raise SystemExit(run_suite(Path("fixtures"), Path("traces")))

Replay is a separate job from grading, and mixing them is how suites become flaky. Freeze a known-good trace once, store it next to the fixture, then on each prompt or model change re-run the same user input through the agent and write a candidate trace beside the golden one. The grader never calls a model; it only compares two files. That split keeps the red-or-green signal explainable when someone asks why CI blocked a “correct” refund payload.

A thin recorder is enough if the agent already logs tool calls. The snippet below is again proposed glue, meant to wrap an existing loop rather than replace an orchestration framework. Persist asked_clarification explicitly, because models often hide a missing question inside a fluent final sentence, and fluent sentences are invisible to schema checks.

# recorder.py — proposed adapter around an existing agent loop
from dataclasses import dataclass, field
from typing import Any, Callable


@dataclass
class TraceRecorder:
    tools: list[dict[str, Any]] = field(default_factory=list)
    asked_clarification: bool = False
    final: dict[str, Any] | None = None

    def on_tool(self, name: str, arguments: dict[str, Any]) -> None:
        self.tools.append({"name": name, "arguments": arguments})

    def on_question(self, text: str) -> None:
        self.asked_clarification = True

    def on_final(self, payload: dict[str, Any]) -> None:
        self.final = payload

    def dump(self) -> dict[str, Any]:
        return {
            "tools": self.tools,
            "asked_clarification": self.asked_clarification,
            "final": self.final,
        }


def replay(user_text: str, run_agent: Callable[[str, TraceRecorder], None]) -> dict[str, Any]:
    rec = TraceRecorder()
    run_agent(user_text, rec)
    return rec.dump()

CI should fail closed on a missing trace as well as a mismatched one. A vanished fixture file is not a pass; it is an untested path that used to be covered. The shell fragment below assumes the agent writes candidate traces into traces/ and that trace_eval.py returns a non-zero status when any case fails. Label this as a suggested workflow until it has been wired to the real runner.

python replay_suite.py --fixtures fixtures --out traces
python trace_eval.py
# exit code 1 means path drift, missing keys, or a vanished golden case

Free model access changes the economics of that loop more than it changes the grader. Path eval only works if it runs on every prompt edit, and paid endpoints make that cadence feel expensive enough to skip. Pointing the replay job at a free model endpoint, then hosting the runner on a free server option, keeps the suite cheap enough to execute while the fixtures stay in git. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project that offers free model access and a free server option, which is enough to host this replay worker without inventing a second paid queue for evaluation traffic.

The product mention is optional infrastructure, not the method. If another free endpoint is already on the team’s network, the same fixtures and the same deterministic grader still apply. What should not be claimed here are model names, token quotas, hardware sizes, or permanence of any free tier, because those figures change and this article does not have a primary-source snapshot of them. Treat availability as a capacity input, then keep the contract in the repository where it can be reviewed.

Path grading is strict on purpose, and that strictness is also the main limitation. Agents that are allowed several valid plans will fail a byte-for-byte tool sequence even when every plan is safe. Creative drafting, open-ended research, and tools whose argument values are timestamps or random ids need looser comparators, such as key-set checks without value hashes. Teams should not use this harness as a quality score for prose, and they should not replace production authorization with a fixture that merely forbids a tool name.

Negative cases belong in the same directory as the happy paths. A refund fixture that requires a question before refund_payment is not complete until a sibling fixture asserts that charge_card never appears when the user only asked for a receipt. Silent regressions often arrive as extra helpfulness: the model still answers, it just starts assuming the missing identifier. Scoring the trace makes that assumption visible as a forbidden tool or a skipped clarification rather than as a slightly different paragraph.

Version the fixtures when the product contract changes, not when a model happens to prefer a different order of reads. If get_order and get_customer become legally interchangeable, encode that as two allowed paths instead of deleting the assertion. A short comment in the fixture id is cheaper than a week of arguing about a flaky eval. The harness is a contract test for behavior the business already agreed on, and it is a weak substitute for product discovery.

Who should skip this approach is as important as who should adopt it. Do not hang path equality on agents whose value is exploration, multi-hop research with unstable source sets, or UI flows that legitimately branch on live inventory. Do not use a free shared endpoint for traces that contain real customer identifiers, because eval traffic is still traffic. Keep fixtures synthetic, keep the grader deterministic, and keep the final schema check as a backstop rather than the only gate.

The core conclusion does not depend on any vendor. If the final payload can be right while the path is wrong, grade the path. Freeze golden traces, compare them with ordinary Python, and run that comparison often enough that a prompt edit cannot land with a silent extra tool call. A free model endpoint and a free server option are one way to pay for that frequency; the fixtures are the part that actually catch the regression.

If a free model endpoint is already in reach, point the replay worker at it and keep the golden traces in the same pull request as the prompt change.