Replay Fixtures Separate Agent Scores From API Weather

A published agent score remains marketing until the tool log is hashed and replayed under isolation controls. Live API calls inject vendor drift, rate-limit noise, and hidden retries that later captions cannot reconstruct. Treating the fixture pack as data, not scenery, is what turns a fluent demo into a measurement. Teams that skip this step compare weather reports from different cities and name the gap a ranking.

Developer demos often treat tool calling as a product feature rather than a measurement surface. That framing is understandable, because an agent invoking an API looks like competence in motion. Competence in motion is still not a dataset, because the world behind the API keeps moving. The same prompt can harvest a new JSON shape or a new auth error while the score prints one decimal.

Load testing a checkout service against the public internet during a sale produces a similar illusion. The latency histogram then describes the sale and the retry budget as much as the service itself. Agent runs that punch live search, tickets, or calendars import the same contamination into the mean. The proposed protocol isolates that weather by replaying a hashed fixture pack and publishing the controls beside the metrics.

Dataset: a fixture pack, not a prompt list

The dataset is a directory of tasks where every tool request already has a recorded response, including failures. Each task file binds a prompt, an allowed tool schema, request fingerprints, and the exact response bytes. Tasks that require a real side effect, such as opening a ticket, do not belong in this pack. Those tasks belong in an operational drill that should never be averaged into a public model score.

A proposed fixture looks like the following JSON document. Fingerprints and schema digests are truncated here so nobody treats them as a downloaded pack.

{
  "task_id": "billing.refund.v3",
  "prompt": "Refund order 1842 if the payment captured and the window is open.",
  "tools": [
    {
      "name": "payments_get",
      "schema_sha256": "sha256:task-local-schema-digest"
    }
  ],
  "calls": [
    {
      "request_fingerprint": "sha256:task-local-call-digest",
      "response_status": 200,
      "response_bytes": "{\"captured\":true,\"window_open\":true}",
      "latency_ms_recorded": 41
    }
  ],
  "expected": {
    "action": "refund",
    "order_id": "1842"
  }
}

The request fingerprint is a canonical hash of method, path, and a sorted JSON body. Cosmetic key order therefore cannot mint a cache miss or a silent live fetch during scoring. Response bytes are stored literally, because pretty-printed JSON and compact JSON are different fixtures under a hash. The pack still needs a manifest that records timeout, retry budget, and every task digest as first-class data.

# proposed: protocol_sheet.py — unexecuted example for a replay pack
import hashlib, json, pathlib, sys

def sha256_file(path: pathlib.Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()

def build_manifest(pack_dir: pathlib.Path) -> dict:
    tasks = sorted(
        path for path in pack_dir.glob("*.json") if path.name != "manifest.json"
    )
    return {
        "protocol": "tool-replay-v1",
        "timeout_s": 30,
        "retries": 0,
        "seed": 20260922,
        "tasks": {path.name: sha256_file(path) for path in tasks},
    }

if __name__ == "__main__":
    pack = pathlib.Path(sys.argv[1])
    manifest = build_manifest(pack)
    payload = json.dumps(manifest, sort_keys=True, indent=2) + "\n"
    out = pack / "manifest.json"
    out.write_text(payload, encoding="utf-8")
    print(sha256_file(out), out)

Running that script produces a manifest hash that later public reports must quote beside the mean.

python protocol_sheet.py ./packs/billing-v3
# prints the manifest digest and path; commit both before any scoring run

If a single fixture byte changes, the hash changes, and yesterday's leaderboard row becomes incomparable rather than quietly stale. Timeout and retry count sit in the same file because they change the metric even when the caption still says accuracy. A score collected with two retries is a different measurement from a score collected with zero retries.

Metrics: a vector with a variance band

The headline number people want is task success, yet success without a failure taxonomy is a labeled coin flip. The proposed script records exact action match, schema-valid calls, unauthorized attempts, and timeout under the frozen budget. Averaging those four fields into one trophy is allowed only after the report prints the whole vector. Early collapse hides agents that succeed by calling tools the published schema never offered to anyone.

Variance belongs in the same paragraph as the mean, because sampling agents are not fully deterministic under replay. The protocol repeats each task a fixed number of times with a documented seed and a bootstrap interval. A ranking that moves less than that interval is a tie, not a launch claim. Publishing the interval is how a group admits noise instead of decorating noise as progress.

# proposed: score_replay.py — unexecuted example, not a published result
import hashlib, json, random, statistics, time
from typing import Callable

class ReplayMismatch(RuntimeError):
    pass

def bootstrap_mean(values, n=1000, seed=20260922):
    rng = random.Random(seed)
    means = []
    for _ in range(n):
        sample = [values[rng.randrange(len(values))] for _ in values]
        means.append(sum(sample) / len(sample))
    means.sort()
    low = means[int(0.025 * n)]
    high = means[int(0.975 * n)]
    return statistics.mean(values), low, high

def fingerprint(method: str, path: str, body: dict) -> str:
    canonical = json.dumps(
        {"method": method, "path": path, "body": body},
        sort_keys=True,
        separators=(",", ":"),
    )
    return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()

def replay_or_raise(task, method, path, body):
    expected = fingerprint(method, path, body)
    for call in task["calls"]:
        if call["request_fingerprint"] == expected:
            return call["response_status"], call["response_bytes"]
    raise ReplayMismatch(f"unrecorded call on {task['task_id']}: {expected}")

def score_task(task, agent_fn: Callable) -> dict:
    started = time.monotonic()
    try:
        result = agent_fn(
            task, lambda method, path, body: replay_or_raise(task, method, path, body)
        )
        elapsed = time.monotonic() - started
        success = int(
            result.get("action") == task["expected"]["action"]
            and result.get("order_id") == task["expected"]["order_id"]
        )
        return {
            "success": success,
            "schema_ok": 1,
            "unauthorized": 0,
            "timeout": int(elapsed > 30),
            "elapsed_s": elapsed,
        }
    except ReplayMismatch:
        return {
            "success": 0,
            "schema_ok": 0,
            "unauthorized": 1,
            "timeout": 0,
            "elapsed_s": time.monotonic() - started,
        }

def summarize(rows):
    successes = [row["success"] for row in rows]
    mean, low, high = bootstrap_mean(successes)
    n = len(rows)
    return {
        "n": n,
        "success_mean": round(mean, 4),
        "success_ci95": [round(low, 4), round(high, 4)],
        "unauthorized_rate": round(sum(row["unauthorized"] for row in rows) / n, 4),
        "timeout_rate": round(sum(row["timeout"] for row in rows) / n, 4),
    }

The ReplayMismatch path is the entire point of the harness, not an error to be softened later. An agent that invents a fourth query has left the dataset even when the extra query looks reasonable. The harness must not silently fetch live bytes in order to be helpful during a public run. Helpful harnesses are the usual way that careful benchmarks slowly decay into product tours with extra decimals.

Controls: isolation that would embarrass a flaky suite

Controls are the unglamorous twin of metrics, and they are what keep the printed numbers from being marketing. The protocol pins the working directory, the timeout clock, and a network allowlist that cannot reach the original APIs. Retry count stays at zero unless the metric name itself includes the retry policy in plain language. If the allowlist is missing, the fixture pack is only a suggestion, and suggestions do not generate rankings.

A second control is runner diversity, because a protocol that only passes on one laptop is a local script. A clean checkout of the same manifest hash on a second host is the minimum replication step worth citing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that second runner when the goal is proving the harness. The comparison that matters is whether two processes print the same digest and overlapping confidence intervals.

# proposed second-runner check after the pack and harness are committed
git clone <harness-repo> && cd <harness-repo>
git checkout <commit-with-manifest>
python protocol_sheet.py ./packs/billing-v3
python - <<'PY'
import hashlib, pathlib
path = pathlib.Path("packs/billing-v3/manifest.json")
print(hashlib.sha256(path.read_bytes()).hexdigest())
PY
# abort the scoring run if this digest does not match the published protocol sheet

Clock control deserves its own paragraph because traces love to print model latency as if it were task difficulty. Recorded fixture latencies may feed a secondary cost axis, but they must not mix into the success mean. Mixing them produces a ranking that rewards whichever replay host had the quieter neighbor that afternoon. That ranking is meteorology again, and meteorology does not belong on an agent leaderboard under any caption.

Why the numbers are not marketing

Marketing numbers omit the protocol sheet, while a measurement quotes the sheet in the same block as the mean. A non-marketing report lists the manifest hash, timeout, retry budget, seed, unauthorized rate, and confidence interval together. It also states the negative space: which live systems were not called and which ranking gaps are smaller than the interval. Tasks excluded for side effects belong in that same block, or readers will assume the pack is complete.

The following report block is the proposed publication unit. Anything shorter should be treated as an anecdote, not as a ranking input.

protocol: tool-replay-v1
manifest_sha256: <digest of billing-v3>
timeout_s: 30
retries: 0
seed: 20260922
repeats_per_task: 5
success_mean: <fill from summarize()>
success_ci95: [<low>, <high>]
unauthorized_rate: <fill from summarize()>
timeout_rate: <fill from summarize()>
live_network: denied
excluded: ticket.write, calendar.create (side-effect drills)

Those figures are illustrative placeholders for the shape of a report, not measurements of any hosted model. Replacing the placeholders without rerunning the harness would recreate the problem this article is trying to kill. The discipline is the sheet, not the decimal, and the sheet is what makes a later audit possible. A decimal without a sheet should be read as an anecdote, however many dashboards repeat it.

Limitations and who should not use this protocol

Replay fixtures measure policy following against a recorded world rather than competence against a living production surface. Fraud patterns and incident response will look too easy or too dated under a frozen tool log. Teams in those settings need operational drills with live canaries, and they should keep those drills off the public average. Mixing the two audiences is how a careful replay score becomes an unsafe claim about production readiness.

The protocol is a poor fit for agents whose value is exploration rather than tool-schema fidelity. A mismatch is scored as unauthorized even when the extra call would have been wise in an open world. Researchers studying open-world browsing should say so instead of forcing that work through a fingerprint table. Cheap extra capacity can also replicate a bad metric at larger scale, and that replication remains marketing.

The practical next step is to write the protocol sheet for one internal pack and refuse unmatched scores. A score that cannot point at a manifest hash should not travel into a comparison table. Readers who later want an extra machine for the replication check can inspect MonkeyCode after that sheet exists. The sheet is the product of the benchmark, and the spare server is only a witness to the hash.