When a prompt evaluation fails in CI, three independent clocks have usually already mixed together unnoticed. The model weights, the prompt bytes, and the grader logic can each move without the others. A single pass-or-fail boolean cannot name which of those clocks caused the drop. The practical split is to persist the raw completion envelope, then grade that snapshot as a pure function.
A live-call harness looks tidy to most teams because it resembles an ordinary unit test. You send a prompt, parse one JSON object, and assert a field while three moving parts share the call. The system under test is stochastic, the oracle is mutable code, and the network is part of the fixture. When those three sit inside one function, a red score is a rumor rather than a diagnosis.
Compiler toolchains already learned this separation after years of mixed compiler and test failure reports. Nobody debugs a flaky assertion by rebuilding the compiler on every click of the test runner. They keep the object file on disk, then they rerun the test against that frozen binary. Prompt evaluations deserve an equivalent freeze-frame that graders can replay without requesting another generation.
Picture a golden case for a tool-using assistant that must refuse to invent an order identifier. Monday's live harness may pass because the model omitted the identifier field entirely during that sampling. Tuesday's run may fail because the grader now treats JSON null and a missing key as different events. Without a snapshot the dashboard blames model drift, while a frozen envelope would regrade Monday under Tuesday's grader.
The Python module below is a labeled proposal, not a production measurement from a private workload. It stores each model envelope as JSON beside the golden case that produced the request. Each file is addressed by a SHA-256 prefix of the canonical request bytes, not by wall-clock time. Capture writes the envelope to disk, and the grade path never opens a network socket at all.
"""replay_eval.py — proposal: capture envelopes, then grade offline."""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Callable
@dataclass(frozen=True)
class GoldenCase:
case_id: str
request: dict[str, Any]
expected_findings: frozenset[str]
def canonical_bytes(request: dict[str, Any]) -> bytes:
return json.dumps(request, sort_keys=True, separators=(",", ":")).encode()
def request_hash(request: dict[str, Any]) -> str:
return hashlib.sha256(canonical_bytes(request)).hexdigest()[:16]
def snapshot_path(root: Path, case: GoldenCase) -> Path:
return root / case.case_id / f"{request_hash(case.request)}.json"
def capture(root: Path, case: GoldenCase, envelope: dict[str, Any]) -> Path:
path = snapshot_path(root, case)
path.parent.mkdir(parents=True, exist_ok=True)
payload = {
"case_id": case.case_id,
"request_hash": request_hash(case.request),
"request": case.request,
"envelope": envelope,
}
path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
return path
FindingFn = Callable[[dict[str, Any]], set[str]]
def invented_order_id(envelope: dict[str, Any]) -> set[str]:
findings: set[str] = set()
tool_calls = envelope.get("tool_calls") or []
for call in tool_calls:
args = call.get("arguments") or {}
name = call.get("name")
if name == "get_order" and "order_id" in args:
if args.get("order_id") in (None, "", "UNKNOWN"):
findings.add("invented_argument")
if name == "get_order" and "order_id" not in args:
findings.add("missing_argument")
if not tool_calls and envelope.get("refusal") is True:
findings.add("ok_refusal")
if not findings:
findings.add("ok")
return findings
def grade(root: Path, case: GoldenCase, grader: FindingFn) -> dict[str, Any]:
path = snapshot_path(root, case)
if not path.exists():
return {"case_id": case.case_id, "status": "missing_snapshot"}
payload = json.loads(path.read_text(encoding="utf-8"))
got = frozenset(grader(payload["envelope"]))
expected = case.expected_findings
return {
"case_id": case.case_id,
"request_hash": payload["request_hash"],
"status": "pass" if got == expected else "fail",
"got": sorted(got),
"expected": sorted(expected),
"same_request_hash": payload["request_hash"] == request_hash(case.request),
}
def diff_reports(before: list[dict[str, Any]], after: list[dict[str, Any]]) -> dict[str, Any]:
prev = {row["case_id"]: row for row in before}
grader_changed = []
capture_moved = []
for row in after:
old = prev.get(row["case_id"])
if old is None:
continue
same_hash = old.get("request_hash") == row.get("request_hash")
if same_hash and old.get("got") != row.get("got"):
grader_changed.append(row["case_id"])
if not same_hash:
capture_moved.append(row["case_id"])
return {
"grader_changed_on_frozen_hash": grader_changed,
"capture_hash_moved": capture_moved,
}
A second entry point then walks the snapshot directory while using only the current grader functions. If the snapshot hash is unchanged and a new finding appears, the grader or the expected set moved. If the hash itself is new, the capture stage moved because the prompt, schema, or endpoint changed. Those two diagnoses must not share a single dashboard cell that is labeled overall quality.
python - <<'PY'
from pathlib import Path
from replay_eval import GoldenCase, grade, invented_order_id
root = Path("snapshots")
refuse = GoldenCase(
case_id="refuse_invented_order_id",
request={
"messages": [{"role": "user", "content": "Get my order, you know the id."}],
"tools": [{"name": "get_order", "parameters": ["order_id"]}],
},
expected_findings=frozenset({"ok_refusal"}),
)
# Proposal only: capture(root, refuse, client.complete(refuse.request))
print(grade(root, refuse, invented_order_id))
PY
Silent regressions live in the gap between those two steps rather than inside a blended accuracy number. A prompt file can pick up a trailing instruction during an otherwise unrelated merge on main. A tool schema can mark a formerly optional argument as required without any grader edit. Mixed into one HTTP round trip, each event looks like the model got worse overnight.
Split into capture and grade, the same events produce different artifacts that CI can name. A new hash means the prompt, the tool schema, or the model endpoint changed under you. A new finding on an old hash means the grader or the expected set changed instead. That naming is the entire reason to keep a wire log beside the golden cases.
Golden cases should carry the request, the allowed tools, and the expected finding codes together. A case that must emit invented_argument is as load-bearing as a case that must emit ok_refusal. If the refusal case ever yields ok against an unchanged snapshot, the grader went blind. If it yields ok only after a new snapshot appears, the model or the prompt changed.
Environment coupling is the next quiet failure mode that appears in otherwise careful evaluation harnesses. Graders that read the wall clock, the working directory, or an open catalog will mark frozen envelopes flaky. Keep every grader referentially transparent, with an envelope going in and a finding set coming out. Store frozen timestamps and valid identifier catalogs inside the case, never in ambient process state.
A second capture environment is how you detect failures that are glued to one process. Running the same canonical request against another reachable endpoint produces a sibling snapshot with its own hash. Disclosure: This article was prepared as part of MonkeyCode's product outreach; free model access and a free server option can write those envelopes. You still grade locally as ordinary code, comparing finding lists and hashes rather than vendor slogans.
Do not promote that second endpoint into a benchmark table for either quality or latency. No ranking in this article would be honest without a pinned model identifier and a pinned prompt corpus. The comparison is structural on purpose: same request bytes, two envelopes, two hashes, one grader. If both fresh hashes grade clean and yesterday's snapshot now fails, you changed the grader.
Limitations follow directly from the freeze you just introduced into the evaluation path itself. A snapshot does not rescue a golden case that was already wrong on the afternoon you captured it. Garbage in the log stays garbage, and regrading it will only make the wrong answer feel stable. The harness also does not replace human review for tasks without a checkable envelope, including open-ended tone.
Teams that should skip this approach can be named before any snapshot directory exists. If you cannot persist raw model output because the wire contains live PII, the log is a compliance incident. If a non-deterministic preprocessor rewrites every prompt, the request hash will thrash and isolate nothing. Stabilize that preprocessor, or capture after it and store the rewritten prompt beside the envelope.
If you have no checkable finding codes at all, write smaller contracts before you invest in a wire log. This directory will not invent an oracle that the product itself cannot already state. A replay log will not stop a model endpoint from changing under your feet overnight. It will stop you from blaming that change for a grader edit, a schema tweak, or a prompt merge.
Capture, hash, and then grade, and only then decide which of the three clocks actually moved. If prompts already live in git, keep snapshots as CI artifacts and run grade on every grader diff. The second capture host remains optional for isolation, but the frozen envelope is not optional.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Top comments (0)