The question is not whether the new snapshot produces different text. It will. The question is whether it produces different text in the places your application reads, and answering that needs a comparison that is not string equality.
Why string comparison gives you nothing
Run one prompt twice against a single model at any temperature above zero and the two outputs differ. Run it against two snapshots and they differ more. A harness that diffs raw strings therefore reports one hundred percent divergence on the first run and is discarded on the second, which is why most version-bump testing stops at somebody eyeballing a dozen outputs.
The fix is to compare the same thing your application compares. Almost every production LLM call has a narrow contract underneath the prose: a JSON object with three required keys, a label from a closed set, a citation list, a yes-or-no decision, a length budget. That contract is what a bump can break and it is comparable exactly. Everything else is style, and style differences between snapshots are expected rather than interesting.
So the harness does not ask “is the output the same”. It asks four narrower questions: does it still parse; is the extracted value the same; is it still within the length and cost envelope; and did it stop for the same reason. That last one is cheap and often the most informative single signal — a finish_reason of length appearing on the new snapshot where the old one returned stop is a truncation regression that a text diff would bury.
Building the prompt set out of real traffic
A hand-written prompt set tests the cases you thought of, which are the cases you already handle. Build the set from logged production inputs instead, and stratify it rather than sampling uniformly — uniform sampling of real traffic gives you a hundred copies of the easy path.
- Take the most frequent input shape, capped at a handful of examples. This is your regression floor: if these move, stop.
- Take every input that has ever produced a parse failure, a retry or a fallback. These sit nearest the boundary and are where a distribution shift shows first.
- Take the longest inputs you serve, because truncation and output-cap behaviour differ per snapshot.
- Take anything that touches a refusal-sensitive domain, since refusals return successfully and will not appear in an error rate.
- Redact before storing, and store the redacted version as the fixture. See redacted fixtures that still reproduce the bug.
Fifty to two hundred cases assembled this way is a more useful instrument than a thousand synthetic ones, and it is small enough to run on every bump without a budget conversation. If you already keep a golden dataset, this is that, sampled for boundary cases rather than for coverage.
The harness
The whole thing is one file. It takes two model identifiers, runs every case against both, extracts the contract, and writes a row per case. Note that it pins temperature to zero for both arms — not because that makes the comparison deterministic (it does not; see the next section) but because it removes one source of variance you are not trying to measure.
# harness.py — run one prompt set against two snapshots.
import json, statistics, sys
from openai import OpenAI
client = OpenAI()
OLD, NEW = sys.argv[1], sys.argv[2] # two dated snapshots, never aliases
def call(model, case):
r = client.chat.completions.create(
model=model,
temperature=0,
max_completion_tokens=case.get("max_tokens", 512),
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": case["system"]},
{"role": "user", "content": case["user"]},
],
)
choice = r.choices[0]
return {
"served": r.model, # resolved snapshot, not what we asked
"finish": choice.finish_reason, # stop | length | tool_calls | content_filter
"out_tokens": r.usage.completion_tokens,
"text": choice.message.content or "",
}
def contract(text):
"""The narrow thing the application actually reads."""
try:
obj = json.loads(text)
except json.JSONDecodeError:
return {"parsed": False}
return {
"parsed": True,
"label": obj.get("label"),
"keys": sorted(obj.keys()),
"n_items": len(obj.get("items", [])),
}
rows = []
for case in json.load(open("cases.json")):
a, b = call(OLD, case), call(NEW, case)
ca, cb = contract(a["text"]), contract(b["text"])
rows.append({
"id": case["id"],
"parse_lost": ca["parsed"] and not cb["parsed"],
"label_moved": ca.get("label") != cb.get("label"),
"keys_moved": ca.get("keys") != cb.get("keys"),
"finish_moved": a["finish"] != b["finish"],
"token_ratio": b["out_tokens"] / max(a["out_tokens"], 1),
})
json.dump(rows, open("report.json", "w"), indent=2)
ratios = [r["token_ratio"] for r in rows]
print("parse lost :", sum(r["parse_lost"] for r in rows), "/", len(rows))
print("label moved:", sum(r["label_moved"] for r in rows), "/", len(rows))
print("finish moved:", sum(r["finish_moved"] for r in rows), "/", len(rows))
print("median output-token ratio new/old:", round(statistics.median(ratios), 3))
The token_ratio column is the one people leave out and then wish they had. It is the cost delta of the bump, measured on your own prompt distribution rather than on a vendor’s example, and it is free to collect while you are already making the calls. A median meaningfully above 1.0 means the new snapshot is more verbose on your traffic and your bill will follow.
The self-comparison baseline
Here is the step that separates a harness you trust from one you argue with. Before comparing old against new, run the old snapshot against itself — the same cases, twice — and produce the same report. That gives you the divergence the model produces with no version change at all: the noise floor.
Without it, every number in the cross-version report is uninterpretable. Three labels moving out of a hundred sounds alarming until the self-comparison also moves three, at which point it is sampling variance and you have learned that those three cases are unstable regardless of version. This is the same reason a regression baseline is re-established rather than assumed, and it is why temperature zero is not deterministic matters practically and not just pedantically.
Report the cross-version divergence against the self-divergence, case by case. Cases that move in both are flaky and belong on a separate list to be fixed as prompts. Cases that move only across versions are the actual finding, and there are usually far fewer of them than the raw number suggested.
Turning the report into a decision
Set the thresholds before you see the output, or you will rationalise whatever you get. Reasonable shape for a gate, with the numbers being yours rather than anybody else’s:
- Any parse loss blocks. A case that produced valid structure and now does not is a hard break; there is no threshold worth setting above zero.
- Label movement blocks above the noise floor. Compare against the self-comparison number, not against zero.
- A new
lengthfinish reason blocks. It means truncation, and truncated JSON is a parse failure waiting for a longer input. - Token ratio is a warning, not a block. It is a budget conversation, and it belongs in the migration note rather than in the gate.
When the gate passes, that is permission to canary, not permission to cut over — the prompt set is boundary cases and real traffic contains shapes you did not sample. Canary releasing the model and keeping the comparison suite around are how the rest of the confidence gets built. Keep the report file from every bump: the sequence of them is the only record you will have of how much a given family moves between snapshots, which is exactly the number you want the next time somebody asks whether a bump is risky.
The awkward part of running both arms is credentials and accounting: two snapshots, often two rate-limit buckets, and a cost figure you want attributed to the harness rather than to production. Routing both arms through one gateway key with a tag per arm makes the token-ratio column fall out of the billing data instead of having to be computed — Multigrid attributes cost per tag, which is the same mechanism behind cost attribution generally.
Top comments (0)