The harness that works is boring: three layers, cheapest first, each one filtering what the next has to look at. The expensive mistake is starting with the clever layer.
What to record for each pair
Every pair needs enough context that a decision made about it three weeks later is still meaningful. Store the request id, the prompt hash, the prompt template id and version, the full raw response from both sides, both usage blocks, both latencies, and the terminating reason from each.
The terminating reason is a field name you have to get right per provider, because it is not the same field. An OpenAI-style chat completion carries finish_reason on each element of choices; Anthropic’s Messages API carries stop_reason on the message. The value vocabularies differ too, so the harness needs a small mapping into your own enum rather than a string comparison across providers. The list of values on one side is in the finish-reason values; write the equivalent for the other side into the mapping and keep it next to the harness.
The prompt set matters more than any of the code below. It must be sampled from production traffic and stratified by request shape, for the same reason the shadow sample is stratified: a hand-written set contains the cases the author thought of, and migrations break on the cases nobody thought of. Redact it, freeze it, and version it, so that a comparison run in October is comparable with one run in August.
The A/A noise floor comes first
Before comparing A against B, compare A against A: run the old provider twice over the same prompt set and score the pairs with exactly the machinery you are about to use on the migration. This costs one extra run and it is the difference between a harness that produces a number and a harness that produces a decision.
The reason is that A and A do not agree either. Sampling is stochastic, and even at temperature zero neither provider guarantees identical output across calls — batching, hardware and routing all introduce variation, which the library covers in why temperature zero is not determinism. So there is a baseline disagreement rate that has nothing to do with the migration. Any A/B difference smaller than the A/A difference is not a signal, and without the floor there is no principled way to say which differences are worth reading.
Report every subsequent number against that floor. “Structural mismatch on 3% of pairs” means nothing until you know the A/A figure is 1%.
Layer 1: structural comparison
Deterministic, cheap, and where most genuine migration defects live. Each check produces a boolean per pair, which makes them aggregatable and makes a gate possible.
- Parse. Did each side produce valid JSON, if JSON was requested? A drop in validity rate is the single most common provider-migration defect and it is visible in the first hundred pairs.
- Schema. Does each response validate against your schema — required fields present, types right, enum values in the allowed set? Enum drift is common and quiet: a field that should be one of four values comes back as a fifth, plausible-looking one.
- Terminating reason. Did both stop for the same reason, after mapping? A response truncated at the token cap on one side and completed naturally on the other is a real difference that every downstream score will misread as mild.
- Tool calls. Same tool selected? Same set of argument keys? Arguments equal after normalising number formatting, key ordering and whitespace? Compare the proposal, never the effect.
- Length ratio. Output tokens on B over output tokens on A. A ratio well away from one is not itself a defect but it is an excellent sort key, and it catches a model that has started padding or truncating.
def structural(a, b, schema):
return {
"parses": json_ok(a) == json_ok(b),
"valid": validates(a, schema) and validates(b, schema),
"stop_same": norm_stop(a) == norm_stop(b),
"tool_same": tool_name(a) == tool_name(b),
"args_same": canonical(tool_args(a)) == canonical(tool_args(b)),
"len_ratio": out_tokens(b) / max(out_tokens(a), 1),
}
These are the checks that should be gates rather than metrics. A structural failure rate above the A/A floor is a blocker; there is no judgement call to make about a response that does not validate.
Layer 2: similarity, and what it does not mean
For free-text answers, embed both responses with the same embedding model and take the cosine between them. Using the same model on both sides is not optional, for the reason set out in why two models’ vectors are not comparable.
Now the part that is misused constantly. A high similarity does not mean the answers agree. Embeddings encode subject matter, and two answers about the same subject sit close together regardless of what they assert. “Yes, this policy covers flood damage” and “No, this policy does not cover flood damage” are nearly identical strings about an identical topic; their embeddings are close, and a similarity threshold will pass the pair. Negation, quantities, dates and names — exactly the content whose correctness matters — are the content embeddings represent worst.
So use the score as a sort key and never as a gate. It is excellent at the thing it is good at: pushing the two hundred least similar pairs to the top of a queue for a human. Complement it with cheap literal checks that catch what it misses — extract numbers, dates, currency amounts and proper nouns from both sides and compare those sets directly. A pair with 0.96 cosine and a different figure in it is precisely the dangerous case, and a regular expression finds it while the embedding cannot.
Layer 3: spending a model on the judgement
Only on the pairs the first two layers flagged, because a judge call per pair over a real prompt set is a meaningful bill and most pairs are uninteresting. Three rules make the judge useful rather than decorative.
Ask a specific question, not a preference. “Which response is better?” produces a position bias and an answer you cannot act on. “Does response B assert any fact that response A does not, and is that fact supported by the provided source text?” produces a labelled defect. Build the question from the failure you actually fear, and see evaluation rubrics for the shape of a rubric that holds up.
Pin the judge and record its identifier with every verdict. A judge on a floating alias drifts under you, and a comparison whose measuring instrument changed halfway through is not a comparison. Use a third model where you can — a judge that is one of the two candidates has an obvious conflict — and present the pair in both orders on a subsample to measure position bias rather than assuming it away.
Calibrate the judge against humans once. Take fifty pairs, have a person label them, and check the agreement rate. If the judge disagrees with your reviewer more than your reviewers disagree with each other, the rubric is the problem, not the models. Evaluation blind spots covers what this misses even when it works.
Turning scores into a work queue
The output of a run should be an ordered list of things to look at, not a dashboard. Sort by structural failure first — those are unambiguous and usually cluster into two or three root causes — then by literal-mismatch flags, then by ascending similarity. Read the top of that list until you stop learning anything new, which in practice happens quickly because failures cluster.
Every disagreement you investigate and resolve becomes a permanent test case, with the prompt, both responses and the verdict, folded into the model-version comparison suite. That is the compounding part: the harness is throwaway, but the cases it surfaces are what makes the next migration cheaper, and the next migration is already on the calendar.
Top comments (0)