The migration looked like an obvious win on economics. In one EvalShift run, the candidate model was:
- 58.8% cheaper
- 75.2% faster
- 0 failed or truncated calls
Run economics. Same 16 calls on both sides, no failures, and every number a finance reviewer cares about moved the right way.
Then the migration report returned:
FAIL
Not because the model crashed. Not because the API changed. Not because the output stopped parsing.
It changed what the agent actually did.
The verdict card. Five of seven budgets passed; the two that did not are the two that describe behavior.
What failed
I replayed 16 real examples against the source and candidate models. Across the run, 13.6% of evaluated comparisons regressed.
The overall regression-rate budget actually passed.
Two other checks did not:
- Tool-selection divergence hit 25%, against a 10% limit.
- The overall equivalence rate fell to 61.4%, below the required 75%.
The written verdict and its findings. The recommendation is not to migrate until the tool-selection and semantic regressions are fixed.
That distinction matters. Looking only at latency, cost, successful requests, or even a few manually inspected outputs would have made this migration look pretty attractive.
The behavioral diff told a different story.
The failures weren't cosmetic
One example asked for an opinion on working late.
The source model answered the question directly.
The candidate instead issued a search_web call and returned no text.
The diff for that example. Source trace on the left, target trace on the right, and the reason it was flagged at the top.
Other cases had the same general problem in different forms:
- an action was reported as completed even though the corresponding tool was never called;
- a request was routed to a different tool;
- an unnecessary tool call appeared where the source answered directly.
These are not necessarily signs that the candidate model is "bad." They show that changing the model changed the application.
For an agent, the model is part of the control flow.
Aggregate quality can hide this
The evaluator breakdown made the tradeoff clearer.
Routing arguments and routing conformance were broadly equivalent in this run, while tool-selection divergence, semantic similarity, and the pairwise equivalence judge showed regressions.
Overall, by evaluator. Two rows say equivalent, three say regressed, and each comes with its own effect size, confidence interval and confidence label.
A single average score would flatten all of that into one number.
For a migration, I care more about the question:
What changed, on which examples, and is that change acceptable for this application?
This is why I built EvalShift
I kept running into model migrations that were tested roughly like this:
change model → try several prompts → outputs look fine → ship.
That works until the difference is something subtle like an extra tool call, a missing action, or different routing behavior.
EvalShift replays the same suite against the source and target model, compares the behavior, and produces the migration decision and individual diffs before the model string gets changed in production. The CLI is local-first and produces a self-contained HTML report; hosted upload is optional. How that differs from an observability tool or a prompt-eval runner is on the comparison pages.
The interesting result from this run wasn't that the candidate was worse.
It was that "59% cheaper and 75% faster" wasn't enough information to decide whether the migration was safe.





Top comments (10)
The verdict is right and the sample cannot support the thresholds. At 16 comparisons, 13.6% is about two items - a 95% interval near [2%, 40%]. The tool-divergence breach is 4 of 16, roughly [7%, 52%], which overlaps the 10% ceiling it is being compared against. So this run establishes "the behaviour changed", not "it breaches 10%".
That distinction changes what you do next. For a go/no-go, 16 is enough to say no. For telling a vendor their regression rate is above a specific number, a 10% budget needs roughly 90 comparisons before the interval is narrower than the limit.
Keep the verdict, loosen the number.
Fair push, and you’re right that the margin is thin. One correction first: the 13.6% and 61.4% aren’t over 16 items. Each example gets scored by several evaluators, so those rates are over 44 scored comparisons (6/44 regressed, 27/44 equivalent). Tool divergence is the one that really is 4/16.
On the interval, EvalShift uses Wilson at 95%. For 4/16 that’s [10.2%, 49.5%], so the lower bound clears the 10% limit by a fifth of a point. Your [7%, 52%] is Clopper-Pearson, and under that it wouldn’t confirm. Equivalence is the same story: 27/44 gives an upper bound of 74.3% against a 75% floor. Both breaches confirmed, both by a hair.
The gate is built around your point. A breach the CI can’t confirm comes out inconclusive, not fail. This one barely made it to fail. For a no-go call I’m fine with that. If I wanted to tell a vendor “your divergence is above 10%”, I’d want five times the suite first, and I should have said so in the post.
Treating an unconfirmed breach as inconclusive rather than fail is the decision that makes a gate trustworthy, and it is rarer than it should be. The alternative - fail on any breach, confirmed or not - trains people to ignore the gate within a month, because at small sample sizes most breaches are noise.
The 74.3% against a 75% floor case is exactly why. The point estimate says breach, the interval says we cannot distinguish it from the floor, and only a human can decide whether to spend the sample finding out. Making that decision explicit in the output - "inconclusive, would need N runs to resolve" - is what turns it from a shrug into an instruction.
On five times the suite: defensible as a rule of thumb for vendor-facing claims, and it has a cleaner form. What you need is not five times the runs, it is a sample size where the 95% lower bound of the measured divergence sits above the floor. For a 10% claim against a 75% floor that is a computable number before you run anything, and putting it in the gate output is what ends the argument about how much testing is enough.
The "I should have said so in the post" is the part I would keep. A post that states its own sample-size limits is the one that survives being cited six months later, when someone quotes the 59% number without the caveat.
Exactly, and the triage side has a break-even you can compute before migrating: savings per call times calls, against regression rate times triage cost per regression. On this run's numbers - about $0.106 saved per call - one hour of rollback triage costs roughly the savings of 600 calls. If a cheaper model buys one triage hour per few hundred runs, the migration is a net loss even though every line on the economics table is green.
So the budget that decides it is regressions per thousand runs, priced in triage minutes. That belongs next to max_tool_divergence, because it is the only line on the report denominated in something the business already counts.
Pricing regressions in triage minutes is a better framing than anything on my report right now. The report knows the cost delta per call and how many comparisons regressed. It doesn’t know what one regression costs you, and only the team has that number. I’d rather take it as an input than guess it. Writing this one down.
Caveat that ties back to your first comment: regressions per thousand runs needs a lot more than 16 examples to be worth multiplying by anything.
The caveat is right, and it is quantifiable - which makes it easier to write down than to argue about.
With zero regressions in 16 runs, the 95% upper bound on the true rate is about 3/16, roughly 19%. So the honest statement from 16 examples is not "low regression rate", it is "we have ruled out rates above about one in five". That is still useful. It just cannot be multiplied by a cost per regression to produce a figure that looks precise.
The sample size then follows from the claim you want to make: about 30 clean runs to rule out 10%, about 300 to rule out 1%, and thousands to say anything about one in a thousand. At that point you have to sample deliberately, because production will not hand you the evidence fast enough.
Which suggests reporting the rate as a bound rather than a point. "0 of 16, so below 19% at 95%" is more honest than "0%" and more useful than dropping the metric. It also answers the question the team will ask next - how many more runs do we need - with something better than a shrug: however many tighten the bound to the size of the decision being made.
The 13.% regression on 16 examples is tricky because two regressions easily sit within noise, but changing what the agent actually does is the part that bites. When a cheaper model silently swaps tool calls or skips validation steps, latency savings get eaten immediately by manual rollback triage.
The skipped action is the one that worried me most. The candidate said it had done the thing and never called the tool. No error, no parse failure, nothing in the latency chart. A user finds that one before you do.
Did the regressions cluster on one kind of input, or spread across the whole set? That decides whether you can send the easy cases to the cheap model and keep the rest on the current one.
I haven’t sliced this run. The suite came straight from captures with no tags, so everything landed in one bucket. Worth noting that one of the regressions was the “easy” case: a plain opinion question where the candidate went and called search_web instead of answering. So I wouldn’t assume easy means safe.
If you tag examples by input type, EvalShift analyzes each tag as its own slice and you can set separate budgets per tag. That’s basically the routing question: pass on chit-chat, fail on actions, and you know what to send where. With 16 examples split three or four ways it’d point you somewhere, not decide it.