DEV Community: Atsushi Hara

A New Model Dropped. Don't Just Swap the ID.

Atsushi Hara — Mon, 21 Sep 2026 02:48:33 +0000

The new model is out. The benchmarks look great. Your provider has already scheduled the sunset date for the one you are running.

So you swap the model ID, run a few requests, see nothing obviously broken, and ship.

I want to argue that this is one of the most common ways LLM-backed systems quietly degrade — and that it tends to happen not because teams are careless, but because "the model works" and "the model is safe to run in production" are two different claims, and only the first one is cheap to check.

Everyone agrees, almost nobody does it

Ask any engineer whether you should evaluate a new model before switching to it. Most will say yes. Then look at what often happens when the deprecation email arrives.

A recent paper on model migration in production systems put the problem plainly: deprecation and migration cycles come around roughly every twelve months, they hit every product built on the retired model, and task-specific evaluation is essential — but a manual evaluation process is too slow and too expensive to fit inside the migration window (Casey et al., 2026). The result is easy to guess. Evaluation tends to get skipped, then done anyway in the last week before the sunset, in a panic, on whatever inputs someone had lying around.

The gap is not one of belief. It is one of process cost. Few people skip a test suite that runs in a minute. Many skip the one that takes three days.

"It works" is a one-dimensional answer to a three-dimensional question

Part of why the swap feels safe is the release announcement. The new model scores higher on the public benchmarks, so it should score higher on your task too. That inference is where things can go wrong.

A new model version might not be a backward-compatible upgrade of the old one. It may be a different model, trained differently, with different habits, and nobody has checked that against your prompt. The prompt you tuned against the old model's habits carries no guarantee with the new one — the same prompt is not promised to perform as well, let alone better. And the reported gains were measured on public datasets that look nothing like your contracts, your invoices, or your support tickets. "Better on the benchmark" and "better on your data" are correlated, not equivalent. The only way to know the second one is to measure it.

When you do measure it, three things can move, and they can move in opposite directions.

Accuracy. The new model may be better on average and worse on the field you care about. It may handle the easy cases identically and change its mind on the ambiguous ones — which are exactly the cases your spot check did not include.

Latency. Newer models are frequently slower per request, particularly the ones that spend tokens reasoning before they answer. A jump in median latency from 1.5 seconds to 3 seconds is easy to miss in a manual test and hard to miss in a UI that blocks on the response.

Token volume. This is the one that tends to surprise people. A new model with the same prompt will often produce longer output: more hedging, more explanation, more fields filled in that you asked it to leave empty. Output tokens are the expensive ones, and they compound. A model that is 20% cheaper per token and produces 80% more of them is a price increase.

Here is the kind of comparison that decides an upgrade — same prompt, same frozen dataset, old model versus new:

                      model-a (current)         model-b (candidate)
accuracy (overall)    91.2%  [89.0 – 93.4]       92.8%  [90.7 – 94.9]
accuracy (weak field) 74.5%  [70.1 – 78.9]       81.0%  [77.0 – 85.0]
latency p50 / p75     1.4s  /  1.9s              2.6s  /  3.8s
output tokens p50     180                        410

Read that table honestly. The headline accuracy gain is inside the noise — the intervals overlap, so you have not shown an improvement. The weak field did get better, and that might be worth a lot. But latency nearly doubled and output volume more than doubled. Whether to ship this is a real decision with real trade-offs, and you cannot make it from "I ran it and it looked fine."

A decision rule you can write down

The point of measuring three dimensions is to have a rule, so the upgrade is not a matter of who argued loudest in the meeting.

Dimension	Compare	Ship if
Accuracy	Confidence intervals, per field	No field's interval moved down below the old one
Latency	p50 and p75, not the mean	Both stay under your budget; p75 is what users feel
Token volume	Median output tokens	Within the budget you set when you priced the feature

Three notes on applying it.

Compare intervals, not point estimates. LLM output is stochastic. A single run of your test set is one sample. Twenty runs give you an interval, and the interval is what tells you whether 91.2% versus 92.8% is signal or noise.

Watch the tails, not the average. Mean latency hides the slow requests. p75 is the request a quarter of your users wait for.

If any dimension regresses, the answer is "not yet", not "no". In my experience, most regressions on a model upgrade turn out to be prompt-model mismatches rather than model failures. The prompt was tuned against the old model's habits. A shorter, stricter output instruction often brings token volume back down; a re-tuned prompt often recovers the field that dropped. But you only know what to fix if you measured it.

Why this stays undone

None of the above is intellectually hard. The reason it tends not to happen is that doing it by hand often looks like this:

Find the last labeled dataset someone built. Discover it lives in a spreadsheet on a laptop and is six months stale.
Write a script to run both models against it, twenty times each, and record accuracy per field, latency per request, and tokens per response.
Compute intervals. Argue about whether the script is computing them correctly.
Paste the numbers into a Slack thread. Lose them.
Repeat in twelve months, from scratch, because nothing was kept.

Each step is a few hours. Together they can add up to a week nobody has during a migration window. So the process gets compressed into "swap the ID and watch the error rate", which catches crashes and can miss everything in the table above.

What we built to make the honest path the easy one

This is the workflow PromptProof exists for.

You keep a labeled dataset in the product, shared with the team, so the domain expert who knows the right answers can maintain it and it does not go stale on someone's laptop. When a new model arrives, you attach the same prompt to the old model and the new one and run a single statistical experiment: 10–50 trials per item, a confidence level you choose, executed in the background.

What comes back is the table above, without the script:

accuracy with confidence intervals, broken down per field, side by side per model
latency at p25 / p50 / p75 per prompt
input and output token volume, as medians and spread, per prompt

It works across OpenAI, Anthropic, and Google models in the same experiment, so "should we move to the new model" and "should we move to a different provider" are the same comparison.

The intent is not that the tool decides for you. It is that the three numbers you need for the decision cost you minutes of setup instead of days, so the evaluation is much more likely to happen before the sunset date instead of after the support tickets.

The short version

A model upgrade is a change to your system. Treat it like one: it needs evidence, not a version bump.
A new version might not be a compatible upgrade — nobody has verified that it is for your prompt. Benchmark gains were measured on public data; yours is not public.
"It works" answers one question. Production needs three: accuracy, latency, token volume.
Compare intervals, watch the tails, budget the tokens.
A regression means "not yet, re-tune the prompt", not "never upgrade".
The reason evaluation gets skipped is usually process cost. Lower the process cost and the evaluation is far more likely to happen.

The next model will probably be better than this one. That is the whole reason to measure before you switch — because the way it is better is often not the way you assumed.

How does your team decide when to move to a new model? I would like to hear what you measure, and what you have been burned by not measuring.

Your Prompt Didn't Change. Your Model Did. Now What?

Atsushi Hara — Thu, 17 Sep 2026 12:25:24 +0000

Every few months the same thing happens.

A provider ships a new model. The one you pinned gets a deprecation date. You open your config, change gpt-4o to whatever is current, run the app once, see reasonable output, and ship it.

Two weeks later someone in support forwards a ticket: a field that used to be extracted correctly is now blank about a fifth of the time.

Nothing in your prompt changed. Nothing in your code changed. The model changed, and you had no test that could have caught it.

Why "I ran it and it looked fine" is not a test

Say you are extracting structured data from long, messy documents — pulling the termination notice period, the auto-renewal terms, and the liability cap out of a 30-page service agreement. The answer is rarely sitting in one labeled box. The notice period is in clause 12, an exception for the initial term is in clause 3, and an amendment on page 27 quietly overrides both. The model has to find the right passages, reconcile them, and return one value.

It is tempting to treat that LLM call like any other function and write one assertion against one input. That breaks down for three reasons.

A wrong answer looks exactly like a right one. If the model returns 60 for the notice period, that is a perfectly plausible number. So is 30, which is what the superseded clause says. You cannot eyeball correctness on this kind of task — "the output looked reasonable" only tells you the model produced something contract-shaped.

LLM outputs are probabilistic. Even at temperature=0, you are not guaranteed identical outputs across runs. Batching, hardware, and silent server-side changes all move the needle. A single passing run is one sample from a distribution you have not measured.

Whole-output equality is the wrong assertion. Your extraction returns eight fields. If one of them regresses and seven improve, a string comparison on the serialized JSON just tells you "different." You need to know which field moved and by how much.

So the naive harness that most of us write first looks like this:

def test_extraction():
    result = extract(prompt, load("service_agreement_001.pdf"))
    assert result["notice_period_days"] == 60
    assert result["governing_law"] == "State of New York"

This is a smoke test. It tells you the pipeline is wired up. It tells you nothing about accuracy, and it will flake often enough that someone eventually marks it @pytest.mark.skip.

What a real prompt regression test needs

Four things, and skipping any one of them puts you back in vibes territory.

A frozen, labeled dataset. Real inputs with ground truth values you agreed on once and stopped editing. Twenty items is enough to start; the point is that it does not drift under you.
Field-level scoring. Accuracy per field, not per document.
Repeated trials. Run the same prompt against the same item many times, because you are estimating a rate, not checking a value.
A confidence interval. So that when you see 89% and 92%, you can say whether that gap is real.

None of this needs much code. Loop over the dataset, run every item twenty times, score each field as a hit or a miss against the ground truth, and put a 95% confidence interval around each field's hit rate. The repeated trials are the part people leave out, and the part that makes the result mean something.

What matters is what comes out the other end. Instead of a green checkmark, you get output that actually supports a decision:

notice_period_days   88.5%  [84.1% – 92.9%]
auto_renewal         94.0%  [91.2% – 96.8%]
liability_cap        71.0%  [65.3% – 76.7%]
governing_law        96.5%  [94.1% – 98.9%]

Now liability_cap is visibly the weak field — no surprise, since it is the one buried in carve-outs and cross-references — and you know the estimate is tight enough to act on.

The decision rule that matters

When you compare the old model to the new one, do not compare point estimates. Compare intervals.

                          model-a              model-b
liability_cap        71.0% [65.3–76.7]   76.0% [70.4–81.6]   → overlapping, no evidence
notice_period_days   88.5% [84.1–92.9]   79.2% [74.0–84.4]   → separated, real regression

The first row looks like a five-point improvement and is not distinguishable from noise. The second row is the one that would have shown up in a support ticket three weeks later.

This single habit — refusing to act on overlapping intervals — removes most of the thrash from prompt work. It stops you from "fixing" a prompt that was never broken, and it stops you from shipping a model swap that quietly cost you nine points on a field that matters.

Where the homegrown harness starts to hurt

You can script all of this in an afternoon. I have written versions of it more than once. What breaks is not the statistics, it is everything around them.

The labels live in a CSV on one engineer's laptop. The domain expert who actually knows what the correct liability_cap is — the person who can read clause 14 and its three exceptions — cannot run Python. Every model comparison means rerunning the whole grid by hand and pasting numbers into Slack. Nobody can answer "what was our accuracy before the upgrade?" because nothing was stored.

That gap is the reason we built PromptProof. It runs statistical experiments as a first-class object: pick a labeled folder, configure 10–50 trials, choose a 90/95/99% confidence level, and it executes every prompt × item × trial combination in the background. You get confidence intervals, a field-level accuracy breakdown, and latency percentiles — and you can run the same prompt across OpenAI, Anthropic, and Google models in one experiment to compare them side by side.

The part that mattered most to us in practice was making the dataset shared. Ground truth labels live with the team, so the person who knows the domain can label, and the person who writes the prompt can test — against the same standard, instead of "it worked on my test set."

Closing the loop with production

A frozen evaluation set catches model upgrades. It does not catch data drift, because your test inputs stopped changing on purpose.

For that you need the other half: sample real production traffic, label it, and fold it back into the evaluation set. PromptProof's monitoring side does this as a loop — ingest production items via API, label ground truth in the UI, copy the labeled data into an experiment folder, and rerun. Regressions show up as an accuracy trend rather than as a ticket.

You can absolutely build this yourself. The point is that "ingest → label → evaluate → deploy → keep watching" is the shape of the thing, and most teams stop after the first arrow.

The short version

If you take one thing from this:

Pin your model IDs, and treat a model bump as a change that requires evidence, not a version bump.
Freeze a labeled dataset before you need it. Twenty real items beats two hundred synthetic ones.
Score per field. Aggregate accuracy hides the regression that will page you.
Run repeated trials. One run is one sample.
Compare intervals, not point estimates. Overlapping means you learned nothing.
Feed production data back in, or your test set will slowly stop resembling reality.

The next deprecation email is already scheduled. The question is only whether you will find out from your test suite or from your users.

What does your prompt regression setup look like? I am curious whether people are mostly hand-rolling this or have landed on something they like — comments welcome.