Your Prompt Didn't Change. Your Model Did. Now What?

Every few months the same thing happens.

A provider ships a new model. The one you pinned gets a deprecation date. You open your config, change gpt-4o to whatever is current, run the app once, see reasonable output, and ship it.

Two weeks later someone in support forwards a ticket: a field that used to be extracted correctly is now blank about a fifth of the time.

Nothing in your prompt changed. Nothing in your code changed. The model changed, and you had no test that could have caught it.

Why "I ran it and it looked fine" is not a test

Say you are extracting structured data from long, messy documents — pulling the termination notice period, the auto-renewal terms, and the liability cap out of a 30-page service agreement. The answer is rarely sitting in one labeled box. The notice period is in clause 12, an exception for the initial term is in clause 3, and an amendment on page 27 quietly overrides both. The model has to find the right passages, reconcile them, and return one value.

It is tempting to treat that LLM call like any other function and write one assertion against one input. That breaks down for three reasons.

A wrong answer looks exactly like a right one. If the model returns 60 for the notice period, that is a perfectly plausible number. So is 30, which is what the superseded clause says. You cannot eyeball correctness on this kind of task — "the output looked reasonable" only tells you the model produced something contract-shaped.

LLM outputs are probabilistic. Even at temperature=0, you are not guaranteed identical outputs across runs. Batching, hardware, and silent server-side changes all move the needle. A single passing run is one sample from a distribution you have not measured.

Whole-output equality is the wrong assertion. Your extraction returns eight fields. If one of them regresses and seven improve, a string comparison on the serialized JSON just tells you "different." You need to know which field moved and by how much.

So the naive harness that most of us write first looks like this:

def test_extraction():
    result = extract(prompt, load("service_agreement_001.pdf"))
    assert result["notice_period_days"] == 60
    assert result["governing_law"] == "State of New York"

This is a smoke test. It tells you the pipeline is wired up. It tells you nothing about accuracy, and it will flake often enough that someone eventually marks it @pytest.mark.skip.

What a real prompt regression test needs

Four things, and skipping any one of them puts you back in vibes territory.

1. A frozen, labeled dataset. Real inputs with ground truth values you agreed on once and stopped editing. Twenty items is enough to start; the point is that it does not drift under you.
2. Field-level scoring. Accuracy per field, not per document.
3. Repeated trials. Run the same prompt against the same item many times, because you are estimating a rate, not checking a value.
4. A confidence interval. So that when you see 89% and 92%, you can say whether that gap is real.

None of this needs much code. Loop over the dataset, run every item twenty times, score each field as a hit or a miss against the ground truth, and put a 95% confidence interval around each field's hit rate. The repeated trials are the part people leave out, and the part that makes the result mean something.

What matters is what comes out the other end. Instead of a green checkmark, you get output that actually supports a decision:

notice_period_days   88.5%  [84.1% – 92.9%]
auto_renewal         94.0%  [91.2% – 96.8%]
liability_cap        71.0%  [65.3% – 76.7%]
governing_law        96.5%  [94.1% – 98.9%]

Now liability_cap is visibly the weak field — no surprise, since it is the one buried in carve-outs and cross-references — and you know the estimate is tight enough to act on.

The decision rule that matters

When you compare the old model to the new one, do not compare point estimates. Compare intervals.

                          model-a              model-b
liability_cap        71.0% [65.3–76.7]   76.0% [70.4–81.6]   → overlapping, no evidence
notice_period_days   88.5% [84.1–92.9]   79.2% [74.0–84.4]   → separated, real regression

The first row looks like a five-point improvement and is not distinguishable from noise. The second row is the one that would have shown up in a support ticket three weeks later.

This single habit — refusing to act on overlapping intervals — removes most of the thrash from prompt work. It stops you from "fixing" a prompt that was never broken, and it stops you from shipping a model swap that quietly cost you nine points on a field that matters.

Where the homegrown harness starts to hurt

You can script all of this in an afternoon. I have written versions of it more than once. What breaks is not the statistics, it is everything around them.

The labels live in a CSV on one engineer's laptop. The domain expert who actually knows what the correct liability_cap is — the person who can read clause 14 and its three exceptions — cannot run Python. Every model comparison means rerunning the whole grid by hand and pasting numbers into Slack. Nobody can answer "what was our accuracy before the upgrade?" because nothing was stored.

That gap is the reason we built PromptProof. It runs statistical experiments as a first-class object: pick a labeled folder, configure 10–50 trials, choose a 90/95/99% confidence level, and it executes every prompt × item × trial combination in the background. You get confidence intervals, a field-level accuracy breakdown, and latency percentiles — and you can run the same prompt across OpenAI, Anthropic, and Google models in one experiment to compare them side by side.

The part that mattered most to us in practice was making the dataset shared. Ground truth labels live with the team, so the person who knows the domain can label, and the person who writes the prompt can test — against the same standard, instead of "it worked on my test set."

Closing the loop with production

A frozen evaluation set catches model upgrades. It does not catch data drift, because your test inputs stopped changing on purpose.

For that you need the other half: sample real production traffic, label it, and fold it back into the evaluation set. PromptProof's monitoring side does this as a loop — ingest production items via API, label ground truth in the UI, copy the labeled data into an experiment folder, and rerun. Regressions show up as an accuracy trend rather than as a ticket.

You can absolutely build this yourself. The point is that "ingest → label → evaluate → deploy → keep watching" is the shape of the thing, and most teams stop after the first arrow.

The short version

If you take one thing from this:

• Pin your model IDs, and treat a model bump as a change that requires evidence, not a version bump.
• Freeze a labeled dataset before you need it. Twenty real items beats two hundred synthetic ones.
• Score per field. Aggregate accuracy hides the regression that will page you.
• Run repeated trials. One run is one sample.
• Compare intervals, not point estimates. Overlapping means you learned nothing.
• Feed production data back in, or your test set will slowly stop resembling reality.

The next deprecation email is already scheduled. The question is only whether you will find out from your test suite or from your users.

---

What does your prompt regression setup look like? I am curious whether people are mostly hand-rolling this or have landed on something they like — comments welcome.