Your prompt A/B test is lying to you (if you keep checking it)

Serhan Yavçin — Wed, 11 Nov 2026 12:26:56 +0000

You rewrote the support prompt. Two days later the dashboard says the new version resolves four points more tickets, and the 95% interval doesn't touch zero. You ship it. A month later the resolution rate is exactly where it was.

The arithmetic was fine. What went wrong was when it was read.

This is a condensed version of a longer guide, with an interactive version of the demo below, sample sizes and a FAQ: A/B testing LLM prompts without false winners.

A 95% interval makes one promise

When there's no real difference, a 95% interval excludes zero only 5% of the time. That promise holds for one look, at a sample size you fixed in advance. A live dashboard recomputes the interval after every batch of results, and every look is a fresh chance for noise to cross the line.

How much does that cost? We simulated 4,000 A/A tests, where both versions are the same prompt at 30% success, and looked after every 100 outcomes per version. A test "found" a difference if its interval ever excluded zero:

Looks	Fixed 95% interval	Always-valid 95% interval
1	4.4%	0.1%
5	13.7%	0.4%
10	19.1%	0.8%
100	38.0%	2.2%

Ten peeks, and one test in five finds a winner that isn't there. Keep looking and it never stops climbing; Armitage, McPherson and Rowe measured the same effect in clinical trials back in 1969.

Prompt tests invite it more than most. The dashboard is live, and the new prompt was written to win.

What peeking looks like

Here is one of those simulated A/A runs. The black line is the new version's lead, the brown lines are the usual 95% interval, and the blue band is an always-valid interval:

Between 1,500 and 2,200 outcomes per version, the usual interval says the new prompt wins by about four points, at seven different looks. Then the lead fades, as noise does. The blue band never leaves zero.

Intervals built to be looked at

The fix is the mixture sequential probability ratio test (mSPRT). Instead of asking "is the difference non-zero at this sample size?", it keeps a running likelihood ratio: how much more likely the data are under some real difference than under none, averaged over the differences you'd plausibly see. When there's no difference, that ratio is a martingale, so the chance that it ever climbs above 1/α is at most α, however often you check (Robbins, 1970). Johari, Koomen, Pekelis and Walsh turned this into p-values and intervals for A/B tests (2022).

With a normal approximation, the 95% always-valid interval for the difference is diff ± r, where:

r = √( V·(V + τ²)/τ² · [ 2·ln(1/α) + ln((V + τ²)/V) ] )
V = s²(A)/n(A) + s²(B)/n(B)

In code, that's one line:

// Half-width of an always-valid interval for mean(B) − mean(A).
// v: the variance of that difference, s²(A)/n(A) + s²(B)/n(B)
// tau: the smallest difference that matters, e.g. 0.05 on a 30% rate
function radius(v: number, tau: number, alpha = 0.05): number {
  const t2 = tau * tau;
  return Math.sqrt(((v * (v + t2)) / t2) * (2 * Math.log(1 / alpha) + Math.log((v + t2) / v)));
}

τ sets the size of difference the test is most sensitive to, so set it to the smallest one you care about.

The price is data. For a real lift from 30% to 35%, it found the difference after a median of 1,400 outcomes per version, and by 2,400 in 80% of tests. A fixed test sized for 80% power needs about 1,400, but only if you look once. Roughly, 1.7 times the data buys the right to look whenever you like and to stop as soon as the answer is clear.

Two more ways a prompt test lies

Randomizing by call. A user who gets the formal prompt for one reply and the casual one for the next has a worse experience than either would give, and their outcomes aren't independent anyway. Hash the user or conversation id with the test's id into a bucket from 0 to 99, and serve the new version to buckets below its share. Same user, same version, no table of assignments, and raising the share only moves users from live to new.

A broken split. You asked for 50/50 and got 10,300 calls on live and 9,700 on the new version? A chi-square test gives χ² = 18 and p ≈ 0.00002: the comparison isn't fair, whatever the metric says. That's a sample ratio mismatch (Fabijan et al., 2019), and in LLM apps the usual suspects are:

the new prompt is longer and times out more, and calls that time out never report an outcome;
an older deployment doesn't know about the test and sends every call to live;
calls missing a variable the new prompt needs fall back to live and are counted there.

Check the split continuously with a strict threshold, such as p < 0.001, and stop reading the metric when it trips.

A checklist for your next prompt test

Randomize by user or conversation, by hash, and keep the share steady.
One primary metric, chosen before the test; tokens and latency as guardrails.
Write down the smallest difference that matters.
Read an always-valid interval, or fix the sample size and look once.
Check the split all the time.
Let the test stop itself on clear harm; ship by hand, after reading real answers.

The full guide goes further: choosing the metric, how many outcomes each effect size needs, changing the share mid-test, the winner's curse, and a FAQ on t-tests, bandits and model-graded scores.

We build PromptHelm, which lets you change the prompts your app sends to an LLM without a redeploy. Its A/B tests work the way this post describes, and the simulations above ran on its statistics code.

References

Armitage, McPherson & Rowe (1969), Repeated significance tests on accumulating data, Journal of the Royal Statistical Society, Series A.
Robbins (1970), Statistical methods related to the law of the iterated logarithm, The Annals of Mathematical Statistics.
Johari, Koomen, Pekelis & Walsh (2022), Always valid inference: continuous monitoring of A/B tests, Operations Research.
Fabijan et al. (2019), Diagnosing sample ratio mismatch in online controlled experiments, KDD.