Most teams shipping AI features are making statistical claims all day without calling them that. An eval score went from 71 to 74. A new prompt "feels better" on a handful of examples. A test showed 60 percent of users preferred version A. Every one of those is a claim about a population made from a sample, which means the rules of sampling apply whether anyone invokes them or not.
Here is the short refresher that catches most of the damage.
Why Small Samples Lie
The core problem is that small samples swing. If you ask 50 users which version they prefer and 30 of them say A, that 60 percent has enough wobble in it that a rerun the following week can land the other way. Nothing changed except which 50 people you happened to ask.
This is not a reason to distrust data, it is a reason to know how much data you have. The law of large numbers says sample averages converge on the true value as the sample grows. Flip a fair coin ten times and seven heads is unremarkable. Flip it ten thousand times and the proportion sits close to 0.50. Your eval set of 40 prompts is the ten flip version of the experiment.
What an Interval Tells You That an Average Cannot
A single number hides its own uncertainty. An interval carries it.
"74 percent" and "74 percent, plus or minus 6" are the same measurement, but only one of them tells you that the 71 percent you measured last month is inside the range and therefore not clearly worse. Teams that report intervals stop shipping changes that were never distinguishable from noise, and they stop rolling back changes that actually worked.
The habit costs nothing. Report the number, then the interval, then the sample size that produced both.
Where Eval Scores Go Wrong
Three failure modes come up over and over in model and prompt work.
The first is peeking. Someone watches a running comparison and stops the moment it crosses the threshold they wanted. Stopping on a high point inflates the effect and quietly destroys the guarantee the test was supposed to provide.
The second is the multiple comparisons problem. Run twenty prompt variants against the same eval set and one of them will look significantly better by chance alone. If you only report that one, you have published noise.
The third is the wrong denominator. Cost per run, accuracy per attempt and latency per call all look reasonable until you remember that failed runs get retried. Cost per successful task is the number that survives contact with production, and it moves differently from the one on the dashboard.
The Takeaway
You do not need graduate statistics to avoid the expensive mistakes. You need four habits: decide the sample size before you look, report intervals rather than bare numbers, count how many comparisons you actually ran, and check that your denominator matches the thing you care about.
Everything else is refinement. These four are the ones that decide whether the number in your changelog means anything.
If you want the underlying material in one place, from descriptive statistics and distributions through hypothesis testing and regression, this guide covers it in plain language: https://www.learnhowtoscience.com/learn-statistics/
Top comments (0)