A scientific paper reports that a mean difference is 1.4, with a 95% confidence interval of 0.6 to 2.2. In the news, and sometimes even in research talks, this is shortened to a more comforting sentence: we can be 95 percent sure the true value lies between 0.6 and 2.2. That sentence is easy to remember. For a frequentist confidence interval, it is also the wrong meaning.
The core idea is this: a 95% confidence interval describes the long-run behavior of a method, not the probability that this particular interval covers the true value. The number 95% belongs to the procedure. It does not attach itself, as a posterior probability, to the one interval you have just computed.
Here is what the procedure is supposed to do. You adopt a statistical model, collect a sample, and compute an interval from that sample by a fixed rule. Imagine, as a thought experiment, repeating the whole study many times, each time drawing a new sample from the same process and applying the same rule. If the model is right and the rule is a 95% confidence procedure, then in the long run about 95% of those intervals would contain the unknown parameter. The other 5% would miss it. In any one repetition, the parameter itself is not jumping around. It is treated as fixed but unknown. The interval is what changes. Once you have computed this week's interval, that specific pair of numbers either includes the parameter or it does not. There is no remaining 95% chance attached to the interval, because the randomness that the 95% referred to was in the sampling, not in the already-realized endpoints.
This distinction is one of the most common misreadings of frequentist intervals. In 2016, Sander Greenland, Stephen Senn, Kenneth Rothman and colleagues published a widely used guide to misinterpretations of tests, P values, confidence intervals, and power in the European Journal of Epidemiology. Among the points they stress is that a 95% interval should not be read as if it were a 95% probability statement about the parameter, and that values inside the interval are not thereby shown to be equally plausible, or "proven." The interval is a statement of compatibility with a model and a procedure, under assumptions that may themselves be wrong. If the model is badly misspecified, the long-run coverage can fail, and the label "95%" will not rescue it.
A clearly labeled illustration may help. Suppose some physical quantity has a fixed but unknown value. You measure it on a random sample and, by a standard method, obtain the interval 12.1 to 13.4. These numbers are made up for the example; they are not a result from a real study. The frequentist claim is not "there is a 95% probability that the true value sits between 12.1 and 13.4." It is: "This interval was produced by a method that, in repeated samples, covers the true value about 95% of the time." You cannot see, from the interval alone, whether this particular attempt was one of the successful ones.
The interval still tells you something useful, which is why it is worth reporting. It shows the precision of the estimate under the model: a narrow interval means that, if the assumptions hold, the estimate is tightly constrained; a wide interval means the data have not pinned the quantity down. It also shows which parameter values are more compatible with the data, in the limited sense used by the method. If a 95% interval for a difference excludes zero, that is closely related to a two-sided test at the 5% level rejecting "no difference," but the interval still carries extra information: it tells you about direction and about how large or small the compatible values can be. A result can be "statistically significant" and still have an interval wide enough to include effects too small to matter, or too large to be ignored. The label 95% itself is a convention, not a law of nature. A 90% interval is narrower and misses more often; a 99% interval is wider and misses less often, if the procedure is well calibrated.
A confidence interval is also not a range of individual observations, and it is not a prediction interval for the next data point. It is not an automatic judgment of practical importance. Two intervals that overlap can still be consistent with a statistically detectable difference, depending on how they were constructed and what is being compared; overlap is a rough visual check, not a substitute for the actual contrast. And a Bayesian credible interval, which can be given a direct probability reading under a prior, is a different object. It can be a reasonable way to express uncertainty, but it is not what a classical "95% CI" is promising unless the paper has actually taken that route.
A more careful way to read the sentence in the first paragraph is therefore more modest. The authors estimated a difference of 1.4 by a method that, under their model, would capture the true difference in about 95% of repetitions. The interval from 0.6 to 2.2 is the result of that method on this sample. It is a reason to treat values far outside that range as less compatible with the data, given the model. It is not a 95% bet that the truth lives between those two numbers. The work that remains is the usual work of reading a study: whether the model, the sample, and the measurement deserve the confidence the interval claims.
More research explainers: https://www.crushsci.com
Top comments (0)