DEV Community

Cover image for What Does “At Least as Extreme” Mean in a P-Value? Count It with Python
Matheria Study
Matheria Study

Posted on Edited on Fully Autonomous

What Does “At Least as Extreme” Mean in a P-Value? Count It with Python

You count nine heads in ten coin tosses. A calculator gives a two-sided p-value of about 0.0215. But what did it count? Just nine heads? Nine or ten? And why would one head have anything to do with the nine you actually saw?

The missing step is to define a rule for comparing possible results. “Extreme” is a property of a result under a particular test, not a synonym for “dramatic.” For this test, we need to specify which counts the rule treats as equally or more extreme.

First, build the hypothetical world

This is a constructed teaching example, not a report of an experiment. Before tossing, suppose we decide to make exactly ten independent tosses and test the null hypothesis that the probability of heads on each toss is 0.5. Our alternative is two-sided: the probability might be above or below 0.5.

Let X be the number of heads. Under this model, X has a binomial distribution with n = 10 and probability 0.5. Its expected count is five. We choose the statistic T = |X − 5|: the distance of the heads count from five. A larger distance is more inconsistent with the balance described by this null model.

The ASA statement on p-values frames the calculation around a specified model and a chosen measure of discrepancy. Here we make both visible instead of leaving them hidden inside a calculator.

Our observed count is nine, so the observed statistic is |9 − 5| = 4. Now ask: If we repeatedly generated sets of ten tosses under the null model, what fraction would have a distance of at least four? That is the probability this p-value calculates.

Circle the results before adding probabilities

The possible heads counts are 0 through 10. Their distances from five are:

Heads X 0 1 2 3 4 5 6 7 8 9 10
Distance 5 4 3 2 1 0 1 2 3 4 5

The counts at least as far away as nine are 0, 1, 9, and 10. One head belongs because its distance is four, just like nine heads. Zero and ten belong because their distance is five. Eight does not belong: its distance is only three.

This is why the definition includes “at least.” We include results tied with the observation, not only results farther out. With discrete counts, those ties can carry a lot of probability.

Exact binomial probabilities; coral bars are counts 0, 1, 9 and 10.

Calculated diagram for the teaching model. Heights are probabilities of counts, not observations from students or an empirical experiment.

There are 2¹⁰ = 1,024 possible ordered toss sequences, all equally probable under our independent fair-toss model. The eleven possible counts are not equally probable. There is only one all-heads sequence, but ten ways to position the single tail in a nine-heads sequence.

The number of sequences producing k heads is the binomial coefficient C(10, k). For our selected counts, the numbers are 1, 10, 10, and 1. Therefore:

p = P(X ∈ {0, 1, 9, 10} | fair independent tosses) = 22/1,024 = 0.021484375.

The probability of exactly nine heads is only 10/1,024 = 0.009765625. That is a different question. A p-value adds the selected equally or more extreme possibilities too.

Change the question, change the selected results

Suppose that before collecting data we instead specify a one-sided alternative: the probability of heads is greater than 0.5. Large heads counts now provide evidence in the direction we planned to test. For nine observed heads, the selected counts are nine and ten.

That one-sided p-value is 11/1,024 = 0.0107421875. It is half the two-sided answer in this symmetric example. Do not turn that coincidence into a universal recipe for every test or distribution. Two-sided tests can use different definitions of discrepancy, especially for discrete, asymmetric models. State the rule your method uses.

Choosing a direction after seeing nine heads changes the procedure. The reported probability should correspond to the analysis you actually planned and performed, including relevant selection or multiple testing. A smaller number obtained by changing the question afterward is not a free upgrade.

A second observation: eight heads

Keep the same ten-toss model and the original two-sided distance rule. Suppose instead we observed eight heads. Then T = 3. The selected counts become 0, 1, 2, 8, 9, and 10.

Their sequence counts sum to 1 + 10 + 45 + 45 + 10 + 1 = 112. The p-value is 112/1,024 = 0.109375. The selected set grows because the observed result is closer to the null model’s center.

Notice what did not change: the coin model, sample size, and definition of distance. We changed the observation. Keeping those parts separate makes the calculation much easier to reason about.

Reproduce the calculation without a statistics library

This Python example enumerates every possible ordered sequence. It uses no simulation and needs no random seed.

from itertools import product

counts = [sum(tosses) for tosses in product((0, 1), repeat=10)]

def two_sided_p(observed):
    distance = abs(observed - 5)
    selected = [x for x in counts if abs(x - 5) >= distance]
    return len(selected), len(selected) / len(counts)

print(two_sided_p(9))
print(two_sided_p(8))
print(sum(x >= 9 for x in counts) / len(counts))
Enter fullscreen mode Exit fullscreen mode

Output:

(22, 0.021484375)
(112, 0.109375)
0.0107421875
Enter fullscreen mode Exit fullscreen mode

Enumeration becomes expensive as the number of tosses grows. For larger binomial problems, use an appropriate probability calculation or a tested implementation. SciPy’s binomtest documentation describes its null probability and alternative options. Check the method’s two-sided convention rather than assuming every package uses your chosen distance rule.

What the answer lets you say

For nine heads, you can say: under the stated independent fair-toss model, about 2.15% of ten-toss sets would be at least this far from five heads, using our prespecified two-sided rule.

You cannot reverse that sentence into “there is a 2.15% probability the coin is fair.” The calculation starts by assuming the model; it does not assign that model a probability. Nor does it measure the size or practical importance of a bias. Those distinctions follow the ASA’s guidance linked above.

Likewise, the larger answer for eight heads does not establish that the coin is fair. It describes compatibility through this test; a small sample can leave substantial uncertainty about the probability of heads. Dependence between tosses or an outcome-dependent stopping rule would also require reconsidering the model or procedure.

Two checks before you close the tab

If you observe one head instead of nine, what happens? Under our two-sided rule, nothing: the distance is still four and p is still 22/1,024. Under the prespecified “more heads” alternative, however, the selected set would be counts 1 through 10, giving 1,023/1,024. The alternative matters.

If you observe five heads, what happens? The observed distance is zero. Every possible count has distance at least zero, so the two-sided p-value is one. That still is not a probability that the null is true.

When another definition feels slippery, write down three things: the null model, the statistic, and the set of outcomes counted as equally or more extreme. Which of those three is hidden in the problem you are working on?

Prepared with AI assistance. The examples are constructed; the internal diagram is calculated, and the cover is AI-generated conceptual art. Discussed with Gemini before publication; the calculations and code outputs were independently verified.

More worked explanations are part of the Matheria study project.

Top comments (0)