How to Calculate Statistical Power and Sample Size
Illustration of Type I error and power in hypothesis testing: the distributions under the null and alternative hypotheses partially overlap, with shaded areas representing the Type I error probability (significance level α) and statistical power. Author: user3000877 (Cross Validated). Source: Wikimedia Commons. License: CC BY-SA 4.0.
Statistical power is the probability that a test will identify an effect as significant when the effect truly exists. Power, significance level, effect size, and sample size constrain one another: fixing three allows the fourth to be calculated, which is the basis of sample size calculations. This article explains how to balance the two types of error, how effect size enters the formula, and how to use power.t.test and G*Power, with a reproducible example requiring 64 participants per group. It also explains why post hoc power should not be used to interpret nonsignificant results.
The Tradeoff Between Power and the Two Types of Error
Hypothesis testing begins by specifying a null hypothesis H₀ (for example, that two groups have the same mean) and an alternative hypothesis H₁ (that the groups truly differ). Rejecting H₀ when it is true is a Type I error, whose probability is denoted by α, the significance level selected in advance. Failing to reject H₀ when an effect truly exists is a Type II error, whose probability is denoted by β. Power is defined as 1 − β, the probability that a test reports a “significant” result when an effect really exists (Cohen, 1992). Think of a test as a smoke alarm: sounding when there is no fire is a Type I error, failing to sound when there is a fire is a Type II error, and power is the probability that it sounds when there really is a fire. Power always refers to a particular assumed true effect: the same study has high power for large effects and low power for small effects. Thus, a statement that “this study has 80% power” must also specify the size of the effect it refers to.
Cohen (1992) described sample size, significance level α, population effect size, and power as four interrelated quantities in statistical inference: for a given test, any one is determined by the other three. When sample size and effect size are fixed, the two types of error trade off against each other. Consider a two-sided t test with 64 participants in each group and a standardized effect size of d = 0.5: at α = 0.05, power is approximately 0.80. Tightening α to 0.01 reduces power to approximately 0.60 under the normal approximation; restoring it to 0.80 requires approximately 95 participants per group, consistent with the values in Cohen’s Table 2. Lakens (2022) gives an example in the opposite direction: with a maximum of 50 participants per group and an expected d = 0.5, α = 0.05 provides only 70% power. If the two types of error are considered equally serious, α can be set to 0.149 to achieve power of 0.851, an approach called compromise power analysis.
80% power is a convention, not a theorem. Cohen (1992) recommended 0.80 as a general convention: substantially lower power makes the risk of Type II error too high, while substantially higher power often requires a sample beyond researchers’ resources. Combined with the commonly used α = 0.05, the ratio of Type II to Type I error risk is 0.20 : 0.05, or 4 : 1. The clinical trial guideline ICH E9 (finalized on February 5, 1998) states that the Type I error probability is usually set at 5% or less and the Type II error probability at 10%–20%. Particularly for trials that are difficult or impossible to repeat, keeping Type II error as low as possible serves the sponsor’s own interests. In some circumstances, other values may also be acceptable or even preferable.
Sample size and effect size jointly determine power: the rows of the nine panels correspond to sample sizes n = 10, 30, 100, and the columns to effects θ = 0.1, 0.3, 0.5. Each panel shows the distributions of the sample mean under the null hypothesis (θ = 0) and the alternative hypothesis. Red shading represents Type I error α, fixed at 0.05, and green shading represents power 1 − β: larger samples produce narrower distributions, while larger effects move the distributions farther apart, both increasing power. The original figure uses a one-sided critical value. Author: Cosmia Nebula. Source: Wikimedia Commons. License: CC BY-SA 4.0.
The Role of Effect Size in the Sample Size Formula
When comparing the means of two independent groups with equal sample sizes and a common standard deviation σ, the normal approximation for the required sample size per group in a two-sided test is (Julious, 2004):
n = 2σ²(z₁₋α/₂ + z₁₋β)² / Δ²
Here, Δ is the between-group mean difference to be detected, and z₁₋α/₂ and z₁₋β are standard normal quantiles (1.960 for a two-sided α = 0.05 and 0.842 for power of 0.80). Defining the standardized effect size as d = Δ/σ simplifies the formula to n = 2(z₁₋α/₂ + z₁₋β)² / d².
Effect size appears squared in the denominator. This yields the most important rule in sample size calculation: halving the effect to be detected quadruples the required sample size. In Cohen’s Table 2, at α = 0.05 and power of 0.80, d = 0.5 requires 64 participants per group, while d = 0.2 requires 393 per group. Their ratio is approximately 6.1, close to (0.5/0.2)² = 6.25. σ also enters the formula as a squared term: if σ is underestimated as half its true value, the calculated sample size is only one quarter of what is actually needed.
The formula above assumes that the variance is known. In actual analysis, the sample variance is used, and the test statistic follows a t distribution; exact power must be calculated iteratively using the noncentral t distribution. Julious (2004) notes that the normal approximation generally yields a sample size only one to two participants smaller than the exact solution and describes Guenther’s correction: add z²₁₋α/₂ / 4 to the normal approximation (approximately 0.96 for a two-sided α = 0.05). Commonly used software generally provides exact results based on the noncentral t distribution.
The hardest quantities to determine in the formula are Δ and σ. ICH E9 requires the trial protocol to state the sample size calculation method, the values used (variance, means, response rates, differences to be detected, and so on), and their justification, and to examine how the sample size changes when these assumptions are violated. Assumptions for confirmatory trials should generally be based on published data or results from earlier trials. The difference to be detected may be the smallest clinically meaningful effect, or the expected effect if it is larger. Lakens (2022) argues that the strongest sample size justification comes from specifying the “smallest effect size of interest” in advance. Simply adopting an effect size from a single study or from literature affected by publication bias often overestimates the effect. Alternatives include bias correction methods or calculations based on the lower bound of a 60% two-sided confidence interval for the effect size (called safeguard power). Cohen’s d = 0.2, 0.5, 0.8 are operational conventions he proposed for “small, medium, and large” effects in the behavioral sciences; they cannot replace judgments about effect sizes in a specific field.
A Reproducible Example and Software Usage
Consider a two-group parallel randomized trial comparing reductions in systolic blood pressure under a new and an existing blood pressure treatment regimen. All the following values are hypothetical and used for illustration: a between-group difference to be detected of Δ = 5 mmHg, standard deviation σ = 10 mmHg, two-sided α = 0.05, power of 0.80, and 1 : 1 allocation between groups.
- Standardized effect size: d = 5 / 10 = 0.5.
- Quantiles: z₀.₉₇₅ = 1.960, z₀.₈₀ = 0.842; their sum is 2.802, which squared gives 7.849.
- Normal approximation: n = 2 × 7.849 / 0.5² = 62.8, rounded up to 63 participants per group.
- Guenther’s correction: 62.8 + 1.960² / 4 = 62.8 + 0.96 = 63.75, rounded to 64 participants per group.
- Exact noncentral t solution: R’s
power.t.testgives n ≈ 63.77, rounded to 64 participants per group and 128 in total, consistent with the 64 participants corresponding to d = 0.5 in Cohen’s Table 2. - If 10% of participants are expected to drop out, calculate 64 / (1 − 0.10) = 71.1 and round up again, recruiting 72 participants per group and 144 in total. ICH E9 requires the protocol to specify how withdrawals and participants who violate the protocol will be handled. If the primary analysis uses the full analysis set, the expected effect size may also need to be reduced.
# Solve for per-group n (two-sample, two-sided t test)
power.t.test(delta = 5, sd = 10, sig.level = 0.05, power = 0.80)
# The same design on the standardized scale gives the same n
power.t.test(delta = 0.5, sd = 1, sig.level = 0.05, power = 0.80)
# Power with 64 per group if the true difference is only 4 mmHg
power.t.test(n = 64, delta = 4, sd = 10, sig.level = 0.05)
The first two lines both yield n ≈ 63.77; the third gives power of approximately 0.61.
power.t.test belongs to the stats package included with R. Its rule is to leave exactly one of the five parameters n, delta, power, sd, and sig.level as NULL, and the function solves for that parameter. n is the number of participants per group; type can be two.sample, one.sample, or paired, and alternative can be two-sided or one-sided. sd and sig.level have default values (1 and 0.05), so they must explicitly be set to NULL to solve for them. By default, a two-sided test counts only the rejection probability in the direction of the true effect; setting strict = TRUE also includes rejection in the opposite direction. G*Power is a free standalone program whose 3rd edition covers the t, F, and χ² test families, as well as z tests and some exact tests (Faul et al., 2007). For this example, select t tests, Means: Difference between two independent means (two groups), and A priori in G*Power, then enter two-sided, d = 0.5, α = 0.05, power of 0.80, and an allocation ratio of 1. This likewise gives 64 participants per group. In addition to a priori analysis, G*Power can perform sensitivity and compromise analyses (Lakens, 2022).
The table below fixes the sample size at 64 participants per group to show how power changes when the true effect differs from the assumption. The values use the normal approximation, power ≈ Φ(d·√(n/2) − z₁₋α/₂), and can be reproduced using any standard normal distribution table. power.t.test uses the t distribution and gives slightly lower results, with differences no greater than 0.01.
| True between-group difference | d | Power at α = 0.05 |
|---|---|---|
| 4 mmHg | 0.4 | 0.62 |
| 5 mmHg | 0.5 | 0.81 |
| 6 mmHg | 0.6 | 0.92 |
A decline in the true difference from just 5 mmHg to 4 mmHg reduces power from approximately eighty percent to approximately sixty percent. Maintaining power of 0.80 for 4 mmHg requires approximately 100 participants per group. This is why ICH E9 requires examining the sensitivity of sample size to departures from assumptions.
The effect of sample size on the power function of a two-sided test: each curve shows how the probability of rejecting the null hypothesis changes with the true parameter value at a different sample size. With larger samples, power rises more quickly as the parameter departs from the null hypothesis. The original figure has German labels, and its parameters do not directly correspond to this article’s example. Author: Qniemiec. Source: Wikimedia Commons. License: CC BY-SA 3.0.
Problems with Post Hoc Power and Alternative Approaches
Post hoc power (also called observed power or retrospective power) is calculated after a study ends by substituting the effect size observed in the sample into the power formula as the true effect. A common use is to interpret a nonsignificant result as follows: if the calculated power is high, “power was sufficient but no difference was found, so H₀ may be true”; if power is low, “the sample was simply too small.” In The American Statistician, Hoenig and Heisey (2001) compiled methodological recommendations endorsing this practice from journals in 19 applied disciplines, at least two of which required it as journal policy. They identified a fundamental logical flaw in the practice. Observed power is a one-to-one function of the p value and provides no new information: for a one-sided z test, observed power is exactly 0.5 when p equals α; for a two-sided test, it is slightly above 0.5. The larger p is, the lower observed power becomes, so nonsignificant results always correspond to relatively low observed power. The reasoning that “high power supports H₀” is also self-contradictory: of two studies with nonsignificant results, the one with higher observed power has a larger test statistic and a smaller p value, which by conventional standards constitutes stronger evidence against H₀. This is like using your exam score after an exam to calculate “whether this exam can measure my ability”: the calculation is entirely determined by the score and adds nothing to its interpretation.
The following table uses a two-sided z test with α = 0.05, calculating observed power = Φ(z − 1.96) + Φ(−z − 1.96), where z is the observed statistic inferred from the p value.
| Observed p value | Observed statistic z | Observed power |
|---|---|---|
| 0.05 | 1.96 | 0.50 |
| 0.20 | 1.28 | 0.25 |
| 0.50 | 0.67 | 0.10 |
Four types of tools are more useful after a study ends. First, report the effect estimate and its confidence interval: Hoenig and Heisey note that once a confidence interval is available, calculating power adds no information. Second, to demonstrate that “no difference of interest exists,” specify equivalence margins in advance and conduct an equivalence test (two one-sided tests). ICH E9 likewise states that it is inappropriate to conclude equivalence or noninferiority solely because a test of the null hypothesis of no difference is nonsignificant. Third, conduct sensitivity power analysis: fix the existing sample size, α, and desired power, and solve for the smallest effect the study can detect. Lakens (2022) gives an example in which, with 15 participants per group, α = 5%, and power of 90%, a study can detect only effects of d ≥ 1.23; this value should be compared with the smallest effect size of interest. Fourth, use the design analysis proposed by Gelman and Carlin (2014): specify a plausible true effect based on external information rather than the study’s own point estimate, and calculate power, the sign error rate (Type S, the probability that a significant estimate has the opposite direction to the true effect), and the exaggeration ratio (Type M, the average ratio of the absolute value of a significant estimate to the true effect). They also provide the R function retrodesign().
The cost of low power extends beyond missed effects. Button et al. (2013) estimated that median power in neuroscience studies was approximately between 8% and 31%. They noted that low power also reduces the likelihood that significant results reflect true effects, leads to inflated effect sizes, and makes results difficult to replicate. Power analysis should take place before a study begins, using justified effect sizes to plan the sample, rather than using the observed effect size after results are available to defend a conclusion.
References
Cohen J. A power primer. Psychological Bulletin, 1992, 112(1): 155–159. doi:10.1037/0033-2909.112.1.155
Julious SA. Tutorial in biostatistics: sample sizes for clinical trials with Normal data. Statistics in Medicine, 2004, 23(12): 1921–1986. doi:10.1002/sim.1783
ICH. E9: Statistical Principles for Clinical Trials (Step 4, 5 February 1998). ICH E9 Guideline PDF
R Core Team. power.t.test: Power calculations for one and two sample t tests (stats package documentation). R documentation
Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 2007, 39(2): 175–191. doi:10.3758/BF03193146
Hoenig JM, Heisey DM. The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician, 2001, 55(1): 19–24. doi:10.1198/000313001300339897
Lakens D. Sample size justification. Collabra: Psychology, 2022, 8(1): 33267. doi:10.1525/collabra.33267
Gelman A, Carlin J. Beyond power calculations: assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 2014, 9(6): 641–651. doi:10.1177/1745691614551642
Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 2013, 14(5): 365–376. doi:10.1038/nrn3475
Research Methods · Rituals can be imitated; understanding cannot be faked
This information was compiled and published by the Scholay One-Stop Academic Platform. Please report any infringement so the content can be removed.
This information is also published on EnterScholar, a community for scholars in the age of intelligence
Originally published at Metho.



Top comments (0)