Why Running Three t-Tests Is a Bug (and How One-Way ANOVA Fixes It)

You have three groups and one question: do their averages differ?

So you do the obvious thing. You run a t-test on A vs B, another on A vs C, and a third on B vs C. One of them comes back at p = .03. You write it up, feel good, and ship it.

Congratulations, you may have just shipped a bug.

This post is about why that bug exists, how one-way ANOVA (analysis of variance) patches it, and how to run, check, and report the whole thing in practice. We'll build the test by hand, verify it in Python and R, break its assumptions on purpose to see what happens, and finish with a calculator you can use to check your own work.

TL;DR

• Many t-tests across many groups inflate your false-positive rate. ANOVA tests all groups with one test at your chosen α.
• ANOVA compares variation between group means to variation within groups. That ratio is the F-statistic.
• A significant F says "something differs". It doesn't say what. That's the job of a post-hoc test like Tukey HSD.
• Unequal variances plus unequal group sizes can quietly wreck classic ANOVA. Welch's ANOVA is the fix.
• Always report an effect size, not just p.

Part 1: The bug (a.k.a. the multiple comparisons problem)

Each t-test at α = .05 accepts a 5% chance of a false alarm when nothing is going on. One test is fine. But comparisons pile up fast:

Groups	Pairwise comparisons	Chance of ≥ 1 false positive (if tests were independent)
3	3	14.3%
4	6	26.5%
5	10	40.1%
6	15	53.7%

The formula behind that table is 1 − 0.95^m, where m is the number of comparisons. But pairwise tests share data (A appears in both A–B and A–C), so they aren't independent and the real inflation is a bit smaller. Let's measure it instead of arguing. Here are four groups drawn from the same population, so any "significant" result is by definition a false alarm:

import itertools
import numpy as np
from scipy import stats

rng = np.random.default_rng(7)
trials, pairwise_fp, anova_fp = 5000, 0, 0

for _ in range(trials):
    groups = [rng.normal(0, 1, 15) for _ in range(4)]   # no true differences
    if any(stats.ttest_ind(a, b).pvalue < .05
           for a, b in itertools.combinations(groups, 2)):
        pairwise_fp += 1
    if stats.f_oneway(*groups).pvalue < .05:
        anova_fp += 1

print(f"any pairwise t-test 'significant': {pairwise_fp / trials:.1%}")
print(f"one-way ANOVA 'significant':       {anova_fp / trials:.1%}")

With that seed I got 20.3% for the six pairwise t-tests and 4.9% for one ANOVA. A "5% error rate" test procedure that errs one time in five is not a 5% procedure.

ANOVA's pitch is simple: ask one question, "is there any difference among these means?", and answer it with one test that keeps its false-positive rate at α.

Part 2: The intuition (no formulas yet)

Picture three study groups. Each person gets an exam score. Two things make scores vary:

1. People in different groups tend to differ. Maybe practice tests really work better. That's the signal.
2. People in the same group differ from each other. Some are tired, some are lucky, some just test badly. That's the noise.

ANOVA asks whether the signal is loud compared with the noise.

Think of three tables at a party. If the tables sit at very different volumes (one is a library, one is a rock concert) while within each table everybody talks at about the same volume, the tables clearly differ. If volume varies wildly within every table, and the table averages are only slightly apart, you can't tell the tables apart from random chatter.

ANOVA turns that into a number:

F = (variation between group means) / (variation within groups)

• F ≈ 1: the group averages differ about as much as you'd expect from noise alone. No evidence of a real difference.
• F ≫ 1: the group averages are far apart relative to the noise. Something is going on.

Under the null hypothesis, F follows an F-distribution. It's right-skewed, can't be negative, and only a large F counts as evidence, so the test is effectively one-tailed. (Under the null the average F isn't exactly 1 either. It's df_within / (df_within − 2), which is about 1.29 for our example below.)

Fun fact: the F is named after Ronald Fisher, who developed ANOVA while analyzing agricultural experiments in the 1920s. The name was later given to the ratio by George Snedecor.

Part 3: Our dataset

Twelve students, three study techniques, four students each:

Flashcards (A)	Summarization (B)	Practice tests (C)
82	75	90
85	72	92
78	80	88
88	77	94
mean 83.25	mean 76.00	mean 91.00
SD 4.27	SD 3.37	SD 2.58

Grand mean (all 12 scores): 1001 / 12 = 83.4167.

⏸️ Pause and predict: will ANOVA be significant? (click to reveal)

Almost certainly. The group means are spread over 15 points (76 to 91), while the typical within-group SD is only 3 to 4 points. The signal is several times louder than the noise. Four students per group is tiny, but the effect is huge. Keep reading to see how huge.

Part 4: The hypotheses

H₀: μ₁ = μ₂ = μ₃                      (all population means are equal)
Hₐ: at least one μᵢ is different      (not all equal)

ANOVA is an omnibus test. A significant result means "not all means are equal", full stop. It doesn't say which groups differ or by how much. Hold that thought for Part 7.

Part 5: Building the F-statistic by hand

ANOVA works by partitioning the total variation into two pieces. All three pieces are "sums of squares" (SS), meaning sums of squared distances.

SS_total   = Σ (every score − grand mean)²
SS_between = Σ nⱼ × (group mean − grand mean)²      ← the signal
SS_within  = Σ (each score − its own group mean)²    ← the noise

SS_total = SS_between + SS_within

Step 1: SS between (the signal)

SS_between = 4(83.25 − 83.4167)² + 4(76.00 − 83.4167)² + 4(91.00 − 83.4167)²
           = 4(0.0278) + 4(55.0069) + 4(57.5069)
           = 450.167

Step 2: SS within (the noise)

Squared distances from each person to their own group's mean:

Flashcards:     (82−83.25)² + (85−83.25)² + (78−83.25)² + (88−83.25)² = 54.75
Summarization:  (75−76)²    + (72−76)²    + (80−76)²    + (77−76)²    = 34.00
Practice tests: (90−91)²    + (92−91)²    + (88−91)²    + (94−91)²    = 20.00

SS_within = 54.75 + 34.00 + 20.00 = 108.75

Step 3: degrees of freedom

df_between = k − 1 = 3 − 1  = 2
df_within  = N − k = 12 − 3 = 9
df_total   = N − 1 = 11

Step 4: mean squares (average squared deviation per df)

MS_between = 450.167 / 2 = 225.083
MS_within  = 108.75  / 9 =  12.083

MS_within is also just the average of the three group variances (18.25, 11.33, 6.67), a pooled estimate of the noise.

Step 5: the F-ratio

F = MS_between / MS_within = 225.083 / 12.083 = 18.63

The ANOVA table

Source	SS	df	MS	F	p
Between groups	450.167	2	225.083	18.63	.0006
Within groups	108.750	9	12.083
Total	558.917	11

Sanity check: 450.167 + 108.750 = 558.917 ✓.

Where does the p-value come from?

The critical value for F with (2, 9) degrees of freedom at α = .05 is 4.256. Our F of 18.63 is more than four times that, so we reject H₀. The exact p-value is the probability of getting an F of 18.63 or larger if H₀ were true: p ≈ .0006.

Conclusion: at least one study technique produces different mean exam scores.

Now you can see why the test is called analysis of variance even though it tests means: it compares two variances.

Fun fact: ANOVA is regression in disguise

A one-way ANOVA is exactly a linear regression with a categorical predictor (dummy-coded groups), and the F-test above is the regression's overall model test. If you already know regression, you already know most of ANOVA.

Part 6: "Significant" is not the same as "big"

The p-value says whether an effect is detectable. It says nothing about size. Report an effect size alongside it.

η² (eta squared)  = SS_between / SS_total = 450.167 / 558.917 = 0.805
ω² (omega squared) = (SS_between − df_between × MS_within) / (SS_total + MS_within)
                   = (450.167 − 2 × 12.083) / (558.917 + 12.083) = 0.746
Cohen's f          = √(η² / (1 − η²)) = 2.03

• η² is the proportion of total variance explained by group membership. Here, about 80%.
• ω² is the same idea, but corrected for the upward bias that η² has in small samples. Prefer it when n is small.
• Cohen's f is a standardized effect for ANOVA. Cohen's rough conventions are 0.10 (small), 0.25 (medium) and 0.40 (large).

Notice that our f of 2.03 is absurdly large. Real-world effects are almost never this big, and that's another reminder this is a toy dataset built to make the arithmetic readable.

Part 7: Okay, but which groups differ?

The F-test said "something's different". To find out what, we run a post-hoc test that compares pairs while controlling the family-wise error rate. The most common choice with equal-ish group sizes is Tukey's HSD (Honestly Significant Difference).

For each pair, Tukey computes a studentized range statistic:

q = (mean_i − mean_j) / √(MS_within / n)

and compares it to a critical value from the studentized range distribution, which depends on the number of groups and df_within. For k = 3 and df = 9 at α = .05, q_crit = 3.948.

A convenient shortcut: any two means further apart than the HSD threshold are significantly different.

HSD = q_crit × √(MS_within / n) = 3.948 × √(12.083 / 4) = 6.863

Comparison	Mean difference	q	Tukey p	Verdict
Practice tests − Summarization	15.00	8.63	.0005	Significant
Practice tests − Flashcards	7.75	4.46	.028	Significant
Flashcards − Summarization	7.25	4.17	.039	Significant

Every difference exceeds 6.863, so all three pairs differ. The 95% simultaneous confidence intervals for the differences: Practice − Flashcards [0.89, 14.61], Flashcards − Summarization [0.39, 14.11], Practice − Summarization [8.14, 21.86]. None of them includes 0.

Why you can't just fall back to plain t-tests

Here's something fun. Run ordinary two-sample t-tests on the same three pairs and apply a Bonferroni correction (multiply each p by 3):

Pair	Raw t-test p	×3 (Bonferroni)	Tukey p
Flashcards vs Summarization	.037	.112	.039
Flashcards vs Practice	.021	.063	.028
Summarization vs Practice	.0004	.0012	.0005

With separate t-tests plus Bonferroni, two of the three comparisons stop being significant. Tukey keeps them. Why? A separate t-test only uses the 8 observations in its two groups (6 degrees of freedom) to estimate the noise. Tukey uses the pooled MS_within from all 12 observations (9 degrees of freedom), a more stable noise estimate, which buys power. (Bonferroni-adjusted t-tests that also use the pooled MS_within land at p ≈ .049, .035 and .0005, close to Tukey.)

Moral: the post-hoc method is part of your analysis, not an afterthought. Choose it before you look at the results, and say which one you used.

Part 8: The assumptions (and what happens when they break)

Classic one-way ANOVA assumes:

1. Independence. Each observation is unrelated to the others. This is by far the most important assumption, and no test can rescue you if it's violated. Measuring the same students repeatedly? Students nested in classrooms? You need a different model (repeated-measures ANOVA, mixed models).
2. Normal residuals. The scores within each group are roughly normal. In practice, check the residuals (each score minus its group mean), pooled together.
3. Equal variances (homogeneity or homoscedasticity). The groups have similar spread.

For our data:

import numpy as np
from scipy import stats

A = np.array([82, 85, 78, 88])
B = np.array([75, 72, 80, 77])
C = np.array([90, 92, 88, 94])
data = [A, B, C]

residuals = np.concatenate([g - g.mean() for g in data])
print(stats.shapiro(residuals))   # W ≈ 0.969, p ≈ .90  → no sign of non-normality
print(stats.levene(*data))        # W ≈ 0.553, p ≈ .59  → no sign of unequal variances

Two honest caveats. First, with n = 4 per group, both tests have almost no power to catch anything, so "not significant" here isn't reassurance. Look at plots (histograms, Q-Q plots, box plots by group). Second, a significant Levene test on a huge sample can flag harmless differences. Judge the size of the variance differences, not just the p-value.

Let's break it on purpose

ANOVA is often described as "robust", which is true under certain conditions. The condition that matters most is balanced group sizes. Watch what unequal variances do when group sizes are also unequal. Here's a Welch ANOVA implementation (it doesn't assume equal variances) followed by a simulation where every population mean is exactly 0, so any rejection is a false alarm:

import numpy as np
from scipy import stats

def welch_anova(*groups):
    k = len(groups)
    n = np.array([len(g) for g in groups], float)
    m = np.array([np.mean(g) for g in groups])
    v = np.array([np.var(g, ddof=1) for g in groups])
    w = n / v
    weighted_mean = (w * m).sum() / w.sum()
    a = (w * (m - weighted_mean) ** 2).sum() / (k - 1)
    lam = ((1 - w / w.sum()) ** 2 / (n - 1)).sum()
    b = 1 + 2 * (k - 2) / (k ** 2 - 1) * lam
    F = a / b
    df2 = (k ** 2 - 1) / (3 * lam)
    return F, k - 1, df2, stats.f.sf(F, k - 1, df2)

rng = np.random.default_rng(11)
sizes, sds = [5, 15, 40], [6, 3, 1]    # smallest group is the noisiest
trials = 10_000
classic = welch = 0
for _ in range(trials):
    groups = [rng.normal(0, s, n) for n, s in zip(sizes, sds)]
    classic += stats.f_oneway(*groups).pvalue < .05
    welch   += welch_anova(*groups)[3] < .05

print(f"classic ANOVA false-positive rate: {classic / trials:.1%}")
print(f"Welch ANOVA false-positive rate:   {welch / trials:.1%}")

I got roughly 39.5% for classic ANOVA and 5.9% for Welch. A nominal 5% test was wrong about 40% of the time.

I also flipped the pairing so the smallest group had the smallest variance (SDs of 1, 3, 6 with the same sizes). Then classic ANOVA became badly conservative (about 0.35% false positives), which means it loses power to detect real effects, while Welch stayed near 4.8%. And with equal group sizes of 20 and SDs of 1, 2, 3, classic ANOVA drifted only to 6.3% (Welch: 5.2%).

The takeaways:

• Balanced designs are forgiving. Equal n protects you against moderate variance differences.
• Unequal n plus unequal variance is the danger zone, and the direction of the mismatch decides whether you get false alarms or lost power.
• Some methodologists recommend using Welch's ANOVA by default since it costs little when variances are equal. At minimum, use it whenever Levene's test or a box plot shows clearly unequal spread.

(Two-group sanity check for the code above: with two groups, welch_anova returns exactly the square of Welch's t-statistic from scipy.stats.ttest_ind(..., equal_var=False), with the same p-value.)

On our toy dataset, Welch's ANOVA gives F(2, 5.77) = 22.79, p = .0018. Same conclusion, slightly different numbers, because it doesn't pool the variances. If you go the Welch route, pair it with Games-Howell post-hoc comparisons instead of Tukey.

And if normality is the problem?

When the data are clearly non-normal (heavy skew, ordinal scales, small samples with outliers), the rank-based alternative is the Kruskal-Wallis test:

print(stats.kruskal(A, B, C))   # H ≈ 8.98, p ≈ .011

Conclusion unchanged. Note that Kruskal-Wallis tests whether the groups' distributions are shifted relative to each other, which is only a statement about means under extra assumptions. Its p-value here (.011) is larger than ANOVA's (.0006) because it throws away magnitude information and keeps only ranks. A good illustration of what that costs you when the parametric assumptions are fine.

Part 9: How many people do you actually need?

Our example had 4 students per group, a demonstration, not a design. Real studies need to ask about power, the probability of detecting an effect that really exists.

from scipy.stats import f, ncf

def anova_power(effect_f, k, n_per_group, alpha=0.05):
    N = k * n_per_group
    df1, df2 = k - 1, N - k
    crit = f.ppf(1 - alpha, df1, df2)
    return 1 - ncf.cdf(crit, df1, df2, effect_f ** 2 * N)

print(anova_power(0.25, 3, 15))   # ≈ 0.28
print(anova_power(0.25, 3, 53))   # ≈ 0.81

For three groups at α = .05 and 80% power:

True effect (Cohen's f)	Needed per group	Total N
0.40 (large)	22	66
0.25 (medium)	53	159
0.10 (small)	323	969

A medium effect with 15 people per group gives you only about a 28% chance of finding it. Underpowered studies don't just miss effects. When they do hit significance, they tend to overestimate the effect size. Decide the sample size before collecting data.

Part 10: Code, from scratch and from libraries

Python: from scratch (matches the tables above)

import numpy as np
from scipy import stats

data = [np.array([82, 85, 78, 88]),
        np.array([75, 72, 80, 77]),
        np.array([90, 92, 88, 94])]

k = len(data)
N = sum(len(g) for g in data)
all_scores = np.concatenate(data)
grand_mean = all_scores.mean()

ss_between = sum(len(g) * (g.mean() - grand_mean) ** 2 for g in data)
ss_within  = sum(((g - g.mean()) ** 2).sum() for g in data)
ss_total   = ((all_scores - grand_mean) ** 2).sum()

df_between, df_within = k - 1, N - k
ms_between = ss_between / df_between
ms_within  = ss_within / df_within
F = ms_between / ms_within
p = stats.f.sf(F, df_between, df_within)

print(f"SS_between={ss_between:.3f}  SS_within={ss_within:.3f}  SS_total={ss_total:.3f}")
print(f"F({df_between}, {df_within}) = {F:.3f}, p = {p:.5f}")
# SS_between=450.167  SS_within=108.750  SS_total=558.917
# F(2, 9) = 18.628, p = 0.00063

Python: library one-liners

print(stats.f_oneway(*data))         # F = 18.628, p = 0.000632
print(stats.tukey_hsd(*data))        # pairwise comparisons with p-values and 95% CIs (SciPy 1.8+)

(If you prefer a DataFrame workflow, statsmodels offers ols(...) with anova_lm, plus pairwise_tukeyhsd, and pingouin has anova, welch_anova and pairwise_gameshowell.)

R

scores <- c(82,85,78,88, 75,72,80,77, 90,92,88,94)
group  <- factor(rep(c("Flashcards", "Summarization", "Practice"), each = 4))

fit <- aov(scores ~ group)
summary(fit)                                   # classic ANOVA table
TukeyHSD(fit)                                  # post-hoc
oneway.test(scores ~ group, var.equal = FALSE) # Welch's ANOVA
kruskal.test(scores ~ group)                   # rank-based alternative
# car::leveneTest(scores ~ group)              # equal-variance check

Excel

1. Enable the Analysis ToolPak (File > Options > Add-ins).
2. Put each group in its own column.
3. Data > Data Analysis > Anova: Single Factor.

Excel gives you the F-test and p-value, but no post-hoc test and no Welch option, so for follow-ups you'll need other software.

SPSS

Analyze > Compare Means > One-Way ANOVA. Put the outcome in Dependent List and the grouping variable in Factor, click Post Hoc and tick Tukey (or Games-Howell if variances differ), and under Options tick Homogeneity of variance test and Welch.

Part 11: Choose your path

1. Are the observations independent? No → stop. You need repeated-measures or mixed models.
2. Are group variances reasonably similar, and are group sizes close? Yes → classic ANOVA is fine. No → Welch's ANOVA.
3. Is the outcome clearly non-normal, ordinal, or full of outliers? Yes, and the sample is small → Kruskal-Wallis, or transform the data.
4. Is F significant? No → don't go fishing with pairwise tests. Report the effect size and a confidence interval, and note that the study may be underpowered. Yes → continue.
5. Which post-hoc? Tukey (classic ANOVA, similar group sizes), Games-Howell (Welch), Bonferroni or Holm (a small set of planned comparisons).
6. Report F, df, p, an effect size, descriptives, and the post-hoc results.

Part 12: The hall of shame (mistakes to avoid)

1. Running multiple t-tests instead of ANOVA. See Part 1.
2. Treating a significant F as "all groups differ". It means at least one pair does.
3. Treating a non-significant F as "all groups are equal". It means you couldn't detect a difference with this sample. Absence of evidence isn't evidence of absence, especially at low power.
4. Skipping the effect size. A tiny difference can be "significant" in a huge sample.
5. Checking normality on the raw pooled data instead of within groups or on residuals. Pooling groups with different means makes even normal data look non-normal.
6. Ignoring independence. The same person measured three times isn't three independent observations.
7. Using one-way ANOVA for repeated measures. Use repeated-measures ANOVA or a mixed model.
8. Choosing the post-hoc test after seeing which one gives you stars. That's p-hacking with extra steps.
9. Fishing in a non-significant ANOVA for "interesting" pairs. If the omnibus test fails, pairwise results are exploratory at best.
10. Not reporting which assumptions you checked and how. Reviewers (and future you) will want to know.

Part 13: Reporting it (APA 7 style)

A one-way ANOVA showed a significant effect of study technique on exam score, F(2, 9) = 18.63, p < .001, η² = .81 (ω² = .75). Tukey HSD post-hoc comparisons indicated that practice tests (M = 91.00, SD = 2.58) produced higher scores than both flashcards (M = 83.25, SD = 4.27, p = .028) and summarization (M = 76.00, SD = 3.37, p < .001), and flashcards outscored summarization (p = .039).

APA style gives p to three decimals with no leading zero, and uses p < .001 for anything smaller. If you used Welch's ANOVA, report it as Welch's F(df₁, df₂) with the adjusted (fractional) denominator degrees of freedom, e.g. F(2, 5.77) = 22.79.

Bonus: what if you only have two groups?

ANOVA with two groups is just a t-test in a trench coat: F = t², with the same p-value. Check it on groups A and C:

r = stats.ttest_ind(A, C)
print(r.statistic ** 2, stats.f_oneway(A, C).statistic)
# both ≈ 9.642, and both p-values ≈ .0210

Skip the arithmetic: a free calculator

When I just want a quick check (or want to see the sums of squares laid out), I use my own free One-Way ANOVA Calculator. It accepts raw data or summary statistics (n, mean and SD per group), shows the full ANOVA table with SS, df and MS, the F-statistic and p-value, a shaded F-distribution curve, Tukey HSD pairwise comparisons, an APA-style write-up, and the step-by-step working. I maintain it, so treat that as full disclosure.

A good test for any ANOVA tool, mine included: enter the three groups from this post. You should see F(2, 9) ≈ 18.63, p ≈ .0006, with SS between ≈ 450.17 and SS within = 108.75, and Tukey flagging all three pairs. If you get something different, the usual suspects are a typo in the data and a mismatch between classic and Welch settings.

More from this series: PASTE_CI_POST_URL_HERE · PASTE_CRONBACH_POST_URL_HERE · PASTE_IQR_POST_URL_HERE

FAQ

Can I use ANOVA with unequal group sizes?
Yes. It's the unequal sizes combined with unequal variances that cause trouble (Part 8). Welch's ANOVA handles both.

What's the difference between one-way and two-way ANOVA?
One-way has a single categorical factor (study technique). Two-way has two (study technique and, say, class year) and can test their interaction, whether the effect of one depends on the level of the other.

Is a significant ANOVA enough, or do I need a post-hoc test?
If you have three or more groups and a significant F, you almost always want a post-hoc test or planned contrasts to say which groups differ.

How many groups can I compare?
As many as you like, but the number of pairwise comparisons grows quickly, and post-hoc tests lose power as it does. Think about what comparisons actually matter before you collect data.

My Levene test is significant. What now?
Check how different the variances (or SDs) actually are and whether the group sizes are balanced. If it's more than a trivial difference, use Welch's ANOVA with Games-Howell.

What if my data aren't normal?
With reasonable sample sizes and balanced groups, ANOVA is fairly tolerant of mild non-normality. With small samples, heavy skew or ordinal data, consider Kruskal-Wallis or a transformation.

Why η² and ω² when I already have p?
p tells you whether an effect is detectable. Effect sizes tell you whether it matters.

What's the null hypothesis, in plain English?
All the populations have the same mean. The test asks whether your sample means are more spread out than chance alone would explain.

Your turn

Take the dataset above and change one thing: make Group B's scores [75, 72, 80, 77] → [84, 80, 87, 85]. Before running anything, predict what happens to F. Does it go up or down? Is Tukey's B–A comparison still significant? Run it in the calculator or in Python, then tell me in the comments whether you called it.

Bonus round: make one group's scores wildly noisier and see how both the ANOVA and Levene's test respond.