You have three or more groups and one number measured on each: exam scores under three study methods, page load times across three CDNs, conversion rates across four landing pages. The natural question is "do the group means differ?"
With two groups you'd use a t-test. With three or more, the standard tool is one-way ANOVA (analysis of variance). This post walks through why it exists, how it works, a worked example you can verify by hand, and how to reproduce it in code.
Why not just run a bunch of t-tests?
Because the false-positive rate adds up. Each t-test at α = 0.05 has a 5% chance of a false alarm when there is no real difference. Run several and the chance that at least one is a false alarm climbs fast.
I simulated this: 5 groups of 10 observations, all drawn from the same normal distribution, so there is nothing to find. Over 10,000 repetitions:
| Approach | Runs with a "significant" result |
|---|---|
| All 10 pairwise t-tests (any p < .05) | 28.6% |
| One-way ANOVA (p < .05) | 4.9% |
The pairwise approach gives you a false finding in more than a quarter of experiments. ANOVA asks one question ("is there any difference at all?") and keeps the error rate near the 5% you chose.
The idea: split the variance into two parts
ANOVA compares two kinds of variation:
- Between-group variation: how far the group means are from the overall (grand) mean. If the groups really differ, this is large.
- Within-group variation: how spread out the individual values are inside each group. This is the natural noise.
If the group means are all equal in the population, then both of these estimate the same noise level, and their ratio should be near 1. If the means differ, the between-group part gets inflated and the ratio gets bigger.
Hypotheses
H0: μ1 = μ2 = ... = μk (all population means are equal)
Ha: at least one mean differs
Note that ANOVA is an omnibus test. A significant result says "something differs", not which groups differ. That's what post-hoc tests are for.
The formulas
Let k be the number of groups, N the total number of observations, n_j the size of group j, x̄_j the group mean, and x̄ the grand mean.
SS_between = Σ n_j × (x̄_j − x̄)²
SS_within = Σ Σ (x_ij − x̄_j)²
SS_total = SS_between + SS_within
df_between = k − 1
df_within = N − k
MS_between = SS_between / df_between
MS_within = SS_within / df_within
F = MS_between / MS_within
Under the null hypothesis, F follows an F-distribution with (k − 1, N − k) degrees of freedom. A large F produces a small p-value.
Worked example
Twelve students are randomly assigned to one of three study methods, four per group. Here are their exam scores:
| Flashcards | Summarizing | Practice tests |
|---|---|---|
| 82 | 75 | 90 |
| 85 | 72 | 92 |
| 78 | 80 | 88 |
| 88 | 77 | 94 |
(Four per group is far too small for real research. It's here so you can check every number by hand.)
Step 1: group sums and means
Flashcards: sum = 333, mean = 83.25
Summarizing: sum = 304, mean = 76.00
Practice: sum = 364, mean = 91.00
Grand sum = 1001, N = 12, grand mean = 83.4167
Step 2: between-group sum of squares
SS_between = 4(83.25 − 83.4167)² + 4(76.00 − 83.4167)² + 4(91.00 − 83.4167)²
= 0.111 + 220.028 + 230.028
= 450.167
Step 3: within-group sum of squares
Flashcards: (−1.25)² + (1.75)² + (−5.25)² + (4.75)² = 54.75
Summarizing: (−1)² + (−4)² + (4)² + (1)² = 34.00
Practice: (−1)² + (1)² + (−3)² + (3)² = 20.00
SS_within = 54.75 + 34.00 + 20.00 = 108.75
Check: SS_total = 450.167 + 108.75 = 558.917, which matches summing squared deviations from the grand mean directly.
Step 4: degrees of freedom and mean squares
df_between = 3 − 1 = 2
df_within = 12 − 3 = 9
MS_between = 450.167 / 2 = 225.083
MS_within = 108.75 / 9 = 12.083
Step 5: the F statistic
F = 225.083 / 12.083 = 18.63
Step 6: significance
With df = (2, 9), the critical F at α = .05 is 4.26. Our F of 18.63 is far beyond it, and the p-value is about .0006. We reject the null: at least one study method has a different mean.
Here is the summary table, in the layout most software prints:
| Source | SS | df | MS | F | p |
|---|---|---|---|---|---|
| Between groups | 450.17 | 2 | 225.08 | 18.63 | .0006 |
| Within groups | 108.75 | 9 | 12.08 | ||
| Total | 558.92 | 11 |
How big is the effect?
A p-value tells you whether there is an effect, not how large it is. The usual effect size for one-way ANOVA is eta squared:
η² = SS_between / SS_total = 450.17 / 558.92 ≈ 0.81
About 81% of the variance in scores is associated with the study method. Eta squared is biased upward in small samples, so many people prefer omega squared, which for this data is about 0.75. Either way, the effect is large (though remember, n = 4 per group).
Which groups differ? Tukey HSD
Now the follow-up. Tukey's Honestly Significant Difference test compares every pair of groups while keeping the overall false-positive rate at 5%.
For a balanced design, two means differ significantly if their gap exceeds:
HSD = q_crit × √(MS_within / n)
For k = 3 groups and df_within = 9, q_crit ≈ 3.948, so:
HSD = 3.948 × √(12.083 / 4) ≈ 6.86
Any pair of means more than 6.86 points apart is significant:
| Comparison | Mean difference | Adjusted p | Significant? |
|---|---|---|---|
| Practice − Flashcards | 7.75 | .028 | Yes |
| Flashcards − Summarizing | 7.25 | .039 | Yes |
| Practice − Summarizing | 15.00 | .0005 | Yes |
So all three methods differ from one another, with practice tests ahead of both and summarizing at the bottom.
Check the assumptions
One-way ANOVA assumes:
- Independence. Observations don't influence each other. This comes from how you collected the data, not from a test.
- Approximate normality within each group (more precisely, of the residuals).
- Equal variances across groups (homogeneity of variance).
For the example data:
- Levene's test for equal variances: p = .59, no evidence of unequal variances.
- Shapiro-Wilk on the residuals: p = .90, no evidence of non-normality.
(With only 12 data points, these tests have very little power, so "no evidence of a problem" is weaker than it sounds. Look at plots too.)
ANOVA is reasonably robust to mild departures, especially when group sizes are equal. If the variances are clearly unequal, switch to:
-
Welch's ANOVA, which doesn't assume equal variances. On our data it gives
F(2, 5.77) = 22.79, p = .0018, the same conclusion. -
Kruskal-Wallis, a rank-based test for when normality is doubtful. Here
H = 8.98, p = .011. It is less powerful than ANOVA when ANOVA's assumptions hold, which is why its p-value is larger.
Code
Python with SciPy
from scipy import stats
flashcards = [82, 85, 78, 88]
summarizing = [75, 72, 80, 77]
practice = [90, 92, 88, 94]
# The omnibus test
F, p = stats.f_oneway(flashcards, summarizing, practice)
print(f"F = {F:.2f}, p = {p:.4f}") # F = 18.63, p = 0.0006
# Post-hoc
print(stats.tukey_hsd(flashcards, summarizing, practice))
# Assumption checks
print(stats.levene(flashcards, summarizing, practice)) # p ≈ 0.59
stats.tukey_hsd needs SciPy 1.8 or later.
Python with statsmodels (full ANOVA table + Tukey)
import pandas as pd
import statsmodels.api as sm
import statsmodels.formula.api as smf
from statsmodels.stats.multicomp import pairwise_tukeyhsd
df = pd.DataFrame({
"score": flashcards + summarizing + practice,
"method": ["flashcards"] * 4 + ["summarizing"] * 4 + ["practice"] * 4,
})
model = smf.ols("score ~ C(method)", data=df).fit()
print(sm.stats.anova_lm(model, typ=2))
print(pairwise_tukeyhsd(df["score"], df["method"]))
Python with pingouin (effect size and Welch in one line each)
import pingouin as pg
print(pg.anova(df, dv="score", between="method", detailed=True)) # includes np2 (η²)
print(pg.welch_anova(df, dv="score", between="method"))
print(pg.pairwise_tukey(df, dv="score", between="method"))
R
scores <- c(82, 85, 78, 88, 75, 72, 80, 77, 90, 92, 88, 94)
method <- factor(rep(c("flashcards", "summarizing", "practice"), each = 4))
fit <- aov(scores ~ method)
summary(fit)
TukeyHSD(fit)
oneway.test(scores ~ method) # Welch's ANOVA (var.equal = FALSE is the default)
kruskal.test(scores ~ method)
Excel
Enable the Analysis ToolPak (File > Options > Add-ins), then Data > Data Analysis > Anova: Single Factor. It returns the ANOVA table with F and p. It does not run Tukey, so for post-hoc comparisons you need another tool.
SPSS
Analyze > Compare Means > One-Way ANOVA, put the outcome in Dependent List and the grouping variable in Factor, then click Post Hoc and tick Tukey.
A neat connection: two groups
ANOVA on two groups is just a t-test in disguise: F = t² and the p-values are identical. Using the flashcards and summarizing groups from above, the pooled two-sample t-test gives t = 2.67, so t² = 7.11, and f_oneway on the same two groups gives F = 7.11 with p = .037. Same answer, two routes.
Try it without writing code
If you want to paste in numbers and get the ANOVA table without setting up a script, this free tool does it:
One-Way ANOVA Calculator: One Way Analysis of Variance Calculator with Steps
Free online One-Way ANOVA calculator. Supports raw and summary statistics, renders step-by-step LaTeX derivations, visualizes F-distribution curves, and computes Tukey HSD comparisons.
takemystatisticsclassforme.us
What it offers:
- Raw data input, or summary statistics (n, mean, standard deviation per group) if you only have what a textbook problem gives you
- A significance level you can set (default 0.05)
- Group descriptive statistics, the full ANOVA summary table, and the F statistic, p-value and critical F
- Tukey HSD pairwise comparisons
- A shaded F-distribution curve, step-by-step math, and an APA-style write-up you can adapt
Paste the three groups from the worked example. You should see F(2, 9) = 18.63, p ≈ .0006, and the three Tukey comparisons above. If you get something different, double-check that each group went in as its own set of values.
Common mistakes
- Running many t-tests instead of ANOVA. See the simulation at the top.
- Stopping at "p < .05". Report an effect size and follow up with a post-hoc test to say which groups differ.
- Skipping post-hoc tests, or using one when the omnibus test isn't significant. Post-hocs are for following up a significant F.
- Ignoring unequal variances and unequal group sizes together. That combination is where standard ANOVA is least reliable. Use Welch's ANOVA.
- Treating repeated measures as independent groups. If the same people appear in every condition, you need repeated-measures ANOVA or a mixed model.
- Reading too much into non-significance. A large p-value with tiny groups usually means low power, not "no effect".
- Tiny samples. Four per group is for textbooks. Plan for enough observations per group to have reasonable power.
- Comparing more than one factor with one-way ANOVA. Two grouping variables call for a two-way ANOVA (or a regression with interaction terms).
How to report it
A concise APA-style write-up of the example:
A one-way ANOVA showed a significant effect of study method on exam score, F(2, 9) = 18.63, p < .001, η² = .81. Tukey HSD comparisons indicated that practice tests (M = 91.00, SD = 2.58) outperformed both flashcards (M = 83.25, SD = 4.27) and summarizing (M = 76.00, SD = 3.37), and flashcards outperformed summarizing (all p < .05).
FAQ
Is ANOVA a test of variances or of means?
Of means. It's called "analysis of variance" because it tests the means by comparing two variance estimates.
What does a large F mean?
The between-group variation is large compared to the within-group noise. Under the null, F hovers around 1.
My ANOVA is significant. Do all the groups differ?
Not necessarily. At least one differs. Use Tukey HSD (or another post-hoc method) to see which pairs.
Do I need equal group sizes?
No, but equal sizes make ANOVA more robust to unequal variances. Tukey's procedure also has a variant for unbalanced groups (Tukey-Kramer).
What if my data are ordinal or clearly non-normal?
Consider the Kruskal-Wallis test.
TL;DR
- Many t-tests inflate false positives. ANOVA tests "any difference" in one go.
-
F = MS_between / MS_within; a big F means the group means are spread further than noise explains. - A significant F is omnibus. Follow up with Tukey HSD and report an effect size (η² or ω²).
- Check independence, normality and equal variances. Use Welch's ANOVA or Kruskal-Wallis when assumptions fail.
- For two groups,
F = t².
If you try the calculator or run the code on your own data, I'd like to hear what you get. Questions about post-hoc choices, unequal variances or effect sizes are welcome in the comments.
Top comments (0)