Two models, the same n test items, a 2×2 table of who got what right. Write p₁₀ for the rows A wins and p₀₁ for the rows B wins, and accA − accB = p₁₀ − p₀₁: the two cells where the models agree cancel. They carry no information about the difference whatsoever.
Condition on the d rows where they disagreed and the null is Binomial(d, ½) exactly — no asymptotics, no nuisance parameters, no p₁₁ anywhere in it. That is McNemar's exact test, and it turns power into a finite sum rather than a simulation:
function powerCond(n, p10, p01, alpha){ // condition on d, then average over it
const delta = p10 + p01;
const w = binomWindow(n, delta);
return powerFromTable(n, delta, rejTable(w.lo + w.v.length - 1, p10 / delta, alpha));
}
Days 66 to 71 were an identification arc — given this file, what can be known at all. This one deliberately steps out, because the question here is not identification but design: how much data to collect, which has an exact answer rather than a bound. https://dev48.infy.uk/ml/day72-model-comparison-power.html
How good the models are does not enter
| world (φ splits the concordant mass) | accuracy A | accuracy B | exact power at n = 600 |
|---|---|---|---|
| 0.99 | 97.6% | 94.6% | 0.830230251229 |
| 0.90 | 89.1% | 86.1% | 0.830230251229 |
| 0.75 | 75.0% | 72.0% | 0.830230251229 |
| 0.55 | 56.2% | 53.2% | 0.830230251229 |
| 0.28 | 30.8% | 27.8% | 0.830230251229 |
That figure is computed by two independent routes — conditional averaging and a direct trinomial sum built from log-gammas — agreeing to 3.0e-14. The sampler cannot separate the worlds either: 60,000 row comparisons, 0 differences, while accuracy itself spans 0.7117.
It is worth having exactly. Two thousand replications read 0.8415, which is 1.3σ away and would have shipped.
Two more that cost data. Ask for 80% power and the textbook formula hands you n = 521; 521 rows deliver 76.43%, and the exact answer is 562. And the fold-level paired t-test, which is how most people actually compare two models, is exactly √c times a real t variate with c = (1 + (k−1)ρ)/(1 − ρ):
| k folds, ρ = 0.5 | nominal | exact type-I error |
|---|---|---|
| 5 | 5% | 32.03% |
| 10 | 5% | 51.24% |
| 20 | 5% | 65.30% |
More folds is worse, which is the opposite of the folklore.
What the measurement contradicted
I built the page expecting a discreteness sawtooth. The conditional level ladder is violently discrete — it dips 215 times as d walks from 1 to 400, because the attainable two-sided levels jump from 0.0078 at d = 8 to 0.0490 at d = 17 — so the power curve in n ought to wobble with it.
It does not. Exact power dips 0 times over 2,999 consecutive sample sizes, and monotonically. The reason is that d is itself random: power at sample size n is a Binomial(n, δ) average over the whole ladder, and averaging a jagged function against a smoothly shifting weight erases the teeth. The negative result is printed on the page rather than quietly dropped, which is the only reason I trust the rest of it.
2,408 assertions in a verifier that extracts the engine by regex and runs it under Node. Self-contained: one file, inline CSS, no external asset of any kind.
Part of a from-scratch series — one idea a day, dependency-free engine: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)