DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

The Bootstrap Covers the Mean 94.5% of the Time and the Maximum 0%. I Measured Both

#datascience #statistics #python #machinelearning

Efron's bootstrap is the best value-for-effort idea in applied statistics. Resample your own data with replacement, recompute the statistic, read the confidence interval off the spread. Four lines, works for statistics nobody has derived a variance for.

It is also routinely sold as assumption-free, which it is not — and the honest way to show that is not a warning paragraph. It is measured coverage.

Everything below runs in your browser: https://dev48.infy.uk/ml/day65-bootstrap.html

The only test that matters 

let covered = 0;
for (let t = 0; t < TRIALS; t++){
  const sample = drawFrom(population, n);      // truth is KNOWN
  const ci = bootstrapCI(sample, statistic, B);
  if (ci.lo <= truth && truth <= ci.hi) covered++;
}
coverage = covered / TRIALS;                   // should be 0.95
Enter fullscreen mode Exit fullscreen mode

Everything else is machinery. This loop is the argument.

statistic population n measured coverage
mean normal 100 94.5%
variance lognormal 10 60.6%
maximum uniform 40 0.0%

A nominal 95% interval that covers 60.6% of the time is not a 95% interval.

The failure that is not fixable

The maximum is not a small-sample wobble. A resample is drawn from the observed values, so:

assert(Math.max(...resample) <= Math.max(...sample));   // ALWAYS
assert(Math.max(...boot) === Math.max(...sample));      // therefore
Enter fullscreen mode Exit fullscreen mode

The bootstrap distribution of the max is pinned at the sample max from above, while the truth is above it. Verified over 200 datasets: 0 resamples ever exceeded the sample maximum, and 63.8% sit exactly on that ceiling — against a theoretical 1 − (1 − 1/n)ⁿ → 1 − 1/e = 63.2%.

Every miss has the truth above the interval; none below. That direction is forced by the structure, and no B fixes it. BCa cannot rescue it either — no reweighting of a bounded distribution reaches past its bound.

The general lesson is worth more than the special case: the bootstrap works when the statistic is a smooth function of the data. Extremes are not smooth. One observation determines the answer.

Three intervals, and they are not variations of each other

Percentile is the obvious one. Basic reflects around the estimate — and the ends swap:

percentile = [lo, hi];
basic      = [2 * theta - hi, 2 * theta - lo];   // note the order
Enter fullscreen mode Exit fullscreen mode

The swap is the idea. If the bootstrap distribution leans right, the estimator probably does too, so the correction leans left. Copy the ends across without swapping and the interval sometimes inverts.

BCa shifts the quantiles using a bias term measured from the bootstrap distribution and an acceleration term from a real jackknife:

const z0 = probit(fraction(boot, b => b < theta));
const jack = sample.map((_, i) => statistic(without(sample, i)));
const a = sum(d³) / (6 * sum(d²)^1.5);
Enter fullscreen mode Exit fullscreen mode

Both are measured from the data, so on a symmetric problem both go to ~0 and BCa collapses to the percentile interval — which is the identity the tests check first. A BCa that differs from percentile on a symmetric problem has a bug in one of its two corrections.

B is not the accuracy dial

Two errors get conflated constantly:

  • Monte Carlo error — you only drew B resamples. Fixed by raising B.
  • Bootstrap approximation error — your sample is not the population. Fixed only by more data.

Measured, on the same problem:

  • B from 200 → 2,000 moves coverage by 1.3 points
  • n from 20 → 200 moves it by 18 points

B ≈ 2,000 is enough. Spend the rest on a bigger sample.

The assertion worth stealing 

assert(coverage(mean, normal, n=100) > 0.93);   // it works where it works
assert(coverage(max,  anything)      < 0.60);   // and fails where it fails
Enter fullscreen mode Exit fullscreen mode

Asserting that a method fails where it should fail is how you find out your implementation is doing the thing you think it is, rather than something that merely produces plausible numbers. If my max coverage had come back at 91%, the bug would have been in my code and I would never have known.

When to reach for it

Smooth statistics — means, ratios, correlations, model metrics — yes, and BCa if the distribution looks skewed. Extremes, minima, maxima, anything determined by one observation — no. Dependent data — not the ordinary bootstrap; resampling rows destroys the dependence structure. Tiny samples — check coverage before you trust it.

Part of a from-scratch series — one ML idea a day, computed rather than quoted: https://dev48.infy.uk/machinelearningfromzero.php

Top comments (0)