Dropout is a training trick. At test time you turn it off and the whole network answers. MC dropout says leave it on, run k forward passes, average them, and read the spread as uncertainty.
Both of those are approximations to the same object — the mean of the dropout ensemble. So I computed that object exactly instead of approximating it: 12 dropout units means 2¹² = 4096 masks, every one enumerated.
👉 Live, the full enumeration in your browser: https://dev48v.infy.uk/dl/day79-dropout-at-inference.html
Turning dropout off does not give you the ensemble mean
With inverted dropout, a kept unit is divided by the keep probability, so the expected post-dropout activation equals the plain activation. That is exactly why "turn dropout off" is supposed to give you the ensemble mean.
It does not, and the control says why:
| network | largest gap | typical output | CONTROL: gap with a linear head |
|---|---|---|---|
| seed 7 | 0.5397 | 0.5537 | 2.75×10⁻¹⁴ |
| seed 101 | 0.4106 | 0.8880 | 2.75×10⁻¹⁴ |
| seed 2027 | 0.2956 | 0.4613 | 2.71×10⁻¹⁴ |
| seed 31337 | 0.8336 | 1.2534 | 1.19×10⁻¹³ |
| seed 90210 | 0.2457 | 0.3972 | 2.51×10⁻¹⁴ |
Make the layer after dropout linear and the expectation passes straight through: E[head(drop)] = head(E[drop]), exactly, and the gap collapses to floating-point zero in all five networks. Put the ReLU back and the gap is a third of the signal.
A bug in the enumeration would not switch itself off when the ReLU does. That is what makes this a property of the architecture rather than a mistake in my code.
The break-even is 13.9 passes
The k-sample mean is unbiased, so its RMS error is σ/√k. The deterministic pass has no variance but a fixed bias. Set them equal:
k* = σ² / bias² = 13.9
And the measured curve crosses exactly where it should:
| stochastic passes k | measured RMS error | vs dropout-off (0.2109) |
|---|---|---|
| 1 | 0.7859 | 273% further |
| 2 | 0.5671 | 169% further |
| 5 | 0.3557 | 69% further |
| 10 | 0.2492 | 18% further |
| 30 | 0.1391 | 34% closer |
| 100 | 0.0792 | 62% closer |
| 1000 | 0.0240 | 89% closer |
Across five networks the break-even lands between 5.8 and 29.8.
The point is not that 30 passes is expensive. It is that 5 and 10 — the numbers that actually appear in code — sit on the wrong side of the line. Averaging a handful of stochastic passes is a more elaborate way of being further from the answer.
The uncertainty moves. The prediction does not.
The keep probability is a training hyperparameter. Sweeping it:
| keep | RMS reported spread | deterministic output |
|---|---|---|
| 0.50 | 0.7865 | identical |
| 0.60 | 0.6568 | identical |
| 0.70 | 0.5391 | identical |
| 0.80 | 0.4222 | identical |
| 0.90 | 0.2893 | identical |
| 0.95 | 0.2021 | identical |
A 3.89× change in the reported uncertainty, and the deterministic output is bit-identical at all 61 inputs — because that pass never sees a mask at all.
Two teams who picked 0.5 and 0.9 will report different uncertainties for the same prediction from the same network, and neither is more right. The answer and the confidence attached to it come from different places: one from the weights, the other from a number chosen during training.
(This one is exact too: at keep ≠ 0.5 the masks are not equally likely, so they are weighted by binomial mass rather than sampled. Still all 4096.)
What I expected to find, and did not
Going in, my hypothesis was that this spread is just a restatement of how large the activations are — that it would track output magnitude and carry nothing else.
| network | r( spread , |output| ) | r( spread , |gap| ) |
|---|---|---|
| seed 7 | 0.7304 | −0.0750 |
| seed 101 | 0.9550 | 0.5912 |
| seed 2027 | 0.9573 | −0.3257 |
| seed 31337 | 0.7092 | 0.8325 |
| seed 90210 | −0.1885 | 0.5506 |
In four of five it holds, at 0.71 to 0.96. In the fifth it is −0.19. One counterexample in five is not a rounding error, so that claim is not established and I am not making it.
The second column is the cleaner negative. The spread does not tell you where the deterministic approximation is worst: −0.33 to +0.83 across five networks — not merely weak, but with an unstable sign. Whatever that number is measuring, it is not the size of the error it is standing next to.
What this does not cover
These networks are not trained — the weights come deterministically from a stated seed, so nothing here says what dropout does to a fitted model. One dropout layer, scalar in and out. No label noise, so the spread is purely the ensemble's own disagreement with no aleatoric term to separate it from. And none of this evaluates MC dropout as a calibration method: no labels, no reliability diagram.
25 in-page checks, 66 verifier assertions, 0 failures.
Top comments (0)