Missing data comes with three names. MCAR: the holes depend on nothing. MAR: the holes depend on something you recorded. MNAR: the holes depend on the value that is missing. The advice is to test which one you have, and the standard test compares the variables you do observe across the missingness pattern.
Here the whole mechanism space is two numbers - beta for how hard missingness leans on x, delta for how far the missing y sit above the recorded ones at the same x - and the routine that produces the observed dataset is not given delta:
sampleBase(n, cfg, beta, seed) // -> { x, R, yObs, yMisBase } NO delta argument
completeY(base, delta) // differs from base ONLY on the missing cells
Two worlds sharing (beta, seed) are handed the same cached base object, so "the observed data is identical" is not something checked afterwards and hoped for - it is the only thing that could have happened. Dependency-free JavaScript, 400 rows, 25 seeds, 250 coverage replications: https://dev48.infy.uk/ml/day68-missing-data.html
One interval, two coverages
Nominal 95%. The MAR truth is 1.000 and the MNAR truth is 1.600, and the intervals are computed once and scored against both.
| method | mean width | MCAR | MAR (world A) | MNAR (world B) |
|---|---|---|---|---|
| complete-case | 0.499 | 96.8% | 0.0% | 32.8% |
| mean imputation | 0.298 | 77.6% | 0.0% | 10.8% |
| multiple imputation (m = 10) | 0.492 | 96.4% | 93.6% | 0.0% |
The method that exists to solve this covers 93.6% of the time in one world and 0.0% in the other, at an identical width. Nothing in the data chose between those two columns.
The diagnostic agrees with itself just as hard. One twin pair screams identically - |t| = 13.59, MCAR rejected 100% of the time - and the other is identically silent at |t| = 0.80 and 4.0%, its noise floor. Comparing observed variables across the missingness pattern is a test of beta and only of beta. A quiet monitor is evidence about beta. It is not evidence about delta.
The ranking is decided by something unobservable
Under the MNAR twin, complete-case lands 0.291 from the truth and multiple imputation lands 0.630. The method everyone is warned off beats the recommended one by better than two to one - because selection on x pushes the complete-case mean up by 0.891 while delta(1 - r) pushes the truth up by 0.600, and the two errors partly cancel. Flip the sign of delta and they add instead. Which estimator wins is settled by a quantity no dataset contains.
What the measurement took away
I wrote meanImpute.mean === completeCase.mean as bit-equality, because it is an exact algebraic identity: the imputed cells sit exactly on the mean, so they cannot move it. It failed. Complete-case adds k recorded values and divides by k; mean imputation adds those k plus (n - k) copies of their average and divides by n. Same number on paper, two summation orders in binary64, 1.6e-15 apart. Fifteen significant figures of agreement and not sixteen. The assertion now runs at 1e-12 and says why, because a === you had to relax is worth keeping visible rather than quietly loosening.
Mean imputation's interval failure is arithmetic, and it is predicted before it is measured: the reported standard error is too small by exactly the response rate, so coverage should land on 2*Phi(1.96r) - 1 = 76.0%, and it measures 77.6%.
406 verifier assertions, 15 of them running in the page. Part of a from-scratch series - one ML idea a day, computed rather than quoted: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)