Seventy days of this series have asked what a column was hiding. This one asks why the row is in the file at all — a claim was filed, a test ordered, a case reviewed. If whatever put the row there also depends on the outcome you are predicting, your sample is not a shrunken copy of the population.
Write s₁ = P(recorded | y=1) and s₀ = P(recorded | y=0). Condition on being recorded and the cell mass cancels, so the odds simply multiply:
q / (1-q) = kappa * eta / (1-eta) kappa = s1 / s0
logit q = logit eta + log kappa -- one constant, added in every cell
Every world, live: https://dev48.infy.uk/ml/day71-outcome-dependent-sampling.html
The move that worked twice does not work here
Day 69 got the Frisch bound and Day 70 got the label-noise rectangle out of one cheap fact: invert the observed law, insist the recovered quantity is a legal probability, and the inequality writes itself. Run it here and it produces nothing. The inverse is η = expit(logit q − log κ), and expit maps the entire real line into (0,1). A sweep of 20,001 worlds across twelve orders of magnitude rejects 0, and an independent 30,001-point sweep rejects 0 as well.
That is a sharper failure than the two before it. No property of the data buys you anything — not more rows, not more features, not better separation. What bounds it is a fact about the design: s₁ ≤ 1 and s₀ ≤ 1 put the prevalence in [fπ′, 1 − f(1−π′)], width exactly 1 − f, one minus the sampling fraction. The recorded base rate cancels out of the width to 0e+0. It is not the sample size that identifies anything, it is the fraction.
The report splits cleanly in half
| quantity | on the sample | in truth |
|---|---|---|
| sensitivity, specificity, LR+, LR−, balanced accuracy, AUC | — | identical to 1.3e-15 |
| precision | 0.6896 | 0.1000 |
| prevalence | 0.5000 | 0.0476 |
| F₁ | 0.7662 | 0.1792 |
| fitted slope β₁ | 2.2000 | 2.2000 |
| fitted intercept β₀ | 0.0000 | −2.9957, exactly −log κ |
Within a class the mix of features is untouched, so everything built only out of p(x|y) is exactly right. Everything that mixes the two classes is wrong by a factor you cannot estimate. The slope comes out at 2.2000 by three routes and the whole distortion lands on the intercept — which is Prentice and Pyke, and the reason case-control studies work at all.
What the measurement contradicted
A 95% Wald interval for precision covers the truth 94.950% of the time. That number is exact: conditional on the denominator the count of true positives is binomial, so the coverage is a double binomial sum, and I computed it two ways that agree to 1e-12.
Having the exact column is what saved it. The simulation read 96.0% at 1,000 replications, which looks like a result about Wald intervals; 95.75% at 2,000, 95.88% at 4,000. Against 94.950% with a standard error of 0.35 points that is two and a half sigma, so it got 100,000 replications on three seed streams — and it was Monte Carlo noise wearing a suit. Without the exact number I would have shipped it.
One consequence worth carrying away: 9,930 of 32,640 model pairs swap order under accuracy on a selected sample, and 0 swap under balanced accuracy, structurally.
One file, vanilla JS, inline CSS, no external asset of any kind.
Part of a from-scratch series — one idea a day, dependency-free engine: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)