Sixty-nine days of this series have taken the target column at face value. It is a recording too — a fraud flag from a rules engine with both kinds of error, a diagnosis code entered under time pressure, "churned" defined by a thirty-day window that some customers straddle.
Two flip rates make the observed law affine in the truth, and the identified set falls straight out of "η has to be a probability":
q = rho0 + delta*eta, delta = 1 - rho0 - rho1
eta >= 0 in every cell <=> rho0 <= min q
eta <= 1 in every cell <=> rho1 <= 1 - max q
Derived rather than quoted — the label-noise analogue of Day 69's Frisch bound, a probability that cannot leave [0,1] in place of a variance that cannot go negative. One dataset then supports a Bayes accuracy of 72.86% or 95.71%, and a true prevalence of 0.333 or 0.667 from a recorded base rate of exactly 0.500. Every world, live: https://dev48.infy.uk/ml/day70-label-noise.html
The ceiling that refuses to move
Same cells, same masses, same ordering. Only how far the observed scores reach changes.
| score range |
A0, the best anyone can measure |
A_max, the largest truth admissible |
set width |
|---|---|---|---|
| 0.20 | 59.14% | 0.957142857142857 | 0.3657 |
| 0.35 | 66.00% | 0.957142857142857 | 0.2971 |
| 0.50 | 72.86% | 0.957142857142857 | 0.2286 |
| 0.70 | 82.00% | 0.957142857142857 | 0.1371 |
| 0.90 | 91.14% | 0.957142857142857 | 0.0457 |
| 0.98 | 94.80% | 0.957142857142857 | 0.0091 |
Identical to 2.2e-16 across all six, and the algebra says it has to be: A0 = qLo + range*G while 1 - 2*qLo = range exactly, so the range cancels and A_max = G. Better separation does not raise the ceiling on what could be true; it lowers the ceiling on how wrong you could be. I built that table expecting two columns to move together and got one that refused to move at all, which is a better result than the one I went looking for.
The expensive failure is not the one I priced
I had the threshold shift down as the damage. At the default it costs 0.0071 of accuracy, because the mass sits at the ends and there is nothing between the two cuts to move. The ranking inversion costs 0.3226: the worst pair wins the leaderboard by 0.0018 while being 0.3226 worse in truth, and it buys that gap purely by predicting positive more often, which is not a measure of quality at all.
Symmetric noise inverts exactly 0 of 32,640 pairs — structurally, because accuracy under noise is then a strictly increasing function of clean accuracy. A sampled zero would have proved nothing, so the page enumerates all 256 classifiers. Asymmetric noise inverts 3,824.
What the measurement contradicted
ρ₀ ≤ min q is a bound, so the estimator everyone reaches for is the smallest fitted score — the anchor point. It is consistent, and the question is which way it is wrong in a finite sample, because that decides whether the reported set is conservative or flattering. At n = 200 it returns 0.0879 against a true 0.2500, lands below the truth in 88% of runs, and reports an identified set 7.9× too narrow. The diagnostic flatters the problem it is diagnosing.
And a null result I nearly shipped as a finding. The corrected 0/1 risk can come out negative, and my first count was 0 of 200 replications, which read as "does not happen". The normal approximation predicts 1.6% at 2,000 test rows, so 0 of 200 is the expected outcome and evidence of nothing. Adding the predicted column turned it into a real result: at 200 test rows it is 23.0% measured against 24.8% predicted, and an error rate below zero happens to roughly one run in four.
15,220 verifier assertions, 31 in-page, 25 seeds, 1,000 coverage replications. One file, inline CSS, no external asset.
Part of a from-scratch series — one idea a day, vanilla JS, one file, dependency-free engine: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)