Fit the scaler after you split is the first rule anybody learns about leakage. This measures what breaking it is actually worth.
Run it: https://dev48.infy.uk/ml/day78-preprocessing-leakage.html
Nothing is estimated
Closed-form ridge regression solved by Gaussian elimination, and every score is the exact risk over a 4,000-row held-out population — not a test-set estimate with a confidence interval around it. 80 rows to work with, 1,200 columns.
The famous mistake is the cheap one
Fitting the scaler on all the data leaks the mean and standard deviation of the held-out rows into training. It is the example in every tutorial.
| true-risk difference, clean vs scaler-leaked | |
|---|---|
| worst across six seeds | 0.000168 |
Under two ten-thousandths. Two means and two standard deviations, computed over a few extra rows, move the fitted model almost not at all. The rule is correct and worth keeping — the reason usually given for it is not what the number says.
The mistake that pays is choosing what to look at
Selecting the top-k features using all the data means the validation rows helped choose the features they are about to score. Across the same six seeds the optimism is never below 0.6381.
That is over 1,000× the scaler's cost.
What it buys is worse than a wrong number
The headline population has no signal in it at all — 1,200 columns of pure noise, nothing to learn.
| reports | true risk | |
|---|---|---|
| honest pipeline | 1.0152 | 1.2327 |
| leaked pipeline | 0.5370 | 1.3184 |
It reports a number that looks like a model which halved the error. There is nothing there to halve.
And the leaked model's true risk is worse — 1.3184 against 1.2327. You pay twice: a flattering number, and a model that earned less than the one you would have had, because selecting on the validation rows picks columns that fit noise those rows happened to carry.
The limit, stated because I expected the opposite
I expected turning the signal up to shrink the gap. It does not — the coefficients grow with the signal count, so the target's variance grows and every error grows with it. Worse for the tidy version: above about four signal columns the comparison stops being stable, and at signal 6 the clean pipeline is the more optimistic of the two. That is the opposite of the finding.
With 80 rows and a 20-row validation split, the optimism of a single fit is noisy, and once the target carries real variance that noise swamps the leak. So the claim is about the low-signal regime and is scoped that way on the page — p large, effect small, which is exactly where feature selection gets reached for. Both ends are checked, so neither can quietly stop being true.
73 verifier asserts, 16 in-page checks, 0 failures.
Top comments (0)