Every other bug in a modelling pipeline announces itself by making the number worse. Leakage makes it better, which is why it ships: nobody investigates a pleasant surprise, and the code that causes it looks like tidy engineering.
scaler = StandardScaler().fit(X) # <- on ALL the data
X = scaler.transform(X)
scores = cross_val_score(model, X, y, cv=5)
Every fold's "held-out" rows contributed to the statistics used to transform the training rows.
Measured against a true holdout the pipeline never sees: https://dev48.infy.uk/ml/day66-cv-leakage.html
The severity ladder is not what people expect
On data with no signal at all — labels random, nothing to learn, so every honest number should read 50%:
| pipeline | cross-validated accuracy |
|---|---|
| everything inside the fold (correct) | 52.4% |
| scale before splitting | 55.0% |
| impute before splitting | 56.4% |
| select features before splitting | 73.7% |
Scaling is mild — two moments from a few extra rows barely move. Selection is catastrophic, and the reason is arithmetic rather than luck: with p features, some correlate with the labels by chance, and choosing them using all the data chooses the ones that fit the rows you are about to score on.
It gets worse with p: 50% at p=50 rising to 78% at p=800. The more thorough your feature engineering, the worse it gets.
The cheapest test for it is three lines
y_shuffled = shuffle(y)
score = your_whole_pipeline(X, y_shuffled)
assert score ≈ chance
Shuffle the labels and re-run everything. There is now provably no signal, so an honest pipeline must score at chance. Measured: honest 45.0%, leaky 69.2%.
No held-out set, no production comparison, no need to know which step is guilty. It should be in every notebook that reports a cross-validated number.
When the splitter itself is the leak
Moving preprocessing inside the fold does not help if the split is wrong. Grouped data (several rows per patient or user) under a random KFold puts the same entity on both sides; time series trained on the future. Both are checked structurally here rather than by score — no group id appears in two folds, no test index is ever smaller than a training index.
Two corrections to my own harness
My first generator gave groups a label but no shared effect, so grouping leaked nothing and the GroupKFold comparison measured noise. Real grouped data always has that structure — it is what makes the group identifiable, and identifiable is what leaks.
And several claims needed averaging over seeds. One 120-row cross-validation has a standard error near 10 points, larger than the effect being measured — which is precisely how a real leak gets waved away as noise.
What fixes it
Everything that learns from data — a mean, a threshold, a vocabulary, a set of selected columns — must learn inside the fold. If it has a .fit(), it belongs in the Pipeline. That object is not for tidiness. It is for correctness.
Part of a from-scratch series — one ML idea a day, computed rather than quoted: https://dev48.infy.uk/machinelearningfromzero.php
Top comments (0)