A model that scores well on the data it was trained on has shown that it can memorise. Cross-validation is the standard way to estimate how it will do on data it has never seen. It only works if one rule holds, though: no information from the validation data may reach the model-building process by any route. Break that rule, which is easy to do without noticing, and your estimate turns optimistic. This post covers k-fold and leave-one-out cross-validation and the leakage patterns that most often break them.
Why not just score on the training set?
The scikit-learn user guide gives the extreme case: a model that simply repeats the labels of samples it has already seen gets a perfect training score and is useless on new data. That is overfitting in its purest form.
The obvious fix is a held-out test set. With small datasets, though, holding data out shrinks the training set further, and the score depends heavily on which rows happened to land in the test split. Cross-validation reuses the data more efficiently and gives a more stable estimate.
The idea is old. Stone (1974), in the Journal of the Royal Statistical Society, Series B, used cross-validation for both choosing a prediction method and assessing it. The Elements of Statistical Learning (Hastie, Tibshirani & Friedman, chapter 7) keeps those two jobs separate:
- Model selection: using CV to pick a model or tune hyperparameters.
- Model assessment: using CV to report the final model's error on new data.
Mixing the two in a single CV loop is one of the leakage sources covered below.
k-fold and leave-one-out
k-fold CV randomly splits the data into k roughly equal folds. In each round, one fold is held out for validation and the model is trained on the other k − 1. Average the k validation scores and you have the CV estimate. Every observation is used for validation exactly once.
Leave-one-out CV (LOOCV) is the special case k = n: each round holds out a single observation.
Choosing k is a bias–variance trade-off. ESL notes that LOOCV is approximately unbiased for expected prediction error, but its n training sets are almost identical, so the estimate can have high variance. It can also be expensive. Five- or ten-fold CV is the common default and usually a sensible compromise.
In scikit-learn:
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LogisticRegression
cv = KFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=cv)
print(scores.mean(), scores.std())
Leakage pattern 1: preprocessing before splitting
This is the classic mistake. You standardise features, impute missing values, or run feature selection on the whole dataset, and only then run cross-validation. Every validation fold has now influenced the means, the imputation values, or the selected features.
Feature selection is the worst offender. ESL section 7.10.2, "The Wrong and Right Way to Do Cross-validation", describes screening thousands of predictors for those most correlated with the label and then cross-validating a classifier on the survivors. Even when labels are pure noise, this procedure can report error rates far below chance. Ambroise and McLachlan (2002) documented the same selection bias in gene-expression studies.
The fix is to put every data-dependent step inside the CV loop. In scikit-learn, that is what Pipeline is for:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
pipe = make_pipeline(
StandardScaler(),
SelectKBest(f_classif, k=20),
LogisticRegression(max_iter=1000),
)
scores = cross_val_score(pipe, X, y, cv=cv) # scaler and selector refit per fold
Leakage pattern 2: tuning and reporting on the same folds
If you try 50 hyperparameter settings and report the best CV score, that score is biased upwards: you picked the winner partly because it got lucky on those folds. Varma and Simon (2006) showed how large this bias can be in small-sample settings.
The remedy is nested cross-validation. An inner loop tunes hyperparameters and an outer loop estimates the performance of the whole tuning procedure:
from sklearn.model_selection import GridSearchCV
inner = KFold(5, shuffle=True, random_state=1)
outer = KFold(5, shuffle=True, random_state=2)
search = GridSearchCV(pipe, {"logisticregression__C": [0.01, 0.1, 1, 10]}, cv=inner)
nested_scores = cross_val_score(search, X, y, cv=outer)
Leakage pattern 3: grouped data split row by row
When several rows come from the same unit (multiple scans per patient, many EEG windows per subject, several photos of the same object), a random row-level split puts the same unit in both training and validation. The model can learn to recognise the individual rather than the pattern you care about. The score then reflects identity matching, not generalisation to new people.
Split by group instead:
from sklearn.model_selection import GroupKFold
scores = cross_val_score(pipe, X, y, groups=subject_ids, cv=GroupKFold(n_splits=5))
A useful sanity check: if the group-wise score falls sharply compared with the row-wise score, your earlier number was mostly measuring leakage.
Leakage pattern 4: time
For time series, a random split lets the model train on the future and predict the past. Features built with rolling windows or "future-aware" aggregates make this worse. Use forward-chaining splits such as TimeSeriesSplit, which only ever validate on data that comes after the training window, and add a gap between them if features look back over several periods.
Leakage pattern 5: features that encode the target
Kaufman, Rosset, Perlich and Stitelman (2012) describe leakage as including information that would not be legitimately available at prediction time. Typical examples are an ID field that correlates with outcome because of how records were created, a "date of treatment" column when predicting diagnosis, or a field that is filled in only after the event you are predicting. No splitting scheme protects you from this. You have to ask, for each feature, "would I actually have this value at the moment of prediction?"
It is not just a Kaggle problem
Kapoor and Narayanan (2023), in Patterns, surveyed published research that used machine learning across many scientific fields. They found leakage to be a widespread cause of overstated results, and proposed "model info sheets" so authors could document how they had ruled it out. If you are using ML in a research setting, it is worth reading their taxonomy before submitting anything.
A short checklist
- Every step that learns from data (scaling, imputation, encoding, feature selection, resampling such as SMOTE) sits inside the CV loop.
- Hyperparameter tuning happens in an inner loop, and the reported score comes from an outer loop or a truly untouched test set.
- If rows share a subject, site, or source, split by group.
- If time matters, validate only on the future.
- Each feature is available at prediction time.
- The test set is used once, at the end.
For a broader look at how validation choices affect what a study can claim, I keep notes on research-method topics like this one on Scholay.
References
- Ambroise, C., & McLachlan, G. J. (2002). Selection bias in gene extraction on the basis of microarray gene-expression data. PNAS, 99(10), 6562–6566.
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning (2nd ed.), chapter 7. Springer.
- Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9).
- Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4).
- Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B, 36(2), 111–147.
- Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91.
Top comments (0)