A while back, I had to retract one of my own results.
A model I'd built was posting a validation score I was genuinely proud of. Then I looked closer: the test set shared rows with the training data. The moment I evaluated on a truly independent set, the improvement vanished. It wasn't a better model. It was leakage.
That stuck with me. Data leakage is the single most common reason ML results turn out to be fiction, and almost nobody checks for it systematically. Here are the five that bite most often, and how to catch them.
- Duplicate rows across train and test
The simplest and most common. If the exact same row is in both sets, you're testing on data the model memorised. Your score goes up; your generalisation doesn't. Fix: de-duplicate before you split, and make sure no test row also exists in train.
- Near-duplicates
Worse, because it hides. Augmented copies, floating-point noise, or the same event logged twice produce rows that aren't byte-identical but are effectively the same. A random split scatters them across both sides. Fix: de-duplicate with a tolerance, and audit your pipeline for copies.
- Temporal (look-ahead) leakage
If your data has time and you split it randomly, your model trains on the future to predict the past. It looks brilliant in validation and falls apart in production, where the future genuinely isn't available. Fix: split by time. Train on the past, test on the future.
- Features that encode the target
A column that's really the answer in disguise: an ID that maps to the label, or a value that was only recorded after the outcome was known. The model learns to read the answer instead of predicting it. Fix: drop it, or replace it with information actually available at prediction time.
- The same group on both sides
Multiple rows per patient, user, or device. Split randomly and the model recognises the subject, not the pattern. It memorises "patient 47 is positive" and can't do anything with patient 48. Fix: use a group-aware split so every group stays entirely on one side.
A real example that still surprises people
Take EEG Eye State, a real 15,000-row dataset from OpenML. It's one continuous recording, so neighbouring samples are almost identical.
Split it the way everyone does by default, a random 80/20, and a gradient boosting model scores 0.971 AUC. Beautiful.
Split it honestly (train on the earlier part of the recording, test on the later part) and it collapses to 0.546, barely better than a coin flip.
That 0.425 AUC was pure leakage. Same data, same model, same features. The only thing that changed was whether the split respected time.
Two tools: one to detect, one to prevent
I got tired of eyeballing this, so I built two small open-source tools that work together.
LeakHound finds the leaks. Point it at your split:
pip install leakhound-ml
leakhound --train train.csv --test test.csv --target label --time-col date --measure-impact
It flags all five leaks above and, with the measure-impact flag, tells you the honest score next to the inflated one. On the EEG example it prints exactly the 0.971 to 0.546 gap. It exits non-zero, so you can drop it into CI and fail a merge that would have shipped a leaky model. It even catches homology leakage in protein and DNA data (sequences that are merely similar, not identical), in pure Python.
safesplit stops the leaks happening in the first place. It makes leakage-safe splits: group-aware, time-aware, and sequence-aware:
pip install safesplit
train, test = safe_split(df, group_col="patient_id") # a group is never on both sides
train, test = safe_split(df, time_col="date") # train on the past, test on the future
train, test = safe_split(df, seq_col="sequence") # keep near-homologous sequences together
One detects, the other prevents, and you can verify one with the other. Both are open source (AGPL):
LeakHound: https://github.com/happyhellpt/leakhound
safesplit: https://github.com/happyhellpt/safesplit
The takeaway
A great validation score is a hypothesis, not a result. Before you trust it, publish it, or ship it, spend the 10 seconds to check whether the split is honest. I've been burned by this in my own work; I'd rather you weren't.
Top comments (0)