Leakage is information in the training features that would not have been available at the moment of the real prediction. It never throws an error, it always improves the offline score, and it is the reason models that validated at 0.94 AUC perform at 0.61 in production. Here are the six constructs that cause almost all of it.
What leakage is, precisely
The test is a single question asked of every feature and every step of the pipeline: could this value have been computed, from data that existed, at the moment the model would really be asked? If the answer requires any qualification, it is leakage.
Two things make it uniquely dangerous among modelling errors. It moves the score in the flattering direction, so nobody investigates. And it is invisible to every check that a normal engineering process runs — the code is correct, the tests pass, the metrics are computed properly. The only thing wrong is what the numbers mean.
1. Preprocessing fitted before the split
# LEAKS
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X) # <-- sees everything
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y)
# also leaks: imputers, PCA, quantile transforms, TF-IDF vocabularies,
# discretisers -- anything with a .fit()
The scaler’s mean and standard deviation are computed over the test rows too, so every training row is standardised using information from the test set. The effect is small on a large dataset and large on a small one — and with an imputer filling missing values from a global median, it is large on any dataset.
The fix is structural rather than a matter of discipline: put every fitted transform inside a Pipeline, and never call fit or fit_transform outside one.
# CORRECT
from sklearn.pipeline import Pipeline
pipe = Pipeline([
("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler()),
("clf", LogisticRegression()),
])
pipe.fit(X_tr, y_tr) # each step fits on train folds only
cross_val_score(pipe, X, y, cv=5) # and refits inside every fold
2. A feature that exists because the outcome did
The subtlest family, because the column names look innocent and the data is real. A value is written into the record as a consequence of the event you are predicting.
-- LEAKS: predicting churn
SELECT
c.customer_id,
c.plan_tier, -- updated to 'cancelled' on churn
c.account_closed_reason, -- non-null only for churners
c.final_invoice_amount, -- exists only if they left
x.churned
FROM customers c
JOIN churn_labels x USING (customer_id)
account_closed_reason is obvious once seen. plan_tier is not, because it is a legitimate feature whose current value happens to encode the outcome. final_invoice_amount is the dangerous kind: a plausible monetary feature that is null for everyone who stayed, so the model learns “not null means churned” and reports 0.99 AUC.
Other instances of the same shape: days_to_resolution in a model predicting whether a ticket will be escalated; discount_applied in a model predicting conversion when the discount is applied at checkout; a risk_band column that a human analyst set after reviewing the case you are trying to score automatically.
The fix is a feature history with validity timestamps and an as-of join, so every feature carries the value it held at the prediction moment — point-in-time correctness is the whole subject of that page. Where no history exists, the column must be dropped; there is no way to reconstruct what was overwritten.
3. A random split on temporal data
# LEAKS
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
shuffle=True, random_state=0)
# on transactions spanning 2024-2026: trains on March, tests on February
The model is evaluated on a period it was trained through. It has seen the fraud ring that operated in that week, the promotion that ran that month, the outage that changed behaviour on that day. None of it will be available when the model is asked about next week.
This is not a small optimism. Any feature with temporal autocorrelation — which is most of them — lets the model interpolate rather than extrapolate, and interpolation is a much easier problem than the one production presents.
# CORRECT
cut = df.event_time.quantile(0.8)
train, test = df[df.event_time < cut], df[df.event_time >= cut]
# for cross-validation, expanding windows rather than folds
from sklearn.model_selection import TimeSeriesSplit
for tr, va in TimeSeriesSplit(n_splits=5).split(X):
...
Note that a temporal split will report a worse score, and that the worse score is the true one. The rolling-origin backtest is the fully developed version of this idea.
4. The same entity on both sides
# LEAKS: one row per user per month, 12 rows per user
X_tr, X_te = train_test_split(panel, test_size=0.2) # splits rows
User 4471 has January to August in train and September to December in test. The model learns user 4471 — their baseline spend, their device, their idiosyncratic pattern — and then recognises them at test time. The score measures memorisation of individuals and predicts nothing about a user the model has never seen, which is the only kind production has.
The same failure appears with near-duplicate rows: a product listed twice with slightly different text, a document that appears in two collections, a customer with two account ids. Deduplicate on content, not on primary key.
# CORRECT
from sklearn.model_selection import GroupShuffleSplit, GroupKFold
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
tr, te = next(gss.split(X, y, groups=panel.user_id))
# and inside cross-validation
for tr, va in GroupKFold(n_splits=5).split(X, y, groups=panel.user_id):
...
When the data is both grouped and temporal — which a customer panel always is — you need both: split by time, and confirm no entity spans the boundary in a way that matters.
5. Target encoding over the whole table
# LEAKS: each row's own label is inside its own feature
df["city_conversion_rate"] = (
df.groupby("city")["converted"].transform("mean")
)
# the same failure, less obviously:
df["user_avg_target"] = df.groupby("user_id")["y"].transform("mean")
df["is_above_segment_median"] = (
df["y"] > df.groupby("segment")["y"].transform("median")
)
For a city with three rows, one third of that feature is the row’s own outcome. For a high-cardinality column — user id, product id, postcode — the group is often a single row and the feature is the label. This produces near-perfect training scores and a model that is worthless, and it is the most common way a Kaggle technique destroys a production project.
The fix is out-of-fold encoding: compute each row’s encoded value from folds that exclude it, with smoothing towards the global mean for small groups.
# CORRECT
from sklearn.model_selection import KFold
import numpy as np
def oof_target_encode(df, col, target, n_splits=5, smoothing=20):
prior = df[target].mean()
out = np.full(len(df), prior, dtype=float)
for tr, va in KFold(n_splits=n_splits, shuffle=True,
random_state=0).split(df):
stats = df.iloc[tr].groupby(col)[target].agg(["mean", "count"])
# shrink small groups towards the prior
enc = ((stats["mean"] * stats["count"] + prior * smoothing)
/ (stats["count"] + smoothing))
out[va] = df.iloc[va][col].map(enc).fillna(prior).to_numpy()
return out
Scikit-learn also ships TargetEncoder, which performs the cross-fitting internally; use it inside a Pipeline so the fold discipline is enforced by the framework rather than by whoever edits the notebook next.
6. Selection and tuning before the split
# LEAKS
from sklearn.feature_selection import SelectKBest, f_classif
X_sel = SelectKBest(f_classif, k=50).fit_transform(X, y) # sees all labels
scores = cross_val_score(model, X_sel, y, cv=5) # meaningless
The selector ranked 5,000 columns by their association with the labels — including the test labels — and kept the 50 that looked best on the whole dataset. Every subsequent cross-validation score is computed on features chosen with knowledge of the validation folds. On a wide dataset of pure noise, this procedure reliably produces a cross-validated AUC well above 0.5.
The same applies to hyperparameter search: tuning against the test set, even by looking at it a dozen times and picking what worked, is leakage through the experimenter. It is why a three-way train, validation and test split exists, and why the test set should be looked at once.
# CORRECT
pipe = Pipeline([
("select", SelectKBest(f_classif, k=50)),
("clf", GradientBoostingClassifier()),
])
scores = cross_val_score(pipe, X, y, cv=5) # selection refits per fold
# and for tuning, nest it
from sklearn.model_selection import GridSearchCV, cross_val_score
inner = GridSearchCV(pipe, param_grid, cv=5)
cross_val_score(inner, X, y, cv=5) # honest generalisation estimate
Two more worth knowing
- Overlapping label and feature windows. Features drawn from days 1–90 and a label defined over days 60–150. The overlap means part of the outcome is inside the feature window. Fix the window definition, not the model — the three-window layout in the churn page is the general form.
- Row order and identifiers. An exported file sorted by class, or an auto-incrementing id that correlates with the label because positives were inserted in a batch. Drop identifier columns unless you can justify each one, and shuffle before you look.
How to catch it
- Be suspicious of a good score. If the first model reaches an AUC far above what the domain plausibly supports, the prior should be leakage rather than talent. Fraud at 0.99, churn at 0.97, default at 0.95 — investigate before celebrating.
- Look at what dominates. Run permutation importance. If one feature carries nearly all the signal, read its definition and its source table before doing anything else. Leakage concentrates.
- Drop the top feature and refit. A genuine model degrades gracefully because the signal is spread. A leaking model collapses to near chance, because there was only ever one thing in it.
- Impose a temporal holdout even when the problem is not temporal. Train on everything before a date, test after it. A large gap between this and a random-split score is the clearest single indicator of leakage available.
- Write the availability date next to each feature. A column in the feature documentation saying when the value becomes known relative to the prediction. Filling it in catches pattern 2 before any code runs, and it is the cheapest control on this list.
- Shadow-score in production before switching. Run the model live without acting on it, log the features it actually receives, and compare their distributions to training. Leakage that survived everything above shows up here as a feature that is systematically different or simply absent.
Top comments (0)