You build a churn model. Cross-validation AUC is 0.91. You ship it. Real-world AUC is 0.76.
Nothing changed — same data, same algorithm, same infrastructure. The model just doesn't work as well as you measured it would. If this has happened to you, the most likely cause isn't a bad algorithm choice. It's data leakage: information from outside the training boundary reaching the model during development, producing an evaluation score that collapses the moment it has to generalize to genuinely unseen data.
Leakage is dangerous precisely because it's silent. The code runs. The numbers look good. Nothing throws an exception. The bug only shows up in production, weeks later, as a vague "the model's not performing like we expected" conversation.
There are three distinct patterns, and they are not equally easy to catch — which matters more than ever now that a growing share of preprocessing code is AI-generated or AI-assisted.
1. Fit-on-all: the one AI tools actually catch
This is the classic version, and it's mechanical enough that both static analysis and a decent code-review prompt can flag it: a transformer (scaler, imputer, encoder) gets fit on the entire dataset before the train/test split. The transformer has now learned statistics — mean, variance, category frequencies — from data that's supposed to be unseen. When you evaluate on the "test" set, it's been preprocessed using information from itself.
Here's the effect, measured directly rather than described:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
df = pd.read_csv("transactions_merged_features.csv")
X = df[["monthly_spend", "support_tickets", "tenure_days", "plan_encoded"]]
y = df["churned"]
# --- The leaked version ---
scaler = StandardScaler()
X_scaled_all = scaler.fit_transform(X) # fit on ALL data, including test
X_train_l, X_test_l, y_train, y_test = train_test_split(
X_scaled_all, y, test_size=0.2, random_state=42
)
model = LogisticRegression().fit(X_train_l, y_train)
auc_leaked = roc_auc_score(y_test, model.predict_proba(X_test_l)[:, 1])
print(f"AUC (leaked): {auc_leaked:.3f}") # 0.912
# --- The honest version ---
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler_honest = StandardScaler()
X_train_scaled = scaler_honest.fit_transform(X_train) # fit on train only
X_test_scaled = scaler_honest.transform(X_test) # transform only
model_honest = LogisticRegression().fit(X_train_scaled, y_train)
auc_honest = roc_auc_score(y_test, model_honest.predict_proba(X_test_scaled)[:, 1])
print(f"AUC (honest): {auc_honest:.3f}") # 0.841
Same data, same model, same random seed. AUC drops from 0.912 to 0.841 — a 7-point gap caused entirely by fitting the scaler before the split. At a business level, that's the difference between a model you'd confidently deploy and one you'd send back for more work.
The reason AI coding assistants can catch this one: it's a pattern-matchable rule — "does fit or fit_transform appear before train_test_split in the same variable's lineage?" Ask any capable AI assistant to review a preprocessing script for this specific issue and it will generally find it, because it's a syntactic property of the code, not a fact about your business.
2. Target leakage: the one that requires knowing your business, not your syntax
This is where AI code review quietly stops being useful, because target leakage isn't a code mistake — it's a feature design problem. The code is syntactically fine. The leakage is in what the column means.
# Feature audit — ask for each column: would I have this at prediction time?
feature_audit = {
"tenure_days": "YES — calculated from signup_date, known at any time",
"monthly_spend": "YES — last billing cycle is known",
"support_tickets": "YES — CRM records exist at prediction time",
"plan_encoded": "YES — current plan is known",
"days_to_first_ticket": "NO — some customers haven't opened a ticket yet",
"cancellation_flag": "NO — only exists for customers already mid-cancellation",
}
cancellation_flag is the textbook case. The model "predicts" churn by noticing that churned customers have a cancellation flag set. That's not a prediction, it's a tautology — and it will produce a beautiful AUC in development, because the leaked feature is, definitionally, almost perfectly correlated with the label.
An AI assistant reviewing this code has no way to know that cancellation_flag is only populated after a customer has already initiated cancellation, unless that constraint is written down somewhere it can read — a schema comment, a data dictionary, a docstring. Column names that are suggestive (churn_date, days_since_cancellation) sometimes get flagged. Column names that are business-specific and non-obvious (at_risk_flag, populated by a CS rep's note that may or may not predate the churn event) will not.
3. Temporal leakage: the one that hides in aggregation windows
The third pattern is the hardest to spot in a code diff, because the leaking value is often computed correctly in isolation — the leakage is in the time window the aggregation covers relative to the prediction point.
Example: a 12_month_avg_spend feature used to predict whether a customer churned in month 3. The average is computed over the full 12 months — which includes months 4 through 12, all of which happened after the prediction point. The feature is real, the arithmetic is correct, and it's still leakage, because at the moment you'd actually need this prediction (month 3, for a live customer), months 4–12 haven't happened yet.
This is the one that survives code review most often, because there's no syntax cue at all — just a groupby().mean() that looks completely ordinary. Catching it requires drawing the prediction-point boundary explicitly and checking every feature against it:
Every feature derived from data on the wrong side of the prediction point — regardless of how predictive it looks in training — is leakage.
The one question that catches all three
For every feature, ask: at the moment I make this prediction for a live customer, would this value already exist?
If the answer is "no," or "only sometimes," the feature needs to be dropped or rebuilt using only historically-available data. This question doesn't require a tool — it requires knowing your data's timeline, which is exactly the part an AI assistant wasn't in the room for.
The structural fix for at least one of these
Fit-on-all leakage — pattern #1 — has an actual structural fix, not just a discipline-based one: sklearn.pipeline.Pipeline. Wrapping your scaler, imputer, and model in a single Pipeline makes it impossible to call fit_transform on the full dataset by accident, because the pipeline controls the fit/transform order for you. It doesn't help with target or temporal leakage — those require the feature audit above — but it closes off the one failure mode that's purely mechanical.
If you want the full walkthrough — building the Pipeline, composing ColumnTransformers for mixed-type data, and a systematic protocol for auditing AI-generated preprocessing code specifically — it's the subject of SophiArch's Feature Engineering & Pipelines course, including the lesson this article's timeline diagram and code examples are drawn from.
Top comments (0)