Beyond the 99% Accuracy Lie: Building a Leak-Free XGBoost Pipeline for Imbalanced Data

Every machine learning beginner hits the same rite of passage. You download a real-world dataset such as credit card fraud, medical diagnostics, or customer churn and train your first classifier, run accuracy evaluation, and see 99.2% accuracy flash across your screen.

For about five minutes, you feel like a genius. Then you print the confusion matrix and realize your model hasn't learned a single thing: it simply predicted the majority class (0) for every single row.

In the real world, balanced datasets rarely exist. The events we actually care about predicting including fraudulent transactions, hardware failures, rare diseases make up less than 1% to 5% of total records. When beginners tackle imbalanced datasets, they usually fall into three classic traps:

1. The Accuracy Illusion & Naive Model Selection

Standard algorithms optimize for overall error reduction. If 98% of your dataset is Class 0 and 2% is Class 1, a model can minimize loss by completely ignoring the minority class. Beginners often cycle through Random Forests, and even raw XGBoost hoping a "stronger" algorithm will magically spot the minority class just to watch their Recall sit stubbornly near zero:

from xgboost import XGBClassifier
from sklearn.metrics import accuracy_score, recall_score

# Naive training on a 98:2 imbalanced dataset
model = XGBClassifier(random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, preds):.4f}") # Output: 0.9850 (Looks great!)
print(f"Recall:   {recall_score(y_test, preds):.4f}")   # Output: 0.0410 (Catastrophic!)

2. Blindly Oversampling (And Causing Data Leakage)

When accuracy fails, the next instinct is to fix the data using oversampling techniques like SMOTE. While SMOTE is powerful, applying it across your dataset before cross-validation bleeds synthetic data into your validation folds thus creating a model that looks incredible in your notebook and collapses in production.

3. Trusting the Default 0.50 Decision Threshold

Calling .predict() in Sklearn or XGBoost applies a rigid 0.50 probability cutoff. On skewed datasets, predicted probabilities for the minority class rarely cross 0.50, meaning you throw away valid detections simply because you left the decision boundary on autopilot.

In this guide, we will walk through how to engineer a production-ready, leak-free XGBoost pipeline from scratch & combining proper SMOTE isolation, intelligent hyperparameter tuning, and mathematical threshold optimization.

---

How SMOTE Rescues Minority Classes (And Where It Goes Wrong)

When a model ignores the minority class because there aren't enough examples to learn from, the most intuitive fix is to balance the dataset.

Basic Random Oversampling simply duplicates existing minority rows. While that balances the numbers, it doesn't add any new information & your decision trees just memorize those exact duplicate points and overfit heavily.

SMOTE (Synthetic Minority Over-sampling Technique) solves this geometrically:

1. It picks a minority class data point at random.
2. It finds its $k$-nearest minority neighbors (usually $k=5$).
3. It draws a line between the point and one of its neighbors and generates a brand-new, synthetic data point somewhere along that line.

Instead of memorizing exact duplicates, the model is forced to learn a broader, smoother decision region around the minority class.

The Silent Killer: SMOTE Data Leakage

Here is where 90% of beginner notebooks fail. They apply SMOTE to the training data before running Cross-Validation:

# ❌ THE WRONG WAY: Causes severe Data Leakage!
from imblearn.over_sampling import SMOTE
from sklearn.model_selection import cross_val_score

smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

# Validation folds inside CV now contain synthetic twins of the training folds!
scores = cross_val_score(model, X_train_smote, y_train_smote, cv=5, scoring='f1')

Why does this break your model?

When you run fit_resample() on X_train first, SMOTE uses data points from the entire set to create synthetic neighbors. When cross_val_score later splits that data into 5 folds, Fold 1 (training) contains points that were directly used to synthesize points in Fold 2 (validation).

Your validation fold is no longer unseen data. Your cross-validation score shoots up to an artificial 0.95, and the moment you test on real, untouched production data, performance craters.

The Fix: Isolating SMOTE Inside imblearn.pipeline

To prevent leakage, SMOTE must be applied strictly inside each individual training fold during cross-validation, leaving the validation fold 100% untouched and aligned with the real-world distribution.

Standard sklearn.pipeline.Pipeline doesn't allow resampling steps because samplers alter the number of rows in X and y. Instead, we use imblearn.pipeline.Pipeline:

# ✅ THE RIGHT WAY: Zero Data Leakage
from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE
from xgboost import XGBClassifier

pipeline = ImbPipeline([
    ('smote', SMOTE(random_state=42)),
    ('xgb', XGBClassifier(random_state=42, eval_metric='logloss'))
])

When you pass this ImbPipeline into cross-validation:

• During each fold split, SMOTE only oversamples the training sub-fold.

• The validation sub-fold is never resampled.

• When calling .predict() on X_test, the pipeline automatically bypasses SMOTE altogether.

Watch Out for "Double Penalization" (SMOTE vs. scale_pos_weight)

If you are using XGBoost, you have probably heard of the scale_pos_weight parameter, which boosts the gradient step for the minority class (typically set to $\frac{\text{Total Negative Examples}}{\text{Total Positive Examples}}$).A common mistake is combining full SMOTE(sampling_strategy=1.0) with a high scale_pos_weight at the same time.

• SMOTE already balances the dataset at the data level (making the ratio 50:50).

• Adding scale_pos_weight on top of a 50:50 dataset forces XGBoost to aggressively over-penalize false negatives at the algorithmic level

The result? Your model becomes paranoid, flagging everything as positive. Recall hits 99%, while Precision drops through the floor. When using SMOTE inside a pipeline, keep scale_pos_weight=1 (or tune SMOTE's sampling_strategy carefully) so your model stays calibrated.

---

Hyperparameter Tuning: Why RandomizedSearchCV Beats GridSearchCV

Even with a leak-free ImbPipeline, out-of-the-box XGBoost parameters (max_depth=6, learning_rate=0.3) are aggressive. On an oversampled dataset, deep trees will quickly memorize the synthetic SMOTE points rather than learning the underlying signal.

To prevent overfitting, we need to tune the tree structure (max_depth, subsample, colsample_bytree) and learning pace (learning_rate, n_estimators).

Beginners usually reach for GridSearchCV, which tests every single combination in a parameter grid. However, when your pipeline runs SMOTE across 5 folds for every combination, a 100-combination grid requires 500 separate oversampling and training runs.

RandomizedSearchCV solves this bottleneck:

• Instead of brute-forcing every combination, it samples a fixed number of settings (n_iter) across a wider parameter space.
• It finds near-optimal hyperparameters in a fraction of the compute time.
• Most importantly, we set scoring='f1' (or 'average_precision') so the search optimizes for minority class performance rather than overall accuracy.

from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold

# Define parameter grid using the 'step_name__parameter' syntax
param_dist = {
    'xgb__n_estimators': [100, 200, 300],
    'xgb__max_depth': [3, 4, 5, 6],
    'xgb__learning_rate': [0.01, 0.05, 0.1, 0.2],
    'xgb__subsample': [0.7, 0.8, 1.0],
    'xgb__colsample_bytree': [0.7, 0.8, 1.0]
}

cv_strategy = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

search = RandomizedSearchCV(
    estimator=pipeline,
    param_distributions=param_dist,
    n_iter=15,
    scoring='f1',
    cv=cv_strategy,
    n_jobs=-1,
    random_state=42
)

search.fit(X_train, y_train)
best_pipeline = search.best_estimator_

Ditching the 0.50 Cutoff: Dynamic Threshold Optimization

Once best_pipeline is trained, calling best_pipeline.predict(X_test) still commits one final mistake: it forces a rigid 0.50 probability threshold.

Under the hood, classifiers don't output 0 or 1 but they output continuous probabilities via .predict_proba(). On heavily imbalanced datasets, your model might assign a 0.38 probability to an actual fraud case. A default 0.50 cutoff throws that prediction in the trash.

Instead of guessing thresholds manually, we can extract the raw probabilities and use the Precision-Recall Curve (precision_recall_curve)to evaluate every possible cutoff simultaneously:

import numpy as np
from sklearn.metrics import precision_recall_curve, classification_report

# 1. Get predicted probabilities for the positive class (Class 1)
y_probs = best_pipeline.predict_proba(X_test)[:, 1]

# 2. Calculate Precision and Recall for all possible thresholds
precisions, recalls, thresholds = precision_recall_curve(y_test, y_probs)

# 3. Compute F1 score at every threshold (adding 1e-10 to prevent division by zero)
f1_scores = (2 * precisions * recalls) / (precisions + recalls + 1e-10)

# 4. Find the exact threshold that maximizes the F1 score
best_idx = np.argmax(f1_scores)
optimal_threshold = thresholds[best_idx]

print(f"Optimal Decision Threshold: {optimal_threshold:.4f}")
print(f"Max F1 Score Achieved:      {f1_scores[best_idx]:.4f}")

# 5. Apply the custom threshold to generate final predictions
final_preds = (y_probs >= optimal_threshold).astype(int)
print(classification_report(y_test, final_preds))

By shifting from a blind 0.50 cutoff to a mathematically optimized threshold, you align the model's decision boundary directly with the true distribution of your minority class & thus maximizing F1 without retraining the model a single time.

---

Putting It All Together: Try the Complete Code

Building a reliable classifier on skewed data isn't about finding a "magic" algorithm but it's about fixing the pipeline mechanics around the model:

1. Isolate Oversampling: Wrap SMOTE inside imblearn.pipeline.Pipeline so validation folds remain 100% untouched during cross-validation.
2. Avoid Double Penalization: Don't stack aggressive scale_pos_weight multipliers on top of a fully oversampled 50:50 dataset.
3. Tune for the Right Metric: Use RandomizedSearchCV optimizing for 'f1' or 'average_precision' rather than raw accuracy.
4. Optimize the Cutoff: Extract raw probabilities with .predict_proba() and let the Precision-Recall curve mathematically lock in your decision threshold.

Run the Interactive Notebook & Explore the Repo

I have packaged the complete, end-to-end implementation—including synthetic imbalanced data generation, baseline comparisons, cross-validated tuning, and Precision-Recall curve visualizations into a public GitHub repository and a one-click Google Colab notebook:

Dot-kirit / xgboost-imbalanced-ml-pipeline

Guideline to tune the model to increase performance according to needs.

XGBoost Pipeline for Imbalanced Data Classification

Open Notebook in Google Colab

An end-to-end Machine Learning pipeline using Scikit-Learn, Imbalanced-Learn (SMOTE), and XGBoost to handle class imbalance and optimize F1-score using custom decision thresholds.

Key Features

• Data Leakage Prevention: Integrates SMOTE inside an imblearn pipeline to ensure oversampling occurs only on training folds during Cross-Validation.
• Hyperparameter Tuning: Uses RandomizedSearchCV to optimize XGBoost hyperparameters (n_estimators, learning_rate, scale_pos_weight, etc.).
• Threshold Optimization: Applies a custom decision threshold (30%) to maximize precision-recall trade-offs.

Tech Stack

• Python 3
• Scikit-Learn
• Imbalanced-Learn
• XGBoost
• Pandas & NumPy

View on GitHub

---

How do you prefer handling extreme class imbalance in your projects, by oversampling with SMOTE, tuning class weights, or calibrating probabilities directly? Drop your thoughts or questions in the comments below!