Berkan Sesen

Posted on Jul 5 • Originally published at sesen.ai

Handling Class Imbalance: Downsampling, SMOTE, and Beyond

#supervisedlearning #discriminative

A fraud detection model scores 99.5% accuracy. Sounds excellent until you check the predictions: it labels every single transaction as legitimate. With only 0.5% of transactions being fraudulent, a classifier that always says "no" is right 99.5% of the time. It catches zero fraudsters.

This is the accuracy paradox: when one class vastly outnumbers the other, a lazy classifier that ignores the minority class can still look great on paper. The fix is not to chase a better algorithm, but to change how you measure success, and sometimes how you prepare the data. By the end of this post, you'll implement five different strategies for handling class imbalance, understand the precision-recall tradeoff each one makes, and know which to reach for in practice.

Four Ways to Rebalance

Before we look at code, watch how four resampling strategies transform the same imbalanced training set. The original data has 1,324 majority samples (blue) and only 76 minority samples (red):

Downsampling discards majority samples to match the minority count (76 each). Upsampling duplicates minority samples to match the majority count (1,324 each). SMOTE creates new synthetic minority samples by interpolating between existing ones. Each approach changes the class balance the model sees during training, but the test set remains untouched.

Let's Build It

Click the badge to run this yourself:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score, precision_score, recall_score
from imblearn.over_sampling import SMOTE
from collections import Counter

np.random.seed(42)

# Create an imbalanced dataset: 95% class 0, 5% class 1
X, y = make_classification(
    n_samples=2000, n_features=20, n_informative=10, n_redundant=5,
    n_classes=2, weights=[0.95, 0.05], flip_y=0.01, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
print(f"Train: {Counter(y_train)}")  # {0: 1324, 1: 76}

# Indices for majority and minority classes
majority_idx = np.where(y_train == 0)[0]
minority_idx = np.where(y_train == 1)[0]

# Strategy 1: Random downsampling
down_idx = np.random.choice(majority_idx, size=len(minority_idx), replace=False)
X_down = np.vstack([X_train[down_idx], X_train[minority_idx]])
y_down = np.concatenate([y_train[down_idx], y_train[minority_idx]])

# Strategy 2: Random upsampling
up_idx = np.random.choice(minority_idx, size=len(majority_idx), replace=True)
X_up = np.vstack([X_train[majority_idx], X_train[up_idx]])
y_up = np.concatenate([y_train[majority_idx], y_train[up_idx]])

# Strategy 3: SMOTE
X_smote, y_smote = SMOTE(random_state=42).fit_resample(X_train, y_train)

# Strategy 4: Class weights (no resampling needed)
# Train and evaluate all strategies
strategies = {
    'Baseline':      (X_train, y_train, {}),
    'Downsample':    (X_down, y_down, {}),
    'Upsample':      (X_up, y_up, {}),
    'SMOTE':         (X_smote, y_smote, {}),
    'Class Weights': (X_train, y_train, {'class_weight': 'balanced'}),
}

for name, (X_tr, y_tr, kwargs) in strategies.items():
    clf = LogisticRegression(max_iter=1000, random_state=42, **kwargs)
    clf.fit(X_tr, y_tr)
    pred = clf.predict(X_test)
    print(f"{name:<15} Prec={precision_score(y_test, pred):.3f}  "
          f"Rec={recall_score(y_test, pred):.3f}  "
          f"F1={f1_score(y_test, pred):.3f}  "
          f"Acc={np.mean(pred == y_test):.3f}")

Train: Counter({0: 1324, 1: 76})
Baseline        Prec=0.545  Rec=0.182  F1=0.273  Acc=0.947
Downsample      Prec=0.185  Rec=0.879  F1=0.305  Acc=0.780
Upsample        Prec=0.222  Rec=0.848  F1=0.352  Acc=0.828
SMOTE           Prec=0.245  Rec=0.818  F1=0.378  Acc=0.852
Class Weights   Prec=0.228  Rec=0.848  F1=0.359  Acc=0.833

The baseline has the highest accuracy (94.7%) but the worst F1 score (0.27). It catches only 18% of minority samples. Every resampling strategy dramatically improves recall (to 82-88%) at the cost of precision. SMOTE achieves the best F1 (0.38) by creating synthetic minority samples that generalise better than simple duplication.

What Just Happened?

The Accuracy Paradox

The baseline model has 94.7% accuracy, barely above the 94.5% you'd get by predicting the majority class for everything. That 0.2 percentage point "improvement" over the trivial classifier is misleading. The resampling strategies all have lower accuracy (78-85%) but higher F1, because they sacrifice correct majority-class predictions to catch more minority samples. With imbalanced data, accuracy is almost useless as a metric.

Precision vs Recall: The Fundamental Tradeoff

Every binary classifier faces a tradeoff between two error types:

Precision answers: "Of all the samples I flagged as positive, how many actually are?" High precision means few false alarms.
Recall answers: "Of all the actual positives, how many did I catch?" High recall means few missed cases.

The F1 score is the harmonic mean of precision and recall:

The harmonic mean punishes extreme imbalances: if either precision or recall is near zero, F1 will also be near zero. This makes it a much better single metric for imbalanced problems than accuracy.

Why Resampling Shifts the Tradeoff

A logistic regression model outputs probabilities. The default decision threshold is 0.5: predict class 1 if $P(\text{class} = 1 \mid \mathbf{x}) > 0.5$ . On imbalanced data, the model rarely outputs probabilities above 0.5 for the minority class, because it has learned that class 1 is rare. Most predictions stay below the threshold, so recall is low.

Resampling changes the class proportions the model sees during training, which shifts the predicted probabilities upward for the minority class. More predictions cross the 0.5 threshold, catching more true positives (higher recall) but also more false positives (lower precision).

What Each Strategy Does

Random downsampling discards majority samples until the classes are balanced. This is the approach from the original R code, which used caret::downSample(). It is fast and simple, but throws away potentially useful data. With only 76 samples per class (152 total), the model has much less to learn from.

Random upsampling duplicates minority samples until the classes are balanced. No data is lost, but the duplicated samples add no new information. The model may overfit to specific minority instances.

SMOTE (Synthetic Minority Oversampling Technique) creates new minority samples by interpolating between existing ones. For each minority sample, it finds its k-nearest minority neighbours and generates synthetic points along the line segments connecting them.

Class weights adjust the loss function instead of the data. Setting class_weight='balanced' in scikit-learn multiplies the loss for each class by $n / (2 \cdot n_k)$ , where $n_k$ is the count of class $k$ . The minority class gets a weight of $1324 / 76 \approx 17.4\times$ higher, making each minority sample count as much as 17 majority samples in the gradient update.

Confusion Matrices: Where the Errors Go

The baseline gets 562 out of 567 negatives right but only 6 out of 33 positives. Downsampling flips the balance: it catches 29 out of 33 positives but misclassifies 128 negatives. SMOTE finds a middle ground: 27 true positives with only 83 false positives.

Going Deeper

How SMOTE Works

SMOTE generates each synthetic sample in three steps:

Pick a minority sample $\mathbf{x}_i$
Find one of its $k$ nearest minority neighbours $\mathbf{x}_{nn}$ (default $k = 5$ )
Create a new sample at a random point on the line segment between them:

This fills in the feature space around existing minority samples rather than simply duplicating them. The synthetic samples are plausible minority examples that the model has not seen before, which reduces overfitting compared to random upsampling.

SMOTE has known failure modes. If minority samples are scattered among majority samples (rather than forming clusters), the interpolated points may land in majority-class territory, creating noisy synthetic examples. Variants like Borderline-SMOTE (only oversample minority points near the decision boundary) and SMOTE-ENN (apply Edited Nearest Neighbours after SMOTE to clean noisy samples) address this.

Precision-Recall Curves: The Full Picture

The single-threshold metrics above (precision, recall, F1) depend on the 0.5 decision threshold. The precision-recall curve traces all possible thresholds, showing the tradeoff across the full range:

The average precision (AP) summarises each curve as a single number. Interestingly, the baseline has the highest AP (0.373), meaning it ranks positives higher on average. Resampling strategies improve recall at a fixed threshold but do not necessarily improve the overall ranking. This is an important nuance: resampling shifts the decision boundary, but it does not change the model's ability to separate classes.

When should you use ROC curves vs precision-recall curves? ROC curves can be misleading on imbalanced data because the false positive rate (x-axis) stays low even with many false positives when the negative class is huge. Precision-recall curves focus entirely on the minority class and are the preferred evaluation tool for imbalanced problems.

The Prentice-Pyke Result: Why Downsampling Works

The original R code referenced a remarkable theoretical result from case-control studies in epidemiology. Prentice & Pyke (1979) proved that for logistic regression, downsampling the majority class preserves all estimated coefficients except the intercept.

This means the model's understanding of which features matter (and by how much) is unaffected by downsampling. Only the baseline probability changes. If you know the true population prevalence, you can correct the intercept analytically:

Where $\pi_1$ is the true prevalence and $\tilde{\pi}_1$ is the prevalence in the downsampled data (0.5 for balanced). This result only holds for logistic regression; it does not generalise to random forests, SVMs, or neural networks.

Which Strategy Should You Use?

There is no universally best approach. The choice depends on your constraints:

Strategy	Best When	Watch Out For
Downsampling	Majority class is very large; compute is limited	Loses useful majority-class information
Upsampling	Minority samples are high-quality; small dataset	Overfitting to duplicated samples
SMOTE	Features are continuous; minority forms clusters	Noisy synthetic samples if classes overlap
Class weights	You want simplicity; no external library needed	Only affects the loss, not the data distribution
Do nothing	You have enough minority samples; using AP or PR-AUC	Threshold-based metrics (F1) will look bad

For most practical problems, start with class_weight='balanced' (simplest, no data manipulation) and compare against SMOTE. If neither satisfies your precision-recall requirements, consider threshold tuning on the original model's predicted probabilities, which can be more effective than any resampling.

Try It Yourself

Tune the threshold. Instead of resampling, train the baseline model and sweep the decision threshold from 0.1 to 0.9. Plot precision and recall as a function of threshold. Can you find a threshold that gives better F1 than SMOTE?
Try Borderline-SMOTE. Replace SMOTE() with BorderlineSMOTE() from imbalanced-learn. Does it improve F1 on this dataset?
Use a tree-based model. Replace logistic regression with RandomForestClassifier. Do the resampling strategies still help, or do tree models handle imbalance better natively?

Where This Comes From

Class imbalance as a research topic gained traction in the late 1990s when machine learning was increasingly applied to fraud detection, medical diagnosis, and fault detection, all domains where the event of interest is rare. The two foundational contributions came from very different fields.

Prentice & Pyke (1979) proved the theoretical basis for downsampling in the context of case-control studies, the standard design in epidemiology for studying rare diseases. Their key result, that retrospective sampling preserves the odds ratio estimates in logistic regression, gave practitioners confidence that downsampling was not merely a computational shortcut but a theoretically sound approach.

"Under a wide variety of designs in which sampling of data records is dependent on the outcome variable, estimation via the prospective logistic model yields valid estimates of the relevant odds ratios."

Prentice, R. L. & Pyke, R. (1979). Logistic Disease Incidence Models and Case-Control Studies. Biometrika, 66(3), 403-411.

Chawla et al. (2002) introduced SMOTE, which became the most widely cited paper on class imbalance handling in machine learning. They demonstrated that creating synthetic minority samples by interpolation outperformed random oversampling across multiple datasets and classifiers. The paper has been cited over 25,000 times.

"Our method of synthetic minority over-sampling technique (SMOTE) works by creating synthetic examples from the minor class instead of by over-sampling with replacement."

Chawla, N. V., Bowyer, K. W., Hall, L. O. & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. JAIR, 16, 321-357.

Frequently Asked Questions

What is class imbalance and why is it a problem?

Class imbalance occurs when one class has far fewer examples than the other (for example, 1% fraud vs 99% legitimate transactions). Standard classifiers optimise overall accuracy, so they learn to predict the majority class almost always, achieving high accuracy while completely failing to detect the rare class you actually care about.

Should I oversample the minority class or downsample the majority class?

It depends on your dataset size. Downsampling is simpler and avoids the risk of overfitting to duplicated examples, but it throws away potentially useful data. SMOTE and other oversampling methods work well when the minority class is very small, but they can create noisy synthetic examples near decision boundaries. Try both and compare using cross-validation.

Why should I use F1-score or AUC instead of accuracy for imbalanced datasets?

Accuracy is misleading when classes are imbalanced. A model that predicts "not fraud" for every transaction achieves 99% accuracy on a 1% fraud dataset but catches zero fraud. F1-score balances precision and recall, while AUC measures performance across all classification thresholds, giving a more honest picture of model quality.

Should I apply SMOTE before or after the train-test split?

Always after the split, and only to the training set. Applying SMOTE before splitting causes synthetic minority examples to leak information from the test set into training, giving overly optimistic performance estimates. This is one of the most common mistakes in handling class imbalance.

Can I combine multiple resampling techniques?

Yes. A common and effective approach is to combine SMOTE (to oversample the minority class) with random undersampling of the majority class. This creates a more balanced dataset without the extreme data loss of pure downsampling or the extreme synthesis of pure oversampling. The imbalanced-learn library supports these pipelines directly.

DEV Community