A model that predicts “no fraud” on every row is 99% accurate on a 1%-fraud dataset. The standard responses are to weight the classes or to resample them, and there is a published finding that for a well-tuned gradient-boosted model, often neither is what you needed.
What imbalance actually breaks
Be specific about the failure, because “the model ignores the minority class” is a symptom with several possible causes and only one of them is imbalance itself.
Gradient boosting minimises a loss summed over rows. With 99,000 negatives and 1,000 positives, the negatives contribute 99% of the gradient, so the fitted function is shaped almost entirely by them. The model is not broken by this — it is doing what you asked. If it outputs a well-calibrated probability of 0.012 for a row, that is useful information. The breakage happens at the point where somebody converts that probability to a label with a threshold of 0.5, which is a default with no justification on imbalanced data, and gets no positives at all.
So there are two separable problems: whether the model has learned the minority class at all, and where the decision threshold sits. Weighting and resampling address the first. Most of the time the second was the real one, and confusing them is why teams apply SMOTE and see nothing improve. Which metric you are even looking at matters here; see classification metrics for why accuracy and ROC-AUC both flatter a model on rare positives and precision-recall AUC does not.
The worked dataset
n = 100,000 rows
positives = 1,000 (1.0%)
negatives = 99,000 (99.0%)
imbalance ratio = 99 : 1
Business context: a positive costs 200 to miss (chargeback),
a false positive costs 5 to review (analyst time).
Cost ratio of FN : FP = 40 : 1
The cost ratio is on the page deliberately, because it is the number that should drive every decision below and is usually the one nobody wrote down. Note that it is an assumption here, chosen to make the arithmetic concrete — your ratio comes from your business and is the input worth arguing about.
Class weighting
Weighting multiplies each row’s contribution to the loss by a per-class factor, so a positive counts for more than a negative. In XGBoost this is one parameter, and the documentation states the conventional value directly: scale_pos_weight controls the balance of positive and negative weights, with a typical value to consider of sum(negative instances) / sum(positive instances).
scale_pos_weight = 99,000 / 1,000 = 99
Effective loss contribution after weighting:
negatives 99,000 * 1 = 99,000
positives 1,000 * 99 = 99,000 → balanced
Three things are worth understanding about what this does. It does not create or duplicate any data, so training time is unchanged and no synthetic point is invented. It interacts with min_child_weight, whose documented default is 1 and which is a minimum sum of instance weights (hessians) in a child — once positives carry weight 99, a single positive row can satisfy a constraint that was intended to require many rows, so leaves can form on almost no evidence. Raising min_child_weight in proportion when you raise the weight is the correction, and it is routinely skipped.
And it destroys calibration. A weighted model’s outputs are probabilities under the reweighted distribution, not the real one, so they systematically overstate the positive rate. If you only need a ranking that is fine. If a downstream system multiplies the probability by an expected loss, it is not, and the model needs recalibrating against a held-out set in the original class proportions. Choosing scale_pos_weight at 99 versus something milder such as the square root of the ratio is a hyperparameter like any other and belongs in the search described in gradient boosting hyperparameter tuning.
SMOTE, and where it goes wrong
SMOTE — Synthetic Minority Over-sampling Technique, from Nitesh Chawla, Kevin Bowyer, Lawrence Hall and Philip Kegelmeyer in the Journal of Artificial Intelligence Research volume 16 (2002) — generates new minority rows rather than duplicating existing ones. For a minority point, take one of its k nearest minority neighbours, and place a synthetic point at a random position on the line segment between them:
x_new = x_i + rand(0, 1) * (x_neighbour - x_i)
To balance the worked dataset at k = 5:
synthetic positives needed = 99,000 - 1,000 = 98,000
each generated from one of 1,000 real seeds
→ ~98 synthetic points per real positive
That last line is the whole problem with applying it here. Ninety-eight synthetic points interpolated from each real one is not new information; it is the convex hull of a thousand points, sampled densely. The original paper combines over-sampling of the minority with under-sampling of the majority precisely to avoid pushing the ratio this far, and its evaluation used C4.5, Ripper and naive Bayes — weak learners by current standards.
Four specific failure modes, all mechanical:
- It interpolates in a metric space you may not have. Nearest neighbours require a distance, so unscaled columns make the neighbour search meaningless and one-hot categoricals produce synthetic points with fractional category membership. Variants exist for mixed data; plain SMOTE on an encoded table quietly does the wrong thing.
- It interpolates across the class boundary. A minority point sitting inside a majority region has minority neighbours on the far side, and the segment between them runs straight through majority territory, seeding synthetic positives where positives do not occur.
- It amplifies label noise. A mislabelled positive becomes a seed and gets replicated dozens of times. See detecting mislabelled rows — cleaning first changes what resampling does.
- Applying it before the split invalidates everything. Synthetic points derived from a row that lands in validation mean the validation set contains interpolations of its own contents. Resample inside the training fold only, every time.
When neither helps
The result that ought to change practice is Yotam Elor and Hadar Averbuch-Elor’s “To SMOTE, or not to SMOTE?” (2022), which ran the balancing question against strong modern classifiers rather than the weak learners earlier studies used. Their conclusion, stated in the abstract, is that the known utility of balancing for weak classifiers holds — and that balancing does not improve prediction performance for the strong ones.
The mechanism is consistent with everything above. A well-tuned gradient-boosted ensemble with sufficient depth can already represent a rare region of the input space; what it lacks is not capacity but a sensible operating point. Balancing changes the class prior the model is fitted under, which shifts the outputs and therefore moves the effective threshold — and if moving the threshold is the actual benefit, you can move the threshold directly, without inventing data or breaking calibration.
The genuine failure that balancing cannot fix is absolute scarcity. A thousand positives spread over forty features is a small sample no matter what fraction of the dataset it is, and interpolating between them adds no information. If the model is bad because there are 30 positives, the answer is more labelled positives, or a reframing as anomaly detection where the minority class is not modelled at all — the approach in anomaly detection.
The thing that usually does help
Train on the natural distribution, keep the probabilities calibrated, and choose the threshold from the cost ratio. With the worked numbers — a false negative costing 200 and a false positive costing 5 — the expected-cost-minimising threshold is where the two are equal:
threshold* = C_FP / (C_FP + C_FN)
= 5 / (5 + 200)
= 0.0244
Alert on any row with p(fraud) > 0.024, not 0.5.
That is a factor of twenty away from the default and requires no weighting, no synthetic data and no retraining. It is derived from the cost assumptions stated at the top of this page, and the arithmetic rather than the specific number is what to keep — substitute your own costs and the same expression gives your threshold.
The order of operations that follows: fit on the real distribution and look at precision-recall AUC rather than accuracy; if the ranking is poor, that is a modelling problem and weighting or resampling is worth a try, evaluated on unresampled validation data; if the ranking is good and the labels are wrong, it was always the threshold. Whatever you choose, the validation set keeps the original class proportions, because a metric computed on a rebalanced set answers a question about a dataset that does not exist.
Top comments (0)