The One-Line Summary: One printable page for the whole boosting family — what each member actually changes about the update rule, a flowchart for picking between AdaBoost, sklearn's GBM, XGBoost, LightGBM and CatBoost, starting hyperparameters with the two that are really one, the losses worth knowing and when to switch, and a symptom-to-fix list for the eight ways a boosted model goes wrong.
The One Line the Whole Family Shares
THE BOOSTING UPDATE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
F_0 = argmin_c sum L(y_i, c)
r_i = -dL(y_i, F(x_i)) / dF(x_i)
h_m = weak learner fit to (x_i, r_i)
F_m = F_(m-1) + lr * h_m
Every member of the family changes ONE of:
- what L is
- what "fit to r" means
- how h_m is grown
- what gets added to the objective
Nothing else. Learn the line, and the library
docs stop being surprising.
Who Changes What
WHAT IT CHANGES vs THE LINE ABOVE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AdaBoost L = exponential. Reweights ROWS instead
of fitting residuals. Equivalent, but
welded to one loss.
GBM (sklearn) any differentiable L. Fits the negative
gradient. The reference implementation.
XGBoost adds SECOND-ORDER gradients (uses the
curvature, not just the slope) and puts
L1/L2 + tree complexity INSIDE the
objective, so pruning is principled.
LightGBM changes how h_m is GROWN: histogram
binning of features, leaf-wise (not
depth-wise) splits. Same maths, much
less of it.
CatBoost changes what the target statistic sees:
ORDERED boosting, so a row's encoding
never uses its own label. Native
categorical handling.
HistGradient sklearn's port of LightGBM's ideas.
Boosting* No extra install. Use this before
reaching for XGBoost.
The Decision Tree
CHOOSING A BOOSTER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Is your data tabular?
|
+- No -> stop. Boosting is not the tool.
| Images/audio/text want neural nets.
|
+- Yes
|
+- Under ~10k rows?
| -> HistGradientBoosting or GBM.
| Tuning matters more than the library.
|
+- 10k - 10M rows?
| -> HistGradientBoosting first.
| XGBoost if you need its ecosystem.
|
+- Over ~10M rows, or many wide categoricals?
|
+- lots of high-cardinality categoricals
| -> CatBoost. Ordered boosting exists
| precisely for this leak.
|
+- otherwise -> LightGBM.
Never start with AdaBoost. It is the ancestor,
not the answer. Read it to understand the family.
Hyperparameters That Matter, In Order
RANK PARAMETER START NOTES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1 learning_rate 0.05 lower = slower
AND better, until
it isn't
1 n_estimators ceiling NOT a target. Use
(same parameter early stopping.
as the rate) lr * n is the
real quantity.
2 max_depth 3 2-6. This is the
interaction order.
(or num_leaves 31 LightGBM's version;
for LightGBM) keep < 2^depth)
3 min_samples_leaf 20 raise on noise
min_child_weight (XGB/LGBM name)
4 subsample 0.8 rows per round.
Nearly free win.
5 colsample_bytree 0.8 columns per tree.
Helps when wide.
6 reg_lambda 1.0 L2. XGB/LGBM only.
reg_alpha 0 L1, for sparsity.
7 loss / objective task the highest-leverage
line people never
touch. See below.
The Losses Worth Knowing
LOSS GRADIENT USE WHEN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
squared_error y - F default. Clean
labels only.
absolute_error sign(y - F) labels are dirty.
MEASURED: 3% of
labels corrupted
took squared from
0.42 R2; absolute
held 0.95.
huber y-F, clipped middle ground.
Got 0.68 on the
same test.
log_loss y - sigmoid(F) classification.
Gives calibrated
probabilities.
quantile asymmetric sign you need a P90,
not a mean.
Boosting vs Bagging, Settled
BOOSTING RANDOM FOREST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
attacks bias variance
trees shallow 2-6 deep, unpruned
built sequentially independently
more trees CAN OVERFIT never harmful
parallel over trees no yes
tuning required real minimal
dirty labels risky tolerant
one-fit answer no yes
usually wins on tabular nothing, but it is
never embarrassing
MEASURED: at lr=1.0 test R2 peaked after ONE tree
and fell to -0.31 by 800. A forest cannot do that.
Eight Ways a Boosted Model Goes Wrong
SYMPTOM -> LIKELY CAUSE / FIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Train loss 0, test terrible -> too many rounds. Early
stop. Boosting really
does overfit.
Grid search takes forever -> you are searching lr
against n_estimators.
They are one number.
Worse than a Random Forest -> lr too high, or depth
too deep. Try 0.05/3.
One outlier dominates -> squared loss chasing
it. Use absolute_error
or huber.
Great CV, bad production -> leakage, or grouped
rows split randomly.
GroupKFold.
Categorical feature ranks
suspiciously high -> target-statistic leak.
CatBoost, or encode
out of fold.
Fit time scales badly -> switch to histogram
binning (LightGBM /
HistGradientBoosting).
Predictions never exceed a
range you have seen -> trees cannot
extrapolate. Still
true here. Ever.
Key Takeaways
One update rule explains the whole family. Each library changes one clause of it; none of them change the shape.
learning_rateandn_estimatorsare one parameter. Measured product held at 4.4 across three rates on classification while the tree count moved 300-fold.Start at
lr=0.05,max_depth=3,subsample=0.8, and early-stop. That configuration is competitive before you have tuned anything.Boosting overfits and forests do not. This is the single most important operational difference between the two families.
The loss is a hyperparameter. Switching to
absolute_erroron 3%-corrupted labels moved R2 from 0.42 to 0.95.Reach for
HistGradientBoostingbefore XGBoost. It ships with sklearn, uses the same histogram trick, and needs no new dependency.
The One-Sentence Summary
Every booster on this page is the same four-line update — start from a constant, take the negative gradient of your chosen loss, fit a shallow tree to it, add a fraction of that tree — and the entire competitive history of XGBoost, LightGBM and CatBoost is three different answers to "which clause of those four lines is the bottleneck on my data?"
What's Next?
- Week 2 recap — tomorrow, tying the boosting posts back to the forest ones.
- XGBoost — second-order gradients and the regularised objective, in detail.
- LightGBM — histogram binning and leaf-wise growth, measured.
- CatBoost — ordered boosting and the target leak it fixes.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
Print this one. The symptom-to-fix table is the part I actually reread.
Questions? Ask in the comments — I read and respond to every one.
Which booster do you reach for by default, and when did you last check that it was still the right one? Mine was XGBoost for years out of habit, and HistGradientBoosting has quietly been enough for most of it. 🗿
Cheat sheets are useful in proportion to how much of the underlying thing you already understand. This one is short because the family is smaller than the marketing suggests — four lines, and a long argument about which of them to optimise.
Top comments (0)