DEV Community: Sachin Kr. Rajput

XGBoost: The Accountant Who Read the Tax Code So Carefully He Found Money Nobody Else Saw

Sachin Kr. Rajput — Fri, 07 Aug 2026 11:47:38 +0000

The One-Line Summary: XGBoost is gradient boosting with three changes that each look like a footnote and together won a decade of competitions — it uses the second derivative so each leaf knows how stiff its correction is rather than only how wrong, it writes the cost of every new leaf directly into the objective so pruning becomes arithmetic instead of a max_depth guess, and it treats a missing value as a direction to be learned rather than a hole to be filled.

The Parable of the Counting House at Anand & Sons

Every autumn the grain merchants of the river towns had to declare what their business was worth, and every autumn they got it wrong. Anand & Sons had solved this the way everyone did: hire twelve clerks, and have each one correct the previous clerk's figure by exactly the amount it was off.

It half worked. It also, twice in living memory, produced a declaration so wrong the family nearly lost the warehouse.

The Old Way: Twelve Clerks, One Instruction

The instruction each clerk received was a single sentence. Find where the last figure was off, and move it by that much.

THE LEDGER, CORRECTED TWELVE TIMES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
line              off by    clerk writes
warehouse rent     -400      +400
sack commission    -400      +400
cartage             -80       +80

Next season:
warehouse rent        0      settled, correct
sack commission    +900      WILDLY over
cartage             -10      nearly right

The rent is a FIXED sum. Move it by 400 and it
moves by 400. Honest work.

The commission is a PERCENTAGE OF TURNOVER. Move
it by 400 and turnover shifts, and the commission
moves by rather more than 400. The clerk overshot,
the next clerk overshot back, and the figure
oscillated for four seasons.

The clerks were not careless. They had been given one number per line — how wrong — and they needed two. Nobody had ever asked how much a line moves when you push it.

The Innovation: A Second Number Per Line

Rukmini was hired to audit the clerks and instead rewrote their instruction. She had read the revenue code end to end, which nobody at Anand & Sons had done in thirty years, and she came back with a question.

"You all know how wrong each line is. Does any of you know how stiff it is?"

She asked every clerk to record two numbers instead of one. How far off the line sits, and how much the total shifts when the line is nudged by a single rupee. The first number is the correction you want. The second is how much of it you dare take.

THE SAME LEDGER, WITH STIFFNESS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
line             off by   stiffness   correction
warehouse rent    -400       1.0        400/1.0 = 400
sack commission   -400       4.0        400/4.0 = 100
cartage            -80       1.0         80/1.0 =  80

The rent takes its full correction. The springy
commission takes a quarter of it, because it will
carry itself the rest of the way.

Divide the error by the stiffness. Nothing else
changed. The oscillation stopped in one season.

The Second Fix: Make Every Line Pay Rent

Within two years the ledger had swollen to nine hundred lines, because a clerk who spots a twelve-rupee discrepancy is rewarded for writing a line about it.

The old remedy was a senior partner who went through in December and struck out lines he judged frivolous. He was often right and never consistent, and no two partners struck out the same lines.

Rukmini found the answer in the code itself — a clause about the cost of maintaining records.

"Charge every new line a fee. Then a line only exists if it is worth more than its fee. We stop arguing about which lines are frivolous, because the ledger tells us."

A FEE PER LINE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Splitting one line into two is worth 112.50
                fee per new line:   20.00
                              net:  +92.50   KEEP

Splitting a different line is worth   2.00
                fee per new line:   20.00
                              net:  -18.00   DROP

The senior partner is out of a job, and the
decision is now the same every December.

That is the whole trick, and it is not a trick. Deciding what to keep stopped being a matter of judgement applied afterwards and became a term inside the sum being minimised.

The Third Fix: A Blank Is Not a Zero

Some suppliers never sent invoices. The clerks wrote in the average invoice for that supplier's trade, which felt responsible and was quietly catastrophic — the suppliers who failed to invoice were disproportionately the ones in trouble, and a blank was the single most informative mark on the page.

Rukmini's rule: do not fill it in. Send every blank the same direction, and let the ledger discover which direction that should be.

WHAT TO DO WITH A BLANK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
FILL IT IN     write the average. The blank now
               looks exactly like an ordinary
               middling supplier. Its information
               is destroyed on purpose.

SEND IT ONE WAY   try both directions once, keep
                  whichever the ledger prefers.
                  A blank stays visible as a blank.

The blanks were not missing data. They were the
data.

What Is XGBoost?

Hard pivot. Every one of Rukmini's three fixes is a named feature.

Gradient boosting, from day 9, builds a running sum $F_{m} = F_{m - 1} + ν h_{m}$ where $h_{m}$ is fit to the negative gradient. XGBoost keeps that skeleton and changes what gets optimised at each round.

The Objective

Write the objective at round $m$ as the loss plus a penalty on the tree itself:

L^{(m)} = i = 1 \sum n L (y_{i}, F_{m - 1} (x_{i}) + f_{m} (x_{i})) + Ω (f_{m})

Ω (f) = γ T + \frac{1}{2} λ j = 1 \sum T w_{j 2}

where $T$ is the number of leaves and $w_{j}$ their values. $γ$ is the fee per line. $λ$ is an L2 penalty on how large a correction any leaf may make.

The Second-Order Step

Now expand the loss to second order in $f_{m}$ — this is the whole difference from a plain GBM:

L^{(m)} ≃ i \sum [g_{i} f_{m} (x_{i}) + \frac{1}{2} h_{i} f_{m} (x_{i})^{2}] + Ω (f_{m})

g_{i} = \frac{\partial L}{\partial F} ∣ ∣ F m - 1, h_{i} = \frac{\partial ^{2} L}{\partial F ^{2}} ∣ ∣ F m - 1

$g_{i}$ is how wrong. $h_{i}$ is how stiff. A plain GBM only ever computes the first.

The Leaf Weight Falls Out

Fix the tree's structure and let $I_{j}$ be the rows landing in leaf $j$ , with $G_{j} = \sum_{i \in I_{j}} g_{i}$ and $H_{j} = \sum_{i \in I_{j}} h_{i}$ . The objective becomes a sum of independent quadratics in each $w_{j}$ , and a quadratic has a closed-form minimum:

w_{j *} = - \frac{G _{j}}{H _{j} + λ}

No line search, no gradient step — the optimal leaf value is an exact formula. Substituting it back gives the quality of a whole tree structure:

L^{*} = - \frac{1}{2} j = 1 \sum T \frac{G _{j 2}}{H _{j} + λ} + γ T

And Therefore the Split Criterion

A split is worth making if it lowers $L^{*}$ . Subtracting the parent from its two children:

Gain = \frac{1}{2} [\frac{G _{L 2}}{H _{L} + λ} + \frac{G _{R 2}}{H _{R} + λ} - \frac{( G _{L} + G _{R} ) ^{2}}{H _{L} + H _{R} + λ}] - γ

This is the payoff. Gini and entropy are heuristics chosen because they behave sensibly. This is derived — it is the exact reduction in the objective you are already minimising, with the cost of the new leaf subtracted. A negative gain means don't split, and that is pre-pruning with a receipt.

Step by Step

Start from a constant $F_{0}$ .
For $m = 1, \dots, M$ :
1. Compute $g_{i}$ and $h_{i}$ for every row.
2. Grow a tree greedily, scoring each candidate split by the Gain formula and refusing any split with Gain $\leq 0$ .
3. Set each leaf to $w_{j *} = - G_{j} / (H_{j} + λ)$ .
4. Update $F_{m} = F_{m - 1} + ν f_{m}$ .
Return $F_{M}$ .

Regularisation You Can Watch Working

The leaf formula, on one leaf holding four rows with residuals 3, 5, 4, 8:

import numpy as np

resid = np.array([3.0, 5.0, 4.0, 8.0])
g = -resid                       # squared error: g = pred - y
G, H = g.sum(), float(len(g))    # h_i = 1 for squared error
print(f"  G = {G:+.1f}   H = {H:.0f}")
print("\n  lambda    w* = -G/(H+lambda)    plain mean residual")
for lam in (0.0, 1.0, 4.0, 12.0, 40.0):
    print(f"  {lam:>6}    {-G/(H+lam):>18.4f}    {resid.mean():.4f}")

  G = -20.0   H = 4

  lambda    w* = -G/(H+lambda)    plain mean residual
     0.0                5.0000    5.0000
     1.0                4.0000    5.0000
     4.0                2.5000    5.0000
    12.0                1.2500    5.0000
    40.0                0.4545    5.0000

At $λ = 0$ the leaf is the mean residual — exactly what a GBM would have written. Every $λ > 0$ pulls it toward zero, and note what it did not touch: the learning rate. These are two independent brakes, which is why XGBoost tolerates deeper trees than a GBM of the same learning_rate.

Now the gain formula deciding two different splits:

import numpy as np

def score(g, lam):
    return g.sum()**2 / (len(g) + lam)

def gain(left, right, lam, gamma):
    gL, gR = -np.array(left), -np.array(right)
    both = np.concatenate([gL, gR])
    return 0.5*(score(gL, lam) + score(gR, lam) - score(both, lam)) - gamma

print("  a split with real separation: [-8,-6] vs [7,9]")
for lam in (0.0, 1.0, 10.0):
    for gm in (0.0, 20.0, 100.0):
        g = gain([-8, -6], [7, 9], lam, gm)
        print(f"    lambda={lam:<5} gamma={gm:<6} gain={g:>+9.4f}  "
              f"{'split' if g > 0 else 'PRUNE'}")

print("\n  a split that is basically noise: [3,5] vs [4,8]")
for lam in (0.0, 1.0):
    g = gain([3, 5], [4, 8], lam, 0.0)
    print(f"    lambda={lam:<5} gamma=0.0    gain={g:>+9.4f}  "
          f"{'split' if g > 0 else 'PRUNE'}")

  a split with real separation: [-8,-6] vs [7,9]
    lambda=0.0   gamma=0.0    gain=+112.5000  split
    lambda=0.0   gamma=20.0   gain= +92.5000  split
    lambda=0.0   gamma=100.0  gain= +12.5000  split
    lambda=1.0   gamma=0.0    gain= +74.9333  split
    lambda=1.0   gamma=20.0   gain= +54.9333  split
    lambda=1.0   gamma=100.0  gain= -25.0667  PRUNE
    lambda=10.0  gamma=0.0    gain= +18.6905  split
    lambda=10.0  gamma=20.0   gain=  -1.3095  PRUNE
    lambda=10.0  gamma=100.0  gain= -81.3095  PRUNE

  a split that is basically noise: [3,5] vs [4,8]
    lambda=0.0   gamma=0.0    gain=  +2.0000  split
    lambda=1.0   gamma=0.0    gain=  -5.3333  PRUNE

Two things worth staring at.

The noise split has a positive gain of 2.0 at lambda=0 and a plain GBM would happily take it. Setting lambda=1 flips it to prune — because each of the two new leaves pays $λ$ in its denominator while the parent paid it once. reg_lambda is not only shrinking leaf values; it is quietly raising the bar for splitting at all.

And the real split survives lambda=10, gamma=0 at a gain of 18.69 but dies at gamma=20. $γ$ is an absolute toll and $λ$ is proportional — they are not interchangeable, and tuning one does not cover the other.

Does Second Order Actually Buy Anything?

The honest test: same trees, same depth, same learning rate, same data. The only difference is whether each leaf is the mean gradient (first order) or $- G / (H + λ)$ (second order). Logistic loss, where $h_{i} = p_{i} (1 - p_{i})$ genuinely varies from row to row.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=3000, n_features=20, n_informative=8,
                           flip_y=0.05, random_state=4)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=4)
sig = lambda z: 1.0 / (1.0 + np.exp(-z))


def boost(order, rounds=200, lr=0.1, depth=3, lam=1.0):
    F = np.zeros(len(ytr))
    Fte = np.zeros(len(Xte))
    curve = []
    for _ in range(rounds):
        p = sig(F)
        g = p - ytr                     # first derivative
        h = p * (1.0 - p)               # second derivative
        t = DecisionTreeRegressor(max_depth=depth,
                                  random_state=0).fit(Xtr, -g)
        leaf, leaf_te = t.apply(Xtr), t.apply(Xte)
        w = {}
        for L in np.unique(leaf):
            m = leaf == L
            w[L] = (-g[m]).mean() if order == 1 \
                else -g[m].sum() / (h[m].sum() + lam)
        F += lr * np.array([w[L] for L in leaf])
        Fte += lr * np.array([w.get(L, 0.0) for L in leaf_te])
        curve.append(Fte.copy())
    return curve


c1, c2 = boost(1), boost(2)
print(" rounds |  first-order  |  second-order")
for r in (10, 25, 50, 100, 200):
    print(f"  {r:>5}  |    {log_loss(yte, sig(c1[r-1])):.4f}     |"
          f"    {log_loss(yte, sig(c2[r-1])):.4f}")

 rounds |  first-order  |  second-order
     10  |    0.5916     |    0.4462
     25  |    0.5056     |    0.3589
     50  |    0.4346     |    0.3322
    100  |    0.3794     |    0.3150
    200  |    0.3436     |    0.3091

Second order at 50 rounds (0.3322) beats first order at 200 rounds (0.3436). I checked how far first order would have to go to catch up: it never reaches 0.3322 within 200 rounds at all. Four times the trees, still behind.

That is Rukmini's first fix, and it is the reason XGBoost felt like a step change rather than an optimisation. Dividing by the summed curvature is not a speed trick — it lands somewhere a first-order method does not reach.

XGBoost from Scratch

Sixty lines, import xgboost used only to check the answer:

import numpy as np, xgboost as xgb
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=600, n_features=8, n_informative=5,
                       noise=8.0, random_state=13)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=13)
LAM, GAMMA, DEPTH, LR, ROUNDS, BASE = 1.0, 0.0, 3, 0.3, 40, 0.0


class Node:
    __slots__ = ("feat", "thr", "left", "right", "val")


def sc(G, H):
    return G * G / (H + LAM)


def build(X, g, h, depth):
    n = Node()
    G, H = g.sum(), h.sum()
    if depth == 0 or len(g) < 2:
        n.feat, n.val = -1, -G / (H + LAM)
        return n
    best, parent = (0.0, -1, 0.0), sc(G, H)
    for f in range(X.shape[1]):
        o = np.argsort(X[:, f], kind="stable")
        cg, ch, xs = np.cumsum(g[o]), np.cumsum(h[o]), X[o, f]
        for i in range(len(xs) - 1):
            if xs[i] == xs[i + 1]:
                continue
            gain = 0.5 * (sc(cg[i], ch[i])
                          + sc(G - cg[i], H - ch[i]) - parent) - GAMMA
            if gain > best[0]:
                best = (gain, f, (xs[i] + xs[i + 1]) / 2.0)
    if best[1] < 0:                      # no split pays for itself
        n.feat, n.val = -1, -G / (H + LAM)
        return n
    n.feat, n.thr = best[1], best[2]
    m = X[:, n.feat] < n.thr
    n.left = build(X[m], g[m], h[m], depth - 1)
    n.right = build(X[~m], g[~m], h[~m], depth - 1)
    return n


def pred1(n, row):
    while n.feat >= 0:
        n = n.left if row[n.feat] < n.thr else n.right
    return n.val


F, trees = np.full(len(ytr), BASE), []
for _ in range(ROUNDS):
    g = F - ytr                          # squared error
    h = np.ones_like(g)
    t = build(Xtr, g, h, DEPTH)
    trees.append(t)
    F += LR * np.array([pred1(t, r) for r in Xtr])

mine = np.full(len(Xte), BASE)
for t in trees:
    mine += LR * np.array([pred1(t, r) for r in Xte])

theirs = xgb.XGBRegressor(
    n_estimators=ROUNDS, learning_rate=LR, max_depth=DEPTH, reg_lambda=LAM,
    gamma=GAMMA, min_child_weight=0, base_score=BASE, reg_alpha=0,
    tree_method="exact").fit(Xtr, ytr).predict(Xte)

print(f"  from scratch R2 = {r2_score(yte, mine):.6f}")
print(f"  xgboost      R2 = {r2_score(yte, theirs):.6f}")
print(f"  correlation     = {np.corrcoef(mine, theirs)[0, 1]:.8f}")
print(f"  RMSE vs xgboost = {np.sqrt(((mine - theirs)**2).mean()):.6f}")
print(f"  y std (test)    = {yte.std():.2f}")

  from scratch R2 = 0.931456
  xgboost      R2 = 0.931628
  correlation     = 0.99999692
  RMSE vs xgboost = 0.251010
  y std (test)    = 110.06

Agreement to 0.0002 R2, with a per-row disagreement of 0.25 on a target whose standard deviation is 110 — about two parts in a thousand. The residual gap is split tie-breaking and float32 binning inside the real thing, exactly as it was for the GBM on day 9.

If you can write that loop, reg_lambda, gamma and min_child_weight stop being knobs you turn hopefully. You know which line of arithmetic each one sits in.

Missing Values Are a Direction, Not a Hole

Rukmini's third fix. 4,000 rows, six columns with values deleted — and the deletions are informative, 40% missing when $y = 1$ versus 10% when $y = 0$ .

import numpy as np, xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss, accuracy_score
from sklearn.impute import SimpleImputer

X, y = make_classification(n_samples=4000, n_features=20, n_informative=8,
                           flip_y=0.05, random_state=6)
rng = np.random.default_rng(6)
Xm = X.copy()
for j in range(6):                                  # informative missingness
    Xm[rng.random(len(y)) < np.where(y == 1, 0.40, 0.10), j] = np.nan
Xtr, Xte, ytr, yte = train_test_split(Xm, y, test_size=0.3, random_state=6)
print(f"  {np.isnan(Xm).mean()*100:.1f}% of all cells are missing")


def fit(a, b):
    return xgb.XGBClassifier(n_estimators=300, learning_rate=0.1,
                             max_depth=4, random_state=0).fit(a, b)


p = fit(Xtr, ytr).predict_proba(Xte)[:, 1]
print(f"  NaN passed through   logloss {log_loss(yte, p):.4f}  "
      f"acc {accuracy_score(yte, p > 0.5):.4f}")

imp = SimpleImputer(strategy="mean").fit(Xtr)
p2 = fit(imp.transform(Xtr), ytr).predict_proba(imp.transform(Xte))[:, 1]
print(f"  mean-imputed first   logloss {log_loss(yte, p2):.4f}  "
      f"acc {accuracy_score(yte, p2 > 0.5):.4f}")
print(f"  imputation cost {log_loss(yte, p2) - log_loss(yte, p):+.4f} logloss")

  7.6% of all cells are missing
  NaN passed through   logloss 0.1909  acc 0.9250
  mean-imputed first   logloss 0.2034  acc 0.9225
  imputation cost +0.0125 logloss

Imputing before fitting made the model worse. Not dramatically — 0.0125 log loss — but it cost something, for extra work, and the direction is the point: mean imputation destroys information on purpose. XGBoost tries both directions for the missing rows at every split and keeps the better one, so a NaN stays distinguishable from an average value all the way through.

The habit to break is df.fillna(df.mean()) as a reflex. If your missingness has any relationship to your target — and in production it usually does, because things go missing when systems are failing — hand the NaNs over and let the split decide.

When Does XGBoost Help Most?

	XGBoost	sklearn GBM	Random Forest
Derivatives used	first and second	first	none
Leaf value	$- G / (H + λ)$ , exact	mean gradient	mean target
Pruning	gain $\leq 0$ , derived	`max_depth` guess	none needed
Missing values	learned direction	must impute	must impute
Regularisation	$λ, γ, α$ in the objective	shrinkage only	n/a
Parallel within a tree	yes	no	yes, across trees
Sensible without tuning	mostly	mostly	yes

Use it for tabular data with real structure, a target worth the last two points of accuracy, and enough rows that tree_method="hist" matters. Don't use it for pixels, audio, raw text, or a first look at a dataset you don't understand yet — a Random Forest will tell you whether there is signal at all, faster and with fewer decisions.

Hyperparameters

Parameter	What it does	Typical range	How to tune
`learning_rate`	fraction of each tree added	0.01–0.1	set low, early-stop
`n_estimators`	rounds	ceiling, not target	`early_stopping_rounds`
`max_depth`	interaction order	3–8	4–6 is the usual sweet spot
`min_child_weight`	minimum $H$ in a leaf	1–20	raise on noisy data
`reg_lambda`	L2 on leaf values	1–10	also raises the split bar
`gamma`	fixed toll per new leaf	0–5	try before raising depth
`subsample`	rows per round	0.6–1.0	0.8
`colsample_bytree`	columns per tree	0.6–1.0	0.8
`tree_method`	split search	`hist`	`hist` unless tiny data
`scale_pos_weight`	class imbalance	$n_{-} / n_{+}$	for skewed binary targets

Note min_child_weight is a floor on the summed hessian, not the row count. On squared error, where every $h_{i} = 1$ , they coincide and everyone learns the wrong intuition. On logistic loss a leaf full of confidently-classified rows has tiny hessians, so min_child_weight=10 may reject a leaf holding hundreds of rows.

Quick Reference Card

XGBOOST: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  Gradient boosting where each round solves a
  regularised second-order problem exactly, instead
  of taking a first-order step.

THE THREE FORMULAS:
  leaf   w* = -G / (H + lambda)
  tree   L* = -0.5 * sum(G_j^2/(H_j+lambda)) + gamma*T
  split  Gain = 0.5*[ G_L^2/(H_L+lam)
                    + G_R^2/(H_R+lam)
                    - G^2/(H+lam) ] - gamma

WHAT EACH KNOB TOUCHES:
  lambda  shrinks leaves AND raises the split bar
  gamma   flat toll per new leaf (pre-pruning)
  alpha   L1, drives leaves to exactly zero
  mcw     floor on summed HESSIAN, not row count

MEASURED:
  2nd order @ 50 rounds  0.3322 logloss
  1st order @ 200 rounds 0.3436  (never catches up)
  NaN passed through     0.1909
  mean-imputed           0.2034

WHEN TO USE:
  + tabular, structured, worth tuning
  + informative missingness
  + you need the last two points
WHEN NOT EFFECTIVE:
  - images, audio, raw text
  - first look at unfamiliar data
  - you cannot early-stop

SKLEARN-COMPATIBLE:
  from xgboost import XGBRegressor, XGBClassifier
  # or sklearn's HistGradientBoosting* for no new dep

Key Takeaways

The second derivative is the headline. Same trees, same rate: second order hit 0.3322 log loss in 50 rounds where first order sat at 0.3436 after 200 and never caught up.
The leaf value is a closed form, not a step. $w^{*} = - G / (H + λ)$ is the exact minimum of a quadratic, which is why there is no line search anywhere in XGBoost.
The split criterion is derived, not chosen. Gain is the actual drop in the objective minus the cost of the new leaf. Gini and entropy are heuristics; this is arithmetic.
reg_lambda does two jobs. It shrinks leaf values and raises the bar for splitting, because two children each pay $λ$ where the parent paid it once. A noise split with gain +2.0 at lambda=0 became -5.33 at lambda=1.
$γ$ and $λ$ are not interchangeable. One is a flat toll, one is proportional. A split survived lambda=10 at gain 18.69 and died at gamma=20.
Don't impute before boosting. Passing NaN through scored 0.1909 against 0.2034 mean-imputed. Missingness is usually correlated with the target, and filling it in erases that on purpose.
min_child_weight is a hessian floor, not a row count. Identical to a row count on squared error, very different on log loss — which is where the confusion always bites.
Sixty lines reproduce it to two parts in a thousand. If you can write the gain formula from memory, every regularisation parameter has an address.

The One-Sentence Summary

XGBoost is what happens when someone reads the objective function as carefully as Rukmini read the tax code: expand the loss to second order and the optimal leaf value becomes an exact formula rather than a step, put the cost of each leaf inside that objective and pruning becomes a subtraction rather than a guess, and treat a missing value as a direction to be learned rather than a hole to be plastered over — three footnotes that between them turned gradient boosting from a good algorithm into the default one.

What's Next?

Now that you know where every XGBoost parameter lives in the maths, you're ready for:

LightGBM — tomorrow. Histogram binning and leaf-wise growth: the same objective, an order of magnitude less arithmetic, and one genuinely dangerous default.
CatBoost — ordered boosting, and the target-encoding leak the other two quietly live with.
Bagging vs Boosting — the settled comparison, measured rather than asserted.
Hyperparameter tuning — what to search when four parameters interact.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the counting house made the gain formula click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever imputed missing values by reflex and never checked what it cost? I did it for years, in a pipeline where the missingness was the strongest feature in the dataset. 🗿

The thing I find genuinely elegant about XGBoost is that it did not invent a new idea. It took gradient boosting and asked what the objective function actually said, in the way that a good accountant reads a statute — not looking for loopholes, just refusing to skim. Second-order terms, a penalty on complexity, a decision rule for missing entries: all three were sitting in the maths, waiting for somebody to be pedantic enough to write them down.

Week 2 Recap: The Week Trees Stopped Voting and Started Correcting Each Other

Sachin Kr. Rajput — Thu, 06 Aug 2026 11:32:43 +0000

The One-Line Summary: Week 2 crossed the line from ensembles that average away variance to ensembles that attack bias one correction at a time — six posts covering out-of-bag error, Extra Trees, AdaBoost, Gradient Boosting, the learning-rate trap and a family cheat sheet, and the honest theme running through all of them is that boosting gives you a better ceiling in exchange for the ability to ruin the model by trying too hard.

The Handoff

Week 1 ended on a single quantity: tree-to-tree correlation, and the fact that the variance of an average has a floor no number of trees can lower.

Week 2 started by finishing that story, then abandoned the whole approach.

THE WEEK IN ONE PICTURE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
BAGGING FAMILY          BOOSTING FAMILY
(days 1-7)              (days 8-11)

many deep trees         many shallow trees
built independently     built in sequence
average them            SUM them
fights variance         fights bias
more trees: safe        more trees: dangerous
one fit, good answer    tuning required

The pivot is a single word:
  average -> sum
Everything else follows from it.

What Each Post Was Actually About

Out-of-Bag Error — Bootstrapping leaves about 36.8% of rows unseen by any given tree, because (1 - 1/n)^n converges to 1/e. Score each row with only the trees that never saw it and you get validation for the price of one fit. Measured: OOB 0.9021, 5-fold 0.9036, test 0.8983, at roughly a fifth of the wall clock.

The part worth keeping is where it lies. With 5 trees, OOB reported 0.7829 for a model that was actually 0.8750. With five visits per patient in the data, OOB reported a perfect 1.0000 against GroupKFold's 0.8620 — because "unseen by this tree" is not the same as "unseen" when the same entity appears five times.

Extra Trees — Stop searching for the best split threshold; draw one at random. Third rung on the same ladder: correlation fell 0.640 → 0.571 → 0.515 across bootstrap, feature subsampling, and random thresholds, single-tree accuracy fell 0.8342 → 0.8204 → 0.8034, and the ensemble went 0.8967 → 0.8983 → 0.9050. Worse trees, better forest, and faster because nothing needs sorting.

AdaBoost — The first algorithm to build the next model because of the last one's mistakes. Reweight the rows you got wrong, fit a stump, weight its vote by how good it was, repeat. It works, and its limitation is that it is welded to exponential loss.

Gradient Boosting — The generalisation that made boosting the default for tabular data. Fit each tree to the negative gradient of any differentiable loss, add a fraction of it. Twenty lines reproduced sklearn to five decimal places of R2; the two diverged at round 23 on exactly one test row out of 240, where a split tie broke the other way.

learning_rate and n_estimators — They are one parameter. Across seven rates the optimal product held at 4.4 on classification while the tree count moved 300-fold. A rate × count grid search is mostly duplicate work.

The family cheat sheet — One update rule, and each library changes exactly one clause of it.

The Two Things That Changed My Mind

Lowest correlation is not the goal. Week 1 built up decorrelation as the thing to chase. Then in the max_features measurements, 'sqrt' on regression produced the lowest tree correlation of any setting — 0.248 — and the worst R2, losing 21 points. Decorrelation is a means. I had been treating it as a score.

Boosting can genuinely destroy a model. I knew this as a caution and had never watched it happen cleanly. At learning_rate=1.0 on a noisy problem, held-out R2 peaked after one tree at 0.186, then fell to -0.31 by 800 trees — worse than predicting the mean for every row. A Random Forest cannot do that no matter how badly you configure it. That asymmetry is the real reason boosting demands more of you.

The Thing I Got Wrong

I published a claim about max_features on regression that I could not reproduce when I went back to re-run it, and I rewrote the post rather than leave it. The corrected version measured three dataset shapes, five seeds each, and the conclusion inverted: the regression default of 1.0 was the most accurate setting every time, and the widely repeated advice to copy 'sqrt' across from the classifier was the worst of four settings on every shape.

Which is a useful thing to have happen in week 2 rather than week 30. The rule I have adopted: no number goes into a post unless the code that produced it ran in the same session I wrote the sentence.

One Idea Per Post, If You Only Keep Six Things

36.8% of rows are out-of-bag for any tree, and that is where free validation comes from.
OOB lies when rows are grouped. oob_score_ of 1.0 is a warning, not an achievement.
Worse trees can make a better forest — Extra Trees, measured three rungs down.
The gradient, not the residual. "Fit the residuals" is true only for squared error.
lr × n_estimators is the quantity that matters. Fix the rate, early-stop the count.
Boosting overfits; forests do not. Everything operational follows from this.

Next Week

Week 3 goes inside the libraries that made boosting win.

XGBoost — second-order gradients, and regularisation moved into the objective so pruning stops being a heuristic.
LightGBM — histogram binning and leaf-wise growth: same maths, an order of magnitude less of it.
CatBoost — ordered boosting, and the target-encoding leak it was built to close.
Stacking — what to do when you have several good models and no idea which to trust.

Let's Connect!

Questions? Ask in the comments — I read and respond to every one, and any question that needs more than a paragraph becomes its own post.

Which of this week's six would you want re-done in more depth? My own vote is the OOB failure modes; the grouped-rows case is the one I have actually been burned by in production. 🗿

Two weeks in, the pattern I did not expect is how often the honest version of a post is less satisfying than the draft. The draft says "here is the trick." The measurement says "here is the trick, and here are the three shapes of data where it costs you." I am keeping the second one.

The Boosting Family Decision Tree: Everything on One Page

Sachin Kr. Rajput — Wed, 05 Aug 2026 08:28:31 +0000

The One-Line Summary: One printable page for the whole boosting family — what each member actually changes about the update rule, a flowchart for picking between AdaBoost, sklearn's GBM, XGBoost, LightGBM and CatBoost, starting hyperparameters with the two that are really one, the losses worth knowing and when to switch, and a symptom-to-fix list for the eight ways a boosted model goes wrong.

The One Line the Whole Family Shares

THE BOOSTING UPDATE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  F_0 = argmin_c  sum L(y_i, c)
  r_i = -dL(y_i, F(x_i)) / dF(x_i)
  h_m = weak learner fit to (x_i, r_i)
  F_m = F_(m-1) + lr * h_m

Every member of the family changes ONE of:
  - what L is
  - what "fit to r" means
  - how h_m is grown
  - what gets added to the objective

Nothing else. Learn the line, and the library
docs stop being surprising.

Who Changes What

                  WHAT IT CHANGES vs THE LINE ABOVE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AdaBoost      L = exponential. Reweights ROWS instead
              of fitting residuals. Equivalent, but
              welded to one loss.

GBM (sklearn) any differentiable L. Fits the negative
              gradient. The reference implementation.

XGBoost       adds SECOND-ORDER gradients (uses the
              curvature, not just the slope) and puts
              L1/L2 + tree complexity INSIDE the
              objective, so pruning is principled.

LightGBM      changes how h_m is GROWN: histogram
              binning of features, leaf-wise (not
              depth-wise) splits. Same maths, much
              less of it.

CatBoost      changes what the target statistic sees:
              ORDERED boosting, so a row's encoding
              never uses its own label. Native
              categorical handling.

HistGradient  sklearn's port of LightGBM's ideas.
Boosting*     No extra install. Use this before
              reaching for XGBoost.

The Decision Tree

CHOOSING A BOOSTER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Is your data tabular?
  |
  +- No  -> stop. Boosting is not the tool.
  |         Images/audio/text want neural nets.
  |
  +- Yes
      |
      +- Under ~10k rows?
      |    -> HistGradientBoosting or GBM.
      |       Tuning matters more than the library.
      |
      +- 10k - 10M rows?
      |    -> HistGradientBoosting first.
      |       XGBoost if you need its ecosystem.
      |
      +- Over ~10M rows, or many wide categoricals?
           |
           +- lots of high-cardinality categoricals
           |    -> CatBoost. Ordered boosting exists
           |       precisely for this leak.
           |
           +- otherwise -> LightGBM.

Never start with AdaBoost. It is the ancestor,
not the answer. Read it to understand the family.

Hyperparameters That Matter, In Order

RANK  PARAMETER          START      NOTES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1     learning_rate      0.05       lower = slower
                                    AND better, until
                                    it isn't

1     n_estimators       ceiling    NOT a target. Use
      (same parameter                early stopping.
       as the rate)                  lr * n is the
                                    real quantity.

2     max_depth          3          2-6. This is the
                                    interaction order.
      (or num_leaves     31         LightGBM's version;
       for LightGBM)                 keep < 2^depth)

3     min_samples_leaf   20         raise on noise
      min_child_weight              (XGB/LGBM name)

4     subsample          0.8        rows per round.
                                    Nearly free win.

5     colsample_bytree   0.8        columns per tree.
                                    Helps when wide.

6     reg_lambda         1.0        L2. XGB/LGBM only.
      reg_alpha          0          L1, for sparsity.

7     loss / objective   task       the highest-leverage
                                    line people never
                                    touch. See below.

The Losses Worth Knowing

LOSS              GRADIENT           USE WHEN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
squared_error     y - F              default. Clean
                                     labels only.

absolute_error    sign(y - F)        labels are dirty.
                                     MEASURED: 3% of
                                     labels corrupted
                                     took squared from
                                     0.42 R2; absolute
                                     held 0.95.

huber             y-F, clipped       middle ground.
                                     Got 0.68 on the
                                     same test.

log_loss          y - sigmoid(F)     classification.
                                     Gives calibrated
                                     probabilities.

quantile          asymmetric sign    you need a P90,
                                     not a mean.

Boosting vs Bagging, Settled

                    BOOSTING       RANDOM FOREST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
attacks             bias           variance
trees               shallow 2-6    deep, unpruned
built               sequentially   independently
more trees          CAN OVERFIT    never harmful
parallel over trees no             yes
tuning required     real           minimal
dirty labels        risky          tolerant
one-fit answer      no             yes
usually wins on     tabular        nothing, but it is
                                   never embarrassing

MEASURED: at lr=1.0 test R2 peaked after ONE tree
and fell to -0.31 by 800. A forest cannot do that.

Eight Ways a Boosted Model Goes Wrong

SYMPTOM                       -> LIKELY CAUSE / FIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Train loss 0, test terrible   -> too many rounds. Early
                                 stop. Boosting really
                                 does overfit.

Grid search takes forever     -> you are searching lr
                                 against n_estimators.
                                 They are one number.

Worse than a Random Forest    -> lr too high, or depth
                                 too deep. Try 0.05/3.

One outlier dominates         -> squared loss chasing
                                 it. Use absolute_error
                                 or huber.

Great CV, bad production      -> leakage, or grouped
                                 rows split randomly.
                                 GroupKFold.

Categorical feature ranks
suspiciously high             -> target-statistic leak.
                                 CatBoost, or encode
                                 out of fold.

Fit time scales badly         -> switch to histogram
                                 binning (LightGBM /
                                 HistGradientBoosting).

Predictions never exceed a
range you have seen           -> trees cannot
                                 extrapolate. Still
                                 true here. Ever.

Key Takeaways

One update rule explains the whole family. Each library changes one clause of it; none of them change the shape.
learning_rate and n_estimators are one parameter. Measured product held at 4.4 across three rates on classification while the tree count moved 300-fold.
Start at lr=0.05, max_depth=3, subsample=0.8, and early-stop. That configuration is competitive before you have tuned anything.
Boosting overfits and forests do not. This is the single most important operational difference between the two families.
The loss is a hyperparameter. Switching to absolute_error on 3%-corrupted labels moved R2 from 0.42 to 0.95.
Reach for HistGradientBoosting before XGBoost. It ships with sklearn, uses the same histogram trick, and needs no new dependency.

The One-Sentence Summary

Every booster on this page is the same four-line update — start from a constant, take the negative gradient of your chosen loss, fit a shallow tree to it, add a fraction of that tree — and the entire competitive history of XGBoost, LightGBM and CatBoost is three different answers to "which clause of those four lines is the bottleneck on my data?"

What's Next?

Week 2 recap — tomorrow, tying the boosting posts back to the forest ones.
XGBoost — second-order gradients and the regularised objective, in detail.
LightGBM — histogram binning and leaf-wise growth, measured.
CatBoost — ordered boosting and the target leak it fixes.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

Print this one. The symptom-to-fix table is the part I actually reread.

Questions? Ask in the comments — I read and respond to every one.

Which booster do you reach for by default, and when did you last check that it was still the right one? Mine was XGBoost for years out of habit, and HistGradientBoosting has quietly been enough for most of it. 🗿

Cheat sheets are useful in proportion to how much of the underlying thing you already understand. This one is short because the family is smaller than the marketing suggests — four lines, and a long argument about which of them to optimise.

learning_rate and n_estimators Are One Parameter, Not Two — Here's the Number That Proves It

Sachin Kr. Rajput — Wed, 05 Aug 2026 08:26:13 +0000

The One-Line Summary: Grid-searching learning_rate against n_estimators wastes most of your compute, because the boosting model that comes out depends almost entirely on their product — measured across seven learning rates, the optimal lr × n_estimators held at 4.4 on classification and converged to about 3.0 on regression, so the right move is to fix the rate as low as your patience allows and let early stopping pick the count.

The Grid You Should Stop Searching

This is the search space almost everyone writes first:

{"learning_rate": [0.3, 0.1, 0.03, 0.01],
 "n_estimators":  [100, 300, 1000, 3000]}

Sixteen combinations. Most of them are the same model fitted at different costs, and yesterday's shrinkage table already hinted at why: the peak moved to 1, 7, 27, and 95 trees as the rate fell 1.0, 0.3, 0.1, 0.03. Those products are 1.0, 2.1, 2.7, 2.9.

So I ran it properly.

The Measurement

600 rows, 40 features, 3 informative, noise 60 — a deliberately hard regression. For each rate, fit 2000 trees and record where held-out R2 actually peaked.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
                       noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)

print("  lr     peak trees   peak R2    lr*trees")
for lr in (1.0, 0.5, 0.3, 0.1, 0.05, 0.03, 0.01):
    m = GradientBoostingRegressor(n_estimators=2000, learning_rate=lr,
                                  max_depth=3, random_state=0).fit(A, a)
    st = np.array([r2_score(b, p) for p in m.staged_predict(B)])
    k = int(st.argmax()) + 1
    print(f"  {lr:<6} {k:>8}   {st[k-1]:>8.4f}   {lr*k:>8.2f}")

  lr     peak trees   peak R2    lr*trees
  1.0           1     0.1860       1.00
  0.5           2     0.2377       1.00
  0.3           7     0.2838       2.10
  0.1          27     0.2717       2.70
  0.05         48     0.2726       2.40
  0.03         95     0.2795       2.85
  0.01        309     0.2795       3.09

The tree count moves by a factor of 309. The product moves from 2.10 to 3.09 once you are at or below lr=0.3 — and it is converging, not drifting.

Classification is tidier still. 1500 rows, 15% label noise, log loss on held-out data:

  lr     peak trees  peak logloss  lr*trees
  1.0            2        0.4231      2.00
  0.3           15        0.3864      4.50
  0.1           44        0.3842      4.40
  0.03         147        0.3840      4.41

4.50, 4.40, 4.41. That is one number, measured three times.

The Direct Test

If the product is what matters, four models with the same product should agree.

  lr x trees = 3.0     fit time   test R2
  lr=0.3   n=10           0.029s   0.2549
  lr=0.1   n=30           0.083s   0.2703
  lr=0.03  n=100          0.273s   0.2786
  lr=0.01  n=300          0.830s   0.2790

They very nearly do — and the gaps say something useful. Dropping from 0.03 to 0.01 bought 0.0004 R2 for 3x the fit time. Dropping from 0.3 to 0.1 bought 0.0154 for 2.9x. The returns are real but they die fast, which is exactly why 0.05–0.1 is the range you see in every serious codebase.

What To Do Instead

Stop searching the rate against the count. Pick the rate from your time budget, then let the count be discovered:

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
                       noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)

m = GradientBoostingRegressor(
        learning_rate=0.05,
        n_estimators=5000,        # a ceiling, not a target
        max_depth=3,
        n_iter_no_change=30,      # stop when held-out loss stalls
        validation_fraction=0.15,
        random_state=0).fit(A, a)
print(m.n_estimators_)            # how many it actually used

Spend the search budget on max_depth, min_samples_leaf and subsample instead. Those change what the model can express. The rate mostly changes how long you wait to get there.

One caveat, because I measured this too. Early stopping is not free:

  lr=0.1   stopped at   44 trees  test R2 0.2158
  lr=0.03  stopped at   76 trees  test R2 0.2597

Both stopped short of the true peak (27 trees / 0.2717 and 95 / 0.2795). With only 420 training rows, a 15% validation slice is 63 rows and it is noisy enough to call "stalled" early. On small data, raise n_iter_no_change, raise validation_fraction, or pick the count with proper cross-validation. On tens of thousands of rows this stops being a problem.

Key Takeaways

lr × n_estimators is approximately invariant. Measured at 4.4 across three rates on classification and converging to ~3.0 on regression, while the tree count changed 300-fold.
The rule breaks at high rates. At lr=1.0 and 0.5 the product collapsed to 1.0 and the peak R2 was 0.186 and 0.238 versus 0.28 lower down. Very large steps overshoot the good solutions entirely — there is no count that rescues them.
Returns below lr=0.03 are negligible. 0.0004 R2 for 3x the compute. Somewhere around 0.05 is the honest default.
A rate × count grid is mostly duplicate work. Fix the rate, put a high ceiling on the count, and early-stop.
Verify early stopping on small data. It stopped at 44 trees when the peak was 27, and at 76 when the peak was 95 — a tiny validation slice cuts both ways.

The One-Sentence Summary

Because each boosting round adds learning_rate × tree to a running sum, halving the rate and doubling the rounds lands in nearly the same place — so learning_rate and n_estimators are two dials wired to one shaft, and the only thing you gain by turning them independently is a longer grid search.

What's Next?

The boosting family cheat sheet — tomorrow. Every knob from this series, plus a flowchart for choosing between AdaBoost, GBM, XGBoost, LightGBM and CatBoost.
XGBoost — second-order gradients and regularisation written into the objective.
LightGBM — histogram binning and leaf-wise growth.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If you have a rate × count grid running right now, kill it and put the compute on max_depth.

Questions? Ask in the comments — I read and respond to every one.

What is the most expensive redundant grid search you have shipped? Mine ran 96 configurations to discover something a single staged-prediction plot would have shown in one fit. 🗿

Boosting is a sum, and the shape of a sum tells you which of its parameters are really independent. Most of my tuning mistakes have come from treating a formula's arguments as separate knobs when the formula had already welded them together.

Gradient Boosting: The Sculptor Who Fixes the Statue by Carving Only Its Mistakes

Sachin Kr. Rajput — Wed, 05 Aug 2026 07:42:33 +0000

The One-Line Summary: Gradient Boosting builds a model as a running sum, where each new small tree is trained not on your labels but on how wrong the sum currently is, and only a fraction of its opinion is added — which turns "fit a model" into "walk downhill in the space of functions," and lets you swap the loss function to change what the algorithm even considers a mistake.

The Parable of the Stoneyard at Halvad

The town of Halvad commissioned a statue of its founder, and the stoneyard sent twelve carvers to bid on the work. Each carver wanted his own block of marble. Each promised a finished figure in ninety days.

The town council, which had no money and less patience, asked whether there was another way.

The Old Way: Twelve Blocks, Twelve Statues

For three generations the yard had worked one way. Give every carver a block, let each work alone, then set the twelve figures in a row and let the council vote for the one they liked best — or, in a famous compromise of 1891, have a stonemason chisel a thirteenth figure that split the difference between all twelve.

It worked. It was expensive, and it had one stubborn flaw.

TWELVE CARVERS, ONE DRAWING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All twelve learned from the same master drawing.
So all twelve made the SAME error at the brow.

  carver 1  brow 4mm too shallow
  carver 2  brow 4mm too shallow
  carver 3  brow 3mm too shallow
  ...
  carver 12 brow 4mm too shallow
  ------------------------------------------
  average   brow ~4mm too shallow

Averaging twelve independent slips cancels them.
Averaging twelve IDENTICAL slips cancels nothing.

Adding a thirteenth carver does not help.
This is the CORRELATION FLOOR, and it is a wall.

Devika, who ran the yard, had spent her career on the other side of that wall. Hire more carvers, get more independence, cancel more slips. She had squeezed everything out of it that could be squeezed.

Her apprentice Anay proposed something the yard considered close to heresy.

The Innovation: Stop Carving Statues

"Why is every carver making a statue? Only the first one needs to. Everyone after that should be carving the difference."

His plan was this. Take one block. Let the first carver rough out something crude — barely a figure, a suggestion of a head and shoulders. Then set the clay model beside it and hand the second carver a caliper instead of a design.

The second carver's job is not to make a statue. It is to walk around the block, measure where stone still stands proud of the model, and remove exactly that much. Nothing else. He never looks at the design. He only looks at the gap.

Then a third carver does the same against the new, smaller gap. Then a fourth.

THE CALIPER READINGS, ROUND BY ROUND
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Eight measuring points on the block. The number is
how far the stone still sits from the model, and
"total error" is the sum of the squared gaps.

start   flat block, cut to average height
        gaps: -9.5 -8.5 -6.5 -2.5 0.5 2.5 9.5 14.5
        total error 518.00

round 1 one cut, the coarsest one available
        total error 518.00 -> 134.00

round 2 gaps: -5.5 -4.5 -2.5 1.5 4.5 6.5 -2.5 2.5
        total error 134.00 -> 50.67

round 3 gaps: -1.3 -0.3 1.7 -1.0 2.0 4.0 -5.0 0.0
        total error  50.67 -> 34.00

round 4 total error  34.00 -> 24.61

Nobody carved a statue. A statue appeared.

Each carver is nearly useless alone. The second one's work, taken by itself, is a few gouges in a rock. But the sum of the block and all the corrections is a figure, and it gets closer every round.

The Part That Went Wrong First

Anay's first attempt destroyed a block, and the reason is the most important thing in this article.

His rule had been: remove everything the caliper says. Full correction, every round. On good marble it worked. On a block with a hidden flaw, the caliper read 9mm at a point where the stone was rotten, the carver took the full 9mm, the surface crumbled past the line, and the next carver read a negative gap and had nothing to add back.

Devika's fix was a rule so cautious it sounded like a joke.

"Take a tenth of what the caliper tells you. Then measure again."

Ten carvers now do the work of one. The figure emerges slowly, and it never overshoots, because no single measurement is ever trusted enough to matter on its own.

FULL TRUST vs A TENTH
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gap reads 9mm at a flawed point

FULL CORRECTION      take 9mm. Surface fails.
                     Gap is now -3mm and there is
                     no putting marble back.

A TENTH              take 0.9mm. Re-measure: 8.1mm.
                     Take 0.9mm. The flaw shows up
                     in the readings long before
                     you have cut through it.

Cost: 10x the passes.
Benefit: no cut is ever a commitment.

The Second Thing That Went Wrong

The clay model had a chipped ear. Nobody noticed for a week.

A caliper reports depth — how far off, in millimetres. At the chipped ear it reported an enormous gap, so every carver in sequence attacked the ear hardest, because that is where the biggest numbers were. They carved a beautiful, faithful reproduction of the damage.

Anay's fix was to change the instrument, not the method. The new instrument reported only direction: proud, or shy. No magnitude. A point that is 90mm off and a point that is 0.9mm off both come back as "proud, take a sliver."

TWO INSTRUMENTS, SAME METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gaps at four points:  -9.5  -2.5  0.5  14.5

DEPTH CALIPER        -9.5  -2.5  0.5  14.5
  reports magnitude. One bad reading dominates
  the whole round.

DIRECTION CALIPER    -1    -1    1    1
  reports only sign. A chipped ear gets the same
  attention as a fingernail.

Same carvers. Same procedure. Different definition
of "mistake" -- and that choice is a PARAMETER.

That is the whole idea. What the caliper measures is the algorithm's definition of a mistake, and you get to choose it.

What Is Gradient Boosting?

Hard pivot. Everything above has a name.

Gradient Boosting builds a prediction as a running sum of small models:

F_{M} (x) = F_{0} (x) + ν m = 1 \sum M h_{m} (x)

where $F_{0}$ is a constant (the mean of $y$ for squared error — the flat block), each $h_{m}$ is a shallow tree, and $ν \in (0, 1]$ is the learning rate, Devika's tenth.

The trick is what $h_{m}$ is trained on. Not $y$ . It is trained on the negative gradient of the loss with respect to the current prediction:

r_{i (m)} = - [\frac{\partial L ( y _{i} , F ( x _{i} ))}{\partial F ( x _{i} )}] F = F m - 1

These are the pseudo-residuals, and they are the caliper reading.

Why "Gradient"

Gradient descent on parameters updates $θ \leftarrow θ - η \nabla_{θ} L$ . Gradient boosting does the same thing one level up: the object being optimised is the function $F$ itself, evaluated at your $n$ training points.

Treat $(F (x_{1}), \dots, F (x_{n}))$ as a point in $R^{n}$ . The loss is a surface over that space. The steepest downhill direction is $- \nabla L$ , which is exactly $r^{(m)}$ above.

But you cannot just take that step, because it only tells you what to do at the $n$ points you have — it says nothing about a new $x$ . So you fit a tree to the gradient. The tree is a generalisable approximation of the downhill direction. Then you take a small step along it.

Boosting is gradient descent where each step has to be a model.

Where the Residuals Come From

For squared error, $L = \frac{1}{2} (y - F)^{2}$ :

- \frac{\partial L}{\partial F} = y - F

The negative gradient is the ordinary residual. This is why every tutorial explains gradient boosting as "fit the residuals" — for one specific loss, that happens to be true. It is a coincidence of squared error, not the definition.

For absolute error, $L = ∣ y - F ∣$ :

- \frac{\partial L}{\partial F} = sign (y - F)

Now the target is $\pm 1$ . Magnitude is gone. That is the direction caliper, and it is not a metaphor — it is the derivative.

working residual r      = [-9.5 -2.5  0.5 14.5]
squared_error  -dL/dF   = [-9.5 -2.5  0.5 14.5]
absolute_error -dL/dF   = [-1. -1.  1.  1.]

For log loss on binary classification the gradient is $y_{i} - p_{i}$ where $p_{i} = σ (F (x_{i}))$ — a probability error, in the space of log-odds. Same machinery, third instrument.

Step by Step

Set $F_{0} (x) = ar g min_{c} \sum_{i} L (y_{i}, c)$ . (Mean for squared error, median for absolute.)
For $m = 1, \dots, M$ :
1. Compute pseudo-residuals $r_{i (m)}$ for every training row.
2. Fit a shallow regression tree $h_{m}$ to $(x_{i}, r_{i (m)})$ .
3. Optionally re-solve each leaf's value to minimise the actual loss inside that leaf, rather than using the mean gradient.
4. Update $F_{m} = F_{m - 1} + ν h_{m}$ .
Return $F_{M}$ .

Step 2.3 is the one nobody mentions. For squared error it changes nothing — the leaf mean already minimises squared loss. For every other loss it matters, and it is why sklearn's absolute-error boosting is not simply "fit trees to signs."

Gradient Boosting with Scikit-Learn

The residual sequence from the parable, with real numbers:

import numpy as np
from sklearn.tree import DecisionTreeRegressor

X = np.array([[1.],[2.],[3.],[4.],[5.],[6.],[7.],[8.]])
y = np.array([2.0, 3.0, 5.0, 9.0, 12.0, 14.0, 21.0, 26.0])

F = np.full_like(y, y.mean())          # the flat block
print(f"F0 = mean(y) = {y.mean():.4f}")
for m in range(1, 5):
    r = y - F                          # the caliper reading
    t = DecisionTreeRegressor(max_depth=1).fit(X, r)
    F = F + 1.0 * t.predict(X)         # full correction, no shrinkage
    print(f"round {m}: residuals {np.round(r, 2)}")
    print(f"         SSE before {np.sum(r**2):.4f} -> "
          f"after {np.sum((y - F)**2):.4f}")

F0 = mean(y) = 11.5000
round 1: residuals [-9.5 -8.5 -6.5 -2.5  0.5  2.5  9.5 14.5]
         SSE before 518.0000 -> after 134.0000
round 2: residuals [-5.5 -4.5 -2.5  1.5  4.5  6.5 -2.5  2.5]
         SSE before 134.0000 -> after 50.6667
round 3: residuals [-1.33 -0.33  1.67 -1.    2.    4.   -5.    0.  ]
         SSE before 50.6667 -> after 34.0000
round 4: residuals [-2.17 -1.17  0.83 -1.83  1.17  3.17 -2.5   2.5 ]
         SSE before 34.0000 -> after 24.6111

Four depth-1 stumps — the weakest models in the library — took the error from 518 to 24.6. Look at round 4's residuals: some got worse than round 3 (point 1 went from -1.33 to -2.17) while the total fell. Boosting does not improve every point every round. It improves the sum.

Gradient Boosting from Scratch

Sixty lines, no gradient boosting import:

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=800, n_features=10, n_informative=6,
                       noise=10.0, random_state=7)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=7)


class MyGradientBoosting:
    """Squared-error gradient boosting. The whole algorithm."""

    def __init__(self, n_estimators=200, learning_rate=0.1, max_depth=3):
        self.n = n_estimators
        self.lr = learning_rate
        self.depth = max_depth

    def fit(self, X, y):
        self.F0 = y.mean()                       # step 1
        F = np.full(len(y), self.F0)
        self.trees = []
        for _ in range(self.n):                  # step 2
            g = y - F                            # 2.1  -dL/dF
            t = DecisionTreeRegressor(
                max_depth=self.depth,
                criterion="friedman_mse",
                random_state=0).fit(X, g)        # 2.2
            F += self.lr * t.predict(X)          # 2.4
            self.trees.append(t)
        return self

    def predict(self, X):
        F = np.full(len(X), self.F0)
        for t in self.trees:
            F += self.lr * t.predict(X)
        return F


mine = MyGradientBoosting().fit(Xtr, ytr)
sk = GradientBoostingRegressor(n_estimators=200, learning_rate=0.1,
                               max_depth=3, random_state=0).fit(Xtr, ytr)
pm, ps = mine.predict(Xte), sk.predict(Xte)

print(f"from scratch R2 = {r2_score(yte, pm):.6f}")
print(f"sklearn      R2 = {r2_score(yte, ps):.6f}")
print(f"correlation between the two = {np.corrcoef(pm, ps)[0, 1]:.8f}")
print(f"y std in the test set = {yte.std():.1f}")
print(f"RMSE between the two models = {np.sqrt(((pm - ps)**2).mean()):.6f}")

from scratch R2 = 0.970898
sklearn      R2 = 0.970890
correlation between the two = 0.99998337
y std in the test set = 108.4
RMSE between the two models = 0.584202

Twenty lines of algorithm, matching the library to five decimal places of R2.

They are not bit-identical, and the reason is worth a paragraph. I checked where they part company:

 n_trees   max|diff|    R2 mine     R2 sklearn
     1      0.000000   0.131791    0.131791
     2      0.000000   0.240005    0.240005
     5      0.000000   0.475405    0.475405
    10      0.000000   0.683293    0.683293
    50      5.615992   0.947227    0.947312
   200      5.615992   0.970898    0.970890

Identical for the first ten rounds. First divergence at round 23, affecting 1 test row out of 240, by 5.6 on a target with a standard deviation of 108 — one split hit a tie between two equally good thresholds and my tree's random state broke it the other way. Then it never grew, because everything after round 23 is fitting residuals that both models agree about.

That is what a correct reimplementation looks like. Not zero difference — a difference you can name.

Shrinkage Is Not Optional

The learning rate is the single most consequential number in boosting. Here is what it does on a deliberately hard problem: 600 rows, 40 features, only 3 informative, noise 60.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
                       noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)

for lr in (1.0, 0.3, 0.1, 0.03):
    m = GradientBoostingRegressor(n_estimators=800, learning_rate=lr,
                                  max_depth=3, random_state=0).fit(A, a)
    st = np.array([r2_score(b, p) for p in m.staged_predict(B)])
    k = st.argmax()
    print(f"lr={lr:<5} peak test R2 {st[k]:.4f} @ {k+1:>3} trees | "
          f"@800 {st[-1]:.4f} | lost {st[k]-st[-1]:.4f}")

lr=1.0   peak test R2 0.1860 @   1 trees | @800 -0.3134 | lost 0.4994
lr=0.3   peak test R2 0.2838 @   7 trees | @800 0.1667 | lost 0.1171
lr=0.1   peak test R2 0.2717 @  27 trees | @800 0.1805 | lost 0.0911
lr=0.03  peak test R2 0.2795 @  95 trees | @800 0.2237 | lost 0.0558

Three things in that table.

Boosting genuinely overfits. At lr=1.0 the model peaks after a single tree and then falls to R2 = -0.31 — worse than predicting the mean for every row. This is the sharpest difference from Random Forests, where adding trees is never harmful. Boosting is not an ensemble that averages away error; it is an optimiser, and optimisers can converge on noise.

Smaller steps land in a better place. Peak R2 goes 0.186 at lr=1.0 to 0.284 at lr=0.3. Full-trust corrections leap over the good solutions.

Smaller steps also decay more gently. Going from peak to 800 trees costs 0.4994 at lr=1.0 and 0.0558 at lr=0.03. Low learning rates are forgiving about where you stop.

And the peak moves in a suspiciously regular way: 1, 7, 27, 95 trees as the rate falls 1.0, 0.3, 0.1, 0.03. That relationship is tomorrow's whole article.

Changing the Loss Changes the Model

The chipped ear, measured. Same algorithm, same trees, same rate — 3% of training labels corrupted by +800, scored against clean test labels.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

X, y = make_regression(n_samples=800, n_features=12, n_informative=6,
                       noise=8.0, random_state=3)
rng = np.random.default_rng(3)
y_bad = y.copy()
y_bad[rng.choice(len(y), 24, replace=False)] += 800.0      # 3% corrupted

Xtr, Xte, ytr, _ = train_test_split(X, y_bad, test_size=0.3, random_state=3)
*_, yte_clean = train_test_split(X, y, test_size=0.3, random_state=3)

print("loss             test MAE (vs clean)   test R2")
for loss in ("squared_error", "absolute_error", "huber"):
    m = GradientBoostingRegressor(loss=loss, n_estimators=300,
                                  learning_rate=0.1, max_depth=3,
                                  random_state=0).fit(Xtr, ytr)
    p = m.predict(Xte)
    print(f"{loss:<16} {mean_absolute_error(yte_clean, p):>13.4f}   "
          f"{r2_score(yte_clean, p):>9.4f}")

loss             test MAE (vs clean)   test R2
squared_error          48.9937      0.4223
absolute_error         19.6481      0.9535
huber                  28.2967      0.6838

R2 went from 0.42 to 0.95 by changing one string. Not more trees, not tuning, not more data — a different definition of "mistake." Squared error chased 24 bad labels and wrecked the model for the other 776. Absolute error saw them as "proud, take a sliver" and moved on.

This is the practical payoff of the gradient framing. AdaBoost was welded to exponential loss. Gradient boosting takes any differentiable loss you can write down, which is why the same three lines handle regression, classification, ranking, and quantile prediction.

When Does Gradient Boosting Help Most?

	Gradient Boosting	Random Forest
Attacks	bias, sequentially	variance, in parallel
Trees	shallow, depth 2–6	deep, unpruned
More trees	can overfit badly	never harmful
Parallel over trees	no, inherently serial	yes
Tuning needed	real; lr × depth × n interact	minimal
Noisy labels	dangerous with squared loss	comparatively robust
Typical tabular winner	yes	rarely

Reach for boosting when you have structured tabular data, a clean-ish target, and the time to tune. Reach for a forest when you want a strong answer in one fit with no thinking. Reach for neither when your features are pixels, audio, or text.

Hyperparameters

Parameter	What it does	Typical range	How to tune
`learning_rate`	fraction of each correction applied	0.01–0.1	set low, then find `n_estimators`
`n_estimators`	number of rounds	100–5000	early stopping, never a fixed guess
`max_depth`	interaction order per tree	2–6	3 is a real default, not a placeholder
`subsample`	rows per round (stochastic GB)	0.5–1.0	0.8 adds variance reduction, nearly free
`min_samples_leaf`	leaf floor	1–20	raise on noisy data
`loss`	definition of a mistake	see above	`absolute_error` when labels are dirty
`n_iter_no_change`	early-stopping patience	10–50	set it; do not tune `n_estimators` by hand
`validation_fraction`	rows held out for stopping	0.1	leave it

Quick Reference Card

GRADIENT BOOSTING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  A running sum of shallow trees, each fit to the
  negative gradient of the loss at the current sum,
  each added at only a fraction of its full value.

THE UPDATE:
  F_0 = argmin_c sum L(y_i, c)
  r_i = -dL(y_i, F(x_i)) / dF(x_i)
  h_m = tree fit to (x_i, r_i)
  F_m = F_(m-1) + lr * h_m

WHAT THE GRADIENT IS:
  squared_error   ->  y - F        (the residual)
  absolute_error  ->  sign(y - F)
  log_loss        ->  y - sigmoid(F)

KEY HYPERPARAMETERS:
  learning_rate     0.01-0.1  (lower = more trees)
  n_estimators      set by early stopping
  max_depth         2-6
  subsample         0.8
  loss              squared / absolute / huber

WHEN TO USE:
  + tabular data with real structure
  + you can afford to tune
  + you need the last two points of accuracy
WHEN NOT EFFECTIVE:
  - dirty labels + squared loss
  - you cannot early-stop
  - images, audio, raw text
  - you need one fit and no tuning (use a forest)

SKLEARN:
  from sklearn.ensemble import GradientBoostingRegressor
  from sklearn.ensemble import GradientBoostingClassifier
  # for anything real, use HistGradientBoosting* instead

Key Takeaways

The new tree is fit to the gradient, not to your labels. "Fit the residuals" is true only for squared error, where the gradient happens to equal the residual.
Boosting is gradient descent in function space. Each step must be a model, because the raw gradient only exists at your training points and cannot generalise to a new row.
The learning rate is not a speed knob, it is a regulariser. At lr=1.0 the test R2 peaked after one tree and fell to -0.31 by 800. At lr=0.03 it peaked at 95 and lost a tenth as much.
Boosting can overfit; forests essentially cannot. More trees is a free improvement in a Random Forest and a real risk here. Early stopping is not optional.
The loss function is a hyperparameter with more leverage than most. Corrupting 3% of labels cost squared error 0.53 R2; switching to absolute_error recovered it, with no other change.
Shallow trees are the point. Depth 3, not depth None. Each tree only has to find one more slice of the remaining error, and a deep tree would take the whole gradient in one bite.
Sixty lines reproduce the library. If you can write the update loop from memory, you know where every hyperparameter acts — and you will believe the results more.
Nothing in this algorithm is parallel over trees. Tree $m$ needs $F_{m - 1}$ . All the speed in XGBoost and LightGBM comes from parallelising inside a tree.

The One-Sentence Summary

Gradient Boosting is the sculptor's insight that only the first worker needs to make a statue and everyone after should carve nothing but the measured gap — formalised as gradient descent in the space of functions, where the step direction is the negative derivative of whatever loss you choose, the step itself must be a shallow tree so it generalises off the training rows, and the step size is deliberately tiny so that no single measurement is ever trusted enough to cut through the stone.

What's Next?

Now that you understand the update rule, you're ready for:

learning_rate and n_estimators are one parameter — tomorrow. The peaks in that shrinkage table landed at 1, 7, 27, and 95 trees for rates of 1.0, 0.3, 0.1, 0.03. That is not a coincidence, and it changes how you search.
XGBoost — second-order gradients, regularisation moved into the objective, and the engineering that won a decade of Kaggle.
LightGBM — histogram binning and leaf-wise growth, for when 800 trees over ten million rows has to finish today.
CatBoost — ordered boosting, which fixes a target leak the other two quietly live with.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the caliper made gradient boosting click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever fixed a model by changing the loss instead of the features? It is the highest-leverage line in my file and the last one I ever think to touch. 🗿

The parable has a detail I did not invent for effect: the carvers who trusted their calipers completely did the most damage. Gradient boosting encodes that as a number between 0.01 and 0.1, and every time I have been tempted to raise it to save wall-clock time, the model has been worse in a way that took me a week to notice.

The Random Forest Series in One Page: Six Posts, One Number That Explains All of Them

Sachin Kr. Rajput — Wed, 05 Aug 2026 07:40:04 +0000

The One-Line Summary: Six posts on Random Forests came down to a single quantity — tree-to-tree correlation — because the variance of an average has a floor at $ρ σ^{2}$ that no number of trees can lower, which is why every knob that helped (columns per split, random thresholds) worked by attacking $ρ$ , why the regression default quietly leaves that knob switched off, and why the importance scores and the free validation score both lie in ways that trace back to the same place.

The Number Underneath All Six Posts

The Random Forests series is done. Here is the whole thing compressed, in the order you'd want to read it, plus the one idea from each post worth keeping.

If you take a single thing away, take this:

THE VARIANCE OF AN AVERAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Var(average) = ρσ²  +  (1-ρ)σ²/N
               │           │
               │           └─ shrinks to 0 with more trees
               └───────────── NEVER shrinks. Ever.

More trees cannot fix correlation.
Only DIVERSITY can fix correlation.

Every post below is a consequence of this line.

1. Random Forest — the blindfolded jury

Bagging plus one restriction: at every split, each tree may only consider a random subset of columns. That handicaps each tree and strengthens the committee, because averaging only cancels errors that point in different directions.

Measured across five generated datasets: raising max_features made individual trees monotonically better (0.727 → 0.848) and tree-to-tree correlation monotonically worse (0.311 → 0.644). Ensemble accuracy peaked in between, on a broad plateau around 6 of 20 features — not exactly on sqrt ≈ 4.47.

Keep: a committee's reliability depends less on how good each member is than on how differently they are wrong.

2. Feature Importance — the company that fired the wrong employee

feature_importances_ sums impurity decrease over the training rows. It pays for opportunity, not value: a column with many possible split points gets many chances to look useful.

A column of random IDs beat a genuinely predictive binary flag, 0.5602 to 0.2722. Pure noise climbed from 0.18% to 57.17% importance on nothing but cardinality.

Keep: permutation importance on held-out rows measures damage rather than activity. And when two columns do the same job, report the group, not the winner — a ranking that flips between runs is the finding.

3. Out-of-Bag Error — the free exam

Each bootstrap sample leaves out $(1 - 1/ n)^{n} \to 1/ e \approx 36.8%$ of rows. That is a limit, not a rule of thumb — measured 0.3679 at n = 10,000. Score each row using only the trees that never saw it and you get validation for the price of one fit.

Measured: OOB 0.9021, 5-fold 0.9036, held-out test 0.8983 — estimates 0.0015 apart, for roughly 5× less wall clock.

Keep: it lies in three specific situations. Too few trees (at B=5, OOB said 0.7829 for a 0.8750 model). Grouped rows — five visits per patient pushed oob_score_ to a perfect 1.0000 against a real 0.8620 under GroupKFold. And heavy imbalance, where 0.9710 accuracy hid 0.1944 recall. Use n_estimators >= 100 or don't quote the number.

4. Extra Trees — the chef who stopped tasting

One change: the split threshold is drawn, not searched. One uniform draw per candidate feature; the impurity criterion still picks the winner among them.

That is the third randomness knob — rows, columns, thresholds — and the ladder is measurable: $ρ$ = 0.640, 0.571, 0.515. Single-tree accuracy fell to 0.8034, the worst of the three, while the ensemble rose to 0.9050, the best of the three. Same trade as post 1, one rung further down. It is also faster, because nothing ever gets sorted.

Keep: you pay in bias, and bias does not average away. Variance reduction compounds over trees; per-tree bias is permanent. That is why the trade has limits.

5. The `max_features` default — and the fix that makes it worse

RandomForestClassifier defaults to 'sqrt'. RandomForestRegressor defaults to 1.0 — every column, every split. Verified from the constructor signature on scikit-learn 1.7.2.

So the regressor ships as plain bagged trees with the diversity knob wide open, and the internet's advice is to copy 'sqrt' across. Measured across three dataset shapes, that advice is backwards. The default won on $R^{2}$ every time (0.8723 / 0.8940 / 0.6828); 'sqrt' lost every time, by 21 points when only 5 of 60 features carried signal. Correlation fell to 0.248 there — the lowest of any setting, and the worst model.

Keep: decorrelation is a means, not a goal. A classifier only needs the majority right, so it can spend per-tree accuracy on diversity. A regressor averages the actual numbers, so per-tree bias goes straight into the mean and never averages away. The default's real cost is ~2x fit time; 0.5 is statistically tied and meaningfully faster.

6. The Cheat Sheet

No new theory. The three knobs, hyperparameter starting ranges split by classification versus regression, a flowchart for choosing an importance method, the RF vs Bagging vs Extra Trees table, and a symptom-to-fix list for eight ways a forest goes wrong in production.

What Changed My Mind

The max_features post was supposed to be a takedown. I expected the regression default to cost real accuracy. It was the most accurate setting on all three shapes I tested. The post now argues the opposite of its first draft, and says so.

'sqrt' is not the answer on regression — it is the trap. I had assumed the classifier default was simply the better choice everywhere. It was the worst of four settings on every shape, catastrophically so on a wide frame with sparse signal. Lowest correlation, worst model: proof that diversity is a cost as well as a benefit.

The Habit Worth Stealing

Average over several seeds before believing any tuning result. A single train/test split moved by as much as the effects I was chasing — half a percent — which means most "setting X is better" claims you'll read, including ones I nearly published, are noise wearing a lab coat.

What's Next

Boosting is already underway. Where forests fight variance by averaging independent trees, boosting attacks bias by training them in sequence, each one paid to fix the last one's mistakes:

AdaBoost — reweight what you got wrong, then weight each learner's vote by how reliable it turned out to be.
Gradient Boosting — that idea generalised: each tree fits the gradient of the loss.
XGBoost — the engineering that made boosting win competitions.
LightGBM, CatBoost, early stopping — and an honest answer to "boosting or forests?" on tabular data.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the correlation floor is the thing that finally made forests click, drop a heart.

Questions? Ask in the comments — I read and respond to every one.

Which of the six surprised you most? For me it was a column of random IDs outranking a genuinely useful flag by two to one, on defaults everybody uses. 🌲

Six posts about forests taught me something about writing: the parts I assumed were too obvious to include turned out to need a diagram, and the parts I expected to be a takedown turned out to be a tie. max_features=1.0 was a footnote in my first draft of post one. It became its own post, then it changed its own conclusion.

Share this with someone who is still adding trees to fix a correlation problem.

The Random Forest Cheat Sheet: Everything on One Page

Sachin Kr. Rajput — Wed, 05 Aug 2026 07:36:33 +0000

The One-Line Summary: Every Random Forest decision from this series compressed onto one printable page — the three randomness knobs and what each one costs, starting hyperparameter ranges split by classification versus regression, which importance method to trust and when, the RF vs Bagging vs Extra Trees table, and a symptom-to-fix list for the eight ways a forest goes wrong in production.

The One Line That Explains Everything Else

THE VARIANCE OF AN AVERAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Var(average) = rho*s2  +  (1-rho)*s2/N
               |             |
               |             +- shrinks to 0 with N
               +--------------- NEVER shrinks

More trees cannot fix correlation.
Only diversity can fix correlation.
But diversity is bought with per-tree accuracy,
and on regression that bill comes due.

The Three Randomness Knobs

KNOB              WHAT IT RANDOMISES        MEASURED rho
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. bootstrap      which ROWS each tree          0.640
                  sees (with replacement)

2. max_features   which COLUMNS a split         0.571
                  may consider                  (sqrt)

3. random splits  which THRESHOLD gets          0.515
                  tried (Extra Trees)

Each rung lowers correlation and lowers
per-tree accuracy. The ensemble peaks in the
middle, not at either end.

Hyperparameters, With Starting Ranges

PARAMETER          CLASSIFICATION      REGRESSION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
n_estimators       300+                300+
                   (not a regulariser - more is
                    never worse, only slower)

max_features       'sqrt' (default)    1.0 (default)
                   try 0.3-0.5         try 0.5-0.7
                   NEVER 1.0 blindly   NEVER 'sqrt'

max_depth          None                None
min_samples_leaf   1-5                 5-20
max_samples        None, or 0.5-0.8 to add diversity
class_weight       'balanced_subsample'    n/a
n_jobs             -1                  -1
random_state       always set it       always set it

The single highest-value line in that table: max_features is the only knob that moves correlation, and its right value differs by problem type. Everything else is comparatively minor.

Which Importance Method?

CHOOSING AN IMPORTANCE METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Do you need a number you would defend
in a review?
  |
  +- No, just a rough sense
  |     -> feature_importances_ (MDI), but know
  |        it is a TRAINING statistic and it
  |        rewards high-cardinality columns
  |
  +- Yes
        |
        +- Are features correlated?
             |
             +- No  -> permutation importance
             |         on HELD-OUT rows,
             |         n_repeats >= 10
             |
             +- Yes -> permutation on held-out rows,
                       then report the GROUP, not
                       the winner. Correlated columns
                       split the credit and the
                       ranking flips on noise.

MEASURED: a column of random IDs scored 0.5602
on MDI against 0.2722 for a genuinely predictive
binary flag. Pure noise went 0.18% -> 57.17%
importance on cardinality alone.

RF vs Bagging vs Extra Trees

                    BAGGING      RF          EXTRA TREES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
rows bootstrapped     yes        yes          optional
columns per split     ALL        subset       subset
threshold             best       best         DRAWN
sorting needed        yes        yes          no
speed                 slow       medium       fastest
measured rho          0.640      0.571        0.515
single-tree acc       0.8342     0.8204       0.8034
ensemble acc          0.8967     0.8983       0.9050

Bagging = RF with max_features off.
Extra Trees = RF with one more knob turned.

Out-of-Bag: Free Validation, Three Traps

OOB IN ONE BLOCK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

(1 - 1/n)^n -> 1/e = 36.8% of rows are out-of-bag
for any given tree. Measured 0.3679 at n=10,000.

Score each row with only the trees that never
saw it: validation for the price of ONE fit.

MEASURED:  OOB 0.9021 | 5-fold 0.9036 | test 0.8983
           ~5x less wall clock than 5-fold

WHERE IT LIES:
  x too few trees    B=5 -> OOB 0.7829 for a
                     0.8750 model. Use B >= 100.
  x grouped rows     5 visits/patient -> OOB
                     1.0000 vs GroupKFold 0.8620
  x heavy imbalance  0.9710 accuracy hiding
                     0.1944 recall

Eight Ways a Forest Goes Wrong

SYMPTOM                        -> LIKELY CAUSE / FIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Great CV, bad production       -> leakage, or grouped
                                  rows split randomly.
                                  Use GroupKFold.

Adding trees changes nothing   -> you are at the rho
                                  floor. Lower
                                  max_features.

Importances differ every run   -> random_state unset,
                                  or correlated
                                  columns tied.

An ID column ranks top         -> MDI cardinality bias.
                                  Switch to permutation
                                  on held-out rows.

oob_score_ = 1.0               -> non-independent rows.
                                  It is not that good.

High accuracy, useless model   -> imbalance. Read
                                  recall / PR-AUC.

Predictions never exceed a     -> trees cannot
range you have seen               extrapolate. Ever.

Regressor slower than expected -> max_features=1.0
                                  default. Try 0.5.

Key Takeaways

Correlation is the ceiling. Everything else in this table is a way of buying diversity, and every purchase has a price.
max_features is the knob that matters — and its correct value differs between classification and regression. Do not copy one to the other.
n_estimators is not a regulariser. Set it high, stop thinking about it.
MDI importance is a training statistic that rewards cardinality. Permutation on held-out rows is the number you defend.
OOB is nearly free and usually accurate — until rows are grouped, trees are few, or classes are skewed.
Trees cannot extrapolate. No amount of tuning changes this.

The One-Sentence Summary

A Random Forest is three randomness knobs stacked on top of bagged trees, and tuning one is really a single decision repeated: how much per-tree accuracy will you trade for tree-to-tree independence — a trade that pays off handsomely on classification, has to be made carefully on regression, and is invisible in every metric except the one number, correlation, that no library reports to you.

What's Next?

The series recap — the one number that ties all six posts together.
Gradient Boosting — the other way to combine trees, attacking bias instead of variance.
XGBoost — the engineering that made boosting win competitions.

Follow me for the next article in the Random Forests Deep Dive series!

Let's Connect!

Print this one. It is the page I actually keep open while tuning.

Questions? Ask in the comments — I read and respond to every one.

What belongs on this page that I left off? The symptom-to-fix table is the part I add to most often. 🌲

Every line in this cheat sheet was a measurement before it was a rule. The ones I inherited from blog posts rather than experiments are exactly the ones that turned out to be wrong.

Don't Copy 'sqrt' Into Your Regressor: What max_features Actually Costs You, Measured

Sachin Kr. Rajput — Wed, 05 Aug 2026 07:31:42 +0000

The One-Line Summary: RandomForestClassifier defaults max_features to 'sqrt' and RandomForestRegressor defaults it to 1.0, and the common advice to "fix" the regressor by copying 'sqrt' across is the one move the numbers actually punish — across three dataset shapes the default won on accuracy every time, 'sqrt' lost by up to 21 points, and what the default really costs you is roughly 2x the fit time and the highest tree correlation of any setting.

Two Defaults, One Library

Check the constructor signatures rather than taking my word for it:

import inspect, sklearn
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

for cls in (RandomForestClassifier, RandomForestRegressor):
    d = inspect.signature(cls.__init__).parameters['max_features'].default
    print(f"{cls.__name__:<24} max_features={d!r}")
print("sklearn", sklearn.__version__)

RandomForestClassifier   max_features='sqrt'
RandomForestRegressor    max_features=1.0
sklearn 1.7.2

So the regressor considers every column at every split. Day one established why that matters: feature subsampling is the knob that decorrelates the trees, and correlation is the floor on the variance of an average. The regressor ships with that knob wide open — which is to say, as plain bagged trees.

The obvious conclusion is that you should close it. That conclusion is wrong, and it took measuring to find out.

The Measurement

Three dataset shapes, four settings, five seeds each, averaged. Correlation is the mean pairwise correlation between individual tree predictions on held-out rows.

MAX_FEATURES ON REGRESSION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

A: p=20, 6 informative, noise=12
  setting      R2       rho    fit s   beats dflt
      1.0   0.8723   0.696    0.48s       -
      0.5   0.8708   0.627    0.28s      1/5
     0.33   0.8473   0.515    0.22s      0/5
     sqrt   0.8036   0.402    0.20s      0/5

B: p=60, 5 informative, noise=8  (sparse signal)
      1.0   0.8940   0.745    1.17s       -
      0.5   0.8855   0.663    0.64s      1/5
     0.33   0.8562   0.542    0.48s      0/5
     sqrt   0.6831   0.248    0.27s      0/5  <- -21 pts

C: p=20, 18 informative, noise=3  (dense signal)
      1.0   0.6828   0.346    0.46s       -
      0.5   0.6820   0.318    0.31s      3/5
     0.33   0.6787   0.278    0.22s      1/5
     sqrt   0.6631   0.240    0.19s      0/5

Every shape: default best on R2, slowest, most correlated.
Every shape: 'sqrt' worst on R2.

Reproduce shape A:

import numpy as np, time
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

for s in [1.0, 0.5, 0.33, 'sqrt']:
    r2, rho, secs = [], [], []
    for sd in range(5):
        X, y = make_regression(n_samples=2000, n_features=20, n_informative=6,
                               noise=12.0, random_state=sd)
        Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=sd)
        t0 = time.perf_counter()
        m = RandomForestRegressor(n_estimators=150, max_features=s,
                                  random_state=42, n_jobs=-1).fit(Xtr, ytr)
        secs.append(time.perf_counter() - t0)
        r2.append(m.score(Xte, yte))
        P = np.array([t.predict(Xte) for t in m.estimators_])
        C = np.corrcoef(P)
        rho.append(np.nanmean(C[np.triu_indices_from(C, k=1)]))
    print(f"{str(s):>6}  R2={np.mean(r2):.4f}  rho={np.mean(rho):.3f}  "
          f"{np.mean(secs):.2f}s")

   1.0  R2=0.8723  rho=0.696  0.48s
   0.5  R2=0.8708  rho=0.627  0.28s
  0.33  R2=0.8473  rho=0.515  0.22s
  sqrt  R2=0.8036  rho=0.402  0.20s

Why Regression Is Not Classification Here

The decorrelation story from day one still holds — lowering max_features really does cut correlation, monotonically, in every shape above. What changes is the price.

A classifier only needs the majority to be right. Individual trees can be sloppy and the vote still lands, so you can spend a lot of per-tree accuracy on diversity and come out ahead.

A regressor averages the actual numbers. Every tree's error goes into the mean, undiluted. Starve a tree of columns and it splits on whatever it was handed, and that bias is baked into the average — and per-tree bias does not average away, however many trees you add. On shape B, 'sqrt' gave each split 4 of 60 columns with only 5 carrying signal, so most trees never saw a useful feature at all. Correlation fell to 0.248, the lowest of any setting, and R2 fell off a cliff with it.

Lowest correlation, worst model. That is the whole lesson: decorrelation is a means, not a goal.

What To Actually Do

Leave the regression default alone unless fit time hurts. It cost 2.0–2.4x the fit time of 0.33 in these runs, and that is the real bill. If you are refitting in a loop or tuning against the clock, 0.5 is the setting to reach for: within 0.002–0.009 R2 of the default in all three shapes, at roughly 1.7x the speed.

Never copy 'sqrt' across from the classifier. Worst of the four on every shape tested, and catastrophic when signal is sparse in a wide frame.

Tune it, don't inherit it. The peak is interior and shape-dependent — 0.5 beat the default 3 times out of 5 on the dense-signal shape and only once on the others. Put it in your search space over [1.0, 0.7, 0.5, 0.33] and let held-out data decide.

Key Takeaways

The two defaults genuinely differ — 'sqrt' for the classifier, 1.0 for the regressor. That asymmetry is deliberate, not an oversight.
The regression default was the most accurate setting in all three shapes tested. Its cost is ~2x fit time and the highest tree correlation, not accuracy.
'sqrt' on regression was the worst setting every time, losing 21 points of R2 when 5 of 60 features carried the signal.
0.5 is the pragmatic compromise — statistically tied with the default, meaningfully faster.
Average over seeds before believing any of this. Single splits moved by more than the gaps between several of these settings.

The One-Sentence Summary

The advice to copy max_features='sqrt' into your regressor is the rare tuning tip that is both widely repeated and measurably backwards: the regression default was the most accurate setting on every dataset shape I tried, 'sqrt' was the least accurate on all of them, and the only thing the default actually costs you is about twice the fit time — so change it to buy speed, never to buy accuracy.

What's Next?

The Random Forest cheat sheet — every knob from this series on one printable page.
The series recap — the one number that explains all six posts.
Gradient Boosting — where the sequential family gets serious.

Follow me for the next article in the Random Forests Deep Dive series!

Let's Connect!

If you have been copying 'sqrt' into regressors, this one is worth a re-run on your own data.

Questions? Ask in the comments — I read and respond to every one.

Has a "well-known" default ever cost you more than the model did? This post began as a takedown of the regression default and ended up defending it. 🌲

I wrote the first draft convinced the default was a bug. Three dataset shapes later the data said the default was fine and the popular fix was the problem. The honest version is less satisfying and considerably more useful.

AdaBoost: The Tutor Who Only Reviews the Questions You Got Wrong

Sachin Kr. Rajput — Wed, 05 Aug 2026 06:52:11 +0000

The One-Line Summary: AdaBoost is the first algorithm in this series that trains its models sequentially rather than in parallel — after every round it raises the weight of the examples it just got wrong, forcing the next weak learner to specialise in exactly those, then weights each learner's vote by how reliable that learner turned out to be, trading bagging's variance reduction for something bagging can never do: reducing bias.

The Parable of Nila and the Eight Weeks

Nila has one workbook with 400 practice problems in it, and eight weeks before the harbour master's navigation test. She solves about six problems in ten. Her family, who have never done this before, decide the sensible thing is to hire help.

They hire twelve tutors.

The Old Way: The Panel of Twelve

Each tutor gets a copy of the workbook, a random slice of it to focus on, and one strict instruction: do not speak to the other eleven. Every Sunday all twelve submit written advice, and Nila does what the majority recommends.

It works, in the way averaging always works. Before the Panel her scores lurched between 41% and 74% depending on which chapter she had revised. After the Panel they sat calmly at 61%, every single week.

Calmly. At 61%. For six weeks.

THE PANEL OF TWELVE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sunday advice, week after week:

  Tutor 1  -> "revise chapters 1-3, the core"
  Tutor 2  -> "revise chapters 1-3"
  Tutor 3  -> "chapters 2-4"
  ...
  Tutor 12 -> "chapters 1-3, and rest well"

MAJORITY: revise chapters 1-3.

Six weeks of that. She can do them in her
sleep.

Problem 230 — tide correction — has been wrong
in all six practice tests. Not one of the
twelve ever mentioned it.

Averaging twelve opinions removed the WOBBLE.
It did not remove the SHARED BLIND SPOT.

The wobble is VARIANCE.
What is left over is BIAS.

This is the wall the Panel could never climb. Twelve helpers working simultaneously, in ignorance of each other, do smooth out each other's quirks. But if all twelve find chapters 1-3 the most obviously teachable material, a thousand of them give the same advice, and problem 230 stays wrong forever.

You cannot average your way out of a mistake everybody is making.

The New Way: One Tutor, Twelve Times

Vera Sandoval charges more than all twelve combined and works differently.

"Don't hire twelve of me on the same morning. Hire me twelve mornings in a row — and let each morning begin by reading the previous morning's wrong answers."

Her method is two ledgers. The first is the question ledger: every problem starts with the same share of Nila's attention, and after each practice test the ones she got wrong get more, the ones she got right get less, and the total stays fixed. Attention is a budget, not a wish.

VERA'S TWO LEDGERS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
LEDGER 1 — attention per problem (starts equal)

  Session 1: 9 of 400 wrong. Those 9 become
             22x as important as the rest.
             In one step.

  Session 2: on the reweighted book, a new
             lesson gets 27 wrong. Those get
             heavier; the old 9 get lighter.

  Session 8: hardest problem carries 55x an
             average one. Easiest, 0.21x.

LEDGER 2 — how loud each lesson is

  fixed almost everything  -> shout   (1.89)
  barely moved the needle  -> whisper (0.35)
  no better than guessing  -> silence

Final advice = every rule, replayed at its own
volume, summed.

The second ledger is the part people miss: Vera does not trust her own sessions equally. Each morning she teaches one rule, then measures how much of the reweighted workbook it fixed. A rule that fixed almost everything gets shouted in the final revision plan; one that barely moved the needle is mentioned quietly; one no better than guessing is struck out, and Vera goes home.

There is a third rule, and it sounds like laziness.

WHY THE SESSIONS MUST BE SHORT:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Vera teaches ONE rule per morning. Never two.

"Why not teach everything on day one?" asks
 Nila's father.

Because if session 1 fixes every problem:

  • session 2 has nothing to correct
  • the question ledger cannot reweight
  • the volume ledger has one entry: infinity
  • you did not hire a tutor twelve times,
    you hired one tutor once

The mistakes ARE the input to the next session.
A session that is too good breaks the chain
that made the chain worth having.

Weak lessons are a REQUIREMENT.

Eight weeks later Nila scores 85%. And one thing went wrong, which matters more than the 85%. Problem 17 has a misprinted answer key, so her correct solution is marked wrong every week. The question ledger, doing exactly what it was built to do, sends more and more attention to problem 17, without limit. In the final week she spends a fifth of her time on a problem that was never solvable.

The ledger cannot tell hard from impossible. That is the deepest flaw in the method.

Why Taking Turns Beats Working in Parallel

THE MATHEMATICS OF TAKING TURNS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  PANEL : twelve helpers at once, none aware
          of the others. Cancels random wobble
          -> VARIANCE. Cannot touch a shared
          blind spot.

  VERA  : one helper twelve times, each turn
          aimed at the last turn's failures.
          Attacks the blind spot -> BIAS.

ALL FOUR MUST HOLD:

  1. Each lesson beats guessing.
  2. Each lesson is WEAK — it leaves mistakes
     for the next one.
  3. Wrong answers get heavier; total
     attention stays fixed.
  4. A lesson's volume is set by what it
     fixed, not by seniority.

Nila's family called this "taking turns."
Machine learning calls it ADABOOST.

Elder Booth closed the Random Forests series by making twelve judges disagree with each other; Vera opens this one by making them wait their turn.

What Is AdaBoost?

AdaBoost (Adaptive Boosting, Freund and Schapire, 1995) builds an ensemble one model at a time. Each model trains on a reweighted copy of the training set that emphasises whatever the ensemble-so-far gets wrong, then votes with a weight set by its own measured reliability. The series so far against the series to come:

	Bagging / RF / Extra Trees	Boosting / AdaBoost
Training order	Parallel, independent	Sequential, each needs the last
Each model sees	A bootstrap of rows	The same rows, reweighted
Goal	Reduce variance	Reduce bias
Base learner	Strong (deep, overfit trees)	Weak (depth-1 stumps)
More models	Never hurts, just plateaus	Can overfit
Parallelisable	Trivially (`n_jobs=-1`)	Not across rounds. Ever.

Row four catches people. In a Random Forest you want each tree deep and overfit, because variance is what averaging removes. In AdaBoost you want each tree almost useless, because leftover error is the fuel for the next round.

The Math

Training data $(x_{i}, y_{i})$ with $y_{i} \in - 1, + 1$ , starting from uniform weights $w_{i (1)} = 1/ n$ . For $m = 1, \dots, M$ :

err m = \sum i w_{i (m)} \cdot 1 [h_{m} (x_{i}) \neq = y_{i}]

α_{m} = \frac{1}{2} ln! (\frac{1 - err _{m}}{err _{m}})

w_{i (m + 1)} \propto w_{i (m)} \cdot exp! (- α_{m} y_{i} h_{m} (x_{i}))

Predict with $H (x) = sign (\sum_{m} α_{m} h_{m} (x))$ .

That exponent collapses to something simple: $y_{i} h_{m} (x_{i})$ is $+ 1$ when the point is correct and $- 1$ when it is wrong, so correct points scale by $e^{- α}$ , wrong points by $e^{+ α}$ , and everything is renormalised to sum to 1. Attention is a budget, exactly as Vera said.

Step by Step

Set every weight to $1/ n$ .
Fit a weak learner on the weighted data.
Compute $err_{m}$ ; if $\geq 0.5$ , stop.
Compute $α_{m}$ .
Scale wrong weights by $e^{α_{m}}$ , right by $e^{- α_{m}}$ , renormalise.
Repeat, then predict with the signed weighted vote.

Alpha Is a Volume Knob With Two Cliffs

$α$ is the only place a learner's quality enters the model.

ALPHA = 0.5 * ln((1 - err) / err)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  err = 0.01   ->  alpha =  2.2976
  err = 0.10   ->  alpha =  1.0986
  err = 0.30   ->  alpha =  0.4236
  err = 0.45   ->  alpha =  0.1003
  err = 0.49   ->  alpha =  0.0200
  err = 0.50   ->  alpha =  0        SILENT
  err = 0.60   ->  alpha = -0.2027   NEGATIVE

TWO CLIFFS:

  err -> 0    alpha -> +infinity. One learner
              drowns out every other; sklearn
              stops and keeps only that one.
              A first stump at err=0 means you
              have no ensemble at all.

  err >= 0.5  alpha <= 0. A coin-flip learner
              earns zero volume and AdaBoost
              halts. (For K classes the bar
              is err < 1 - 1/K.)

Note the asymmetry that trips people up: err = 0 is not a triumph, it is a failure. A learner that nails the weighted training set in round one hands you infinite weight, ends boosting, and leaves you one overfit tree wearing an ensemble's name. That is precisely why the base learner must be weak.

The Loss Function Nobody Mentions

This is what turns AdaBoost from a clever heuristic into a member of a family. With running score $F_{M} (x) = \sum_{m = 1 M} α_{m} h_{m} (x)$ , AdaBoost is greedily minimising

L = i = 1 \sum n exp! (- y_{i} F_{M} (x_{i}))

which is exponential loss. Two consequences fall out.

First, fix $h_{m}$ and solve $\partial L / \partial α_{m} = 0$ : the answer is $\frac{1}{2} ln \frac{1 - err}{err}$ . The alpha formula is not a tuned design choice, it is the exact minimiser. Nobody picked it; it fell out.

Second, the weight $w_{i}$ at round $m$ is proportional to $exp (- y_{i} F_{m - 1} (x_{i}))$ . A point's weight IS its current exponential loss. On the wrong side with margin $- 3$ it carries $e^{3} \approx 20$ ; at margin $- 10$ , $e^{10} \approx 22, 026$ . Exponential loss punishes confident mistakes with no upper bound, and the reweighting inherits that unboundedness exactly. That is the misprinted answer key, stated in calculus.

The bridge to tomorrow: if AdaBoost is stagewise minimisation of exponential loss, what happens when you swap in log loss, squared error, or Huber — anything differentiable — and fit each learner to the negative gradient instead? That generalisation is Gradient Boosting.

200 Useless Stumps Beat One Good Tree

We'll use make_hastie_10_2, the benchmark AdaBoost was originally demonstrated on. Its label is a nonlinear function of ten Gaussian features, which makes a single axis-aligned stump close to worthless.

import numpy as np
from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Never pass algorithm=... (deprecated in 1.6, gone in 1.8).
# The base learner arg is `estimator`, not `base_estimator`.
stump = DecisionTreeClassifier(max_depth=1, random_state=42).fit(X_tr, y_tr)
tree = DecisionTreeClassifier(random_state=42).fit(X_tr, y_tr)
rf = RandomForestClassifier(n_estimators=200, random_state=42,
                            n_jobs=-1).fit(X_tr, y_tr)
ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=200, learning_rate=1.0, random_state=42,
).fit(X_tr, y_tr)

print(f"one decision stump (depth 1)   : {stump.score(X_te, y_te):.4f}")
print(f"one full-depth decision tree   : {tree.score(X_te, y_te):.4f}")
print(f"RandomForest, 200 trees        : {rf.score(X_te, y_te):.4f}")
print(f"AdaBoost, 200 stumps           : {ada.score(X_te, y_te):.4f}")
print()

# staged_score replays the ensemble one round at a time
tr = list(ada.staged_score(X_tr, y_tr))
te = list(ada.staged_score(X_te, y_te))
print(f"{'stumps used':>12} {'train':>8} {'test':>8}")
for r in [1, 5, 25, 100, 200]:
    print(f"{r:>12} {tr[r - 1]:>8.4f} {te[r - 1]:>8.4f}")

one decision stump (depth 1)   : 0.5183
one full-depth decision tree   : 0.7183
RandomForest, 200 trees        : 0.8417
AdaBoost, 200 stumps           : 0.8467

 stumps used    train     test
           1   0.5621   0.5183
           5   0.6043   0.5500
          25   0.7229   0.6717
         100   0.8436   0.8083
         200   0.8964   0.8467

Sit with that. A coin flip scores 0.5000; one stump scores 0.5183, beating chance by 1.8 points — very nearly worthless. Two hundred of those worthless things stacked in sequence score 0.8467, beating a fully grown tree (0.7183) and a 200-tree Random Forest (0.8417) on identical data.

The climb is worth memorising: 0.5183, 0.5500, 0.6717, 0.8083, 0.8467. Slow, then fast, then grinding. Nothing happened in a leap; every point was bought by one stump correcting the residue of the previous 199.

Watching the Weights Move

The experiment nobody runs: implement the reweighting loop by hand on 400 points and just print the weights, round by round. Note flip_y=0.02 — we mislabel about 2% of the data deliberately, because every real workbook has misprints.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier

X, y = make_classification(
    n_samples=400, n_features=2, n_informative=2, n_redundant=0,
    n_clusters_per_class=1, class_sep=2.0, flip_y=0.02, random_state=42,
)
y = np.where(y == 0, -1, 1)          # AdaBoost's native label space

n = len(X)
uniform = 1.0 / n
w = np.full(n, uniform)              # every problem equally important
history = [w.copy()]

print(f"{'round':>6} {'weighted err':>13} {'alpha':>8} {'# wrong':>9}")
for rnd in range(1, 9):
    stump = DecisionTreeClassifier(max_depth=1, random_state=42)
    stump.fit(X, y, sample_weight=w)              # teach to the ledger
    wrong = stump.predict(X) != y
    err = w[wrong].sum()
    alpha = 0.5 * np.log((1 - err) / err)
    print(f"{rnd:>6} {err:>13.4f} {alpha:>8.4f} {wrong.sum():>9}")
    w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0))   # wrong -> heavier
    w /= w.sum()                                         # budget stays fixed
    history.append(w.copy())

H = np.array(history)
hardest = np.argsort(-H[-1])[:3]     # 3 heaviest points at the end
easiest = np.argsort(H[-1])[:3]      # 3 lightest points at the end
cols = list(hardest) + list(easiest)

print()
print("SAMPLE WEIGHT, AS A MULTIPLE OF THE STARTING WEIGHT")
print(f"{'round':>6}" + "".join(f"{'#' + str(i):>10}" for i in cols))
for r in range(9):
    print(f"{r:>6}" + "".join(f"{H[r, i] / uniform:>10.2f}" for i in cols))

 round  weighted err    alpha   # wrong
     1        0.0225   1.8857         9
     2        0.1431   0.8950        27
     3        0.2744   0.4863       243
     4        0.3004   0.4226       205
     5        0.3377   0.3369       205
     6        0.2919   0.4432         9
     7        0.3349   0.3429       199
     8        0.3329   0.3476       205

SAMPLE WEIGHT, AS A MULTIPLE OF THE STARTING WEIGHT
 round      #230      #271      #158      #328      #171       #19
     0      1.00      1.00      1.00      1.00      1.00      1.00
     1     22.22     22.22     22.22      0.51      0.51      0.51
     2     77.66     77.66     12.97      0.30      0.30      0.30
     3     53.51     53.51      8.93      0.21      0.21      0.21
     4     38.25     38.25     14.87      0.34      0.34      0.34
     5     28.87     56.63     11.22      0.26      0.26      0.26
     6     49.46     97.02     19.23      0.18      0.18      0.18
     7     73.83     72.95     14.46      0.14      0.14      0.14
     8     55.34     54.67     21.72      0.21      0.21      0.21

There is the tutor's ledger, printed. Three passes.

Round 1 is violent. The first stump's weighted error is only 0.0225, earning a loud $α = 1.8857$ — and a loud alpha means a savage reweighting. The nine points it missed jump from 1.00 to 22.22 times their starting weight in one step. Overnight, exactly as Vera does it.

The right-hand three columns move in perfect lockstep — 0.51, 0.30, 0.21, 0.34, identical for #328, #171 and #19. Not a coincidence: all three were classified correctly in every round, so all three were multiplied by the same $e^{- α_{m}}$ every time. They are the chapters Nila can do in her sleep, and AdaBoost has stopped spending time on them.

Now #230. Up to 77.66x, down to 28.87x when a stump finally gets it right, back up to 73.83x when the next one misses again — it oscillates because the ensemble keeps almost-solving it. By round 8 the heaviest-to-lightest spread is roughly 260-fold: same 400 rows, wildly unequal attention.

Also notice # wrong. Round 1 misses 9 points; round 3 misses 243 of 400. The later stumps are terrible by any ordinary standard — but they beat a coin flip on the reweighted data, and that is the only test that matters.

AdaBoost from Scratch

Forty-odd lines of NumPy, and the loop is exactly the six steps above.

import numpy as np
from sklearn.datasets import make_hastie_10_2
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split


class MyAdaBoost:
    """SAMME AdaBoost for binary labels in {-1, +1}."""

    def __init__(self, n_estimators=200, learning_rate=1.0, random_state=42):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.random_state = random_state

    def fit(self, X, y):
        n = len(X)
        w = np.full(n, 1.0 / n)                    # all points equally urgent
        self.learners_, self.alphas_ = [], []
        for m in range(self.n_estimators):
            stump = DecisionTreeClassifier(max_depth=1,
                                           random_state=self.random_state)
            stump.fit(X, y, sample_weight=w)               # 1. weighted fit
            wrong = stump.predict(X) != y
            err = float(w[wrong].sum())                    # 2. weighted error
            if err >= 0.5:                                 # no better than luck
                break
            if err <= 0.0:                                 # perfect -> alpha inf
                self.learners_, self.alphas_ = [stump], [1.0]
                break
            alpha = self.learning_rate * 0.5 * np.log((1.0 - err) / err)  # 3.
            w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0))   # 4. reweight
            w /= w.sum()                                         # 5. renorm
            self.learners_.append(stump)
            self.alphas_.append(alpha)
        return self

    def decision_function(self, X):
        return sum(a * lrn.predict(X)
                   for a, lrn in zip(self.alphas_, self.learners_))

    def predict(self, X):
        return np.where(self.decision_function(X) >= 0, 1.0, -1.0)

    def score(self, X, y):
        return float((self.predict(X) == y).mean())


X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

mine = MyAdaBoost(n_estimators=200, random_state=42).fit(X_tr, y_tr)
theirs = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
                            n_estimators=200, learning_rate=1.0,
                            random_state=42).fit(X_tr, y_tr)

print(f"MyAdaBoost        test accuracy: {mine.score(X_te, y_te):.4f}")
print(f"sklearn AdaBoost  test accuracy: {theirs.score(X_te, y_te):.4f}")
print("predictions agreeing on test set: "
      f"{(mine.predict(X_te) == theirs.predict(X_te)).mean():.1%}")
print("first 5 alphas, mine x2 :",
      " ".join(f"{2 * a:.4f}" for a in mine.alphas_[:5]))
print("first 5 alphas, sklearn :",
      " ".join(f"{a:.4f}" for a in theirs.estimator_weights_[:5]))

MyAdaBoost        test accuracy: 0.8467
sklearn AdaBoost  test accuracy: 0.8467
predictions agreeing on test set: 100.0%
first 5 alphas, mine x2 : 0.2499 0.1353 0.1654 0.1491 0.2021
first 5 alphas, sklearn : 0.2499 0.1353 0.1654 0.1491 0.2021

Not "close." Identical — same accuracy, same prediction on all 600 test points.

The x2 is the one real discrepancy between textbook AdaBoost and sklearn. sklearn's SAMME uses $α_{m} = lr \cdot ln! (\frac{1 - err}{err} \cdot (K - 1))$ , which at $K = 2$ is exactly twice the classical $\frac{1}{2} ln \frac{1 - err}{err}$ . Doubling every alpha cannot change the sign of the weighted vote, so predictions are untouched — and the weight update matches too: mine scales wrong points by $e^{α}$ and right ones by $e^{- α}$ (ratio $e^{2 α}$ ), sklearn scales wrong by $e^{2 α}$ and leaves right alone (same ratio), and renormalisation erases the difference. Hence exact agreement rather than approximate.

No hidden machinery in the library version. Five lines in a loop.

The Defining Weakness: Hard vs Impossible

Now the misprinted answer key, in code. Take cleanly separable data, flip 10% of the training labels, and ask two questions: where does AdaBoost spend its attention, and what does that cost?

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=2000, n_features=10, n_informative=10, n_redundant=0,
    n_clusters_per_class=1, class_sep=1.4, flip_y=0.0, random_state=42,
)
y = np.where(y == 0, -1, 1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

n = len(y_tr)
bad = np.random.default_rng(42).choice(n, int(0.10 * n), replace=False)
y_bad = y_tr.copy()
y_bad[bad] = -y_bad[bad]                       # 10% misprinted answer keys
is_bad = np.zeros(n, bool)
is_bad[bad] = True

w = np.full(n, 1.0 / n)
print("Share of AdaBoost's attention on the 10% corrupted labels")
for r in range(1, 101):
    s = DecisionTreeClassifier(max_depth=1, random_state=42)
    s.fit(X_tr, y_bad, sample_weight=w)
    wrong = s.predict(X_tr) != y_bad
    err = w[wrong].sum()
    alpha = 0.5 * np.log((1 - err) / err)
    w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0))
    w /= w.sum()
    if r in (1, 5, 10, 25, 50, 100):
        print(f"  after round {r:>3}: {w[is_bad].sum():>6.1%} of total weight")

print()
print(f"{'labels flipped':>15} {'AdaBoost':>10} "
      f"{'RandomForest':>13} {'errors A/RF':>14}")
for frac in [0.0, 0.10, 0.20]:
    yb = y_tr.copy()
    if frac:
        i = np.random.default_rng(42).choice(n, int(frac * n), replace=False)
        yb[i] = -yb[i]
    A = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
                           n_estimators=200, random_state=42).fit(X_tr, yb)
    R = RandomForestClassifier(n_estimators=200, random_state=42,
                               n_jobs=-1).fit(X_tr, yb)
    ea = (A.predict(X_te) != y_te).sum()
    er = (R.predict(X_te) != y_te).sum()
    print(f"{frac:>14.0%} {A.score(X_te, y_te):>10.4f} "
          f"{R.score(X_te, y_te):>13.4f} {str(ea) + ' / ' + str(er):>14}")

Share of AdaBoost's attention on the 10% corrupted labels
  after round   1:  16.8% of total weight
  after round   5:  37.8% of total weight
  after round  10:  39.9% of total weight
  after round  25:  44.3% of total weight
  after round  50:  45.0% of total weight
  after round 100:  43.7% of total weight

 labels flipped   AdaBoost  RandomForest    errors A/RF
            0%     0.9950        0.9983          3 / 1
           10%     0.9783        0.9983         13 / 1
           20%     0.9417        0.9850         35 / 9

Two findings, and the first explains the second.

By round 25, AdaBoost spends 44% of its attention on 10% of the data — and every one of those points has a wrong label. It is not confused; it is working as designed. Those points stay misclassified, so their exponential loss grows, so their weight grows, so every later stump is fitted to please them. Nila, spending a fifth of her final week on problem 17.

The cost is a 4-to-13x multiplication of test errors. On clean data both models are effectively perfect (3 and 1 errors out of 600). Corrupt 10% of training labels and Random Forest does not notice — still 1 error — while AdaBoost jumps to 13. At 20%, Random Forest makes 9 and AdaBoost makes 35.

Why is the forest immune? Bagging averages over noisy points and never singles them out. A mislabeled row lands in roughly 63% of bootstraps and is outvoted in all of them, carrying the same weight from the first tree to the five-hundredth. Bagging has no mechanism for obsession. AdaBoost is built out of one.

ADABOOST: THE HONEST ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
REACH FOR IT WHEN:
  ✓ Labels are clean and trustworthy
  ✓ Tabular data, moderate size
  ✓ A single model badly underfits (high bias)
  ✓ You want an interpretable additive model
      (depth-1 stumps -> one term per feature)
  ✓ You have almost nothing to tune

LOOK ELSEWHERE WHEN:
  ✗ Labels are noisy or crowd-sourced
      -> Random Forest, or a robust loss
  ✗ Heavy outliers you cannot clean
  ✗ You need training parallelism
      (rounds are strictly sequential)
  ✗ You want last-percent tabular accuracy
      -> LightGBM / XGBoost / CatBoost

THE ONE-LINE RULE:
  Audit your labels BEFORE you use AdaBoost.
  It will find every mistake you made and
  build a model around it.

learning_rate and n_estimators Are One Dial

learning_rate multiplies every alpha, shrinking each round's contribution. Smaller steps mean each stump says less, so you need more of them.

from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

rounds = [25, 50, 100, 200]
print(f"{'lr':>6}" + "".join(f"{'M=' + str(r):>9}" for r in rounds))
for lr in [1.0, 0.5, 0.2, 0.1]:
    a = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
                           n_estimators=200, learning_rate=lr,
                           random_state=42).fit(X_tr, y_tr)
    st = list(a.staged_score(X_te, y_te))
    print(f"{lr:>6.1f}" + "".join(f"{st[r - 1]:>9.4f}" for r in rounds))

    lr     M=25     M=50    M=100    M=200
   1.0   0.6717   0.7667   0.8083   0.8467
   0.5   0.6067   0.6600   0.7150   0.7817
   0.2   0.5083   0.5617   0.6283   0.6733
   0.1   0.4950   0.5183   0.5617   0.6200

Read along a row and accuracy climbs with more rounds; read down a column and it falls with a smaller rate. lr=0.1 at 200 rounds (0.6200) loses to lr=1.0 at 25 rounds (0.6717) — shrinking the steps 10x costs more than 8x the rounds can repay here. So never tune these two separately: fix learning_rate, then push n_estimators until a validation curve stops improving.

The Hyperparameters That Matter

Parameter	What it controls	Sensible range	Notes
`estimator`	The weak learner	`DecisionTreeClassifier(max_depth=1)`	Depth = max interaction order. Depth 1 is purely additive and cannot learn interactions; try 2-3 if you need them.
`n_estimators`	Boosting rounds	50-500	Unlike a forest, more can overfit.
`learning_rate`	Shrinkage on every alpha	0.05-1.0	Halve one, roughly double the other.
`algorithm`	Legacy SAMME switch	do not pass it	Deprecated in sklearn 1.6, removed in 1.8.
`base_estimator`	—	—	Removed. Use `estimator`.

Two anti-patterns. A deep tree as the base estimator defeats the whole mechanism: weighted error crashes toward zero, alpha spikes, and you have paid for an ensemble to get one overfit tree. Boosting before you have looked at your labels is worse — every other model here quietly tolerates 5% label noise, while AdaBoost finds it, weights it, and enshrines it.

Quick Reference Card

ADABOOST: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  Weak learners trained IN SEQUENCE, each one
  fitted to the mistakes of the last. Reduces
  BIAS. (Bagging is parallel -> VARIANCE.)

THE FIVE STEPS:
  1. w = 1/n for every sample
  2. fit weak learner on weighted data
  3. err = weighted misclassification rate
  4. alpha = 0.5 * ln((1 - err) / err)
  5. wrong *= e^alpha, right *= e^-alpha,
     renormalise -> back to step 2

PREDICT:  sign( sum_m alpha_m * h_m(x) )

THE LOSS IT MINIMISES:
  L = sum_i exp(-y_i * F(x_i))
  A sample's weight IS its exponential loss.
  Unbounded -> that is the whole weakness.

HYPERPARAMETERS:
  estimator      = DecisionTree(max_depth=1)
  n_estimators   = 50-500 (CAN overfit)
  learning_rate  = 0.05-1.0 (one dial with
                   n_estimators, not two)

USE IT:    ✓ clean labels, high bias, tabular
AVOID IT:  ✗ noisy labels, outliers

SKLEARN:
  from sklearn.ensemble import AdaBoostClassifier
  # estimator=..., NOT base_estimator=...
  # never pass algorithm=...

Key Takeaways

Boosting is sequential; bagging is parallel. Random Forest trains independent trees to cancel noise and cut variance; AdaBoost trains dependent stumps, each aimed at the last one's failures, and cuts bias. Picking the wrong one is a category error, not a tuning mistake.
Misclassified samples get heavier and the total stays fixed. Nine points missed in round one jumped to 22.22x their starting weight in one step, while points never wrong decayed in lockstep to 0.21x.
alpha = 0.5 * ln((1 - err) / err) is not a design choice — it is the exact analytical minimiser of exponential loss. Zero at err = 0.5, negative above it, unbounded as err approaches 0.
Weak learners are a requirement, not a tolerance. A learner with err = 0 earns infinite weight and ends boosting after one model. Stumps that are too strong buy you one overfit tree at 200x the price.
Two hundred near-useless models beat one good one. One stump scored 0.5183 against a coin flip's 0.5000; two hundred scored 0.8467, beating a full tree (0.7183) and a 200-tree Random Forest (0.8417).
A sample's weight IS its exponential loss, and that loss is unbounded — so a persistently misclassified point's weight grows without limit. That is why label noise is the defining weakness: with 10% of labels flipped, AdaBoost spent 44% of its attention on the corrupted points and made 13 test errors where Random Forest made 1.
learning_rate and n_estimators are one parameter. Smaller alphas say less per round, so you need more rounds; lr=0.1 at 200 rounds lost to lr=1.0 at 25.

The One-Sentence Summary

AdaBoost works because Vera was right that twelve helpers on the same morning can only cancel each other's random errors while one helper twelve mornings running can fix a shared blind spot — so it trains each weak learner on data where yesterday's mistakes are worth exponentially more, sets each lesson's volume to the exact minimiser of exponential loss, and turns a stump that barely beats a coin flip into a model that beats a Random Forest; and it fails for the very same reason, because a ledger that escalates whatever stays wrong cannot tell a hard problem from a misprinted answer key.

What's Next?

Now that you understand how boosting trades parallelism for the ability to attack bias, you're ready for:

Gradient Boosting — tomorrow. Swap exponential loss for any differentiable loss, fit each learner to the negative gradient, and you get the algorithm that runs most of tabular ML.
XGBoost — second-order gradients, regularisation written into the objective, and the engineering that won a decade of Kaggle.
LightGBM — histogram binning and leaf-wise growth, which is how you boost ten million rows without waiting until Thursday.
CatBoost — ordered boosting and native categorical handling, built to fix the target leakage the other two quietly live with.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If Vera's two ledgers made AdaBoost click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

When did you last audit your labels before blaming your model? Roughly half of every "boosting overfits" I have investigated turned out to be "boosting found the annotation mistakes nobody checked for." 🎯

There is something uncomfortably human in AdaBoost. It is the most conscientious algorithm in classical machine learning — it never lets a mistake go — and that conscientiousness is exactly what destroys it when one of the mistakes was never a mistake at all. I have managed engineers like this, and I have been one. The fix was never to care less. It was to learn the difference between a hard problem and a wrong answer key.

Share this with someone still trying to fix bias by adding more trees. Tell them the trees need to take turns.

Extra Trees: The Chef Who Stopped Tasting the Soup and Somehow Cooked Faster and Better

Sachin Kr. Rajput — Tue, 04 Aug 2026 06:54:36 +0000

The One-Line Summary: Extra Trees is a Random Forest that stops searching for the best split point — at every node it draws one random threshold per candidate feature and keeps the best of those few candidates, which pushes tree-to-tree correlation below anything feature subsampling alone can reach, and because nothing ever gets sorted, it trains several times faster while usually scoring the same or better.

The Parable of the Copper Ladle

Two posts ago we blindfolded a jury. Yesterday we found a free exam hiding in the leftovers. Today we leave the courthouse and walk into a kitchen.

The Copper Ladle serves exactly one dish. Every evening one hundred cooks each make one pot of it, and at seven o'clock all hundred pots are poured into a single copper cauldron. What comes out of that cauldron is what the city eats. No diner has ever tasted one cook's pot — only the blend.

The Era of Careful Tasting

Head chef Fen Okoro trained every cook the same way, and she was proud of it: season in six stages, and at each stage taste properly — forty spoonfuls, amounts side by side — then commit to whichever tastes best right now.

It was rigorous. By August it was also a disaster.

TUESDAY SERVICE AT THE COPPER LADLE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
100 cooks. 100 pots. One cauldron at seven.
The city eats the BLEND, never a single pot.

House rule: 6 seasoning stages. At each stage,
taste 40 spoonfuls, keep the best amount.

  Cook   1  stage 1: tastes 40 -> 9g salt
  Cook   2  stage 1: tastes 40 -> 9g salt
   ...
  Cook 100  stage 1: tastes 40 -> 9g salt

Cauldron tonight:  9.0g salt per bowl
The dish wants:    6.0g salt per bowl

24,000 tastings. One mistake, made 100 times.

Every cook landed on nine grams for the same reason: at stage one the pot is at a rolling boil, and boiling soup hides salt. A hundred tongues were fooled in the same direction, so a hundred cooks overshot by the same three grams.

They were measuring carefully. That was the problem.

WHY THE BLEND DIDN'T SAVE THEM:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Blending only fixes mistakes that point in
DIFFERENT directions.

  100 pots, each +3g too salty
    -> cauldron: +3g too salty

  100 pots: -4, +2, -1, +5, -3, +1, ...
    -> cauldron: +0.2g. Nearly perfect.

Careful tasting did not make the pots good.
It made them IDENTICAL.

Meanwhile the queue reached the tram stop. Twenty-four thousand tastings a night is not cooking; it is auditing.

The Dishwasher's Proposal

Teo had washed pots at the Copper Ladle for eleven years. He had watched four thousand services from the sink and had never once been asked his opinion, which is roughly how long it takes to develop a good one.

"What if nobody tastes anything — and every cook seasons on a dice roll?"

Fen asked him to leave. He came back next morning with a written rule: at each stage grab three jars at random, take one blind pinch from each — whatever size your hand closes on, no measuring, no tasting — smell them, keep the most promising, move on. One pinch per jar, no second attempt.

THE DICE RULE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Per stage: 3 random jars, ONE blind pinch each,
keep the best-smelling one. Then move on.

  Cook 1: salt 2g | pepper 4g | thyme 1g
            -> keeps salt 2g
  Cook 2: salt 11g | cumin 3g | bay 2g
            -> keeps cumin 3g
   ...

Salt across the 100 pots: 1g ... 12g. All over.
Average in the cauldron: 6.4g per bowl  ✓

Tastings tonight: 0.
Service ran three times faster.

Fen's objection was the obvious one: you have made every single cook worse. Teo agreed instantly. Not one pot was as good as a carefully tasted pot, and several were frankly bad.

But they were bad in a hundred different directions, and the cauldron does not serve pots. It serves their average.

Why the Sloppy Kitchen Won

THE ARITHMETIC OF SLOPPINESS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A blend is only as good as two things:

  1. How good each pot is      (competence)
  2. How DIFFERENTLY each is wrong
                               (independence)

Careful tasting maximised #1, destroyed #2.
The dice rule gives up a little #1 to buy an
enormous amount of #2.

  competence per pot:  down     ▼
  independence:        way up   ▲▲▲

And it SAVES time: tasting forty spoonfuls
was the expensive part.

The kitchen called it THE DICE RULE.
Machine learning calls it EXTRA TREES.

The soup got better and the kitchen got faster. Those are not usually the same decision — and everything below is that sentence, written in mathematics and Python.

What Is Extra Trees?

Extra Trees — Extremely Randomized Trees, from Geurts, Ernst and Wehenkel (2006) — is a Random Forest with one change at the split-finding step.

An ordinary tree, at every node, sorts each candidate feature's values, walks every cut point between them, scores each and keeps the winner. That is the exhaustive tasting, and it is where nearly all of training time goes. Extra Trees refuses to do it.

At each node	Random Forest	Extra Trees
Candidate features	random subset, size `m`	random subset, size `m`
Thresholds per feature	every midpoint	one, drawn uniformly
Split chosen	best of all candidates	best of the `m` random ones
Rows per tree	bootstrap sample	all rows (`bootstrap=False`)

Row three is the part people mangle: Extra Trees is not a random tree. It still scores candidates with the same impurity criterion and keeps the best — it just has m candidates instead of thousands. Teo's cooks smelled all three pinches; they never got a second pinch.

The Third Knob

THE THREE KNOBS OF RANDOMNESS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  KNOB 1  ROWS       bootstrap per tree
                     -> bagging

  KNOB 2  COLUMNS    random feature subset at
                     every split (max_features)
                     -> random forest

  KNOB 3  THRESHOLDS one random cut per feature
                     instead of the best cut
                     -> extra trees

All three do one job: push rho, the tree-to-
tree correlation, down.

Var(avg) = rho*sigma^2 + (1-rho)*sigma^2/N
           └── the floor. More trees never
               touch it. Only diversity does.

For an average of $N$ predictors with variance $σ^{2}$ and correlation $ρ$ :

Var (f ˉ) = ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}

The second term dies as trees are added; the first does not. Feature subsampling attacks $ρ$ by hiding columns. Random thresholds attack it differently: two trees that pick the same root feature still cut it in different places, because each drew its own threshold from a continuum.

What It Costs: Bias

A greedy split maximises impurity reduction on the node's data; a random one does not, so each Extra Tree fits the signal slightly worse — higher bias. That matters because of an asymmetry people forget:

Variance averages away. A hundred noisy trees, averaged, are far less noisy.
Bias does not. The bias of the average is the average of the biases.

So Extra Trees pays permanent per-tree bias to buy variance and correlation reduction that compound over n_estimators. Usually profitable — below we measure where it stops being.

Step by Step

For $b = 1$ to $B$ :

Take the whole training set (no bootstrap, by default).
Grow a tree. At every node: draw $m$ features at random; for each, draw one threshold $t_{f} \sim U (min, max)$ from that feature's range among this node's samples; score the $m$ candidates with Gini or MSE; keep the best, split, recurse.
Stop on the usual rules — pure node, min_samples_leaf, max_depth.

Predict by vote or mean. Because $min$ and $max$ are node-local, the threshold distribution narrows automatically as you descend.

Measuring the Third Knob

Turn all three knobs on the same data and watch tree quality, correlation, accuracy and wall-clock time move together.

import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

def tree_correlation(model, X):
    """Mean pairwise correlation between individual tree predictions."""
    P = np.array([t.predict(X) for t in model.estimators_])
    C = np.corrcoef(P)
    return float(np.nanmean(C[np.triu_indices_from(C, k=1)]))

models = [
    ("1 knob  bagging", RandomForestClassifier(
        n_estimators=200, max_features=None, random_state=42, n_jobs=1)),
    ("2 knobs RF     ", RandomForestClassifier(
        n_estimators=200, max_features="sqrt", random_state=42, n_jobs=1)),
    ("3 knobs ET     ", ExtraTreesClassifier(
        n_estimators=200, max_features="sqrt", random_state=42, n_jobs=1)),
]

print(f"{'model':<16}{'single':>8}{'rho':>7}{'test':>8}"
      f"{'nodes':>7}{'depth':>7}{'fit s':>7}")
for name, m in models:
    t0 = time.perf_counter()
    m.fit(X_tr, y_tr)
    fit_s = time.perf_counter() - t0
    single = np.mean([t.score(X_te, y_te) for t in m.estimators_])
    nodes = np.mean([t.tree_.node_count for t in m.estimators_])
    depth = np.mean([t.get_depth() for t in m.estimators_])
    print(f"{name:<16}{single:>8.4f}{tree_correlation(m, X_te):>7.3f}"
          f"{m.score(X_te, y_te):>8.4f}{nodes:>7.0f}{depth:>7.1f}{fit_s:>7.2f}")

model             single    rho    test  nodes  depth  fit s
1 knob  bagging   0.8342  0.640  0.8967    153   12.8   1.72
2 knobs RF        0.8204  0.571  0.8983    219   14.4   0.45
3 knobs ET        0.8034  0.515  0.9050    696   25.5   0.17

Six columns, one story told six ways:

Individual trees get worse — 0.8342, 0.8204, 0.8034. Extra Trees builds the weakest tree of the three.
Correlation drops — 0.640, 0.571, 0.515. The third knob buys another 0.056 on top of subsampling.
The ensemble gets better anyway — 0.8967, 0.8983, 0.9050. Worst trees, best committee.
Fit time collapses — 1.72s, 0.45s, 0.17s, single-threaded: 2.6x faster than the forest, 10x faster than bagging.

The last two columns are the ones nobody warns you about: 696 nodes at depth 25.5 versus the forest's 219 at 14.4. A uniform threshold often peels off a handful of samples instead of halving them, so the tree needs far more levels to reach purity.

So Extra Trees is much cheaper to train and modestly more expensive to serve — 8.5 ms (RF) versus 11.0 ms (ET) to predict the 600-row test set, at 3x the nodes in memory. Worth knowing if you train on a schedule and serve under an SLO.

The `bootstrap=False` Default

RandomForestClassifier bootstraps by default. ExtraTreesClassifier does not.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

# The default: every tree sees all 1400 training rows.
et = ExtraTreesClassifier(n_estimators=200, random_state=42, n_jobs=-1)
et.fit(X_tr, y_tr)
print(f"bootstrap=False (default)  train={et.score(X_tr, y_tr):.4f}  "
      f"test={et.score(X_te, y_te):.4f}  oob=n/a")

# Ask for a free validation score anyway, and sklearn stops you.
try:
    ExtraTreesClassifier(n_estimators=200, oob_score=True,
                         random_state=42).fit(X_tr, y_tr)
except ValueError as e:
    print(f"oob_score=True + bootstrap=False ->")
    print(f"    ValueError: {e}")

# Flip the knob and out-of-bag comes back.
et_b = ExtraTreesClassifier(n_estimators=200, bootstrap=True, oob_score=True,
                            random_state=42, n_jobs=-1)
et_b.fit(X_tr, y_tr)
print(f"bootstrap=True             train={et_b.score(X_tr, y_tr):.4f}  "
      f"test={et_b.score(X_te, y_te):.4f}  oob={et_b.oob_score_:.4f}")

bootstrap=False (default)  train=1.0000  test=0.9050  oob=n/a
oob_score=True + bootstrap=False ->
    ValueError: Out of bag estimation only available if bootstrap=True
bootstrap=True             train=1.0000  test=0.8983  oob=0.8957

Why the default is right. Randomness is a budget, not a virtue. A bootstrap sample holds about 63.2% of the distinct rows, so it costs every tree a third of its data — pure bias, paid for diversity. Extra Trees already gets its diversity from the thresholds, so it declines to pay: 0.9050 against 0.8983.

Why train accuracy is 1.0000. No bootstrap, no depth limit, so every tree memorises the training set. A training score on an Extra Trees model tells you nothing.

When to flip it to True. You want oob_score — no bootstrap means no held-out rows, and sklearn raises rather than guess. You want max_samples. Or your labels are noisy, and having each bad row missing from ~37% of trees is free robustness.

Where the Speedup Comes From

It is not a micro-optimisation. It is an algorithmic step deleted.

WHERE THE TIME GOES, PER NODE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
n = samples at this node, m = candidate features

RANDOM FOREST
  for each of m features:
      SORT the n values     <- O(n log n)
      scan every midpoint   <- O(n)
  cost: O(m * n log n)

EXTRA TREES
  for each of m features:
      min and max           <- O(n)
      draw ONE threshold    <- O(1)
      score ONE partition   <- O(n)
  cost: O(m * n)

The sort is the whole difference.

In theory that is a factor of $lo g n$ . In practice the midpoint scan is the branch-heavy inner loop CPUs hate (helping Extra Trees) while its deeper trees mean more nodes to visit (hurting it), and on tabular data the fight settles at 2–4x in Extra Trees' favour — 2.6x above, wider on wide data. A RandomizedSearchCV therefore gets ~2.5x the configurations per hour, often worth more than the model difference itself.

The Noise Sensitivity Trade-Off

So far Extra Trees looks free. It is not — and the surprise is which mess breaks it. It handles one kind of noise better than Random Forest, and the other kind worse.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=10, n_informative=4,
                           n_redundant=0, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)
rf  = lambda: RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1)
et  = lambda: ExtraTreesClassifier(n_estimators=100, random_state=42, n_jobs=-1)
etl = lambda: ExtraTreesClassifier(n_estimators=100, min_samples_leaf=5,
                                   random_state=42, n_jobs=-1)
etm = lambda: ExtraTreesClassifier(n_estimators=100, max_features=0.5,
                                   random_state=42, n_jobs=-1)

print("JUNK COLUMNS BOLTED ON  (mean of 2 noise draws)")
print(f"{'junk':>6}{'RF':>9}{'ET':>9}{'ET-RF':>9}{'ETmf.5':>9}")
for extra in [0, 25, 100, 200]:
    a, b, c = [], [], []
    for s in range(2):
        g = np.random.default_rng(200 + s)
        Ztr = np.hstack([X_tr, g.normal(size=(len(X_tr), extra))]) if extra else X_tr
        Zte = np.hstack([X_te, g.normal(size=(len(X_te), extra))]) if extra else X_te
        a.append(rf().fit(Ztr, y_tr).score(Zte, y_te))
        b.append(et().fit(Ztr, y_tr).score(Zte, y_te))
        c.append(etm().fit(Ztr, y_tr).score(Zte, y_te))
    print(f"{extra:>6}{np.mean(a):>9.4f}{np.mean(b):>9.4f}"
          f"{np.mean(b)-np.mean(a):>+9.4f}{np.mean(c):>9.4f}")

print("FLIPPED TRAINING LABELS  (mean of 3 noise draws)")
print(f"{'flip%':>6}{'RF':>9}{'ET':>9}{'ET-RF':>9}{'ETleaf5':>9}")
for frac in [0.0, 0.15, 0.30]:
    a, b, c = [], [], []
    for s in range(3):
        g = np.random.default_rng(300 + s)
        yn = y_tr.copy()
        k = int(frac * len(yn))
        if k:
            yn[g.choice(len(yn), k, replace=False)] ^= 1
        a.append(rf().fit(X_tr, yn).score(X_te, y_te))
        b.append(et().fit(X_tr, yn).score(X_te, y_te))
        c.append(etl().fit(X_tr, yn).score(X_te, y_te))
    print(f"{frac*100:>5.0f}%{np.mean(a):>9.4f}{np.mean(b):>9.4f}"
          f"{np.mean(b)-np.mean(a):>+9.4f}{np.mean(c):>9.4f}")

JUNK COLUMNS BOLTED ON  (mean of 2 noise draws)
  junk       RF       ET    ET-RF   ETmf.5
     0   0.8733   0.8700  -0.0033   0.8900
    25   0.8283   0.7875  -0.0408   0.8817
   100   0.7458   0.7267  -0.0192   0.8658
   200   0.7250   0.6825  -0.0425   0.8475
FLIPPED TRAINING LABELS  (mean of 3 noise draws)
 flip%       RF       ET    ET-RF  ETleaf5
    0%   0.8733   0.8700  -0.0033   0.8567
   15%   0.8278   0.8389  +0.0111   0.8328
   30%   0.7606   0.7700  +0.0094   0.7900

Noise in the features hurts Extra Trees more. Bolt 200 columns of Gaussian garbage onto a 10-feature problem and default Extra Trees falls to 0.6825 while the forest holds 0.7250. A random cut on a genuinely informative feature often produces a mediocre gain, and a mediocre gain does not stand out against the lucky gains junk columns throw up by chance. Exhaustive search gives real signal its best shot.

Noise in the labels hurts it less. Flip 30% of training labels and Extra Trees edges ahead — same mechanism, opposite sign: a random cut cannot chase an individual mislabelled point the way a greedy one can.

TWO KINDS OF NOISE, TWO ANSWERS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NOISY FEATURES (junk columns)
  ET loses to RF        ✗
  FIX: raise max_features
       0.6825 -> 0.8475 at 200 junk columns

NOISY LABELS (flipped y)
  ET beats RF           ✓
  FIX: raise min_samples_leaf
       0.7700 -> 0.7900 at 30% flipped

Random thresholds are bad at finding signal
AND bad at memorising noise.

Now look at the ETmf.5 column, the most useful number here: 0.8475 with 200 junk columns, against 0.7250 for the forest and 0.6825 for default Extra Trees. The 'sqrt' default was starving it — with 210 columns it draws 14 candidates, and 14 single random thresholds cannot find a good split. Give it more draws per node and it recovers, then wins outright, still as the faster model. In a Random Forest max_features mostly controls decorrelation; in Extra Trees it also controls how much search happens at all, so its sensible range sits higher.

Extra Trees for Regression

The trade lands differently here. Piecewise-constant regressors have visible steps, and averaging many differently placed steps smooths the surface — random thresholds place them in far more distinct positions than greedy splits.

import time
import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor
from sklearn.model_selection import train_test_split

def rho_of(model, X):
    P = np.array([t.predict(X) for t in model.estimators_])
    C = np.corrcoef(P)
    return float(np.nanmean(C[np.triu_indices_from(C, k=1)]))

for noise in [1.0, 5.0]:
    X, y = make_friedman1(n_samples=2000, n_features=10, noise=noise,
                          random_state=42)
    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3,
                                              random_state=42)
    print(f"--- friedman1, target noise sigma={noise} ---")
    print(f"{'model':<15}{'single R2':>11}{'rho':>7}{'ens R2':>9}{'fit s':>7}")
    for name, m in [
        ("RF mf=1.0 (def)", RandomForestRegressor(
            n_estimators=150, random_state=42, n_jobs=1)),
        ("RF mf=0.3      ", RandomForestRegressor(
            n_estimators=150, max_features=0.3, random_state=42, n_jobs=1)),
        ("ET mf=1.0 (def)", ExtraTreesRegressor(
            n_estimators=150, random_state=42, n_jobs=1)),
        ("ET mf=0.3      ", ExtraTreesRegressor(
            n_estimators=150, max_features=0.3, random_state=42, n_jobs=1)),
    ]:
        t0 = time.perf_counter()
        m.fit(X_tr, y_tr)
        fit_s = time.perf_counter() - t0
        single = np.mean([t.score(X_te, y_te) for t in m.estimators_])
        print(f"{name:<15}{single:>11.4f}{rho_of(m, X_te):>7.3f}"
              f"{m.score(X_te, y_te):>9.4f}{fit_s:>7.2f}")

--- friedman1, target noise sigma=1.0 ---
model            single R2    rho   ens R2  fit s
RF mf=1.0 (def)     0.6273  0.751   0.8590   0.77
RF mf=0.3           0.4725  0.602   0.8412   0.31
ET mf=1.0 (def)     0.6075  0.721   0.8745   0.34
ET mf=0.3           0.3484  0.498   0.8372   0.17
--- friedman1, target noise sigma=5.0 ---
model            single R2    rho   ens R2  fit s
RF mf=1.0 (def)    -0.1515  0.406   0.4103   0.80
RF mf=0.3          -0.2160  0.328   0.4122   0.32
ET mf=1.0 (def)    -0.1884  0.368   0.4179   0.33
ET mf=0.3          -0.3094  0.252   0.4025   0.16

At low target noise Extra Trees wins on both axes: R² 0.8745 against 0.8590, in 0.34s against 0.77s. At high noise the ensembles converge (0.4179 against 0.4103) while the speed gap holds.

The row that should stop you is ET mf=1.0 at sigma 5.0: single-tree R² is -0.1884. Every tree is worse than predicting the mean, and the ensemble of them explains 42% of the variance. That is the whole series in one line — averaging does not need good members, only members whose errors point in different directions.

Note what max_features=0.3 did: the lowest $ρ$ in the table (0.498) and a worse ensemble. Diversity is not the objective; accuracy is. And a nuance for anyone who read the max_features post — 1.0 is the notorious bad default for RandomForestRegressor, but for ExtraTreesRegressor it is defensible, since the thresholds already supply the decorrelation subsampling would have to. The one regression default worth leaving alone.

Extra Trees from Scratch

Now the part that makes "stopped tasting the soup" concrete: two split-finders in NumPy, run on the same node with the same candidate features, printing what each chose and what it cost.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

def gini(y):
    if len(y) == 0: return 0.0
    p = np.bincount(y, minlength=2) / len(y)
    return 1.0 - (p ** 2).sum()

def gain(y, mask):
    """Impurity drop from splitting y into mask / ~mask."""
    n, nl = len(y), int(mask.sum())
    if nl == 0 or nl == n: return 0.0
    return gini(y) - (nl/n)*gini(y[mask]) - ((n-nl)/n)*gini(y[~mask])

def best_threshold_split(X, y, feats):
    """RANDOM FOREST style: taste every midpoint of every candidate feature."""
    best, tried = (-1.0, None, None), 0
    for f in feats:
        col = X[:, f]
        vals = np.unique(col)
        for thr in (vals[:-1] + vals[1:]) / 2.0:
            tried += 1
            g = gain(y, col <= thr)
            if g > best[0]: best = (g, f, thr)
    return best, tried

def random_threshold_split(X, y, feats, rng):
    """EXTRA TREES style: ONE uniform draw per candidate feature. No tasting."""
    best, tried = (-1.0, None, None), 0
    for f in feats:
        col = X[:, f]
        thr = rng.uniform(col.min(), col.max())   # <- the entire difference
        tried += 1
        g = gain(y, col <= thr)
        if g > best[0]: best = (g, f, thr)
    return best, tried

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, _, y_tr, _ = train_test_split(X, y, test_size=0.3, random_state=42)
feats = np.array([1, 4, 7, 12, 15])       # the 5 candidate features at this node

(gb, fb, tb), nb = best_threshold_split(X_tr, y_tr, feats)
(gr, fr, tr), nr = random_threshold_split(X_tr, y_tr, feats,
                                          np.random.default_rng(42))
print(f"node: {len(y_tr)} samples, gini={gini(y_tr):.4f}, "
      f"candidates={feats.tolist()}")
print(f"BEST  : feature {fb:>2}  thr={tb:+.4f}  gain={gb:.4f}  "
      f"({nb} splits tried)")
print(f"RANDOM: feature {fr:>2}  thr={tr:+.4f}  gain={gr:.4f}  "
      f"({nr} splits tried)")
print(f"        random keeps {gr/gb:.1%} of the gain "
      f"for {nr/nb:.3%} of the work")

g, chosen = [], []
for s in range(100):
    (gg, ff, _), _ = random_threshold_split(X_tr, y_tr, feats,
                                            np.random.default_rng(s))
    g.append(gg); chosen.append(int(ff))
g = np.array(g)
print(f"100 seeds: mean gain={g.mean():.4f} ({g.mean()/gb:.0%} of best), "
      f"worst={g.min():.4f}, best={g.max():.4f}")
print(f"           root feature chosen: {sorted(set(chosen))} "
      f"-> {len(set(chosen))} different roots")

node: 1400 samples, gini=0.4999, candidates=[1, 4, 7, 12, 15]
BEST  : feature  7  thr=+0.3604  gain=0.1281  (6995 splits tried)
RANDOM: feature  1  thr=+1.9664  gain=0.0197  (5 splits tried)
        random keeps 15.4% of the gain for 0.071% of the work
100 seeds: mean gain=0.0697 (54% of best), worst=0.0022, best=0.1243
           root feature chosen: [1, 7, 15] -> 3 different roots

This is the article in one output block.

The greedy finder tried 6,995 candidate splits to land on feature 7 at +0.3604, gain 0.1281. The random finder tried five, and with random_state=42 it drew badly: gain 0.0197, fifteen percent of what was available. That split is terrible — and that is not a flaw in the demo, that is the algorithm. One Extra Tree is permitted to be terrible.

Then read the last two lines. Across 100 seeds the random finder averages 54% of the best available gain, and the winning feature itself moves: three different features take the root, where the greedy finder returns feature 7 every time, in every tree, forever. Feature subsampling can only vary the root by hiding feature 7; random thresholds make the winner vary even when every tree sees everything.

Here is the same idea grown into an ensemble — pure NumPy, no bootstrap — checked against sklearn.

import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.model_selection import train_test_split

def gini(y):
    p = np.bincount(y, minlength=2) / len(y)
    return 1.0 - (p ** 2).sum()

class MyExtraTree:
    """One Extra Tree: at each node, ONE random threshold per candidate feature."""

    def __init__(self, k=5, min_leaf=3, rng=None):
        self.k, self.min_leaf, self.rng = k, min_leaf, rng

    def fit(self, X, y):
        self.root = self._grow(X, y)
        return self

    def _leaf(self, y):
        return np.bincount(y, minlength=2) / len(y)

    def _grow(self, X, y):
        if len(y) < 2 * self.min_leaf or len(np.unique(y)) == 1:
            return self._leaf(y)
        best = None
        for f in self.rng.choice(X.shape[1], self.k, replace=False):
            col = X[:, f]
            lo, hi = col.min(), col.max()
            if hi <= lo:
                continue
            thr = self.rng.uniform(lo, hi)          # no search, one draw
            mask = col <= thr
            nl = int(mask.sum())
            if nl < self.min_leaf or len(y) - nl < self.min_leaf:
                continue
            g = (gini(y) - (nl/len(y))*gini(y[mask])
                 - ((len(y)-nl)/len(y))*gini(y[~mask]))
            if best is None or g > best[0]:
                best = (g, f, thr, mask)            # best of the random draws
        if best is None:
            return self._leaf(y)
        _, f, thr, mask = best
        return (f, thr, self._grow(X[mask], y[mask]),
                self._grow(X[~mask], y[~mask]))

    def predict_proba(self, X):
        out = np.zeros((len(X), 2))
        stack = [(self.root, np.arange(len(X)))]
        while stack:
            node, idx = stack.pop()
            if not isinstance(node, tuple):
                out[idx] = node
                continue
            f, thr, left, right = node
            m = X[idx, f] <= thr
            stack.append((left, idx[m]))
            stack.append((right, idx[~m]))
        return out


X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

t0 = time.perf_counter()
rng = np.random.default_rng(42)
forest = [MyExtraTree(k=5, min_leaf=3, rng=rng).fit(X_tr, y_tr)   # no bootstrap
          for _ in range(50)]
proba = np.mean([t.predict_proba(X_te) for t in forest], axis=0)
mine = float((proba.argmax(axis=1) == y_te).mean())
print(f"MyExtraTree x50 (pure NumPy) : {mine:.4f}   "
      f"built in {time.perf_counter() - t0:.2f}s")

sk = ExtraTreesClassifier(n_estimators=50, max_features=5, min_samples_leaf=3,
                          random_state=42, n_jobs=-1).fit(X_tr, y_tr)
print(f"sklearn ExtraTreesClassifier : {sk.score(X_te, y_te):.4f}")

MyExtraTree x50 (pure NumPy) : 0.8933   built in 0.63s
sklearn ExtraTreesClassifier : 0.8917

Within noise of the library, with no sorting anywhere in it — fifty fully grown trees over 1,400 rows in 0.63 seconds of interpreted Python. That is how little work remains once you delete the threshold search.

When Extra Trees Wins, and When It Doesn't

Situation	Pick	Why
Training time matters	Extra Trees	2–4x faster fit, no sort per node
Smooth regression target	Extra Trees	Random cut positions average smoother
Noisy or mislabelled `y`	Extra Trees	Random cuts can't chase bad labels
Many pure-junk features	RF, or ET with high `max_features`	Greedy search defends real signal
Signal at a sharp exact threshold	Random Forest	A uniform draw rarely lands on it
Skewed features, extreme outliers	Random Forest	Uniform draws land in empty space
`oob_score` out of the box	Random Forest	ET needs `bootstrap=True` first
Tight serving latency or memory	Random Forest	ET trees: 3x nodes, 1.8x depth

The APIs are identical and the two disagree by a point or two in unpredictable directions, so when in doubt, fit both and keep the winner. It is one import and, for Extra Trees, a third of the time.

Extra Trees Hyperparameters

Parameter	What it does	Sensible range	Notes
`n_estimators`	Trees	300–1000	Not a regularizer. ET is cheap — be generous
`max_features`	Candidates per split	`'sqrt'`–`0.5` (clf), `1.0` (reg)	Tune first. Higher than RF: one shot per feature
`bootstrap`	Sample rows	`False` (default)	`True` for `oob_score`, `max_samples`, noisy labels
`min_samples_leaf`	Min per leaf	1 (clf), 5–20 (reg)	The real regularizer, and your label-noise defence
`max_depth`	Depth cap	`None`, or 20–40	ET runs deep (25.5 vs 14.4 here). Cap if size matters
`criterion`	Impurity	`'gini'` / `'squared_error'`	Still used — ET scores its random candidates
`n_jobs`	Parallelism	`-1`	Free, always

One anti-pattern: copying your Random Forest's max_features straight into Extra Trees. Related parameters, different jobs, and we measured the bill — 0.6825 versus 0.8475 on the junk-column test.

Quick Reference Card

EXTRA TREES: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHAT IT IS:
  Random Forest, but each split threshold is DRAWN AT
  RANDOM (one per candidate feature) instead of searched.
  Best of the m random candidates still wins.

THE THREE KNOBS:
  rows       -> bootstrap        (bagging)
  columns    -> max_features     (random forest)
  thresholds -> random cuts      (extra trees)  <- new

THE TRADE:
  bias per tree     ▲ slightly (bias does NOT average away)
  correlation rho   ▼▼  measured 0.571 -> 0.515
  fit time          ▼▼▼ measured 2.6x faster
  model size        ▲▲  696 nodes vs 219

DEFAULTS TO KNOW:
  bootstrap    = False   (whole dataset per tree!)
  oob_score    -> ValueError unless bootstrap=True
  train score  -> always 1.0000, tells you nothing
  max_features = 'sqrt' (clf) / 1.0 (reg, and fine here)

WHEN IT WINS:
  ✓ Fit time matters / big hyperparameter search
  ✓ Smooth regression targets
  ✓ Noisy or mislabelled y
WHEN IT LOSES:
  ✗ Many pure-junk features (raise max_features!)
  ✗ Signal at a sharp exact threshold
  ✗ Tight serving latency or memory

SKLEARN:
  from sklearn.ensemble import (
      ExtraTreesClassifier, ExtraTreesRegressor)

Key Takeaways

One change: the threshold is drawn, not searched. One uniform draw per candidate feature, inside that node's range, and the best of those few wins.
It is not a random tree. The impurity criterion still picks the winner — Teo's cooks smelled all three pinches, they just never got a second pinch.
This is the third knob: rows, columns, thresholds. We measured the ladder: $ρ$ = 0.640, 0.571, 0.515.
Weaker trees, stronger ensemble — again. Single-tree accuracy fell to 0.8034, worst of the three, while the ensemble rose to 0.9050, best of the three.
You pay in bias, and bias does not average away. Variance reduction compounds over trees; per-tree bias is permanent. That is why the trade has limits.
bootstrap=False is the default and it is right. The thresholds already supply diversity, so no tree gives up a third of its rows. Flip it for oob_score, max_samples, or noisy labels.
The speedup is structural: no search means no sort per node, $O (mn)$ instead of $O (mn lo g n)$ — 2.6x here, and ~2.5x the hyperparameter search per hour.
The noise trade-off cuts both ways. ET lost under junk features (0.6825 vs 0.7250), won under flipped labels (0.7700 vs 0.7606), and raising max_features fixed the first case outright, to 0.8475.

The One-Sentence Summary

Extra Trees is Teo's dice rule written in NumPy: stop tasting forty spoonfuls at every stage, take one blind pinch from each of a few random jars and keep the best-smelling one, and you get a hundred sloppier pots whose mistakes point in a hundred different directions — which is exactly why the blend beats a hundred identically over-salted pots, and why deleting the most expensive step in tree building makes the model faster and, far more often than anyone expects, more accurate.

What's Next?

Every model in this series has been fighting variance — bagging, feature subsampling, random thresholds, all of it averaging noise away. Tomorrow the strategy inverts.

AdaBoost — the pivot from variance reduction to bias reduction: not independent trees averaged in parallel, but a sequence of deliberately weak stumps, each built to fix what the last got wrong. Everything you learned about decorrelation stops applying.
Gradient Boosting — that insight generalised: each tree fits the gradient of the loss, turning boosting into gradient descent in function space.
XGBoost — the engineering that made boosting win competitions: second-order gradients, regularised leaf weights, sparsity-aware splits, and histogram binning (another way of refusing to sort).
The Boosting series — LightGBM, CatBoost, early stopping, learning rates, and an honest answer to "boosting or forests?" on tabular data.

Follow me for the next article in the Random Forests Deep Dive series!

Let's Connect!

If the Copper Ladle made Extra Trees click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Do you actually fit both RandomForest and ExtraTrees before choosing? I do, every time, and Extra Trees wins more often than its reputation suggests — usually while I am still waiting for the forest to finish. 🥄

There is a version of engineering discipline that is really just expensive habit. The Copper Ladle tasted forty spoonfuls a stage because tasting felt like rigour, and it took a dishwasher to notice that all that rigour was producing one hundred copies of the same mistake, slowly. The best optimisation I ever shipped was not a faster algorithm — it was deleting a step everybody assumed had to be there.

Share this with the teammate still waiting for a Random Forest to finish training. Tell them to try the one that stopped tasting.

Out-of-Bag Error: The Free Exam You Didn't Know You Were Already Taking

Sachin Kr. Rajput — Mon, 03 Aug 2026 06:54:33 +0000

The One-Line Summary: Because each tree in a Random Forest trains on a bootstrap sample, exactly $(1 - 1/ n)^{n} \to 1/ e \approx 36.8%$ of rows are invisible to any given tree, so you can score every row using only the trees that never saw it — a validation estimate that costs one fit instead of five, on an exam nobody had to set a single question aside for.

The Parable of the School That Never Set an Exam

The village of Predicta, having fixed its jury by blindfolding every judge, turned to its school — and hit a problem the jury never had. How do you find out whether the teaching worked?

Schoolmaster Alder kept a thousand question cards in a wooden bin. Real problems from real harvests: how to split a disputed field, what a wet season costs in grain. Students learned by working through them. At term's end, Alder needed a number to put in front of the village council.

The Sealed Cupboard

For years Alder did the obvious thing. Before term began he counted out two hundred cards, put them in a cupboard, and locked it. Nobody studied those. At term's end he unlocked it and made the students answer them cold.

Honest. Also expensive, in a way it took him a decade to notice.

THE SEALED CUPBOARD:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Bin: 1000 cards.
  800 -> teaching
  200 -> locked away, untouched, all term

Cost 1: students never learn from 200 real
        problems. They finish the term WORSE
        than they had to be.

Cost 2: the exam is only 200 questions. Draw an
        unlucky 200 and the school looks awful.
        Alder ran it twice, different cards:
          cupboard A -> 71 of 200 correct
          cupboard B -> 84 of 200 correct
        Same students, same teaching. 13 points.

A clerk proposed the fix: seal a DIFFERENT 200
five times, teach the whole term five times,
average the five numbers.

It worked. It cost five terms.

This is the price of an HONEST HOLD-OUT.

Alder accepted that price the way you accept weather. One term for a shaky number, or five for a steady one. Pick.

Nim Notices the Ledger

The person who broke this open was not a teacher. It was Nim, fourteen, whose entire job was the bin.

Here is how a student got their practice stack. Nim reached in, pulled a card, copied it onto a slate, and dropped the original back in the bin. Then again. A thousand times. A thousand slates from a bin that never got smaller — which means some cards got copied three times, and some never got copied at all.

Nim kept a ledger of every draw, because otherwise he lost count. Nobody had ever asked to read it.

NIM'S LEDGER  (Tova's stack, shrunk to 10 cards)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Draws, in order:  1  3  3  6  7  9  1 10  7  3

  card  1  ■■       card  6  ■
  card  2  -        card  7  ■■
  card  3  ■■■      card  8  -
  card  4  -        card  9  ■
  card  5  -        card 10  ■

10 draws produced only 6 distinct cards.
Tova has NEVER SEEN cards 2, 4, 5, 8.

For Tova — and Tova alone — those four are a
sealed cupboard. Nobody locked it.
It locked itself.

Nim brought the ledger to Alder and asked the question that ended the sealed cupboard forever.

"Why are we locking cards away, when every student already has hundreds they have never once seen — and a different set for each one?"

Alder did not follow at first. Nim explained it card by card.

GRADING CARD 47, THE FREE WAY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Twelve students. Nim reads his ledger:

  Tova   drew it 2x    -> DISQUALIFIED, she studied it
  Bram   never drew it -> eligible  ✓
  Ilse   drew it 1x    -> DISQUALIFIED
  Ozan   never drew it -> eligible  ✓
  ... (of 12, four never drew card 47)

Ask ONLY those four. They are answering a
question they were never taught.

  Bram ✓   Ozan ✓   Vell ✗   Hedda ✓
  -> 3 of 4. Card 47 counts as CORRECT.

Repeat for all 1000 cards. Each is graded by a
different quarter of the class.

Cards sealed away:      0
Terms of teaching used: 1

Alder ran it that autumn and got 84 correct in every hundred. Then, out of pure suspicion, he ran the old five-cupboard ritual too — five full terms of work — and got 84 as well.

He had been paying five terms for a number sitting in a fourteen-year-old's notebook the whole time.

Why the Cupboard Locks Itself

What unsettled Alder was that the fraction never moved. Every student, every term, roughly the same share of the bin went unseen.

THE ARITHMETIC OF THE MISSING THIRD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

One draw from a bin of n cards.
Chance it is NOT card 47:        1 - 1/n

The card goes back in, so the next draw has the
same chance. And the next. n draws in total:

  P(card 47 never drawn) = (1 - 1/n)^n

Watch that as the bin grows:

     n = 10    ->  0.349
     n = 100   ->  0.366
     n = 1000  ->  0.3677
     n -> huge ->  0.3679

It does not drift. It CONVERGES.
The limit is 1/e = 0.367879...

  IN  a student's stack:  ≈ 63.2% of the bin
  OUT of it:              ≈ 36.8%

Two things the village lived off for years:

  1. Every student arrives with a free exam
     attached, about 368 cards long.
  2. Every free exam is DIFFERENT, so pooling
     them covers the whole bin.

Predicta called it Nim's Third.
Machine learning calls it OUT-OF-BAG ERROR.

Bootstrap sampling was never just a way to make trees different. It was quietly generating a held-out set for every tree, and throwing it away.

What Is Out-of-Bag Error?

Every tree in a Random Forest trains on a bootstrap sample: $n$ rows drawn from your $n$ training rows with replacement. Some rows land in it twice. Some never land at all.

The rows a tree never saw are that tree's out-of-bag (OOB) rows, and for those rows the tree is an honest grader — asked about data it was not fit on. Out-of-bag error flips that from tree-wise to row-wise: for each row, collect the trees that excluded it, average only those, compare against the truth.

Parable	Random Forest
Card in the bin	Training row
Student	Tree
Nim's ledger	Bootstrap indices per tree
Cards never drawn	That tree's OOB rows
Polling only the ignorant	OOB prediction for a row
The school's honest score	`oob_score_`
Five cupboards, five terms	5-fold CV

The Math

Fix a row $i$ and a tree $b$ . Each bootstrap draw picks row $i$ with probability $1/ n$ , so it misses row $i$ with probability $1 - 1/ n$ . Draws are independent, and there are $n$ :

P (row i \in / bag b) = (1 - \frac{1}{n})^{n}

Take logs and expand $ln (1 - x) = - x - \frac{x ^{2}}{2} - \frac{x ^{3}}{3} - \dots$ with $x = 1/ n$ :

n ln! (1 - \frac{1}{n}) = - 1 - \frac{1}{2 n} - \frac{1}{3 n ^{2}} - \dots ⟶ - 1

So it converges to $e^{- 1} \approx 0.3679$ — and from below: small $n$ leaves out slightly less. Two corollaries you'll use constantly:

With $B$ trees, the number grading row $i$ is $Binomial (B, 1/ e)$ — mean $0.368 B$ .
The probability row $i$ is graded by no tree is $(1 - 1/ e)^{B} = 0.63 2^{B}$ .

That second one is where OOB goes wrong. We'll come back to it.

The Algorithm

Fit the forest normally, recording the bootstrap indices each tree used.
For each row $i$ , find $S_{i} = b : i \in / bag_{b}$ .
If $S_{i}$ is empty, skip the row — and be nervous.
Otherwise average the trees in $S_{i}$ : soft-vote probabilities for classification, mean for regression.
Score those pooled predictions against $y$ .

Note step 4. OOB uses roughly $0.368 B$ trees per row, not $B$ . You are estimating a smaller forest than the one you shipped.

First, Let's Verify the 36.8%

No theory. Just draw cards and count.

# Empirically verifying the 36.8% -- no theory, just counting.
import numpy as np

rng = np.random.default_rng(42)
TRIALS = 2000

print(f"{'n':>7} {'measured':>10} {'(1-1/n)^n':>11} {'1/e':>8} {'diff':>8}")
print("-" * 48)
for n in [5, 10, 25, 100, 500, 2000, 10000]:
    left_out = []
    for _ in range(TRIALS):
        draw = rng.integers(0, n, n)          # bootstrap: n draws WITH replacement
        seen = np.zeros(n, dtype=bool)
        seen[draw] = True
        left_out.append(1.0 - seen.mean())    # fraction of rows never drawn
    measured = float(np.mean(left_out))
    theory = (1 - 1 / n) ** n
    print(f"{n:>7} {measured:>10.4f} {theory:>11.4f} "
          f"{1/np.e:>8.4f} {abs(measured - 1/np.e):>8.4f}")

print(f"\n1/e = {1/np.e:.6f}  ->  {100/np.e:.2f}% of rows sit out every tree.")

      n   measured   (1-1/n)^n      1/e     diff
------------------------------------------------
      5     0.3288      0.3277   0.3679   0.0391
     10     0.3488      0.3487   0.3679   0.0191
     25     0.3602      0.3604   0.3679   0.0077
    100     0.3663      0.3660   0.3679   0.0016
    500     0.3675      0.3675   0.3679   0.0004
   2000     0.3680      0.3678   0.3679   0.0001
  10000     0.3679      0.3679   0.3679   0.0000

1/e = 0.367879  ->  36.79% of rows sit out every tree.

measured tracks (1-1/n)^n to four decimals at every size, and both walk up to $1/ e$ from below. By $n = 25$ you're within 0.008 of the limit. For any dataset you actually care about, "36.8% of rows are out-of-bag" is exact enough to plan with.

Out-of-Bag Error in Scikit-Learn

One keyword argument. That is the whole API.

import time
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

def fresh():
    return RandomForestClassifier(n_estimators=200, oob_score=True,
                                  random_state=42, n_jobs=-1)

# --- Route 1: OOB. One fit. The score falls out of it for free.
t0 = time.perf_counter()
rf = fresh().fit(X_tr, y_tr)
oob_time = time.perf_counter() - t0
oob = rf.oob_score_

# --- Route 2: 5-fold cross-validation. Five fits.
t0 = time.perf_counter()
cv = cross_val_score(fresh(), X_tr, y_tr, cv=5, scoring="accuracy")
cv_time = time.perf_counter() - t0

# --- Route 3: the held-out test set we are not allowed to peek at.
test = rf.score(X_te, y_te)

print(f"OOB score          : {oob:.4f}   ({oob_time:.2f}s, 1 fit)")
print(f"5-fold CV mean     : {cv.mean():.4f}   ({cv_time:.2f}s, 5 fits)"
      f"  +/- {cv.std():.4f}")
print(f"Held-out test score: {test:.4f}   (the ground truth)")
print(f"\nfolds: {np.round(cv, 4)}")
print(f"OOB is off by      : {abs(oob - test):.4f}")
print(f"5-fold is off by   : {abs(cv.mean() - test):.4f}")
print(f"Time saved         : {cv_time / oob_time:.1f}x")

OOB score          : 0.9021   (0.28s, 1 fit)
5-fold CV mean     : 0.9036   (1.44s, 5 fits)  +/- 0.0132
Held-out test score: 0.8983   (the ground truth)

folds: [0.9143 0.8893 0.9143 0.9143 0.8857]
OOB is off by      : 0.0038
5-fold is off by   : 0.0052
Time saved         : 5.1x

Sit with those three numbers. 0.9021, 0.9036, 0.8983. The spread across all three is under half a point — smaller than the spread between individual CV folds (0.8857 to 0.9143). And on this run OOB was the closer of the two to the held-out truth.

One fit versus five — 5.1x on this run. Exactly the multiple you'd predict: cross-validation's cost model is trivial, $k$ folds means $k$ forests.

The Three Attributes You Need

Attribute	What it holds
`oob_score=True`	Turns the machinery on. Requires `bootstrap=True`.
`oob_score_`	Accuracy for classifiers, $R^{2}$ for regressors. Higher is better.
`oob_decision_function_`	Pooled OOB probabilities, `(n, n_classes)`. `oob_prediction_` on regressors.

oob_decision_function_ is the one people ignore and shouldn't. oob_score_ collapses everything into one accuracy number, often the wrong one. The decision function hands you raw per-row probabilities, so you can compute ROC-AUC, log loss, balanced accuracy or a calibration curve — from a single fit, with no leakage. Since sklearn 1.3 you can skip that step and pass a callable straight to oob_score, e.g. oob_score=roc_auc_score.

Three traps:

ExtraTreesClassifier defaults to bootstrap=False. oob_score=True alone raises; set both. (More tomorrow.)
max_samples changes the fraction. Drawing $f \cdot n$ samples makes the OOB fraction $e^{- f}$ , not $e^{- 1}$ . max_samples=0.5 leaves out 60.7% of rows.
OOB only covers what happens inside fit. Scale or impute beforehand and OOB will launder that leakage. Cross-validation inside a Pipeline will not.

The Trees That Grade You Are Fewer Than You Think

Each row is scored by only ~36.8% of your trees. With 200 that is ~74 graders: plenty. With 10 it is ~4, and some rows get none at all.

import warnings
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

print(f"{'trees':>6} {'oob_score_':>11} {'test':>8} {'oob-test':>9} "
      f"{'rows w/o OOB':>13} {'warned':>7}")
print("-" * 60)
for B in [5, 10, 25, 50, 100, 200]:
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        rf = RandomForestClassifier(n_estimators=B, oob_score=True,
                                    random_state=42, n_jobs=-1).fit(X_tr, y_tr)
        warned = any("OOB" in str(w.message) for w in caught)
    df = rf.oob_decision_function_
    orphans = int(np.isnan(df).any(axis=1).sum() + (df.sum(axis=1) == 0).sum())
    test = rf.score(X_te, y_te)
    print(f"{B:>6} {rf.oob_score_:>11.4f} {test:>8.4f} "
          f"{rf.oob_score_ - test:>+9.4f} {orphans:>13} {str(warned):>7}")

print(f"\nP(a row is OOB for no tree at all) = 0.632^B")
for B in [5, 10, 25, 50]:
    print(f"  B={B:<4} -> {0.632**B:.2e}  ~= {0.632**B * 1400:.1f} of 1400 rows")

 trees  oob_score_     test  oob-test  rows w/o OOB  warned
------------------------------------------------------------
     5      0.7829   0.8750   -0.0921           154    True
    10      0.8479   0.8983   -0.0505            16    True
    25      0.8893   0.9000   -0.0107             0   False
    50      0.8964   0.9000   -0.0036             0   False
   100      0.9000   0.8983   +0.0017             0   False
   200      0.9021   0.8983   +0.0038             0   False

P(a row is OOB for no tree at all) = 0.632^B
  B=5    -> 1.01e-01  ~= 141.2 of 1400 rows
  B=10   -> 1.02e-02  ~= 14.2 of 1400 rows
  B=25   -> 1.04e-05  ~= 0.0 of 1400 rows
  B=50   -> 1.09e-10  ~= 0.0 of 1400 rows

At five trees, OOB reports 0.7829 for a forest whose real accuracy is 0.8750. Nine points pessimistic — and it says so, in a UserWarning everybody scrolls past. 154 of 1,400 rows had no eligible grader at all, where theory predicted 141.

READING THE TREE-COUNT TABLE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Each row is graded by ~0.368 x B trees.

  B=5    ->  ~2 graders  measuring a 2-TREE
                         forest. Useless.
  B=25   ->  ~9 graders  bias mostly gone
  B=200  ->  ~74 graders OOB ≈ test  ✓

The bias has a DIRECTION: always pessimistic.
A 74-tree vote is worse than a 200-tree vote,
so OOB understates your real forest.

RULES OF THUMB:
  ✗ B < 25    OOB is noise. Ignore it.
  ✓ B >= 100  OOB is trustworthy.
  ✓ B >= 250  bias below measurement noise.

"Some inputs do not have OOB scores" is not a
bug. It is too few trees. Add trees.

The direction of that bias is the useful part: too few trees makes your model look worse than it is, never better. A conservative lie is survivable. Next, the lies that go the other way.

When Out-of-Bag Error Lies

OOB rests on one assumption: a row that is out-of-bag is genuinely unseen. Break it and the number turns actively harmful — it stays confident while becoming wrong.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import balanced_accuracy_score, roc_auc_score
from sklearn.model_selection import GroupKFold, cross_val_score

rng = np.random.default_rng(42)

# ---- LIE #1: rows that are not independent (400 patients, 5 visits each)
Xb, yb = make_classification(n_samples=400, n_features=20, n_informative=5,
                             n_redundant=5, random_state=42)
groups = np.repeat(np.arange(400), 5)
X = np.repeat(Xb, 5, axis=0) + rng.normal(0, 0.01, (2000, 20))  # near-twins
y = np.repeat(yb, 5)

rf = RandomForestClassifier(n_estimators=200, oob_score=True,
                            random_state=42, n_jobs=-1).fit(X, y)
grouped = cross_val_score(
    RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1),
    X, y, cv=GroupKFold(n_splits=5), groups=groups).mean()

print("LIE #1 -- grouped rows (5 near-duplicate visits per patient)")
print(f"  oob_score_          : {rf.oob_score_:.4f}   <- looks fantastic")
print(f"  GroupKFold accuracy : {grouped:.4f}   <- the honest number")
print(f"  OOB overstates by   : {rf.oob_score_ - grouped:+.4f}")

# ---- LIE #2: heavy class imbalance
Xi, yi = make_classification(n_samples=2000, n_features=20, n_informative=5,
                             n_redundant=5, weights=[0.97, 0.03],
                             flip_y=0.01, random_state=42)
rfi = RandomForestClassifier(n_estimators=200, oob_score=True,
                             random_state=42, n_jobs=-1).fit(Xi, yi)
proba = rfi.oob_decision_function_
oob_pred = rfi.classes_[proba.argmax(axis=1)]

print(f"\nLIE #2 -- imbalance ({np.bincount(yi)[1]} positives of {len(yi)})")
print(f"  oob_score_ (accuracy)   : {rfi.oob_score_:.4f}   <- 'great model!'")
print(f"  always-predict-majority : {(yi == 0).mean():.4f}   <- the dumb baseline")
print(f"  OOB balanced accuracy   : {balanced_accuracy_score(yi, oob_pred):.4f}")
print(f"  OOB recall on positives : {(oob_pred[yi == 1] == 1).mean():.4f}")
print(f"  OOB ROC-AUC             : {roc_auc_score(yi, proba[:, 1]):.4f}")

LIE #1 -- grouped rows (5 near-duplicate visits per patient)
  oob_score_          : 1.0000   <- looks fantastic
  GroupKFold accuracy : 0.8620   <- the honest number
  OOB overstates by   : +0.1380

LIE #2 -- imbalance (72 positives of 2000)
  oob_score_ (accuracy)   : 0.9710   <- 'great model!'
  always-predict-majority : 0.9640   <- the dumb baseline
  OOB balanced accuracy   : 0.5972
  OOB recall on positives : 0.1944
  OOB ROC-AUC             : 0.8440

Lie #1 is a perfect 1.0000. Every row correct — and the reason has nothing to do with the model being good. When patient 219's third visit is out-of-bag for a tree, that patient's other four visits are almost certainly in-bag. The tree memorised a near-identical row. OOB thinks it is testing on unseen data; it is testing on a photocopy. Hold out whole patients with GroupKFold and the truth is 0.8620 — fourteen points lower. Time series is the same disease with a clock: yesterday and today are near-twins, so bootstrapping smears the future into the past. Use TimeSeriesSplit.

Lie #2 is quieter and therefore worse. oob_score_ says 0.9710. But predicting "negative" for all 2,000 rows scores 0.9640 — the model beat a constant by 0.7 points. Its recall on the class you built the thing to find is 0.1944: it misses four of every five positives. The AUC of 0.8440 says the ranking has real signal; the default 0.5 threshold throws it away. oob_score_, being plain accuracy, hid all of that behind one flattering number.

THE THREE WAYS OOB LIES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

1. TOO FEW TREES         direction: pessimistic
   Rows get 0-4 graders. Noisy, understated.
   Fix: n_estimators >= 100, and read the
        "Some inputs do not have OOB" warning.

2. NON-INDEPENDENT ROWS  direction: OPTIMISTIC
   Panels, repeat customers, multi-visit
   patients, time series, duplicate rows.
   An OOB row's twin is in-bag -> leakage.
   Measured: 1.0000 vs 0.8620 real.
   Fix: GroupKFold / TimeSeriesSplit. OOB
        CANNOT be repaired here — bootstrap
        has no idea groups exist.

3. CLASS IMBALANCE       direction: meaningless
   oob_score_ is plain ACCURACY. At 97:3 the
   trivial model already scores 0.9640.
   Measured: 0.9710 accuracy, but recall on
   positives = 0.1944.
   Fix: score oob_decision_function_ yourself,
        or pass a callable to oob_score.

The honest summary:
  OOB assumes rows are INDEPENDENT and that you
  have ENOUGH TREES. It never checks either.

Out-of-Bag Error from Scratch

Nim's insight was that the ledger was the whole trick. So let's write the ledger: bootstrap by hand, remember who saw what, score each row using only the trees blind to it.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier


def oob_score_from_scratch(X, y, n_estimators=200, seed=42):
    """The whole trick: keep the ledger of who saw what."""
    rng = np.random.default_rng(seed)
    n, classes = len(X), np.unique(y)
    votes = np.zeros((n, len(classes)))   # summed probs from EXCLUDING trees
    n_graders = np.zeros(n, dtype=int)    # how many trees never saw each row

    for _ in range(n_estimators):
        in_bag_idx = rng.integers(0, n, n)          # bootstrap, WITH replacement
        seen = np.zeros(n, dtype=bool)
        seen[in_bag_idx] = True
        oob = ~seen                                 # <-- the 36.8%

        tree = DecisionTreeClassifier(
            max_features="sqrt", random_state=int(rng.integers(1 << 31))
        ).fit(X[in_bag_idx], y[in_bag_idx])

        # credit only the rows this tree was blind to
        cols = np.searchsorted(classes, tree.classes_)
        votes[np.ix_(oob, cols)] += tree.predict_proba(X[oob])
        n_graders[oob] += 1

    scored = n_graders > 0                          # rows with >=1 honest grader
    pred = classes[votes[scored].argmax(axis=1)]
    return (pred == y[scored]).mean(), scored.sum(), n_graders.mean()


X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
                           n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)

mine, scored, avg = oob_score_from_scratch(X_tr, y_tr, 200, seed=42)
theirs = RandomForestClassifier(n_estimators=200, oob_score=True,
                                random_state=42, n_jobs=-1).fit(X_tr, y_tr)

print(f"my OOB score        : {mine:.4f}")
print(f"sklearn oob_score_  : {theirs.oob_score_:.4f}")
print(f"difference          : {abs(mine - theirs.oob_score_):.4f}")
print(f"rows scored         : {scored} of {len(X_tr)}")
print(f"avg graders per row : {avg:.1f} of 200  ({avg / 200:.1%})")

my OOB score        : 0.9029
sklearn oob_score_  : 0.9021
difference          : 0.0007
rows scored         : 1400 of 1400
avg graders per row : 73.8 of 200  (36.9%)

Seven ten-thousandths apart — different random draws, same estimator. There is no hidden machinery in oob_score_; it is ~seen and a running tally.

And look at the last line, the article closing its own loop: 73.8 graders out of 200 — 36.9%. Not because I asked for it. Because $1/ e$ is what falls out of drawing $n$ times from $n$ with replacement, whether you notice or not.

Three details that matter more than they look:

seen[in_bag_idx] = True handles duplicate indices for free — assigning True twice is still True. That's why the ledger is a boolean mask, not a counter.
votes accumulates probabilities, not hard labels. Soft voting is what sklearn uses for oob_decision_function_; hard voting gives a different number.
np.searchsorted(classes, tree.classes_) exists because a bootstrap sample can miss a rare class entirely, leaving that tree's predict_proba with fewer columns than the forest has classes. On balanced data it never fires. On a 99:1 problem it will, and silently corrupts your columns.

When Should You Use OOB?

Situation	Use OOB?	Why
Quick check while iterating	Yes	Free, one fit, instant feedback
Small data you can't split	Yes	Every row trains and validates
Slow fit, tight compute	Yes	5-fold is 5x cost, same answer
Coarse `max_features` sweeps	Carefully	Fine for ranking only
Grouped / time-series data	No	Broken. Use `GroupKFold`
Imbalanced target	Decision function	`oob_score_` is accuracy
Tuning a whole pipeline	No	Upstream leakage hides
The number in the report	No	Use a test set touched once

OOB is your development loop, cross-validation is your decision procedure, the test set is your evidence. Three jobs, and OOB is very good at the first.

The Knobs That Affect OOB

Parameter	Effect on OOB	Guidance
`oob_score`	Turns it on; `True` or a callable (≥ 1.3)	Callable when accuracy is wrong
`bootstrap`	Must be `True`	`False` means no OOB at all
`n_estimators`	Drives the bias (~`0.368 * B` graders)	≥ 100 to trust, ≥ 250 to forget
`max_samples`	OOB fraction becomes $e^{- f}$	`0.5` leaves out 60.7%
`class_weight`	`'balanced_subsample'` reweights per bag	Changes predictions, not the metric
`warm_start`	Adding trees recomputes OOB	Watch OOB stabilise cheaply

max_samples is the trap. A genuinely useful lever — smaller bootstraps mean faster fits, more diverse trees and more OOB rows per tree — but the 36.8% rule of thumb stops applying the moment you touch it.

Quick Reference Card

OUT-OF-BAG ERROR: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

WHAT IT IS:
  Bootstrap leaves ~36.8% of rows out of each tree.
  Score every row using ONLY the trees that missed it.
  A validation set that assembles itself, for free.

THE ONE FORMULA:
  P(row not in bag) = (1 - 1/n)^n  ->  1/e = 0.3679

  in-bag  ≈ 63.2%      graders per row ≈ 0.368 x B
  out     ≈ 36.8%      P(no grader)    = 0.632^B

COST:
  OOB = 1 fit        k-fold = k fits
  measured: 0.9021 OOB, 0.9036 5-fold,
            0.8983 test.  ~5x faster.

TRUST IT WHEN:
  ✓ n_estimators >= 100 (>= 250 ideally)
  ✓ Rows are genuinely independent
  ✓ Classes balanced, or you score
    oob_decision_function_ yourself
  ✓ No preprocessing outside the forest

IT LIES WHEN:
  ✗ Few trees      -> pessimistic  (0.78 vs 0.88)
  ✗ Grouped rows   -> OPTIMISTIC   (1.00 vs 0.86)
  ✗ Time series    -> OPTIMISTIC, badly
  ✗ Imbalance      -> meaningless  (0.97 = baseline)
  ✗ bootstrap=False-> does not exist

SKLEARN:
  RandomForestClassifier(
      n_estimators=300, oob_score=True,
      bootstrap=True, n_jobs=-1)
  -> .oob_score_              (acc / R^2)
  -> .oob_decision_function_  (n, n_classes)
  -> .oob_prediction_         (regressors)

Key Takeaways

Bootstrap leaves out $(1 - 1/ n)^{n} \to 1/ e \approx 36.8%$ of rows per tree — a limit, not an approximation. Measured 0.3679 at $n = 10, 000$ .
A row's OOB prediction uses only the trees that never saw it. Row-wise, not tree-wise. That flip is the entire algorithm.
OOB costs one fit; $k$ -fold costs $k$ . Measured ~5x, for estimates 0.0015 apart — and OOB was closer to the truth.
Each row is graded by only ~ $0.368 B$ trees, so OOB estimates a smaller forest than you shipped and runs pessimistic. At $B = 5$ : 0.7829 for a 0.8750 model.
n_estimators >= 100, or don't quote the number. $P (no grader) = 0.63 2^{B}$ — 10% of rows at 5 trees, one in a hundred thousand at 25.
Non-independent rows break OOB silently and optimistically. Five visits per patient pushed oob_score_ to a perfect 1.0000 against a real 0.8620. Bootstrap has no concept of a group.
oob_score_ is plain accuracy — at 97:3 it scored 0.9710 while recall on positives was 0.1944. Use oob_decision_function_.
OOB is a development loop, not evidence. Iterate on OOB, decide with cross-validation, report the test set.

The One-Sentence Summary

Out-of-bag error works because Nim was right: the moment you draw $n$ cards from a bin of $n$ with replacement, mathematics quietly locks away $1/ e$ of the bin — a different 36.8% for every tree — so the honest exam you were about to pay for by sealing data away and refitting five times has already been written and graded, and all you have to do is read the ledger of who saw what.

What's Next?

Now that you can validate a forest for free, you're ready for:

Extra Trees — tomorrow. A third source of randomness: stop searching for the best split and pick thresholds at random. Faster and often better — and its bootstrap=False default means OOB is off until you say otherwise.
AdaBoost — the first great alternative to averaging. Build trees in sequence and make each obsess over what the last got wrong.
Gradient Boosting — the same idea reframed as gradient descent in function space, each tree fitting the residuals of everything before it.
The Boosting series — where our whole variance-reduction story flips into a bias-reduction story, and every bagging intuition has to be re-earned.

Follow me for the next article in the Random Forests Deep Dive series!

Let's Connect!

If Nim's ledger made out-of-bag error click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a model on an OOB score? I have, twice — and once it was panel data with repeat customers. A beautiful 0.97 that fell apart in week one. Group your folds. 📚

What I keep coming back to is that Nim wasn't smarter than the schoolmaster. He just happened to be holding the ledger, and he was curious enough to read it. Most of the free information in an engineering organisation works like this: already generated, already written down by whoever does the unglamorous part of the job, and thrown away because nobody senior enough to act on it has ever looked.

Share this with the next person who runs 5-fold CV on a 500-tree forest and waits. Tell them to check the ledger first.

Feature Importance in Random Forests: The Company That Fired the Wrong Employee Because It Measured the Wrong Thing

Sachin Kr. Rajput — Fri, 31 Jul 2026 07:23:33 +0000

The One-Line Summary: feature_importances_ measures how much each column reduced impurity on the training rows, which quietly rewards columns with many possible split points instead of columns that actually predict anything — so a column of random IDs can outrank a genuinely useful binary flag, exactly like a company promoting the man who signs everything and firing the woman who prevents every disaster.

The Parable of the Company That Counted Signatures

Last time, a village blindfolded twelve judges and got better verdicts out of worse jurors. This time, a freight company spends eleven years not noticing that its one unarguable number measures the opposite of what it thinks.

Harrowgate & Vale moves cargo. Three hundred and forty staff, four warehouses, one rule everybody knows by heart: the Ledger decides.

The Ledger

Every time a job gets done, whoever did it signs the Ledger. At year end the partners count signatures. Most signatures, biggest bonus. Fewest signatures, cleaned desk. Nobody can argue with arithmetic.

THE LEDGER, YEAR ELEVEN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NAME      SIGNATURES   WHAT THEY SIGN FOR
Devon         38,412   every parcel that arrives
Marcus         4,510   the dock log, odd weeks
Wei            4,488   the dock log, even weeks
Otis           1,204   whichever floor he's on
Priya             96   the manifest, before a truck
                       leaves
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Devon signs for every parcel that comes through the door. He inspects nothing, decides nothing. His signature is on everything, which is another way of saying it tells you nothing: he signed all of them, the good and the ruined alike.

Priya does one thing. Twice a week she reads a departing truck's manifest against what is on the truck and writes one word: checked. Ninety-six words a year. When Priya checks a truck, it arrives whole. The fortnight she had shingles, three trucks went out wrong and Harrowgate lost its second-largest customer.

Marcus and Wei share the dock log on alternating weeks. Between them they cover every shift, and Marcus writing dock clear means identically the same thing as Wei writing it.

Otis signs for whichever floor he is standing on. Nobody knows why.

WHAT THE LEDGER DECIDED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Devon    38,412 signatures  →  DIRECTOR OF OPERATIONS
Marcus    4,510 signatures  →  "middling. let him go."
Wei       4,488 signatures  →  "middling. let him go."
Priya        96 signatures  →  "contributes nothing."

Promoted: the man whose signature means nothing.
Fired:    the woman who prevented every disaster,
          and BOTH people who ran the dock.

Three months later, four trucks left with the wrong manifests, the dock ran two days behind for a quarter, and Devon — now Director of Operations — requisitioned a larger stamp.

Shuffle Week

Ines Vaugh had been a junior partner for six weeks and had already learned not to attack the Ledger directly. So she attacked the question instead.

"Stop counting what people touch. Start measuring what breaks when what they do stops making sense."

Pick one person. For one week, secretly scramble their output: every mark they make gets randomly reassigned to a different job. Devon's signatures land on the wrong parcels. Priya's checked marks land on trucks she never looked at. Nothing else changes — same people, same routines, same number of marks. The marks just stop meaning anything.

Then count how many shipments go wrong, on next month's work — not on the eleven years of Ledger the company had already written about itself.

SHUFFLE WEEK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
whose marks were  shipments
scrambled         gone wrong   verdict
nobody (baseline)         29   —
Devon                     29   no damage at all
Otis                      30   no damage at all
Marcus                    30   no damage at all
Wei                       30   no damage at all
Priya                     71   SEVERE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Devon's marks were noise for eleven years, and
scrambling noise produces noise. Nothing moved.
Priya's one word a week was holding up the roof.

But read the dock rows literally and you get
"neither Marcus nor Wei matters" — wrong. Nothing
broke because the office simply read the OTHER
man's entries instead.

  scramble one  →  no damage
  scramble both →  the dock stops

The Ledger had ranked Devon four hundred times above Priya. Shuffle Week ranked Priya first and Devon at zero — and still fired both dock men.

Two people doing one job each look replaceable, because each one is replaceable, right up to the moment you replace both. So Ines added a second, expensive test, the empty chair: send the person home for a month and run the company without them. And when two people share a job, empty the chair, not the person.

Why the Ledger Lied

WHAT THE LEDGER WAS ACTUALLY MEASURING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The Ledger counted OPPORTUNITIES TO SIGN.

  Devon had 38,412 chances to be involved.
  Priya had 96.

Someone who touches everything will always
outscore someone who touches one thing, even when
the one thing is the only thing that matters.

Three lessons, one story:

  1. Counting activity is not measuring value.
       → this is IMPURITY-BASED IMPORTANCE (MDI)
  2. Break someone on purpose, then measure the
     damage — on work nobody has seen yet.
       → this is PERMUTATION IMPORTANCE
  3. Two people doing the same job each look
     replaceable, and both get fired.
       → this is the CORRELATED FEATURE problem

Everything below is those three sentences, written in Python.

What Is Feature Importance?

A trained Random Forest hands you feature_importances_ for free: one non-negative number per column, summing to 1.0. It looks like an answer. It is a measurement, and you need to know what got measured. Three different questions hide under "how important is this feature":

Question	Method	Measured on
How much did this column reduce training impurity?	MDI / Gini importance	training rows
How much worse is the model if this column becomes nonsense?	Permutation importance (MDA)	held-out rows
How much worse would a model be that never saw this column?	Drop-column importance	held-out rows, after refitting

feature_importances_ answers the first. Almost nobody wants the first.

The Math of MDI

Gini impurity of a node $t$ is $H (t) = 1 - \sum_{k} p_{k} (t)^{2}$ , and a split's impurity decrease is

Δ i (t) = H (t) - \frac{n _{L}}{n _{t}} H (t_{L}) - \frac{n _{R}}{n _{t}} H (t_{R})

MDI for feature $j$ sums that over every node in every tree that split on $j$ , weighted by the samples reaching each node, then averages over the $B$ trees:

MDI (j) = \frac{1}{B} b = 1 \sum B t \in T_{b} v (t) = j \sum \frac{n _{t}}{n} Δ i (t)

sklearn then normalizes the vector to sum to 1 — which is why MDI feels like a percentage of credit, and why people quote it to stakeholders.

Notice what is not in that formula: no held-out data. Not one row. Every quantity — $n_{t}$ , $p_{k} (t)$ , the chosen threshold — comes from the rows the tree was grown on. MDI is a training score, and we would never report training accuracy to a stakeholder.

Where the Bias Comes From

WHERE THE CARDINALITY BIAS COMES FROM
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A feature with u distinct values offers u-1
candidate thresholds at every single node.

  binary flag     u = 2      →     1 threshold
  floor number    u = 3      →     2 thresholds
  4-level code    u = 4      →     3 thresholds
  unique id       u = 1400   →  1399 thresholds

The tree keeps the BEST of those candidates.

Best-of-1399 lucky splits beats best-of-1 honest
split on the training rows almost every time —
and the winner's luck is recorded as impurity
reduction, which is reported as IMPORTANCE.

MDI pays for OPPORTUNITY and calls it VALUE.

That is Devon's 38,412 signatures. Two biases, one root:

Cardinality bias. High-cardinality columns get more chances to look good: IDs, timestamps, hashes, customer numbers, latitudes.
High-variance / in-sample bias. Impurity decrease is scored on the same rows used to choose the split, so any column that helps the tree memorize gets paid. Fully-grown trees make it worse: deep nodes hold few samples, and almost anything shatters them.

Permutation Importance, Formally

Fit once. Score on held-out data for a baseline $s$ . Then, for feature $j$ , shuffle that one column and re-score:

PI (j) = s - \frac{1}{R} r = 1 \sum R s! (f, X_{perm (j, r)}, y)

Shuffling breaks the relationship between column $j$ and $y$ while leaving $j$ 's marginal distribution untouched. If the model leaned on $j$ , the score falls. If it didn't, nothing happens — and if $j$ was actively misleading the model, the score rises and importance goes negative. Negative importance is a feature, not a bug: the column is worse than useless.

Step by Step

Fit the forest on the training set. Never refit — you are auditing this exact model.
Score it on held-out rows for your baseline $s$ .
For each feature $j$ , for each of $R$ repeats: shuffle column $j$ , keep $y$ where it is, score, record $s - s_{perm}$ .
Average over the repeats. Keep the standard deviation.
Sort. Anything within a couple of standard deviations of zero is noise, and say so out loud.

Feature Importance with Scikit-Learn

Harrowgate & Vale as a dataset. Five columns, five employees:

priya_check — a binary flag that genuinely drives the outcome
marcus_log — a 4-level code that genuinely drives the outcome
wei_log — a byte-for-byte duplicate of marcus_log
devon_seq — 2,000 unique integers, pure noise, no relationship to the label
floor_no — 3 levels, also pure noise

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

FEATURES = ["priya_check", "marcus_log", "wei_log", "devon_seq", "floor_no"]

def build_review_data(n=2000, seed=42):
    """Five 'employees'. Only two of them actually do anything."""
    rng = np.random.default_rng(seed)
    priya_check = rng.integers(0, 2, n)      # BINARY, genuinely predictive
    marcus_log  = rng.integers(0, 4, n)      # 4 levels, genuinely predictive
    wei_log     = marcus_log.copy()          # EXACT duplicate of Marcus
    devon_seq   = rng.permutation(n)         # 2000 unique values, PURE NOISE
    floor_no    = rng.integers(1, 4, n)      # 3 levels, pure noise
    log_effect  = np.array([-1.5, -0.45, 0.45, 1.5])
    logit = 3.0 * priya_check + log_effect[marcus_log] - 1.5
    y = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
    X = np.column_stack([priya_check, marcus_log, wei_log,
                         devon_seq, floor_no]).astype(float)
    return X, y

X, y = build_review_data()
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
rf = RandomForestClassifier(n_estimators=200, random_state=42,
                            n_jobs=-1).fit(X_tr, y_tr)

print(f"train accuracy: {rf.score(X_tr, y_tr):.4f}")
print(f"test  accuracy: {rf.score(X_te, y_te):.4f}")
print()
print("THE PERFORMANCE REVIEW (feature_importances_)")
for i in np.argsort(rf.feature_importances_)[::-1]:
    bar = "#" * int(round(rf.feature_importances_[i] * 50))
    print(f"  {FEATURES[i]:>11}  {rf.feature_importances_[i]:.4f}  {bar}")

train accuracy: 1.0000
test  accuracy: 0.7083

THE PERFORMANCE REVIEW (feature_importances_)
    devon_seq  0.5602  ############################
  priya_check  0.2722  ##############
   marcus_log  0.0711  ####
      wei_log  0.0663  ###
     floor_no  0.0303  ##

The most important feature in this model, by a factor of two, is a column of random integers. devon_seq holds 56% of the total, and it has no relationship to the label whatsoever — rng.permutation, drawn independently of y. The genuinely predictive binary flag comes second at 27%. And marcus_log and wei_log — a column that genuinely matters, present twice — sit near the bottom at 0.0711 and 0.0663. Middling. Let them go.

The model got promoted to production and fired Priya.

Note train accuracy: 1.0000 beside test accuracy: 0.7083. The forest memorized the training set perfectly, and devon_seq is how. MDI is not measuring prediction; it is measuring participation in memorization.

The Same Model, Audited Properly

Same forest, same rows. We change only the question.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

FEATURES = ["priya_check", "marcus_log", "wei_log", "devon_seq", "floor_no"]

def build_review_data(n=2000, seed=42):
    rng = np.random.default_rng(seed)
    priya, marcus = rng.integers(0, 2, n), rng.integers(0, 4, n)
    devon, floor = rng.permutation(n), rng.integers(1, 4, n)
    logit = 3.0 * priya + np.array([-1.5, -0.45, 0.45, 1.5])[marcus] - 1.5
    y = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
    # marcus appears twice: Marcus and Wei do the identical job
    return np.column_stack([priya, marcus, marcus,
                            devon, floor]).astype(float), y

X, y = build_review_data()
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
rf = RandomForestClassifier(n_estimators=200, random_state=42,
                            n_jobs=-1).fit(X_tr, y_tr)

mdi = rf.feature_importances_
tr = permutation_importance(rf, X_tr, y_tr, n_repeats=10,
                            random_state=42, n_jobs=-1)
te = permutation_importance(rf, X_te, y_te, n_repeats=10,
                            random_state=42, n_jobs=-1)

print(f"{'feature':>11} {'MDI':>8} {'perm TRAIN':>11} {'perm TEST':>11} {'+/-':>7}")
for i in np.argsort(te.importances_mean)[::-1]:
    print(f"{FEATURES[i]:>11} {mdi[i]:>8.4f} {tr.importances_mean[i]:>+11.4f} "
          f"{te.importances_mean[i]:>+11.4f} {te.importances_std[i]:>7.4f}")

    feature      MDI  perm TRAIN   perm TEST     +/-
priya_check   0.2722     +0.2926     +0.1488  0.0172
   floor_no   0.0303     +0.1921     -0.0053  0.0126
  devon_seq   0.5602     +0.2821     -0.0133  0.0167
 marcus_log   0.0711     +0.1452     -0.0258  0.0087
    wei_log   0.0663     +0.1381     -0.0263  0.0096

perm TEST is the column that tells the truth. Exactly one feature is positive: priya_check, at +0.1488 with a standard deviation of 0.0172 — eight sigma from zero, unambiguous. devon_seq scores -0.0133: shuffling the random ID improves held-out accuracy, because the model was memorizing through it. The feature MDI called most important is the one the model would be better off without.

perm TRAIN is the trap inside the fix. Permutation importance inherits the sins of whatever rows you run it on. On the training set, devon_seq scores +0.2821 — neck and neck with the real signal at +0.2926 — and even floor_no, three levels of pure noise, scores +0.1921. Of course: the model memorized through those columns, so scrambling them destroys the memorization. Permutation importance on training data is a different way of computing the same lie. The held-out set is not a nice-to-have; it is the entire mechanism.

Watching Cardinality Buy Importance

Keep a single genuinely predictive binary feature, add exactly one noise feature, and vary nothing but the number of distinct values that noise may take.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
n = 2000
priya_check = rng.integers(0, 2, n)              # the real signal, 2 levels
logit = 3.0 * priya_check - 1.5
y = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)

print(f"{'noise levels':>12} {'noise MDI':>10} {'priya MDI':>10} {'test acc':>9}")
for k in [2, 5, 20, 100, 500, 2000]:
    noise = rng.integers(0, k, n)                # pure noise, k distinct values
    X = np.column_stack([priya_check, noise]).astype(float)
    X_tr, X_te, y_tr, y_te = train_test_split(
        X, y, test_size=0.3, random_state=42, stratify=y
    )
    rf = RandomForestClassifier(n_estimators=200, random_state=42,
                                n_jobs=-1).fit(X_tr, y_tr)
    a, b = rf.feature_importances_
    print(f"{k:>12} {b:>10.4f} {a:>10.4f} {rf.score(X_te, y_te):>9.4f}")

noise levels  noise MDI  priya MDI  test acc
           2     0.0018     0.9982    0.8033
           5     0.0160     0.9840    0.8033
          20     0.0772     0.9228    0.8033
         100     0.2800     0.7200    0.8000
         500     0.5118     0.4882    0.7367
        2000     0.5717     0.4283    0.6850

Nothing about that column's relationship to the label changed across those six rows — it is uniform random garbage in all of them. Its apparent importance went from 0.18% to 57.17%, a 300-fold swing, bought entirely with distinct values.

Now the last column. Test accuracy holds flat at 0.8033 while the noise stays coarse, then collapses to 0.6850 once it is fine enough to memorize with. The feature MDI promotes hardest is the one doing the most damage — the same mechanism read from both ends.

The Correlated Pair That Fools Everything

The hard case: the one that survives switching to permutation importance, and the one I have watched cost real teams real accuracy.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

def build_review_data(n=2000, seed=42):
    rng = np.random.default_rng(seed)
    priya, marcus = rng.integers(0, 2, n), rng.integers(0, 4, n)
    devon, floor = rng.permutation(n), rng.integers(1, 4, n)
    logit = 3.0 * priya + np.array([-1.5, -0.45, 0.45, 1.5])[marcus] - 1.5
    y = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
    return np.column_stack([priya, marcus, marcus,
                            devon, floor]).astype(float), y

X, y = build_review_data()

def train_on(cols, seeds=(42, 43, 44)):
    """Refit from scratch on a column subset -> (mean acc, first model)."""
    a, b, c, d = train_test_split(X[:, cols], y, test_size=0.3,
                                  random_state=42, stratify=y)
    ms = [RandomForestClassifier(n_estimators=200, random_state=s,
                                 n_jobs=-1).fit(a, c) for s in seeds]
    return float(np.mean([m.score(b, d) for m in ms])), ms[0]

# --- 1. what a duplicate does to MDI ---
_, both = train_on([0, 1, 2, 3, 4])
_, solo = train_on([0, 1, 3, 4])          # Wei deleted from the payroll
m, w = both.feature_importances_[1], both.feature_importances_[2]
print("MDI for the log-keeping job:")
print(f"  two people share it: marcus={m:.4f} + wei={w:.4f} = {m + w:.4f}")
print(f"  one person does it:   marcus={solo.feature_importances_[1]:.4f}")

# --- 2. drop-column importance: the only honest audit ---
full, _ = train_on([0, 1, 2, 3, 4])
print(f"\nDROP-COLUMN IMPORTANCE   (baseline test acc = {full:.4f})")
for label, cols in [("fire priya_check", [1, 2, 3, 4]),
                    ("fire marcus_log ", [0, 2, 3, 4]),
                    ("fire wei_log    ", [0, 1, 3, 4]),
                    ("fire devon_seq  ", [0, 1, 2, 4]),
                    ("fire floor_no   ", [0, 1, 2, 3]),
                    ("fire BOTH logs  ", [0, 3, 4])]:
    acc, _ = train_on(cols)
    print(f"  {label}  acc={acc:.4f}  importance={full - acc:+.4f}")

MDI for the log-keeping job:
  two people share it: marcus=0.0711 + wei=0.0663 = 0.1374
  one person does it:   marcus=0.1470

DROP-COLUMN IMPORTANCE   (baseline test acc = 0.7083)
  fire priya_check  acc=0.5628  importance=+0.1456
  fire marcus_log   acc=0.7083  importance=+0.0000
  fire wei_log      acc=0.7083  importance=+0.0000
  fire devon_seq    acc=0.7750  importance=-0.0667
  fire floor_no     acc=0.7017  importance=+0.0067
  fire BOTH logs    acc=0.6433  importance=+0.0650

MDI splits the credit almost exactly in half. One column doing the log job scores 0.1470. Two identical columns doing it score 0.0711 and 0.0663 — summing to 0.1374, the same total, halved. Duplicate a column ten times and you have ten features that all look like rounding error. The information did not get less useful. It got divided.

Permutation importance doesn't split the credit, it destroys it. In the previous table marcus_log scored -0.0258 and wei_log -0.0263 on held-out data. Not small — negative. Shuffling Marcus does nothing, because the forest reads Wei instead; shuffling Wei does nothing, because the forest reads Marcus. An analyst reading that table drops both, ships it, and loses accuracy.

Drop-column gets it right — but only if you drop the group. Firing marcus_log costs +0.0000. Firing wei_log costs +0.0000 (identical, because it is literally the same column). Firing both costs +0.0650, the second-biggest number in the analysis. One at a time, free. Together, expensive.

And the punchline the Ledger never saw coming: firing devon_seq scores -0.0667. Deleting the model's "most important feature" makes it 6.7 points more accurate, 0.7083 to 0.7750. Devon was not useless. Devon was the most expensive employee at Harrowgate & Vale.

THE HIERARCHY OF HONESTY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                    devon    priya   marcus     wei
MDI (train)        0.5602   0.2722   0.0711  0.0663
                     ↑ #1     ↑ #2   ↑ weak  ↑ weak

permutation (test)-0.0133  +0.1488  -0.0258 -0.0263
                   ↑ zero     ↑ #1  ↑ negative!

drop-column       -0.0667  +0.1456  +0.0000 +0.0000
drop the GROUP          —        —   +0.0650 (!)

  MDI          got EVERYTHING wrong.
  permutation  got the noise right, the pair wrong.
  drop-group   got all of it right, for p+1 fits.

How to Actually Handle Correlated Features

Compute the Spearman rank correlation matrix of your features.
Hierarchically cluster on 1 - |rho| (scipy.cluster.hierarchy.ward, then fcluster).
Keep one representative per cluster, or permute and drop whole clusters.
Report importance per cluster, not per column.

If you take one habit from this article, take that one. It converts an unanswerable question ("is Marcus important?") into an answerable one ("is the dock log important?").

Permutation Importance from Scratch

Twelve lines of NumPy. No gradients, no impurity bookkeeping, no access to the model's internals — just predict, a shuffle, and subtraction. Which is why it works on anything: forests, boosters, neural nets, ONNX blobs behind an HTTP call.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split

FEATURES = ["priya_check", "marcus_log", "wei_log", "devon_seq", "floor_no"]

def my_permutation_importance(model, X, y, n_repeats=10, seed=42):
    """Permutation importance in 12 lines. This is the whole algorithm."""
    rng = np.random.default_rng(seed)
    baseline = float((model.predict(X) == y).mean())   # score once, honestly
    scores = np.zeros((X.shape[1], n_repeats))
    Xp = X.copy()                                     # never touch caller's X
    for j in range(X.shape[1]):
        original = Xp[:, j].copy()
        for r in range(n_repeats):
            Xp[:, j] = rng.permutation(original)      # break j <-> y only
            shuffled = float((model.predict(Xp) == y).mean())
            scores[j, r] = baseline - shuffled        # how much did we lose?
        Xp[:, j] = original                           # put the column back
    return scores.mean(axis=1), scores.std(axis=1)

def build_review_data(n=2000, seed=42):
    rng = np.random.default_rng(seed)
    priya, marcus = rng.integers(0, 2, n), rng.integers(0, 4, n)
    devon, floor = rng.permutation(n), rng.integers(1, 4, n)
    logit = 3.0 * priya + np.array([-1.5, -0.45, 0.45, 1.5])[marcus] - 1.5
    y = (rng.random(n) < 1.0 / (1.0 + np.exp(-logit))).astype(int)
    return np.column_stack([priya, marcus, marcus,
                            devon, floor]).astype(float), y

X, y = build_review_data()
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)
rf = RandomForestClassifier(n_estimators=200, random_state=42,
                            n_jobs=-1).fit(X_tr, y_tr)

mine, mine_sd = my_permutation_importance(rf, X_te, y_te, n_repeats=10, seed=42)
ref = permutation_importance(rf, X_te, y_te, n_repeats=10,
                             random_state=42, n_jobs=-1)

print(f"{'feature':>11} {'mine':>9} {'sklearn':>9} {'diff':>8}")
for i in range(X.shape[1]):
    print(f"{FEATURES[i]:>11} {mine[i]:>+9.4f} {ref.importances_mean[i]:>+9.4f} "
          f"{abs(mine[i] - ref.importances_mean[i]):>8.4f}")
print(f"\nmax absolute disagreement: "
      f"{np.max(np.abs(mine - ref.importances_mean)):.4f}")
print(f"monte-carlo noise (sklearn std): {ref.importances_std.mean():.4f}")

    feature      mine   sklearn     diff
priya_check   +0.1518   +0.1488   0.0030
 marcus_log   -0.0272   -0.0258   0.0013
    wei_log   -0.0263   -0.0263   0.0000
  devon_seq   -0.0045   -0.0133   0.0088
   floor_no   -0.0077   -0.0053   0.0023

max absolute disagreement: 0.0088
monte-carlo noise (sklearn std): 0.0130

The largest disagreement between my twelve lines and sklearn's implementation is 0.0088 — smaller than sklearn's own repeat-to-repeat standard deviation of 0.0130. Same estimator, different seeds. sklearn.inspection parallelizes, supports arbitrary scoring, and handles pandas; that is the entire difference.

That 0.0130 deserves a stare. Your importances have error bars about as wide as the gap between third and fifth place, and n_repeats=5 on a small test set will reorder your leaderboard between runs. Always print the standard deviation. If you are going to delete a column because of a number, earn it with n_repeats=30.

When Does Each Method Help Most?

FEATURE IMPORTANCE: THE HONEST ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
REACH FOR MDI WHEN:
  ✓ You want a zero-cost first glance
  ✓ All features share similar cardinality
  ✓ You will verify before acting on it
  ✗ Never when a high-cardinality column exists
  ✗ Never in a slide shown to a stakeholder

REACH FOR PERMUTATION IMPORTANCE WHEN:
  ✓ You need a number you can defend
  ✓ You care about a specific metric (AUC, recall)
  ✓ Refitting the model is expensive
  ✓ You have a genuine held-out set
  ✗ Not when features are strongly correlated

REACH FOR DROP-COLUMN WHEN:
  ✓ You are actually deciding what to delete
  ✓ p+1 refits fit in your compute budget
  ✓ You drop correlated GROUPS, not members

NONE OF THEM TELL YOU:
  ✗ Causation. Not even slightly.
  ✗ Direction of effect (does it push up or down?)
  ✗ Anything about ONE specific prediction
  ✗ Whether a feature would help a DIFFERENT model

Two caveats. Permutation importance evaluates the model off the data manifold: shuffle height independently of weight and you invent 140 cm adults weighing 110 kg, so the model gets scored on inputs that cannot exist. That is a genuine objection and the reason conditional permutation schemes exist; for most tabular work it is second-order, and for tightly coupled physical quantities it is not.

And none of this explains a prediction. Everything here is global: this column matters to this model overall. When a declined loan applicant asks why their application failed, global importance is the wrong tool — that is what SHAP values are for, per-row additively-decomposed attributions with real axioms behind them, and the subject of its own article later in this series. Global importance ranks columns; SHAP explains rows.

The Knobs That Change the Answer

Parameter	Where	What it does	Guidance
`n_repeats`	`permutation_importance`	shuffles per feature	10 to look, 30+ to delete
`scoring`	`permutation_importance`	which metric gets damaged	Set it explicitly. Accuracy-based importance on an imbalanced problem is meaningless.
`max_samples`	`permutation_importance`	subsample the eval rows	Lower for speed on big test sets
`random_state`	`permutation_importance`	the shuffle order	`42`, or your leaderboard moves
`n_jobs`	`permutation_importance`	parallelism	`-1`. Embarrassingly parallel.
`max_depth`, `min_samples_leaf`	the forest	how deep MDI's in-sample bias reaches	Shallower trees shrink the bias, never remove it
`max_features`	the forest	which columns get a chance per split	Low values spread MDI across a correlated group

One anti-pattern worth naming: permutation_importance(rf, X_train, y_train), because that is the variable in scope. We measured the cost — pure noise at +0.2821, a hair below the real signal. That is not a milder version of the right answer. It is the wrong answer with extra compute.

Quick Reference Card

FEATURE IMPORTANCE: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE THREE METHODS:
  MDI          impurity drop on TRAIN rows.
               free (already computed), badly biased.
  PERMUTATION  score drop after shuffling column j
               on HELD-OUT rows. R·p predicts, no refit.
  DROP-COLUMN  refit without column j. exact, p+1 fits.

MDI's TWO BIASES:
  cardinality  u distinct values → u-1 thresholds →
               more chances to get lucky → more credit
  in-sample    it is a training score wearing a hat

WHAT BREAKS MDI *AND* PERMUTATION:
  correlated features. MDI splits the credit,
  permutation masks it (each twin covers the other).
  Fix: cluster on Spearman rho, drop whole groups.

DEFAULTS THAT ARE ACTUALLY GOOD:
  n_repeats = 10 to look, 30 to decide
  scoring   = whatever you optimise in production
  evaluate  = ALWAYS on held-out rows
  report    = mean AND standard deviation

WHAT NOT TO DO:
  ✗ Ship feature_importances_ to a stakeholder
  ✗ Run permutation importance on the training set
  ✗ Drop a correlated group one member at a time
  ✗ Read any of it as causation

SKLEARN:
  from sklearn.inspection import permutation_importance
  rf.feature_importances_   # MDI — handle with care

Key Takeaways

feature_importances_ is a training-set statistic. It sums impurity decrease over the rows the tree was grown on and never looks at held-out data. You would not quote training accuracy; stop quoting this.
MDI is biased toward high cardinality. $u$ distinct values buy $u - 1$ thresholds per node, and the tree keeps the luckiest. Pure noise climbed from 0.18% to 57.17% importance on nothing but distinct values.
A random ID beat a genuinely predictive binary flag, 0.5602 to 0.2722. Not a contrived edge case — an unpruned forest, one unique-per-row column, the defaults everyone uses.
Permutation importance fixes it, but only on held-out rows. The same audit on training data scored the noise column +0.2821 against +0.2926 for the real signal. The held-out set is the mechanism, not a formality.
Negative importance is information. devon_seq scored -0.0133 on permutation and -0.0667 on drop-column: actively harmful, and deleting it moved accuracy from 0.7083 to 0.7750.
Correlated features break MDI by division and permutation by masking. Two identical columns split 0.1470 of MDI into 0.0711 and 0.0663, and each scored negative under permutation because the other covered for it.
Dropping one of a correlated pair costs nothing; dropping both costs everything. +0.0000 each, +0.0650 together. Drop groups, never members.
Drop-column importance is ground truth and costs $p + 1$ refits — cheap when you are about to delete columns from production. And no global method explains a single prediction; that is SHAP's job.

The One-Sentence Summary

feature_importances_ counts how often and how usefully each column got to help memorize the training set, so it pays for opportunity rather than value and will hand its top rank to a column of random IDs while a genuinely decisive binary flag looks like rounding error — audit your model the way Ines audited Harrowgate & Vale: scramble one column at a time on rows nobody has seen, believe the damage rather than the activity, and when two columns do the same job, empty the chair instead of firing the person.

What's Next?

Now that you know your forest's self-assessment is unreliable, you're ready for:

Out-of-Bag Error — tomorrow's deep dive. Bootstrapping leaves roughly 36.8% of rows out of every tree, so you already own a free held-out set living inside your model — and it is the cheapest legitimate place to compute the permutation importance you just learned.
Extra Trees — a third source of randomness: stop searching for the best split and pick thresholds at random. Faster, sometimes better, and it moves the cardinality bias in an interesting direction.
AdaBoost — the algorithm that made weak learners fashionable, where trees grow in sequence and each is paid to fix the last one's mistakes.
The max_features default — Friday's short post, and the most expensive one-word default in scikit-learn regression. For classification it is 'sqrt'. For regression it is not.

Follow me for the next article in the Random Forests Deep Dive series!

Let's Connect!

If Harrowgate & Vale's Ledger made feature importance click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a feature-importance chart you'd now take back? I have. A customer-ID column once made it into a "top drivers of churn" deck of mine, and a very kind data scientist asked, in front of everyone, what it would mean to increase a customer's ID. 📋

Every organisation I have worked in has had a Ledger — a number easy to count, impossible to argue with, and quietly measuring the wrong thing. Story points. Lines of code. Tickets closed. Messages sent. The failure mode is always identical: the metric rewards whoever touches the most surface area, and the person holding the whole thing together with one decision a week gets a note about visibility. Machine learning did not invent this mistake. It just made it fast enough to watch.

Share this with whoever on your team is about to delete a feature because the bar chart was short. Ask them to empty the chair first.