Gradient Boosting: The Sculptor Who Fixes the Statue by Carving Only Its Mistakes

The One-Line Summary: Gradient Boosting builds a model as a running sum, where each new small tree is trained not on your labels but on how wrong the sum currently is, and only a fraction of its opinion is added — which turns "fit a model" into "walk downhill in the space of functions," and lets you swap the loss function to change what the algorithm even considers a mistake.

---

The Parable of the Stoneyard at Halvad

The town of Halvad commissioned a statue of its founder, and the stoneyard sent twelve carvers to bid on the work. Each carver wanted his own block of marble. Each promised a finished figure in ninety days.

The town council, which had no money and less patience, asked whether there was another way.

---

The Old Way: Twelve Blocks, Twelve Statues

For three generations the yard had worked one way. Give every carver a block, let each work alone, then set the twelve figures in a row and let the council vote for the one they liked best — or, in a famous compromise of 1891, have a stonemason chisel a thirteenth figure that split the difference between all twelve.

It worked. It was expensive, and it had one stubborn flaw.

TWELVE CARVERS, ONE DRAWING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All twelve learned from the same master drawing.
So all twelve made the SAME error at the brow.

  carver 1  brow 4mm too shallow
  carver 2  brow 4mm too shallow
  carver 3  brow 3mm too shallow
  ...
  carver 12 brow 4mm too shallow
  ------------------------------------------
  average   brow ~4mm too shallow

Averaging twelve independent slips cancels them.
Averaging twelve IDENTICAL slips cancels nothing.

Adding a thirteenth carver does not help.
This is the CORRELATION FLOOR, and it is a wall.

Devika, who ran the yard, had spent her career on the other side of that wall. Hire more carvers, get more independence, cancel more slips. She had squeezed everything out of it that could be squeezed.

Her apprentice Anay proposed something the yard considered close to heresy.

---

The Innovation: Stop Carving Statues

"Why is every carver making a statue? Only the first one needs to. Everyone after that should be carving the difference."

His plan was this. Take one block. Let the first carver rough out something crude — barely a figure, a suggestion of a head and shoulders. Then set the clay model beside it and hand the second carver a caliper instead of a design.

The second carver's job is not to make a statue. It is to walk around the block, measure where stone still stands proud of the model, and remove exactly that much. Nothing else. He never looks at the design. He only looks at the gap.

Then a third carver does the same against the new, smaller gap. Then a fourth.

THE CALIPER READINGS, ROUND BY ROUND
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Eight measuring points on the block. The number is
how far the stone still sits from the model, and
"total error" is the sum of the squared gaps.

start   flat block, cut to average height
        gaps: -9.5 -8.5 -6.5 -2.5 0.5 2.5 9.5 14.5
        total error 518.00

round 1 one cut, the coarsest one available
        total error 518.00 -> 134.00

round 2 gaps: -5.5 -4.5 -2.5 1.5 4.5 6.5 -2.5 2.5
        total error 134.00 -> 50.67

round 3 gaps: -1.3 -0.3 1.7 -1.0 2.0 4.0 -5.0 0.0
        total error  50.67 -> 34.00

round 4 total error  34.00 -> 24.61

Nobody carved a statue. A statue appeared.

Each carver is nearly useless alone. The second one's work, taken by itself, is a few gouges in a rock. But the sum of the block and all the corrections is a figure, and it gets closer every round.

---

The Part That Went Wrong First

Anay's first attempt destroyed a block, and the reason is the most important thing in this article.

His rule had been: remove everything the caliper says. Full correction, every round. On good marble it worked. On a block with a hidden flaw, the caliper read 9mm at a point where the stone was rotten, the carver took the full 9mm, the surface crumbled past the line, and the next carver read a negative gap and had nothing to add back.

Devika's fix was a rule so cautious it sounded like a joke.

"Take a tenth of what the caliper tells you. Then measure again."

Ten carvers now do the work of one. The figure emerges slowly, and it never overshoots, because no single measurement is ever trusted enough to matter on its own.

FULL TRUST vs A TENTH
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gap reads 9mm at a flawed point

FULL CORRECTION      take 9mm. Surface fails.
                     Gap is now -3mm and there is
                     no putting marble back.

A TENTH              take 0.9mm. Re-measure: 8.1mm.
                     Take 0.9mm. The flaw shows up
                     in the readings long before
                     you have cut through it.

Cost: 10x the passes.
Benefit: no cut is ever a commitment.

---

The Second Thing That Went Wrong

The clay model had a chipped ear. Nobody noticed for a week.

A caliper reports depth — how far off, in millimetres. At the chipped ear it reported an enormous gap, so every carver in sequence attacked the ear hardest, because that is where the biggest numbers were. They carved a beautiful, faithful reproduction of the damage.

Anay's fix was to change the instrument, not the method. The new instrument reported only direction: proud, or shy. No magnitude. A point that is 90mm off and a point that is 0.9mm off both come back as "proud, take a sliver."

TWO INSTRUMENTS, SAME METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gaps at four points:  -9.5  -2.5  0.5  14.5

DEPTH CALIPER        -9.5  -2.5  0.5  14.5
  reports magnitude. One bad reading dominates
  the whole round.

DIRECTION CALIPER    -1    -1    1    1
  reports only sign. A chipped ear gets the same
  attention as a fingernail.

Same carvers. Same procedure. Different definition
of "mistake" -- and that choice is a PARAMETER.

That is the whole idea. What the caliper measures is the algorithm's definition of a mistake, and you get to choose it.

---

What Is Gradient Boosting?

Hard pivot. Everything above has a name.

Gradient Boosting builds a prediction as a running sum of small models:

$$F_M(x) = F_0(x) + \nu \sum_{m=1}^{M} h_m(x)$$

where $F_0$ is a constant (the mean of $y$ for squared error — the flat block), each $h_m$ is a shallow tree, and $\nu \in (0, 1]$ is the learning rate, Devika's tenth.

The trick is what $h_m$ is trained on. Not $y$ . It is trained on the negative gradient of the loss with respect to the current prediction:

$$r_i^{(m)} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]{F = F{m-1}}$$

These are the pseudo-residuals, and they are the caliper reading.

Why "Gradient"

Gradient descent on parameters updates $\theta \leftarrow \theta - \eta \nabla_\theta L$ . Gradient boosting does the same thing one level up: the object being optimised is the function $F$ itself, evaluated at your $n$ training points.

Treat $\big(F(x_1), \dots, F(x_n)\big)$ as a point in $\mathbb{R}^n$ . The loss is a surface over that space. The steepest downhill direction is $-\nabla L$ , which is exactly $r^{(m)}$ above.

But you cannot just take that step, because it only tells you what to do at the $n$ points you have — it says nothing about a new $x$ . So you fit a tree to the gradient. The tree is a generalisable approximation of the downhill direction. Then you take a small step along it.

Boosting is gradient descent where each step has to be a model.

Where the Residuals Come From

For squared error, $L = \tfrac{1}{2}(y - F)^2$ :

$$-\frac{\partial L}{\partial F} = y - F$$

The negative gradient is the ordinary residual. This is why every tutorial explains gradient boosting as "fit the residuals" — for one specific loss, that happens to be true. It is a coincidence of squared error, not the definition.

For absolute error, $L = |y - F|$ :

$$-\frac{\partial L}{\partial F} = \text{sign}(y - F)$$

Now the target is $\pm 1$ . Magnitude is gone. That is the direction caliper, and it is not a metaphor — it is the derivative.

working residual r      = [-9.5 -2.5  0.5 14.5]
squared_error  -dL/dF   = [-9.5 -2.5  0.5 14.5]
absolute_error -dL/dF   = [-1. -1.  1.  1.]

For log loss on binary classification the gradient is $y_i - p_i$ where $p_i = \sigma(F(x_i))$ — a probability error, in the space of log-odds. Same machinery, third instrument.

Step by Step

1. Set $F_0(x) = \arg\min_c \sum_i L(y_i, c)$ . (Mean for squared error, median for absolute.)
2. For $m = 1, \dots, M$ :
   1. Compute pseudo-residuals $r_i^{(m)}$ for every training row.
   2. Fit a shallow regression tree $h_m$ to $(x_i, r_i^{(m)})$ .
   3. Optionally re-solve each leaf's value to minimise the actual loss inside that leaf, rather than using the mean gradient.
   4. Update $F_m = F_{m-1} + \nu\, h_m$ .
3. Return $F_M$ .

Step 2.3 is the one nobody mentions. For squared error it changes nothing — the leaf mean already minimises squared loss. For every other loss it matters, and it is why sklearn's absolute-error boosting is not simply "fit trees to signs."

---

Gradient Boosting with Scikit-Learn

The residual sequence from the parable, with real numbers:

import numpy as np
from sklearn.tree import DecisionTreeRegressor

X = np.array([[1.],[2.],[3.],[4.],[5.],[6.],[7.],[8.]])
y = np.array([2.0, 3.0, 5.0, 9.0, 12.0, 14.0, 21.0, 26.0])

F = np.full_like(y, y.mean())          # the flat block
print(f"F0 = mean(y) = {y.mean():.4f}")
for m in range(1, 5):
    r = y - F                          # the caliper reading
    t = DecisionTreeRegressor(max_depth=1).fit(X, r)
    F = F + 1.0 * t.predict(X)         # full correction, no shrinkage
    print(f"round {m}: residuals {np.round(r, 2)}")
    print(f"         SSE before {np.sum(r**2):.4f} -> "
          f"after {np.sum((y - F)**2):.4f}")

F0 = mean(y) = 11.5000
round 1: residuals [-9.5 -8.5 -6.5 -2.5  0.5  2.5  9.5 14.5]
         SSE before 518.0000 -> after 134.0000
round 2: residuals [-5.5 -4.5 -2.5  1.5  4.5  6.5 -2.5  2.5]
         SSE before 134.0000 -> after 50.6667
round 3: residuals [-1.33 -0.33  1.67 -1.    2.    4.   -5.    0.  ]
         SSE before 50.6667 -> after 34.0000
round 4: residuals [-2.17 -1.17  0.83 -1.83  1.17  3.17 -2.5   2.5 ]
         SSE before 34.0000 -> after 24.6111

Four depth-1 stumps — the weakest models in the library — took the error from 518 to 24.6. Look at round 4's residuals: some got worse than round 3 (point 1 went from -1.33 to -2.17) while the total fell. Boosting does not improve every point every round. It improves the sum.

---

Gradient Boosting from Scratch

Sixty lines, no gradient boosting import:

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=800, n_features=10, n_informative=6,
                       noise=10.0, random_state=7)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=7)


class MyGradientBoosting:
    """Squared-error gradient boosting. The whole algorithm."""

    def __init__(self, n_estimators=200, learning_rate=0.1, max_depth=3):
        self.n = n_estimators
        self.lr = learning_rate
        self.depth = max_depth

    def fit(self, X, y):
        self.F0 = y.mean()                       # step 1
        F = np.full(len(y), self.F0)
        self.trees = []
        for _ in range(self.n):                  # step 2
            g = y - F                            # 2.1  -dL/dF
            t = DecisionTreeRegressor(
                max_depth=self.depth,
                criterion="friedman_mse",
                random_state=0).fit(X, g)        # 2.2
            F += self.lr * t.predict(X)          # 2.4
            self.trees.append(t)
        return self

    def predict(self, X):
        F = np.full(len(X), self.F0)
        for t in self.trees:
            F += self.lr * t.predict(X)
        return F


mine = MyGradientBoosting().fit(Xtr, ytr)
sk = GradientBoostingRegressor(n_estimators=200, learning_rate=0.1,
                               max_depth=3, random_state=0).fit(Xtr, ytr)
pm, ps = mine.predict(Xte), sk.predict(Xte)

print(f"from scratch R2 = {r2_score(yte, pm):.6f}")
print(f"sklearn      R2 = {r2_score(yte, ps):.6f}")
print(f"correlation between the two = {np.corrcoef(pm, ps)[0, 1]:.8f}")
print(f"y std in the test set = {yte.std():.1f}")
print(f"RMSE between the two models = {np.sqrt(((pm - ps)**2).mean()):.6f}")

from scratch R2 = 0.970898
sklearn      R2 = 0.970890
correlation between the two = 0.99998337
y std in the test set = 108.4
RMSE between the two models = 0.584202

Twenty lines of algorithm, matching the library to five decimal places of R2.

They are not bit-identical, and the reason is worth a paragraph. I checked where they part company:

 n_trees   max|diff|    R2 mine     R2 sklearn
     1      0.000000   0.131791    0.131791
     2      0.000000   0.240005    0.240005
     5      0.000000   0.475405    0.475405
    10      0.000000   0.683293    0.683293
    50      5.615992   0.947227    0.947312
   200      5.615992   0.970898    0.970890

Identical for the first ten rounds. First divergence at round 23, affecting 1 test row out of 240, by 5.6 on a target with a standard deviation of 108 — one split hit a tie between two equally good thresholds and my tree's random state broke it the other way. Then it never grew, because everything after round 23 is fitting residuals that both models agree about.

That is what a correct reimplementation looks like. Not zero difference — a difference you can name.

---

Shrinkage Is Not Optional

The learning rate is the single most consequential number in boosting. Here is what it does on a deliberately hard problem: 600 rows, 40 features, only 3 informative, noise 60.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score

X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
                       noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)

for lr in (1.0, 0.3, 0.1, 0.03):
    m = GradientBoostingRegressor(n_estimators=800, learning_rate=lr,
                                  max_depth=3, random_state=0).fit(A, a)
    st = np.array([r2_score(b, p) for p in m.staged_predict(B)])
    k = st.argmax()
    print(f"lr={lr:<5} peak test R2 {st[k]:.4f} @ {k+1:>3} trees | "
          f"@800 {st[-1]:.4f} | lost {st[k]-st[-1]:.4f}")

lr=1.0   peak test R2 0.1860 @   1 trees | @800 -0.3134 | lost 0.4994
lr=0.3   peak test R2 0.2838 @   7 trees | @800 0.1667 | lost 0.1171
lr=0.1   peak test R2 0.2717 @  27 trees | @800 0.1805 | lost 0.0911
lr=0.03  peak test R2 0.2795 @  95 trees | @800 0.2237 | lost 0.0558

Three things in that table.

Boosting genuinely overfits. At lr=1.0 the model peaks after a single tree and then falls to R2 = -0.31 — worse than predicting the mean for every row. This is the sharpest difference from Random Forests, where adding trees is never harmful. Boosting is not an ensemble that averages away error; it is an optimiser, and optimisers can converge on noise.

Smaller steps land in a better place. Peak R2 goes 0.186 at lr=1.0 to 0.284 at lr=0.3. Full-trust corrections leap over the good solutions.

Smaller steps also decay more gently. Going from peak to 800 trees costs 0.4994 at lr=1.0 and 0.0558 at lr=0.03. Low learning rates are forgiving about where you stop.

And the peak moves in a suspiciously regular way: 1, 7, 27, 95 trees as the rate falls 1.0, 0.3, 0.1, 0.03. That relationship is tomorrow's whole article.

---

Changing the Loss Changes the Model

The chipped ear, measured. Same algorithm, same trees, same rate — 3% of training labels corrupted by +800, scored against clean test labels.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

X, y = make_regression(n_samples=800, n_features=12, n_informative=6,
                       noise=8.0, random_state=3)
rng = np.random.default_rng(3)
y_bad = y.copy()
y_bad[rng.choice(len(y), 24, replace=False)] += 800.0      # 3% corrupted

Xtr, Xte, ytr, _ = train_test_split(X, y_bad, test_size=0.3, random_state=3)
*_, yte_clean = train_test_split(X, y, test_size=0.3, random_state=3)

print("loss             test MAE (vs clean)   test R2")
for loss in ("squared_error", "absolute_error", "huber"):
    m = GradientBoostingRegressor(loss=loss, n_estimators=300,
                                  learning_rate=0.1, max_depth=3,
                                  random_state=0).fit(Xtr, ytr)
    p = m.predict(Xte)
    print(f"{loss:<16} {mean_absolute_error(yte_clean, p):>13.4f}   "
          f"{r2_score(yte_clean, p):>9.4f}")

loss             test MAE (vs clean)   test R2
squared_error          48.9937      0.4223
absolute_error         19.6481      0.9535
huber                  28.2967      0.6838

R2 went from 0.42 to 0.95 by changing one string. Not more trees, not tuning, not more data — a different definition of "mistake." Squared error chased 24 bad labels and wrecked the model for the other 776. Absolute error saw them as "proud, take a sliver" and moved on.

This is the practical payoff of the gradient framing. AdaBoost was welded to exponential loss. Gradient boosting takes any differentiable loss you can write down, which is why the same three lines handle regression, classification, ranking, and quantile prediction.

---

When Does Gradient Boosting Help Most?

	Gradient Boosting	Random Forest
Attacks	bias, sequentially	variance, in parallel
Trees	shallow, depth 2–6	deep, unpruned
More trees	can overfit badly	never harmful
Parallel over trees	no, inherently serial	yes
Tuning needed	real; lr × depth × n interact	minimal
Noisy labels	dangerous with squared loss	comparatively robust
Typical tabular winner	yes	rarely

Reach for boosting when you have structured tabular data, a clean-ish target, and the time to tune. Reach for a forest when you want a strong answer in one fit with no thinking. Reach for neither when your features are pixels, audio, or text.

---

Hyperparameters

Parameter	What it does	Typical range	How to tune
learning_rate	fraction of each correction applied	0.01–0.1	set low, then find n_estimators
n_estimators	number of rounds	100–5000	early stopping, never a fixed guess
max_depth	interaction order per tree	2–6	3 is a real default, not a placeholder
subsample	rows per round (stochastic GB)	0.5–1.0	0.8 adds variance reduction, nearly free
min_samples_leaf	leaf floor	1–20	raise on noisy data
loss	definition of a mistake	see above	absolute_error when labels are dirty
n_iter_no_change	early-stopping patience	10–50	set it; do not tune n_estimators by hand
validation_fraction	rows held out for stopping	0.1	leave it

---

Quick Reference Card

GRADIENT BOOSTING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  A running sum of shallow trees, each fit to the
  negative gradient of the loss at the current sum,
  each added at only a fraction of its full value.

THE UPDATE:
  F_0 = argmin_c sum L(y_i, c)
  r_i = -dL(y_i, F(x_i)) / dF(x_i)
  h_m = tree fit to (x_i, r_i)
  F_m = F_(m-1) + lr * h_m

WHAT THE GRADIENT IS:
  squared_error   ->  y - F        (the residual)
  absolute_error  ->  sign(y - F)
  log_loss        ->  y - sigmoid(F)

KEY HYPERPARAMETERS:
  learning_rate     0.01-0.1  (lower = more trees)
  n_estimators      set by early stopping
  max_depth         2-6
  subsample         0.8
  loss              squared / absolute / huber

WHEN TO USE:
  + tabular data with real structure
  + you can afford to tune
  + you need the last two points of accuracy
WHEN NOT EFFECTIVE:
  - dirty labels + squared loss
  - you cannot early-stop
  - images, audio, raw text
  - you need one fit and no tuning (use a forest)

SKLEARN:
  from sklearn.ensemble import GradientBoostingRegressor
  from sklearn.ensemble import GradientBoostingClassifier
  # for anything real, use HistGradientBoosting* instead

---

Key Takeaways

1. The new tree is fit to the gradient, not to your labels. "Fit the residuals" is true only for squared error, where the gradient happens to equal the residual.

2. Boosting is gradient descent in function space. Each step must be a model, because the raw gradient only exists at your training points and cannot generalise to a new row.

3. The learning rate is not a speed knob, it is a regulariser. At lr=1.0 the test R2 peaked after one tree and fell to -0.31 by 800. At lr=0.03 it peaked at 95 and lost a tenth as much.

4. Boosting can overfit; forests essentially cannot. More trees is a free improvement in a Random Forest and a real risk here. Early stopping is not optional.

5. The loss function is a hyperparameter with more leverage than most. Corrupting 3% of labels cost squared error 0.53 R2; switching to absolute_error recovered it, with no other change.

6. Shallow trees are the point. Depth 3, not depth None. Each tree only has to find one more slice of the remaining error, and a deep tree would take the whole gradient in one bite.

7. Sixty lines reproduce the library. If you can write the update loop from memory, you know where every hyperparameter acts — and you will believe the results more.

8. Nothing in this algorithm is parallel over trees. Tree $m$ needs $F_{m-1}$ . All the speed in XGBoost and LightGBM comes from parallelising inside a tree.

---

The One-Sentence Summary

Gradient Boosting is the sculptor's insight that only the first worker needs to make a statue and everyone after should carve nothing but the measured gap — formalised as gradient descent in the space of functions, where the step direction is the negative derivative of whatever loss you choose, the step itself must be a shallow tree so it generalises off the training rows, and the step size is deliberately tiny so that no single measurement is ever trusted enough to cut through the stone.

---

What's Next?

Now that you understand the update rule, you're ready for:

1. learning_rate and n_estimators are one parameter — tomorrow. The peaks in that shrinkage table landed at 1, 7, 27, and 95 trees for rates of 1.0, 0.3, 0.1, 0.03. That is not a coincidence, and it changes how you search.
2. XGBoost — second-order gradients, regularisation moved into the objective, and the engineering that won a decade of Kaggle.
3. LightGBM — histogram binning and leaf-wise growth, for when 800 trees over ten million rows has to finish today.
4. CatBoost — ordered boosting, which fixes a target leak the other two quietly live with.

Follow me for the next article in the Boosting: The Complete Guide series!

---

Let's Connect!

If the caliper made gradient boosting click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever fixed a model by changing the loss instead of the features? It is the highest-leverage line in my file and the last one I ever think to touch. 🗿

---

The parable has a detail I did not invent for effect: the carvers who trusted their calipers completely did the most damage. Gradient boosting encodes that as a number between 0.01 and 0.1, and every time I have been tempted to raise it to save wall-clock time, the model has been worse in a way that took me a week to notice.