Bagging vs Boosting: The Orchestra That Plays Together vs The Relay Team That Runs One at a Time

The One-Line Summary: Bagging and boosting both turn mediocre trees into strong models, from exactly opposite directions — the orchestra plays every part at once and averages away the wobble, the relay hands each runner only the distance the previous one failed to cover. Measured on identical data they finish within 0.08 of the same total error, and yet removing a single tree costs the orchestra 0.0014 and costs the relay 0.2361.

---

The Parable of the Amberton Festival

Every autumn Amberton holds two contests on the same field: the orchestra plays at noon, and the relay runs at four. For thirty years the town treated them as unrelated amusements. Then a bandmaster who also kept the running club's stopwatch noticed that both were answers to the same question — how do you get a hundred imperfect people to produce one correct result? — and that the two answers were exact opposites.

---

The Orchestra: Forty Players, One Note

Not one of the forty violinists is perfectly in tune. Ask any of them to play alone and you will hear it. Some run a little sharp, some a little flat, and no two are wrong in the same way on the same evening.

Played together, something strange happens. The sharpness of one player sits on top of the flatness of another and the two quietly cancel. Nobody is corrected. Nobody listens to anybody else. They simply all play at once, and the hall hears a note steadier than any single instrument producing it.

THE ORCHESTRA — FORTY PLAYERS, ONE NOTE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The note on the page:  440

  violin  1  plays  443   (+3 sharp)
  violin  2  plays  437   (-3 flat)
  violin  3  plays  441   (+1 sharp)
  violin  4  plays  436   (-4 flat)
     ...
  violin 40  plays  442   (+2 sharp)

  what the hall hears:  440.1

No player was corrected. No player heard another.
Each was wrong; the wrongness pointed in random
directions and cancelled.

Add a 41st violin and the hall gets steadier still.
It can never get worse.

This is called VARIANCE REDUCTION.

The important detail is the one that sounds like a limitation: no violinist is trying to fix anyone. The cancellation is a free side effect of independence.

---

The Relay: Four Runners, One Race

The relay is built on the opposite principle. Nobody runs the whole race.

The first runner covers most of the distance, badly. The second does not start at the beginning — he starts exactly where the first one stopped, and covers most of what is left. The third takes the remainder of that. Each runner's job is defined entirely by the failure of the runner before.

"Why should the second man repeat the first man's ground? Give him only the distance the first man failed to cover."

THE RELAY — EACH RUNNER TAKES WHAT IS LEFT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Distance to the line:  1000 m

  runner 1  covers 700   →  300 left
  runner 2  covers 210   →   90 left
  runner 3  covers  63   →   27 left
  runner 4  covers  19   →    8 left

Runner 2 never attempts the first 700 m. That part
is finished. He is handed 300 m of pure failure and
nothing else.

Reorder them and it is not the same race — runner
4's leg exists only because runners 1-3 already ran.

This is called FITTING THE RESIDUAL.

Four runners got within 8 m. Forty violinists would still be averaging.

---

The Two Years It Went Wrong

Amberton remembers two festivals badly, and each one broke a different method.

In one year the orchestra tuned from a piano that had been left by a window all winter and had gone flat. All forty players tuned to it faithfully. All forty played the same wrong note, and the average of forty identical mistakes is that mistake. A forty-first violin would not have helped.

In another year a groundsman set the first marker forty metres short. Runner one stopped where the marker said. Runner two measured his leg from that point, honestly and carefully, and so did three and four. Every correction after the first was faithful to a false premise, and the team ran, confidently and in perfect form, to the wrong place.

TWO WAYS TO FAIL
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE FLAT PIANO
  Every violin tuned from the same flat piano.
  All forty played 436. The average was 436.
  A 41st violin also played 436.
  ✗ Averaging cannot remove an error that everyone
    shares. This is CORRELATION, and it is the
    floor the orchestra can never get under.

THE SHORT MARKER
  The first marker was set 40 m short.
  Runner 2 measured his leg from it. So did 3 and 4.
  Each correction was faithful to a false start.
  ✗ The relay never questions the baton. It
    inherits it. This is NOISE SENSITIVITY.

---

Why Each One Works

THE MATHEMATICS OF AVERAGING VS CORRECTING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AVERAGING N members whose errors have variance σ²
and average pairwise correlation ρ:

    variance  =  ρσ²  +  (1-ρ)σ²/N
                  ↑            ↑
             shared error   private error
             never shrinks  → 0 as N grows

  REQUIREMENTS
  1. Each member is fit independently.
  2. Members must DISAGREE (low ρ), or step 1
     bought you nothing.
  3. Each member should be LOW BIAS — averaging
     removes variance and never touches bias.

CORRECTING: member m is fit to what m-1 left over.

    F_m(x) = F_{m-1}(x) + lr · h_m(x)

  REQUIREMENTS
  1. Order is fixed. Members are not swappable.
  2. Each member should be WEAK — a strong one
     eats the whole residual and the chain stops.
  3. There is no ρ floor to worry about. There is
     also no protection from a wrong target.

Two methods, two entirely different failure modes, and — this is the part almost nobody checks — very often the same final accuracy.

---

What Are Bagging and Boosting?

Bagging (bootstrap aggregating) fits $N$ copies of the same learner on $N$ different bootstrap resamples of the training set, then averages their predictions. The members never see each other. The fit is embarrassingly parallel.

Boosting fits members in sequence. Each one is trained on the errors the current ensemble still makes, and is added to the ensemble with a small weight. The fit is inherently serial.

The Math

For bagging, the variance of the average of $N$ predictors with individual variance $\sigma^2$ and mean pairwise correlation $\rho$ is:

$$\text{Var}(\bar{f}) = \rho\sigma^2 + \frac{1 - \rho}{N}\sigma^2$$

The second term vanishes as $N \to \infty$ . The first does not. Everything interesting about bagging is a fight to lower $\rho$ , which is exactly what random forests do by sampling features at each split.

For boosting, the model is a stagewise additive expansion. At stage $m$ we fit a new learner to the negative gradient of the loss and take a damped step:

$$F_m(x) = F_{m-1}(x) + \nu \cdot h_m(x), \qquad h_m \approx -\frac{\partial L}{\partial F_{m-1}}$$

There is no $\rho$ in that expression, because independence was never the point.

Step by Step

Bagging

1. Draw a bootstrap sample of size $n$ with replacement.
2. Fit a deep, unpruned learner on it.
3. Repeat $N$ times, in any order, on any number of machines.
4. Average (regression) or vote (classification).

Boosting

1. Start with a constant prediction, usually the mean.
2. Compute what the current ensemble still gets wrong.
3. Fit a small, deliberately weak learner to that.
4. Add it, scaled by the learning rate. Return to step 2.

---

The Decomposition That Settles It

The textbook line is "bagging reduces variance, boosting reduces bias." That is a claim about numbers, so let us get the numbers. Sixty independent training sets, the same test set, and a decomposition of the error of four models into the part that is systematic and the part that is jitter.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor

N_FITS, N_TRAIN, N_TEST = 60, 300, 500
X_test, y_true = make_friedman1(n_samples=N_TEST, noise=0.0, random_state=7)

def decompose(make_model, label):
    preds = np.zeros((N_FITS, N_TEST))
    for i in range(N_FITS):
        Xtr, ytr = make_friedman1(n_samples=N_TRAIN, noise=1.0,
                                  random_state=1000 + i)
        preds[i] = make_model(i).fit(Xtr, ytr).predict(X_test)
    mean_pred = preds.mean(axis=0)
    bias2 = float(np.mean((mean_pred - y_true) ** 2))
    var = float(np.mean(preds.var(axis=0)))
    mse = float(np.mean((preds - y_true) ** 2))
    print(f"{label:<26} bias2 {bias2:7.4f}   "
          f"variance {var:7.4f}   total {mse:7.4f}")

print("BIAS-VARIANCE DECOMPOSITION")
print("60 independent training sets, Friedman #1")
print("-" * 58)
decompose(lambda i: DecisionTreeRegressor(max_depth=None, random_state=42),
          "single deep tree")
decompose(lambda i: BaggingRegressor(DecisionTreeRegressor(max_depth=None),
          n_estimators=100, random_state=42, n_jobs=-1), "bagging, 100 deep trees")
decompose(lambda i: GradientBoostingRegressor(n_estimators=100, max_depth=1,
          learning_rate=0.1, random_state=42), "boosting, 100 stumps")
decompose(lambda i: DecisionTreeRegressor(max_depth=1, random_state=42),
          "single stump")

BIAS-VARIANCE DECOMPOSITION
60 independent training sets, Friedman #1
----------------------------------------------------------
single deep tree           bias2  3.2389   variance  8.1118   total 11.3507
bagging, 100 deep trees    bias2  3.6628   variance  1.0594   total  4.7222
boosting, 100 stumps       bias2  4.1052   variance  0.5443   total  4.6495
single stump               bias2 16.1720   variance  1.9015   total 18.0735

Read each ensemble against its own base learner, which is the comparison people usually skip.

Bagging took a deep tree and cut its variance from 8.1118 to 1.0594 — a factor of 7.7 — while bias went slightly up, from 3.2389 to 3.6628. That is the orchestra exactly: it did nothing whatsoever about the note being wrong, and everything about the wobble.

Boosting took a stump and cut its bias from 16.1720 to 4.1052, a factor of 3.9. That is the relay: a runner who could only ever cover a fraction of the track, turned into a team that reaches the line.

And here is the part the textbook line gets wrong. Boosting also cut variance, from 1.9015 to 0.5443, by a factor of 3.5. Shrinkage and weak learners are a regularizer, not just a bias fix. "Boosting reduces bias" is directionally right and quantitatively incomplete.

Then look at the two totals: 4.7222 and 4.6495. Two philosophies, opposite mechanisms, opposite failure modes, and they arrive within 0.08 of each other.

---

Are the Members Interchangeable?

The parable claims the orchestra's players are swappable and the relay's runners are not. That is testable. Build both from scratch, confirm they match the library, then remove exactly one tree from a hundred and see what it costs.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

class MyBagging:
    """Resample, fit deep trees independently, average."""
    def __init__(self, n=100, seed=42):
        self.n, self.seed = n, seed
    def fit(self, X, y):
        rng = np.random.RandomState(self.seed)
        self.trees_ = []
        for _ in range(self.n):
            idx = rng.randint(0, len(X), len(X))            # bootstrap
            t = DecisionTreeRegressor(max_depth=None, random_state=0)
            self.trees_.append(t.fit(X[idx], y[idx]))
        return self
    def predict(self, X):
        return np.mean([t.predict(X) for t in self.trees_], axis=0)

class MyBoosting:
    """Fit each tree to what is LEFT OVER."""
    def __init__(self, n=100, lr=0.1, depth=3, seed=42):
        self.n, self.lr, self.depth, self.seed = n, lr, depth, seed
    def fit(self, X, y):
        self.f0_ = y.mean()
        resid = y - self.f0_
        self.trees_ = []
        for _ in range(self.n):
            t = DecisionTreeRegressor(max_depth=self.depth,
                                      random_state=self.seed)
            t.fit(X, resid)
            resid = resid - self.lr * t.predict(X)          # still wrong
            self.trees_.append(t)
        return self
    def predict(self, X):
        return self.f0_ + self.lr * np.sum(
            [t.predict(X) for t in self.trees_], axis=0)

X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

bag = MyBagging(100).fit(Xtr, ytr)
boo = MyBoosting(100, 0.1, 3).fit(Xtr, ytr)
skl_b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=100,
                         random_state=42, n_jobs=-1).fit(Xtr, ytr)
skl_g = GradientBoostingRegressor(n_estimators=100, max_depth=3,
                                  learning_rate=0.1, random_state=42).fit(Xtr, ytr)

print("FROM SCRATCH vs SCIKIT-LEARN   (test MSE)")
print("-" * 52)
print(f"  bagging   mine {mean_squared_error(yte, bag.predict(Xte)):8.4f}"
      f"   sklearn {mean_squared_error(yte, skl_b.predict(Xte)):8.4f}")
print(f"  boosting  mine {mean_squared_error(yte, boo.predict(Xte)):8.4f}"
      f"   sklearn {mean_squared_error(yte, skl_g.predict(Xte)):8.4f}")

def bag_pred(trees): return np.mean([t.predict(Xte) for t in trees], axis=0)
def boo_pred(trees): return boo.f0_ + boo.lr * np.sum(
                            [t.predict(Xte) for t in trees], axis=0)
full_b = mean_squared_error(yte, bag_pred(bag.trees_))
full_g = mean_squared_error(yte, boo_pred(boo.trees_))

print("\nREMOVING ONE MEMBER FROM 100   (test MSE)")
print("-" * 52)
print(f"{'':<22}{'bagging':>12}{'boosting':>12}")
print(f"{'all 100':<22}{full_b:>12.4f}{full_g:>12.4f}")
for name, keep in [("drop the 1st tree",   list(range(1, 100))),
                   ("drop the 50th tree",  [i for i in range(100) if i != 49]),
                   ("drop the 100th tree", list(range(0, 99)))]:
    print(f"{name:<22}"
          f"{mean_squared_error(yte, bag_pred([bag.trees_[i] for i in keep])):>12.4f}"
          f"{mean_squared_error(yte, boo_pred([boo.trees_[i] for i in keep])):>12.4f}")

FROM SCRATCH vs SCIKIT-LEARN   (test MSE)
----------------------------------------------------
  bagging   mine   3.6307   sklearn   3.5487
  boosting  mine   2.0786   sklearn   2.0770

REMOVING ONE MEMBER FROM 100   (test MSE)
----------------------------------------------------
                           bagging    boosting
all 100                     3.6307      2.0786
drop the 1st tree           3.6293      2.3147
drop the 50th tree          3.6295      2.1008
drop the 100th tree         3.6310      2.0806

Thirty lines each, and both land on the library: boosting at 2.0786 against scikit-learn's 2.0770.

Now the removals. For bagging, dropping the first, the fiftieth or the hundredth tree moves test error by at most 0.0014, and dropping the last tree (3.6310) was very slightly worse than dropping the first (3.6293) — which is noise, and that is the point. The members are genuinely interchangeable. There is no first violin.

For boosting, dropping the first tree costs 0.2361 and dropping the hundredth costs 0.0020. The same operation is 118 times more expensive at the front of the chain than at the back. Tree one is running the first 700 metres. Tree one hundred is running the last eight.

One honest correction while I was building this. My first version shuffled the members and announced that boosting was "destroyed" — and printed a number identical to the unshuffled one. Of course it did: prediction is a sum, and addition commutes. Order matters while fitting, not while predicting. The removal test is the right experiment; the shuffle test was me describing a result I expected instead of reading the one I got.

---

The Mirror Image: How Deep Should the Trees Be?

If bagging fixes variance and boosting fixes bias, then they should want opposite base learners. Bagging should want deep trees — low bias, high variance, exactly the raw material averaging can improve. Boosting should want shallow ones — leave it something to correct.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

print("BASE-LEARNER DEPTH")
print("test MSE, 100 estimators each, mean of 3 seeds")
print("-" * 50)
print(f"{'max_depth':>10}  {'bagging':>10}  {'boosting':>10}")
for d in [1, 2, 3, 5, 8, None]:
    bag, boo = [], []
    for s in range(3):
        b = BaggingRegressor(DecisionTreeRegressor(max_depth=d),
                             n_estimators=100, random_state=s, n_jobs=-1).fit(Xtr, ytr)
        g = GradientBoostingRegressor(n_estimators=100, max_depth=d,
                                      learning_rate=0.1, random_state=s).fit(Xtr, ytr)
        bag.append(mean_squared_error(yte, b.predict(Xte)))
        boo.append(mean_squared_error(yte, g.predict(Xte)))
    print(f"{str(d):>10}  {np.mean(bag):>10.4f}  {np.mean(boo):>10.4f}")

BASE-LEARNER DEPTH
test MSE, 100 estimators each, mean of 3 seeds
--------------------------------------------------
 max_depth     bagging    boosting
         1     19.5944      5.0660
         2     13.2544      2.6711
         3      8.9423      2.0769
         5      5.6241      2.0196
         8      3.9657      3.0942
      None      3.6156      7.1105

Two curves running in opposite directions.

Bagging improves monotonically with depth and is best at unlimited: 19.5944 with stumps, 3.6156 with fully grown trees. Give the orchestra players who can actually play; averaging will handle the wobble.

Boosting is U-shaped. Best at depth 5 (2.0196), and unlimited depth is a disaster at 7.1105 — worse than boosting stumps. A fully grown tree consumes the entire residual on the first pass, leaves nothing for anyone else, and what it consumed included the noise.

The practical rule falls straight out: if you are bagging, stop pruning. If you are boosting, keep the trees small and buy your capacity with more of them instead.

---

Where Boosting Breaks

Two failure modes, both predicted by the parable, both measurable.

import numpy as np
from sklearn.datasets import make_classification, make_friedman1
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              BaggingRegressor, GradientBoostingRegressor)
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = make_classification(n_samples=2000, n_features=20, n_informative=8,
                           n_redundant=2, flip_y=0.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

print("LABEL-NOISE SENSITIVITY   (test accuracy, mean of 5 seeds)")
print("-" * 64)
print(f"{'flipped':>8}  {'bagging (RF)':>14}  {'boosting (GB)':>15}  {'gap':>7}")
rows = []
for frac in [0.00, 0.05, 0.10, 0.20, 0.30]:
    rf_s, gb_s = [], []
    for seed in range(5):
        rng = np.random.RandomState(seed)
        y_noisy = ytr.copy()
        n_flip = int(frac * len(y_noisy))
        if n_flip:
            idx = rng.choice(len(y_noisy), n_flip, replace=False)
            y_noisy[idx] = 1 - y_noisy[idx]
        rf_s.append(RandomForestClassifier(n_estimators=200, random_state=seed,
                    n_jobs=-1).fit(Xtr, y_noisy).score(Xte, yte))
        gb_s.append(GradientBoostingClassifier(n_estimators=200, max_depth=3,
                    random_state=seed).fit(Xtr, y_noisy).score(Xte, yte))
    rf_m, gb_m = np.mean(rf_s), np.mean(gb_s)
    rows.append((rf_m, gb_m))
    print(f"{frac:>7.0%}  {rf_m:>14.4f}  {gb_m:>15.4f}  {rf_m-gb_m:>+7.4f}")
print(f"\nclean -> 30% noise:  bagging loses {rows[0][0]-rows[-1][0]:.4f},"
      f"  boosting loses {rows[0][1]-rows[-1][1]:.4f}")

# Does adding more members ever hurt?
X, y = make_friedman1(n_samples=400, noise=5.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=42)
print("\nADDING MORE ESTIMATORS   (small noisy sample)")
print("-" * 56)
print(f"{'n_estimators':>13}  {'bagging':>10}  {'boosting':>10}")
for n in [10, 100, 1000, 3000]:
    b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=n,
                         random_state=42, n_jobs=-1).fit(Xtr, ytr)
    g = GradientBoostingRegressor(n_estimators=n, max_depth=3,
                                  learning_rate=0.1, random_state=42).fit(Xtr, ytr)
    print(f"{n:>13}  {mean_squared_error(yte, b.predict(Xte)):>10.4f}"
          f"  {mean_squared_error(yte, g.predict(Xte)):>10.4f}")
g = GradientBoostingRegressor(n_estimators=3000, max_depth=3, learning_rate=0.1,
                              random_state=42).fit(Xtr, ytr)
errs = [mean_squared_error(yte, p) for p in g.staged_predict(Xte)]
print(f"\nboosting best at n={int(np.argmin(errs))+1} (MSE {min(errs):.4f}),"
      f" ends at n=3000 (MSE {errs[-1]:.4f})")

LABEL-NOISE SENSITIVITY   (test accuracy, mean of 5 seeds)
----------------------------------------------------------------
 flipped    bagging (RF)    boosting (GB)      gap
     0%          0.8560           0.8317  +0.0243
     5%          0.8480           0.8113  +0.0367
    10%          0.8350           0.8067  +0.0283
    20%          0.8013           0.7467  +0.0547
    30%          0.7427           0.6840  +0.0587

clean -> 30% noise:  bagging loses 0.1133,  boosting loses 0.1477

ADDING MORE ESTIMATORS   (small noisy sample)
--------------------------------------------------------
 n_estimators     bagging    boosting
           10     39.5639     37.1438
          100     36.5054     43.4739
         1000     35.7662     48.4598
         3000     35.5388     48.4704

boosting best at n=19 (MSE 36.1841), ends at n=3000 (MSE 48.4704)

Noise. Going from clean labels to 30% flipped, bagging gives up 0.1133 of accuracy and boosting gives up 0.1477. The gap between them widens from +0.0243 to +0.0587. A mislabelled row is a residual that never goes away, so boosting keeps sending runners at it. Bagging simply averages over it. This is the short marker, and it is why boosting is the wrong first choice on hand-labelled data.

More members. Bagging on a small noisy sample goes 39.5639 → 35.5388 as it grows from 10 to 3,000 trees. Monotone, and it never gets worse, exactly as the variance formula promises. Boosting starts at 37.1438 and climbs to 48.4704. Its best model was 19 trees; running it to 3,000 made it a third worse than its own optimum.

That asymmetry is the single most useful thing on this page. n_estimators is a capacity knob for boosting and a smoothness knob for bagging. One of them needs early stopping and a validation set. The other you can set to "more" and walk away.

---

When Does Each Help Most?

	Bagging	Boosting
Attacks	Variance	Bias (and some variance)
Base learner	Deep, unpruned	Shallow, depth 3–6
Fitting	Parallel	Serial
More members	Never hurts	Overfits eventually
Noisy labels	Degrades gracefully	Chases the noise
Needs early stopping	No	Yes
Tuning effort	Almost none	Real
Typical accuracy, clean tabular	Good	Usually better

Reach for bagging when labels are noisy or hand-made, when you want a defensible baseline in one line, when you have cores and no time, or when nobody will be around to babysit a validation curve.

Reach for boosting when labels are trustworthy, when the last two points of accuracy are worth real tuning effort, and when you can hold out a validation set to stop on.

---

Bagging and Boosting Hyperparameters

Parameter	Bagging	Boosting
n_estimators	100–1000, more is safe	tune with early stopping
max_depth	None	3–6
learning_rate	n/a	0.01–0.1, lower needs more trees
max_features	sqrt lowers ρ	mild regularizer
subsample	bootstrap is the mechanism	0.5–0.8 adds randomness
What to tune first	almost nothing	learning_rate × n_estimators

---

Quick Reference Card

BAGGING vs BOOSTING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT THEY ARE:
  Bagging  — fit members independently, average them.
  Boosting — fit each member to the leftover error.

THE FORMULAS:
  bagging   Var = ρσ² + (1-ρ)σ²/N
  boosting  F_m = F_{m-1} + lr · h_m

MEASURED HERE (Friedman #1):
  deep tree     bias 3.2389  var 8.1118
  bagged        bias 3.6628  var 1.0594   (var ÷7.7)
  stump         bias 16.1720 var 1.9015
  boosted       bias 4.1052  var 0.5443   (bias ÷3.9)

DROP ONE TREE OF 100:
  bagging   costs 0.0014      boosting costs 0.2361

WHEN TO USE:
  ✓ bagging  — noisy labels, no time, no babysitting
  ✓ boosting — clean labels, tuning budget, val set
WHEN NOT EFFECTIVE:
  ✗ bagging  — members all wrong the same way (high ρ)
  ✗ boosting — mislabelled rows, deep trees, no early stop

SKLEARN:
  from sklearn.ensemble import RandomForestRegressor
  from sklearn.ensemble import GradientBoostingRegressor

---

Key Takeaways

1. They attack different halves of the error — bagging cut a deep tree's variance from 8.1118 to 1.0594 while nudging bias slightly up; boosting cut a stump's bias from 16.1720 to 4.1052.

2. "Boosting reduces bias" is incomplete — boosting also cut variance by a factor of 3.5, from 1.9015 to 0.5443. Shrinkage and weak learners regularize.

3. They arrive at nearly the same place — 4.7222 against 4.6495 total error, from opposite directions.

4. Bagging's members are interchangeable, boosting's are a chain — removing one tree of a hundred costs bagging at most 0.0014, and costs boosting 0.2361 at the front versus 0.0020 at the back.

5. They want opposite base learners — bagging is best with unlimited depth (3.6156), boosting is best at depth 5 (2.0196) and is worse than useless at unlimited depth (7.1105).

6. n_estimators means two different things — bagging improved all the way to 3,000 trees; boosting's optimum was 19 and by 3,000 it was a third worse.

7. Boosting pays more for bad labels — 30% flipped labels cost bagging 0.1133 accuracy and boosting 0.1477, with the gap widening monotonically.

8. Compare each ensemble to its own base learner — against each other the two totals look interchangeable, and the entire mechanism is invisible.

---

The One-Sentence Summary

Bagging and boosting are the orchestra and the relay — forty players sounding one note by cancelling each other's independent mistakes, against four runners each handed only the distance the last one failed to cover — and the measurements say the orchestra divides variance by 7.7 while barely touching bias, the relay divides bias by 3.9, they finish within 0.08 of the same total error, and the only way to tell them apart from the outside is to remove one member and watch it cost the orchestra 0.0014 and the relay 0.2361.

---

What's Next?

Now that you can tell the two families apart, you're ready for:

1. XGBoost vs LightGBM in 10 minutes — tomorrow. Same data, same tuning budget, honest numbers.
2. The hyperparameter tuning cheat sheet — everything that matters, on one page.
3. Stacking — what to do when you have several good models and no idea which to trust.
4. Blending and voting — the two ways a committee can agree, and why one of them cheats.

Follow me for the next article in the Boosting: The Complete Guide series!

---

Let's Connect!

If the orchestra and the relay made ensembles click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Which one do you reach for first? I default to bagging for the baseline and only spend the tuning budget on boosting once I've seen how clean the labels are — because a random forest has never once surprised me in production, and gradient boosting has. ⚖️

---

What strikes me about these two, after measuring them properly, is that the family argument was never about accuracy — they land in the same place. It is about what you are willing to be responsible for. Bagging asks for independence and gives you a model that cannot be made worse by adding to it. Boosting asks for attention and gives you a model that will follow you anywhere, including off a cliff, because every tree after the first is an act of faith in the ones before it. One is a crowd, the other is an argument. The crowd is safer; the argument is usually right; and the engineering judgment is knowing which of those you need this week.

---

Forward this to the teammate who set n_estimators=5000 on a gradient booster and went to lunch.