The One-Line Summary: AdaBoost is the first algorithm in this series that trains its models sequentially rather than in parallel — after every round it raises the weight of the examples it just got wrong, forcing the next weak learner to specialise in exactly those, then weights each learner's vote by how reliable that learner turned out to be, trading bagging's variance reduction for something bagging can never do: reducing bias.
The Parable of Nila and the Eight Weeks
Nila has one workbook with 400 practice problems in it, and eight weeks before the harbour master's navigation test. She solves about six problems in ten. Her family, who have never done this before, decide the sensible thing is to hire help.
They hire twelve tutors.
The Old Way: The Panel of Twelve
Each tutor gets a copy of the workbook, a random slice of it to focus on, and one strict instruction: do not speak to the other eleven. Every Sunday all twelve submit written advice, and Nila does what the majority recommends.
It works, in the way averaging always works. Before the Panel her scores lurched between 41% and 74% depending on which chapter she had revised. After the Panel they sat calmly at 61%, every single week.
Calmly. At 61%. For six weeks.
THE PANEL OF TWELVE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sunday advice, week after week:
Tutor 1 -> "revise chapters 1-3, the core"
Tutor 2 -> "revise chapters 1-3"
Tutor 3 -> "chapters 2-4"
...
Tutor 12 -> "chapters 1-3, and rest well"
MAJORITY: revise chapters 1-3.
Six weeks of that. She can do them in her
sleep.
Problem 230 — tide correction — has been wrong
in all six practice tests. Not one of the
twelve ever mentioned it.
Averaging twelve opinions removed the WOBBLE.
It did not remove the SHARED BLIND SPOT.
The wobble is VARIANCE.
What is left over is BIAS.
This is the wall the Panel could never climb. Twelve helpers working simultaneously, in ignorance of each other, do smooth out each other's quirks. But if all twelve find chapters 1-3 the most obviously teachable material, a thousand of them give the same advice, and problem 230 stays wrong forever.
You cannot average your way out of a mistake everybody is making.
The New Way: One Tutor, Twelve Times
Vera Sandoval charges more than all twelve combined and works differently.
"Don't hire twelve of me on the same morning. Hire me twelve mornings in a row — and let each morning begin by reading the previous morning's wrong answers."
Her method is two ledgers. The first is the question ledger: every problem starts with the same share of Nila's attention, and after each practice test the ones she got wrong get more, the ones she got right get less, and the total stays fixed. Attention is a budget, not a wish.
VERA'S TWO LEDGERS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
LEDGER 1 — attention per problem (starts equal)
Session 1: 9 of 400 wrong. Those 9 become
22x as important as the rest.
In one step.
Session 2: on the reweighted book, a new
lesson gets 27 wrong. Those get
heavier; the old 9 get lighter.
Session 8: hardest problem carries 55x an
average one. Easiest, 0.21x.
LEDGER 2 — how loud each lesson is
fixed almost everything -> shout (1.89)
barely moved the needle -> whisper (0.35)
no better than guessing -> silence
Final advice = every rule, replayed at its own
volume, summed.
The second ledger is the part people miss: Vera does not trust her own sessions equally. Each morning she teaches one rule, then measures how much of the reweighted workbook it fixed. A rule that fixed almost everything gets shouted in the final revision plan; one that barely moved the needle is mentioned quietly; one no better than guessing is struck out, and Vera goes home.
There is a third rule, and it sounds like laziness.
WHY THE SESSIONS MUST BE SHORT:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Vera teaches ONE rule per morning. Never two.
"Why not teach everything on day one?" asks
Nila's father.
Because if session 1 fixes every problem:
• session 2 has nothing to correct
• the question ledger cannot reweight
• the volume ledger has one entry: infinity
• you did not hire a tutor twelve times,
you hired one tutor once
The mistakes ARE the input to the next session.
A session that is too good breaks the chain
that made the chain worth having.
Weak lessons are a REQUIREMENT.
Eight weeks later Nila scores 85%. And one thing went wrong, which matters more than the 85%. Problem 17 has a misprinted answer key, so her correct solution is marked wrong every week. The question ledger, doing exactly what it was built to do, sends more and more attention to problem 17, without limit. In the final week she spends a fifth of her time on a problem that was never solvable.
The ledger cannot tell hard from impossible. That is the deepest flaw in the method.
Why Taking Turns Beats Working in Parallel
THE MATHEMATICS OF TAKING TURNS:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PANEL : twelve helpers at once, none aware
of the others. Cancels random wobble
-> VARIANCE. Cannot touch a shared
blind spot.
VERA : one helper twelve times, each turn
aimed at the last turn's failures.
Attacks the blind spot -> BIAS.
ALL FOUR MUST HOLD:
1. Each lesson beats guessing.
2. Each lesson is WEAK — it leaves mistakes
for the next one.
3. Wrong answers get heavier; total
attention stays fixed.
4. A lesson's volume is set by what it
fixed, not by seniority.
Nila's family called this "taking turns."
Machine learning calls it ADABOOST.
Elder Booth closed the Random Forests series by making twelve judges disagree with each other; Vera opens this one by making them wait their turn.
What Is AdaBoost?
AdaBoost (Adaptive Boosting, Freund and Schapire, 1995) builds an ensemble one model at a time. Each model trains on a reweighted copy of the training set that emphasises whatever the ensemble-so-far gets wrong, then votes with a weight set by its own measured reliability. The series so far against the series to come:
| Bagging / RF / Extra Trees | Boosting / AdaBoost | |
|---|---|---|
| Training order | Parallel, independent | Sequential, each needs the last |
| Each model sees | A bootstrap of rows | The same rows, reweighted |
| Goal | Reduce variance | Reduce bias |
| Base learner | Strong (deep, overfit trees) | Weak (depth-1 stumps) |
| More models | Never hurts, just plateaus | Can overfit |
| Parallelisable | Trivially (n_jobs=-1) |
Not across rounds. Ever. |
Row four catches people. In a Random Forest you want each tree deep and overfit, because variance is what averaging removes. In AdaBoost you want each tree almost useless, because leftover error is the fuel for the next round.
The Math
The Math
Training data with , starting from uniform weights . For :
Predict with .
That exponent collapses to something simple: is when the point is correct and when it is wrong, so correct points scale by , wrong points by , and everything is renormalised to sum to 1. Attention is a budget, exactly as Vera said.
Step by Step
- Set every weight to .
- Fit a weak learner on the weighted data.
- Compute ; if , stop.
- Compute .
- Scale wrong weights by , right by , renormalise.
- Repeat, then predict with the signed weighted vote.
Alpha Is a Volume Knob With Two Cliffs
is the only place a learner's quality enters the model.
ALPHA = 0.5 * ln((1 - err) / err)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
err = 0.01 -> alpha = 2.2976
err = 0.10 -> alpha = 1.0986
err = 0.30 -> alpha = 0.4236
err = 0.45 -> alpha = 0.1003
err = 0.49 -> alpha = 0.0200
err = 0.50 -> alpha = 0 SILENT
err = 0.60 -> alpha = -0.2027 NEGATIVE
TWO CLIFFS:
err -> 0 alpha -> +infinity. One learner
drowns out every other; sklearn
stops and keeps only that one.
A first stump at err=0 means you
have no ensemble at all.
err >= 0.5 alpha <= 0. A coin-flip learner
earns zero volume and AdaBoost
halts. (For K classes the bar
is err < 1 - 1/K.)
Note the asymmetry that trips people up: err = 0 is not a triumph, it is a failure. A learner that nails the weighted training set in round one hands you infinite weight, ends boosting, and leaves you one overfit tree wearing an ensemble's name. That is precisely why the base learner must be weak.
The Loss Function Nobody Mentions
This is what turns AdaBoost from a clever heuristic into a member of a family. With running score , AdaBoost is greedily minimising
which is exponential loss. Two consequences fall out.
First, fix and solve : the answer is . The alpha formula is not a tuned design choice, it is the exact minimiser. Nobody picked it; it fell out.
Second, the weight at round is proportional to . A point's weight IS its current exponential loss. On the wrong side with margin it carries ; at margin , . Exponential loss punishes confident mistakes with no upper bound, and the reweighting inherits that unboundedness exactly. That is the misprinted answer key, stated in calculus.
The bridge to tomorrow: if AdaBoost is stagewise minimisation of exponential loss, what happens when you swap in log loss, squared error, or Huber — anything differentiable — and fit each learner to the negative gradient instead? That generalisation is Gradient Boosting.
200 Useless Stumps Beat One Good Tree
We'll use make_hastie_10_2, the benchmark AdaBoost was originally demonstrated on. Its label is a nonlinear function of ten Gaussian features, which makes a single axis-aligned stump close to worthless.
import numpy as np
from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, random_state=42
)
# Never pass algorithm=... (deprecated in 1.6, gone in 1.8).
# The base learner arg is `estimator`, not `base_estimator`.
stump = DecisionTreeClassifier(max_depth=1, random_state=42).fit(X_tr, y_tr)
tree = DecisionTreeClassifier(random_state=42).fit(X_tr, y_tr)
rf = RandomForestClassifier(n_estimators=200, random_state=42,
n_jobs=-1).fit(X_tr, y_tr)
ada = AdaBoostClassifier(
estimator=DecisionTreeClassifier(max_depth=1),
n_estimators=200, learning_rate=1.0, random_state=42,
).fit(X_tr, y_tr)
print(f"one decision stump (depth 1) : {stump.score(X_te, y_te):.4f}")
print(f"one full-depth decision tree : {tree.score(X_te, y_te):.4f}")
print(f"RandomForest, 200 trees : {rf.score(X_te, y_te):.4f}")
print(f"AdaBoost, 200 stumps : {ada.score(X_te, y_te):.4f}")
print()
# staged_score replays the ensemble one round at a time
tr = list(ada.staged_score(X_tr, y_tr))
te = list(ada.staged_score(X_te, y_te))
print(f"{'stumps used':>12} {'train':>8} {'test':>8}")
for r in [1, 5, 25, 100, 200]:
print(f"{r:>12} {tr[r - 1]:>8.4f} {te[r - 1]:>8.4f}")
one decision stump (depth 1) : 0.5183
one full-depth decision tree : 0.7183
RandomForest, 200 trees : 0.8417
AdaBoost, 200 stumps : 0.8467
stumps used train test
1 0.5621 0.5183
5 0.6043 0.5500
25 0.7229 0.6717
100 0.8436 0.8083
200 0.8964 0.8467
Sit with that. A coin flip scores 0.5000; one stump scores 0.5183, beating chance by 1.8 points — very nearly worthless. Two hundred of those worthless things stacked in sequence score 0.8467, beating a fully grown tree (0.7183) and a 200-tree Random Forest (0.8417) on identical data.
The climb is worth memorising: 0.5183, 0.5500, 0.6717, 0.8083, 0.8467. Slow, then fast, then grinding. Nothing happened in a leap; every point was bought by one stump correcting the residue of the previous 199.
Watching the Weights Move
The experiment nobody runs: implement the reweighting loop by hand on 400 points and just print the weights, round by round. Note flip_y=0.02 — we mislabel about 2% of the data deliberately, because every real workbook has misprints.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(
n_samples=400, n_features=2, n_informative=2, n_redundant=0,
n_clusters_per_class=1, class_sep=2.0, flip_y=0.02, random_state=42,
)
y = np.where(y == 0, -1, 1) # AdaBoost's native label space
n = len(X)
uniform = 1.0 / n
w = np.full(n, uniform) # every problem equally important
history = [w.copy()]
print(f"{'round':>6} {'weighted err':>13} {'alpha':>8} {'# wrong':>9}")
for rnd in range(1, 9):
stump = DecisionTreeClassifier(max_depth=1, random_state=42)
stump.fit(X, y, sample_weight=w) # teach to the ledger
wrong = stump.predict(X) != y
err = w[wrong].sum()
alpha = 0.5 * np.log((1 - err) / err)
print(f"{rnd:>6} {err:>13.4f} {alpha:>8.4f} {wrong.sum():>9}")
w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0)) # wrong -> heavier
w /= w.sum() # budget stays fixed
history.append(w.copy())
H = np.array(history)
hardest = np.argsort(-H[-1])[:3] # 3 heaviest points at the end
easiest = np.argsort(H[-1])[:3] # 3 lightest points at the end
cols = list(hardest) + list(easiest)
print()
print("SAMPLE WEIGHT, AS A MULTIPLE OF THE STARTING WEIGHT")
print(f"{'round':>6}" + "".join(f"{'#' + str(i):>10}" for i in cols))
for r in range(9):
print(f"{r:>6}" + "".join(f"{H[r, i] / uniform:>10.2f}" for i in cols))
round weighted err alpha # wrong
1 0.0225 1.8857 9
2 0.1431 0.8950 27
3 0.2744 0.4863 243
4 0.3004 0.4226 205
5 0.3377 0.3369 205
6 0.2919 0.4432 9
7 0.3349 0.3429 199
8 0.3329 0.3476 205
SAMPLE WEIGHT, AS A MULTIPLE OF THE STARTING WEIGHT
round #230 #271 #158 #328 #171 #19
0 1.00 1.00 1.00 1.00 1.00 1.00
1 22.22 22.22 22.22 0.51 0.51 0.51
2 77.66 77.66 12.97 0.30 0.30 0.30
3 53.51 53.51 8.93 0.21 0.21 0.21
4 38.25 38.25 14.87 0.34 0.34 0.34
5 28.87 56.63 11.22 0.26 0.26 0.26
6 49.46 97.02 19.23 0.18 0.18 0.18
7 73.83 72.95 14.46 0.14 0.14 0.14
8 55.34 54.67 21.72 0.21 0.21 0.21
There is the tutor's ledger, printed. Three passes.
Round 1 is violent. The first stump's weighted error is only 0.0225, earning a loud — and a loud alpha means a savage reweighting. The nine points it missed jump from 1.00 to 22.22 times their starting weight in one step. Overnight, exactly as Vera does it.
The right-hand three columns move in perfect lockstep — 0.51, 0.30, 0.21, 0.34, identical for #328, #171 and #19. Not a coincidence: all three were classified correctly in every round, so all three were multiplied by the same every time. They are the chapters Nila can do in her sleep, and AdaBoost has stopped spending time on them.
Now #230. Up to 77.66x, down to 28.87x when a stump finally gets it right, back up to 73.83x when the next one misses again — it oscillates because the ensemble keeps almost-solving it. By round 8 the heaviest-to-lightest spread is roughly 260-fold: same 400 rows, wildly unequal attention.
Also notice # wrong. Round 1 misses 9 points; round 3 misses 243 of 400. The later stumps are terrible by any ordinary standard — but they beat a coin flip on the reweighted data, and that is the only test that matters.
AdaBoost from Scratch
Forty-odd lines of NumPy, and the loop is exactly the six steps above.
import numpy as np
from sklearn.datasets import make_hastie_10_2
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
class MyAdaBoost:
"""SAMME AdaBoost for binary labels in {-1, +1}."""
def __init__(self, n_estimators=200, learning_rate=1.0, random_state=42):
self.n_estimators = n_estimators
self.learning_rate = learning_rate
self.random_state = random_state
def fit(self, X, y):
n = len(X)
w = np.full(n, 1.0 / n) # all points equally urgent
self.learners_, self.alphas_ = [], []
for m in range(self.n_estimators):
stump = DecisionTreeClassifier(max_depth=1,
random_state=self.random_state)
stump.fit(X, y, sample_weight=w) # 1. weighted fit
wrong = stump.predict(X) != y
err = float(w[wrong].sum()) # 2. weighted error
if err >= 0.5: # no better than luck
break
if err <= 0.0: # perfect -> alpha inf
self.learners_, self.alphas_ = [stump], [1.0]
break
alpha = self.learning_rate * 0.5 * np.log((1.0 - err) / err) # 3.
w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0)) # 4. reweight
w /= w.sum() # 5. renorm
self.learners_.append(stump)
self.alphas_.append(alpha)
return self
def decision_function(self, X):
return sum(a * lrn.predict(X)
for a, lrn in zip(self.alphas_, self.learners_))
def predict(self, X):
return np.where(self.decision_function(X) >= 0, 1.0, -1.0)
def score(self, X, y):
return float((self.predict(X) == y).mean())
X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)
mine = MyAdaBoost(n_estimators=200, random_state=42).fit(X_tr, y_tr)
theirs = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
n_estimators=200, learning_rate=1.0,
random_state=42).fit(X_tr, y_tr)
print(f"MyAdaBoost test accuracy: {mine.score(X_te, y_te):.4f}")
print(f"sklearn AdaBoost test accuracy: {theirs.score(X_te, y_te):.4f}")
print("predictions agreeing on test set: "
f"{(mine.predict(X_te) == theirs.predict(X_te)).mean():.1%}")
print("first 5 alphas, mine x2 :",
" ".join(f"{2 * a:.4f}" for a in mine.alphas_[:5]))
print("first 5 alphas, sklearn :",
" ".join(f"{a:.4f}" for a in theirs.estimator_weights_[:5]))
MyAdaBoost test accuracy: 0.8467
sklearn AdaBoost test accuracy: 0.8467
predictions agreeing on test set: 100.0%
first 5 alphas, mine x2 : 0.2499 0.1353 0.1654 0.1491 0.2021
first 5 alphas, sklearn : 0.2499 0.1353 0.1654 0.1491 0.2021
Not "close." Identical — same accuracy, same prediction on all 600 test points.
The x2 is the one real discrepancy between textbook AdaBoost and sklearn. sklearn's SAMME uses
, which at
is exactly twice the classical
. Doubling every alpha cannot change the sign of the weighted vote, so predictions are untouched — and the weight update matches too: mine scales wrong points by
and right ones by
(ratio
), sklearn scales wrong by
and leaves right alone (same ratio), and renormalisation erases the difference. Hence exact agreement rather than approximate.
No hidden machinery in the library version. Five lines in a loop.
The Defining Weakness: Hard vs Impossible
Now the misprinted answer key, in code. Take cleanly separable data, flip 10% of the training labels, and ask two questions: where does AdaBoost spend its attention, and what does that cost?
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import AdaBoostClassifier, RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(
n_samples=2000, n_features=10, n_informative=10, n_redundant=0,
n_clusters_per_class=1, class_sep=1.4, flip_y=0.0, random_state=42,
)
y = np.where(y == 0, -1, 1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)
n = len(y_tr)
bad = np.random.default_rng(42).choice(n, int(0.10 * n), replace=False)
y_bad = y_tr.copy()
y_bad[bad] = -y_bad[bad] # 10% misprinted answer keys
is_bad = np.zeros(n, bool)
is_bad[bad] = True
w = np.full(n, 1.0 / n)
print("Share of AdaBoost's attention on the 10% corrupted labels")
for r in range(1, 101):
s = DecisionTreeClassifier(max_depth=1, random_state=42)
s.fit(X_tr, y_bad, sample_weight=w)
wrong = s.predict(X_tr) != y_bad
err = w[wrong].sum()
alpha = 0.5 * np.log((1 - err) / err)
w = w * np.exp(alpha * np.where(wrong, 1.0, -1.0))
w /= w.sum()
if r in (1, 5, 10, 25, 50, 100):
print(f" after round {r:>3}: {w[is_bad].sum():>6.1%} of total weight")
print()
print(f"{'labels flipped':>15} {'AdaBoost':>10} "
f"{'RandomForest':>13} {'errors A/RF':>14}")
for frac in [0.0, 0.10, 0.20]:
yb = y_tr.copy()
if frac:
i = np.random.default_rng(42).choice(n, int(frac * n), replace=False)
yb[i] = -yb[i]
A = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
n_estimators=200, random_state=42).fit(X_tr, yb)
R = RandomForestClassifier(n_estimators=200, random_state=42,
n_jobs=-1).fit(X_tr, yb)
ea = (A.predict(X_te) != y_te).sum()
er = (R.predict(X_te) != y_te).sum()
print(f"{frac:>14.0%} {A.score(X_te, y_te):>10.4f} "
f"{R.score(X_te, y_te):>13.4f} {str(ea) + ' / ' + str(er):>14}")
Share of AdaBoost's attention on the 10% corrupted labels
after round 1: 16.8% of total weight
after round 5: 37.8% of total weight
after round 10: 39.9% of total weight
after round 25: 44.3% of total weight
after round 50: 45.0% of total weight
after round 100: 43.7% of total weight
labels flipped AdaBoost RandomForest errors A/RF
0% 0.9950 0.9983 3 / 1
10% 0.9783 0.9983 13 / 1
20% 0.9417 0.9850 35 / 9
Two findings, and the first explains the second.
By round 25, AdaBoost spends 44% of its attention on 10% of the data — and every one of those points has a wrong label. It is not confused; it is working as designed. Those points stay misclassified, so their exponential loss grows, so their weight grows, so every later stump is fitted to please them. Nila, spending a fifth of her final week on problem 17.
The cost is a 4-to-13x multiplication of test errors. On clean data both models are effectively perfect (3 and 1 errors out of 600). Corrupt 10% of training labels and Random Forest does not notice — still 1 error — while AdaBoost jumps to 13. At 20%, Random Forest makes 9 and AdaBoost makes 35.
Why is the forest immune? Bagging averages over noisy points and never singles them out. A mislabeled row lands in roughly 63% of bootstraps and is outvoted in all of them, carrying the same weight from the first tree to the five-hundredth. Bagging has no mechanism for obsession. AdaBoost is built out of one.
ADABOOST: THE HONEST ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
REACH FOR IT WHEN:
✓ Labels are clean and trustworthy
✓ Tabular data, moderate size
✓ A single model badly underfits (high bias)
✓ You want an interpretable additive model
(depth-1 stumps -> one term per feature)
✓ You have almost nothing to tune
LOOK ELSEWHERE WHEN:
✗ Labels are noisy or crowd-sourced
-> Random Forest, or a robust loss
✗ Heavy outliers you cannot clean
✗ You need training parallelism
(rounds are strictly sequential)
✗ You want last-percent tabular accuracy
-> LightGBM / XGBoost / CatBoost
THE ONE-LINE RULE:
Audit your labels BEFORE you use AdaBoost.
It will find every mistake you made and
build a model around it.
learning_rate and n_estimators Are One Dial
learning_rate multiplies every alpha, shrinking each round's contribution. Smaller steps mean each stump says less, so you need more of them.
from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
X, y = make_hastie_10_2(n_samples=2000, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)
rounds = [25, 50, 100, 200]
print(f"{'lr':>6}" + "".join(f"{'M=' + str(r):>9}" for r in rounds))
for lr in [1.0, 0.5, 0.2, 0.1]:
a = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
n_estimators=200, learning_rate=lr,
random_state=42).fit(X_tr, y_tr)
st = list(a.staged_score(X_te, y_te))
print(f"{lr:>6.1f}" + "".join(f"{st[r - 1]:>9.4f}" for r in rounds))
lr M=25 M=50 M=100 M=200
1.0 0.6717 0.7667 0.8083 0.8467
0.5 0.6067 0.6600 0.7150 0.7817
0.2 0.5083 0.5617 0.6283 0.6733
0.1 0.4950 0.5183 0.5617 0.6200
Read along a row and accuracy climbs with more rounds; read down a column and it falls with a smaller rate. lr=0.1 at 200 rounds (0.6200) loses to lr=1.0 at 25 rounds (0.6717) — shrinking the steps 10x costs more than 8x the rounds can repay here. So never tune these two separately: fix learning_rate, then push n_estimators until a validation curve stops improving.
The Hyperparameters That Matter
| Parameter | What it controls | Sensible range | Notes |
|---|---|---|---|
estimator |
The weak learner | DecisionTreeClassifier(max_depth=1) |
Depth = max interaction order. Depth 1 is purely additive and cannot learn interactions; try 2-3 if you need them. |
n_estimators |
Boosting rounds | 50-500 | Unlike a forest, more can overfit. |
learning_rate |
Shrinkage on every alpha | 0.05-1.0 | Halve one, roughly double the other. |
algorithm |
Legacy SAMME switch | do not pass it | Deprecated in sklearn 1.6, removed in 1.8. |
base_estimator |
— | — |
Removed. Use estimator. |
Two anti-patterns. A deep tree as the base estimator defeats the whole mechanism: weighted error crashes toward zero, alpha spikes, and you have paid for an ensemble to get one overfit tree. Boosting before you have looked at your labels is worse — every other model here quietly tolerates 5% label noise, while AdaBoost finds it, weights it, and enshrines it.
Quick Reference Card
ADABOOST: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
Weak learners trained IN SEQUENCE, each one
fitted to the mistakes of the last. Reduces
BIAS. (Bagging is parallel -> VARIANCE.)
THE FIVE STEPS:
1. w = 1/n for every sample
2. fit weak learner on weighted data
3. err = weighted misclassification rate
4. alpha = 0.5 * ln((1 - err) / err)
5. wrong *= e^alpha, right *= e^-alpha,
renormalise -> back to step 2
PREDICT: sign( sum_m alpha_m * h_m(x) )
THE LOSS IT MINIMISES:
L = sum_i exp(-y_i * F(x_i))
A sample's weight IS its exponential loss.
Unbounded -> that is the whole weakness.
HYPERPARAMETERS:
estimator = DecisionTree(max_depth=1)
n_estimators = 50-500 (CAN overfit)
learning_rate = 0.05-1.0 (one dial with
n_estimators, not two)
USE IT: ✓ clean labels, high bias, tabular
AVOID IT: ✗ noisy labels, outliers
SKLEARN:
from sklearn.ensemble import AdaBoostClassifier
# estimator=..., NOT base_estimator=...
# never pass algorithm=...
Key Takeaways
Boosting is sequential; bagging is parallel. Random Forest trains independent trees to cancel noise and cut variance; AdaBoost trains dependent stumps, each aimed at the last one's failures, and cuts bias. Picking the wrong one is a category error, not a tuning mistake.
Misclassified samples get heavier and the total stays fixed. Nine points missed in round one jumped to 22.22x their starting weight in one step, while points never wrong decayed in lockstep to 0.21x.
alpha = 0.5 * ln((1 - err) / err)is not a design choice — it is the exact analytical minimiser of exponential loss. Zero aterr = 0.5, negative above it, unbounded aserrapproaches 0.Weak learners are a requirement, not a tolerance. A learner with
err = 0earns infinite weight and ends boosting after one model. Stumps that are too strong buy you one overfit tree at 200x the price.Two hundred near-useless models beat one good one. One stump scored 0.5183 against a coin flip's 0.5000; two hundred scored 0.8467, beating a full tree (0.7183) and a 200-tree Random Forest (0.8417).
A sample's weight IS its exponential loss, and that loss is unbounded — so a persistently misclassified point's weight grows without limit. That is why label noise is the defining weakness: with 10% of labels flipped, AdaBoost spent 44% of its attention on the corrupted points and made 13 test errors where Random Forest made 1.
learning_rateandn_estimatorsare one parameter. Smaller alphas say less per round, so you need more rounds;lr=0.1at 200 rounds lost tolr=1.0at 25.
The One-Sentence Summary
AdaBoost works because Vera was right that twelve helpers on the same morning can only cancel each other's random errors while one helper twelve mornings running can fix a shared blind spot — so it trains each weak learner on data where yesterday's mistakes are worth exponentially more, sets each lesson's volume to the exact minimiser of exponential loss, and turns a stump that barely beats a coin flip into a model that beats a Random Forest; and it fails for the very same reason, because a ledger that escalates whatever stays wrong cannot tell a hard problem from a misprinted answer key.
What's Next?
Now that you understand how boosting trades parallelism for the ability to attack bias, you're ready for:
- Gradient Boosting — tomorrow. Swap exponential loss for any differentiable loss, fit each learner to the negative gradient, and you get the algorithm that runs most of tabular ML.
- XGBoost — second-order gradients, regularisation written into the objective, and the engineering that won a decade of Kaggle.
- LightGBM — histogram binning and leaf-wise growth, which is how you boost ten million rows without waiting until Thursday.
- CatBoost — ordered boosting and native categorical handling, built to fix the target leakage the other two quietly live with.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If Vera's two ledgers made AdaBoost click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
When did you last audit your labels before blaming your model? Roughly half of every "boosting overfits" I have investigated turned out to be "boosting found the annotation mistakes nobody checked for." 🎯
There is something uncomfortably human in AdaBoost. It is the most conscientious algorithm in classical machine learning — it never lets a mistake go — and that conscientiousness is exactly what destroys it when one of the mistakes was never a mistake at all. I have managed engineers like this, and I have been one. The fix was never to care less. It was to learn the difference between a hard problem and a wrong answer key.
Share this with someone still trying to fix bias by adding more trees. Tell them the trees need to take turns.
Top comments (0)