The One-Line Summary: Week 2 crossed the line from ensembles that average away variance to ensembles that attack bias one correction at a time — six posts covering out-of-bag error, Extra Trees, AdaBoost, Gradient Boosting, the learning-rate trap and a family cheat sheet, and the honest theme running through all of them is that boosting gives you a better ceiling in exchange for the ability to ruin the model by trying too hard.
The Handoff
Week 1 ended on a single quantity: tree-to-tree correlation, and the fact that the variance of an average has a floor no number of trees can lower.
Week 2 started by finishing that story, then abandoned the whole approach.
THE WEEK IN ONE PICTURE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
BAGGING FAMILY BOOSTING FAMILY
(days 1-7) (days 8-11)
many deep trees many shallow trees
built independently built in sequence
average them SUM them
fights variance fights bias
more trees: safe more trees: dangerous
one fit, good answer tuning required
The pivot is a single word:
average -> sum
Everything else follows from it.
What Each Post Was Actually About
Out-of-Bag Error — Bootstrapping leaves about 36.8% of rows unseen by any given tree, because (1 - 1/n)^n converges to 1/e. Score each row with only the trees that never saw it and you get validation for the price of one fit. Measured: OOB 0.9021, 5-fold 0.9036, test 0.8983, at roughly a fifth of the wall clock.
The part worth keeping is where it lies. With 5 trees, OOB reported 0.7829 for a model that was actually 0.8750. With five visits per patient in the data, OOB reported a perfect 1.0000 against GroupKFold's 0.8620 — because "unseen by this tree" is not the same as "unseen" when the same entity appears five times.
Extra Trees — Stop searching for the best split threshold; draw one at random. Third rung on the same ladder: correlation fell 0.640 → 0.571 → 0.515 across bootstrap, feature subsampling, and random thresholds, single-tree accuracy fell 0.8342 → 0.8204 → 0.8034, and the ensemble went 0.8967 → 0.8983 → 0.9050. Worse trees, better forest, and faster because nothing needs sorting.
AdaBoost — The first algorithm to build the next model because of the last one's mistakes. Reweight the rows you got wrong, fit a stump, weight its vote by how good it was, repeat. It works, and its limitation is that it is welded to exponential loss.
Gradient Boosting — The generalisation that made boosting the default for tabular data. Fit each tree to the negative gradient of any differentiable loss, add a fraction of it. Twenty lines reproduced sklearn to five decimal places of R2; the two diverged at round 23 on exactly one test row out of 240, where a split tie broke the other way.
learning_rate and n_estimators — They are one parameter. Across seven rates the optimal product held at 4.4 on classification while the tree count moved 300-fold. A rate × count grid search is mostly duplicate work.
The family cheat sheet — One update rule, and each library changes exactly one clause of it.
The Two Things That Changed My Mind
Lowest correlation is not the goal. Week 1 built up decorrelation as the thing to chase. Then in the max_features measurements, 'sqrt' on regression produced the lowest tree correlation of any setting — 0.248 — and the worst R2, losing 21 points. Decorrelation is a means. I had been treating it as a score.
Boosting can genuinely destroy a model. I knew this as a caution and had never watched it happen cleanly. At learning_rate=1.0 on a noisy problem, held-out R2 peaked after one tree at 0.186, then fell to -0.31 by 800 trees — worse than predicting the mean for every row. A Random Forest cannot do that no matter how badly you configure it. That asymmetry is the real reason boosting demands more of you.
The Thing I Got Wrong
I published a claim about max_features on regression that I could not reproduce when I went back to re-run it, and I rewrote the post rather than leave it. The corrected version measured three dataset shapes, five seeds each, and the conclusion inverted: the regression default of 1.0 was the most accurate setting every time, and the widely repeated advice to copy 'sqrt' across from the classifier was the worst of four settings on every shape.
Which is a useful thing to have happen in week 2 rather than week 30. The rule I have adopted: no number goes into a post unless the code that produced it ran in the same session I wrote the sentence.
One Idea Per Post, If You Only Keep Six Things
- 36.8% of rows are out-of-bag for any tree, and that is where free validation comes from.
-
OOB lies when rows are grouped.
oob_score_of 1.0 is a warning, not an achievement. - Worse trees can make a better forest — Extra Trees, measured three rungs down.
- The gradient, not the residual. "Fit the residuals" is true only for squared error.
-
lr × n_estimatorsis the quantity that matters. Fix the rate, early-stop the count. - Boosting overfits; forests do not. Everything operational follows from this.
Next Week
Week 3 goes inside the libraries that made boosting win.
- XGBoost — second-order gradients, and regularisation moved into the objective so pruning stops being a heuristic.
- LightGBM — histogram binning and leaf-wise growth: same maths, an order of magnitude less of it.
- CatBoost — ordered boosting, and the target-encoding leak it was built to close.
- Stacking — what to do when you have several good models and no idea which to trust.
Let's Connect!
Questions? Ask in the comments — I read and respond to every one, and any question that needs more than a paragraph becomes its own post.
Which of this week's six would you want re-done in more depth? My own vote is the OOB failure modes; the grouped-rows case is the one I have actually been burned by in production. 🗿
Two weeks in, the pattern I did not expect is how often the honest version of a post is less satisfying than the draft. The draft says "here is the trick." The measurement says "here is the trick, and here are the three shapes of data where it costs you." I am keeping the second one.
Top comments (0)