The One-Line Summary: Six posts on Random Forests came down to a single quantity — tree-to-tree correlation — because the variance of an average has a floor at that no number of trees can lower, which is why every knob that helped (columns per split, random thresholds) worked by attacking , why the regression default quietly leaves that knob switched off, and why the importance scores and the free validation score both lie in ways that trace back to the same place.
The Number Underneath All Six Posts
The Random Forests series is done. Here is the whole thing compressed, in the order you'd want to read it, plus the one idea from each post worth keeping.
If you take a single thing away, take this:
THE VARIANCE OF AN AVERAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Var(average) = ρσ² + (1-ρ)σ²/N
│ │
│ └─ shrinks to 0 with more trees
└───────────── NEVER shrinks. Ever.
More trees cannot fix correlation.
Only DIVERSITY can fix correlation.
Every post below is a consequence of this line.
1. Random Forest — the blindfolded jury
Bagging plus one restriction: at every split, each tree may only consider a random subset of columns. That handicaps each tree and strengthens the committee, because averaging only cancels errors that point in different directions.
Measured across five generated datasets: raising max_features made individual trees monotonically better (0.727 → 0.848) and tree-to-tree correlation monotonically worse (0.311 → 0.644). Ensemble accuracy peaked in between, on a broad plateau around 6 of 20 features — not exactly on sqrt ≈ 4.47.
Keep: a committee's reliability depends less on how good each member is than on how differently they are wrong.
2. Feature Importance — the company that fired the wrong employee
feature_importances_ sums impurity decrease over the training rows. It pays for opportunity, not value: a column with many possible split points gets many chances to look useful.
A column of random IDs beat a genuinely predictive binary flag, 0.5602 to 0.2722. Pure noise climbed from 0.18% to 57.17% importance on nothing but cardinality.
Keep: permutation importance on held-out rows measures damage rather than activity. And when two columns do the same job, report the group, not the winner — a ranking that flips between runs is the finding.
3. Out-of-Bag Error — the free exam
Each bootstrap sample leaves out of rows. That is a limit, not a rule of thumb — measured 0.3679 at n = 10,000. Score each row using only the trees that never saw it and you get validation for the price of one fit.
Measured: OOB 0.9021, 5-fold 0.9036, held-out test 0.8983 — estimates 0.0015 apart, for roughly 5× less wall clock.
Keep: it lies in three specific situations. Too few trees (at B=5, OOB said 0.7829 for a 0.8750 model). Grouped rows — five visits per patient pushed oob_score_ to a perfect 1.0000 against a real 0.8620 under GroupKFold. And heavy imbalance, where 0.9710 accuracy hid 0.1944 recall. Use n_estimators >= 100 or don't quote the number.
4. Extra Trees — the chef who stopped tasting
One change: the split threshold is drawn, not searched. One uniform draw per candidate feature; the impurity criterion still picks the winner among them.
That is the third randomness knob — rows, columns, thresholds — and the ladder is measurable: = 0.640, 0.571, 0.515. Single-tree accuracy fell to 0.8034, the worst of the three, while the ensemble rose to 0.9050, the best of the three. Same trade as post 1, one rung further down. It is also faster, because nothing ever gets sorted.
Keep: you pay in bias, and bias does not average away. Variance reduction compounds over trees; per-tree bias is permanent. That is why the trade has limits.
5. The max_features default — and the fix that makes it worse
RandomForestClassifier defaults to 'sqrt'. RandomForestRegressor defaults to 1.0 — every column, every split. Verified from the constructor signature on scikit-learn 1.7.2.
So the regressor ships as plain bagged trees with the diversity knob wide open, and the internet's advice is to copy 'sqrt' across. Measured across three dataset shapes, that advice is backwards. The default won on
every time (0.8723 / 0.8940 / 0.6828); 'sqrt' lost every time, by 21 points when only 5 of 60 features carried signal. Correlation fell to 0.248 there — the lowest of any setting, and the worst model.
Keep: decorrelation is a means, not a goal. A classifier only needs the majority right, so it can spend per-tree accuracy on diversity. A regressor averages the actual numbers, so per-tree bias goes straight into the mean and never averages away. The default's real cost is ~2x fit time; 0.5 is statistically tied and meaningfully faster.
6. The Cheat Sheet
No new theory. The three knobs, hyperparameter starting ranges split by classification versus regression, a flowchart for choosing an importance method, the RF vs Bagging vs Extra Trees table, and a symptom-to-fix list for eight ways a forest goes wrong in production.
What Changed My Mind
The max_features post was supposed to be a takedown. I expected the regression default to cost real accuracy. It was the most accurate setting on all three shapes I tested. The post now argues the opposite of its first draft, and says so.
'sqrt' is not the answer on regression — it is the trap. I had assumed the classifier default was simply the better choice everywhere. It was the worst of four settings on every shape, catastrophically so on a wide frame with sparse signal. Lowest correlation, worst model: proof that diversity is a cost as well as a benefit.
The Habit Worth Stealing
Average over several seeds before believing any tuning result. A single train/test split moved by as much as the effects I was chasing — half a percent — which means most "setting X is better" claims you'll read, including ones I nearly published, are noise wearing a lab coat.
What's Next
Boosting is already underway. Where forests fight variance by averaging independent trees, boosting attacks bias by training them in sequence, each one paid to fix the last one's mistakes:
- AdaBoost — reweight what you got wrong, then weight each learner's vote by how reliable it turned out to be.
- Gradient Boosting — that idea generalised: each tree fits the gradient of the loss.
- XGBoost — the engineering that made boosting win competitions.
- LightGBM, CatBoost, early stopping — and an honest answer to "boosting or forests?" on tabular data.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If the correlation floor is the thing that finally made forests click, drop a heart.
Questions? Ask in the comments — I read and respond to every one.
Which of the six surprised you most? For me it was a column of random IDs outranking a genuinely useful flag by two to one, on defaults everybody uses. 🌲
Six posts about forests taught me something about writing: the parts I assumed were too obvious to include turned out to need a diagram, and the parts I expected to be a takedown turned out to be a tie. max_features=1.0 was a footnote in my first draft of post one. It became its own post, then it changed its own conclusion.
Share this with someone who is still adding trees to fix a correlation problem.
Top comments (0)