The One-Line Summary: Random Forest is bagging plus one extra restriction — at every single split, each tree is only allowed to consider a random subset of features, which forces the trees to disagree with each other, and because bagging's variance reduction depends on how uncorrelated the trees are, deliberately handicapping every tree makes the forest stronger than the sum of its parts.
The Parable of the Village That Fixed Its Jury
Last time we left the village of Predicta, they had just invented the jury. Twelve judges instead of one. Vote. Take the majority. It worked — verdicts stopped swinging with the weather.
For about a year.
The Problem Nobody Expected
THE NEW COMPLAINT:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Case: "Did Farmer Tom steal the grain?"
The evidence room contains 20 items. But one of
them is a signed confession sitting on the table.
Judge 1: reads the confession -> Guilty
Judge 2: reads the confession -> Guilty
Judge 3: reads the confession -> Guilty
...
Judge 12: reads the confession -> Guilty
VOTE: 12-0 Guilty.
Twelve judges. One opinion.
Elder Booth was troubled. She had promised the village that twelve independent minds would cancel out each other's mistakes. But every judge walked into the evidence room, saw the most obvious item, and stopped looking.
WHAT SHE REALISED:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The jury was never really twelve juries.
It was ONE jury, repeated twelve times.
If all twelve make the same mistake,
averaging twelve mistakes gives you...
the same mistake.
Averaging only cancels errors that POINT IN
DIFFERENT DIRECTIONS.
Identical judges have identical errors.
Identical errors don't cancel. They ADD UP.
And the confession, it turned out, had been forged.
The Strange Solution
Elder Booth proposed something the village council found insulting.
"What if we blindfold every judge — and blindfold each one differently?"
Her rule was simple and, on its face, absurd. Before each judge examined the evidence, a clerk would randomly cover up most of the evidence room. Judge 4 might only be allowed to see the ledger and the cart tracks. Judge 9 might only see the witness statements and the barn lock.
The council objected: you are making every judge worse. Booth agreed. That was the point.
THE BLINDFOLD RULE:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
20 pieces of evidence. Each judge sees 4, at random.
Judge 1 sees: confession, ledger, lock, tracks
Judge 2 sees: witness A, barn, weather, cart
Judge 3 sees: ledger, weather, witness B, sacks
Judge 4 sees: tracks, sacks, witness A, boots
...
Judge 2 has NEVER SEEN the forged confession.
Judge 3 has NEVER SEEN the forged confession.
Judge 4 has NEVER SEEN the forged confession.
They are forced to reason from other things.
Each judge is now WEAKER than before.
The jury is STRONGER than before.
Fifteen of the twenty judges never saw the confession at all. They reasoned from cart tracks and the barn lock and the weather that night, and eleven of them concluded Tom was somewhere else entirely.
Verdict: not guilty. Correct, this time.
Why Weaker Judges Made a Better Jury
THE ARITHMETIC OF DISAGREEMENT:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A jury's reliability depends on TWO things:
1. How good each judge is (competence)
2. How much they DISAGREE (independence)
The old jury maximised #1 and destroyed #2.
Every judge was excellent, and every judge
was excellent in exactly the same way.
The blindfold trades a little of #1
for a great deal of #2.
competence: slightly down ▼
independence: hugely up ▲▲▲
Net effect: the jury gets BETTER.
The village called this the BLINDFOLD PARADOX.
Machine learning calls it RANDOM FOREST.
That trade — a small loss in individual quality for a large gain in diversity — is the entire idea. Everything below is that sentence, written in mathematics and Python.
What Is a Random Forest?
A Random Forest is a bagging ensemble of decision trees with one additional source of randomness: feature subsampling at every split.
That's it. Two knobs of randomness, stacked:
| Source of randomness | Where it comes from | What it does |
|---|---|---|
| Bootstrap sampling | Each tree trains on a random sample of rows, drawn with replacement | Makes trees see different examples |
| Feature subsampling | At each split, the tree considers only a random subset of columns | Makes trees see different signals |
Bagging gives you the first. Random Forest adds the second — and the second is what makes it work on real data.
Why the Second Knob Matters So Much
Recall the variance of an average of models, each with variance and pairwise correlation :
Stare at that first term. As
, the second term vanishes — but
does not.
THE CORRELATION FLOOR:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Var(average) = ρσ² + (1-ρ)σ²/N
│ │
│ └─ shrinks to 0 with more trees
└───────────── NEVER shrinks. Ever.
σ² = 1.0, and you train 10,000 trees:
ρ = 0.9 -> variance floor = 0.90 (useless)
ρ = 0.5 -> variance floor = 0.50
ρ = 0.2 -> variance floor = 0.20
ρ = 0.05 -> variance floor = 0.05 (excellent)
More trees cannot fix correlation.
Only DIVERSITY can fix correlation.
This is the whole reason Random Forest exists. Plain bagged trees on real data are highly correlated, because one or two dominant features get chosen as the root split in nearly every tree. You can add ten thousand trees and never get past that floor.
Feature subsampling attacks directly. If a tree is only shown 4 of 20 features at a split, then even the strongest feature is unavailable 80% of the time, and the tree is forced to find independent structure.
The Algorithm
For to :
- Draw a bootstrap sample of size from the training data.
- Grow a tree on that sample. At every node:
- draw features at random from the available ( ),
- choose the best split among only those ,
- split, and recurse.
- Grow until the stopping rule (usually: until pure, or
min_samples_leaf).
Predict by majority vote (classification) or mean (regression).
Note step 2 carefully: the subset is redrawn at each node, not once per tree. This is the detail people get wrong.
Proving the Correlation Story in Code
Let's not take the theory on faith. We'll measure
directly for bagged trees versus a Random Forest on the same data.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
X, y = make_classification(
n_samples=2000, n_features=20, n_informative=5,
n_redundant=5, random_state=42,
)
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, random_state=42
)
def mean_tree_correlation(model, X):
"""Average pairwise correlation between individual tree predictions."""
preds = np.array([t.predict(X) for t in model.estimators_])
C = np.corrcoef(preds)
off_diag = C[np.triu_indices_from(C, k=1)]
return float(np.nanmean(off_diag))
# max_features=None -> every split sees all 20 features = plain bagging
bag = RandomForestClassifier(
n_estimators=200, max_features=None, random_state=42, n_jobs=-1
).fit(X_tr, y_tr)
# max_features='sqrt' -> every split sees ~4 features = Random Forest
rf = RandomForestClassifier(
n_estimators=200, max_features="sqrt", random_state=42, n_jobs=-1
).fit(X_tr, y_tr)
for name, m in [("Bagged trees (all features)", bag),
("Random Forest (sqrt) ", rf)]:
rho = mean_tree_correlation(m, X_te)
single = np.mean([t.score(X_te, y_te) for t in m.estimators_])
print(f"{name} rho={rho:.3f} "
f"avg single tree={single:.4f} ensemble={m.score(X_te, y_te):.4f}")
Bagged trees (all features) rho=0.640 avg single tree=0.8342 ensemble=0.8967
Random Forest (sqrt) rho=0.571 avg single tree=0.8204 ensemble=0.8983
Read those numbers slowly, because they are the entire article:
- The average individual tree in the Random Forest is worse (0.8204 vs 0.8342). The blindfold hurt each judge, exactly as promised.
- Tree-to-tree correlation dropped (0.571 vs 0.640).
- The ensemble is better (0.8983 vs 0.8967).
The ensemble gap is small on a single dataset — a handful of test rows. Hold that thought; we'll widen it into a proper measurement in a moment, because one split is not evidence.
Worse members, better committee. That is not a paradox once you have seen the variance formula — it is arithmetic.
Random Forest from Scratch
Fifty lines, NumPy only, using sklearn's tree as the base learner so we're isolating exactly the two randomness knobs.
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
class MyRandomForest:
"""Bootstrap rows + random features per split. That's the whole model."""
def __init__(self, n_estimators=200, max_features="sqrt", random_state=0):
self.n_estimators = n_estimators
self.max_features = max_features
self.random_state = random_state
def fit(self, X, y):
rng = np.random.default_rng(self.random_state)
n = len(X)
self.trees_ = []
for b in range(self.n_estimators):
idx = rng.integers(0, n, n) # bootstrap WITH replacement
tree = DecisionTreeClassifier(
max_features=self.max_features, # redrawn at EVERY split
random_state=int(rng.integers(1e9)),
).fit(X[idx], y[idx])
self.trees_.append(tree)
self.classes_ = np.unique(y)
return self
def predict_proba(self, X):
votes = np.zeros((len(X), len(self.classes_)))
for tree in self.trees_: # soft voting
votes += tree.predict_proba(X)
return votes / len(self.trees_)
def predict(self, X):
return self.classes_[self.predict_proba(X).argmax(axis=1)]
def score(self, X, y):
return float((self.predict(X) == y).mean())
X, y = make_classification(n_samples=2000, n_features=20, n_informative=5,
n_redundant=5, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=42)
mine = MyRandomForest(n_estimators=200, random_state=42).fit(X_tr, y_tr)
theirs = RandomForestClassifier(n_estimators=200, max_features="sqrt",
random_state=42, n_jobs=-1).fit(X_tr, y_tr)
print(f"MyRandomForest: {mine.score(X_te, y_te):.4f}")
print(f"sklearn RandomForest: {theirs.score(X_te, y_te):.4f}")
MyRandomForest: 0.9083
sklearn RandomForest: 0.8983
Within noise of each other, as it should be. There is no hidden magic in the library version — max_features on a DecisionTreeClassifier already does per-split subsampling, and the loop above supplies the bootstrap.
Watching Diversity Buy Accuracy
One more experiment, because this curve is worth having in your head. Sweep max_features from 1 to all 20 and watch three quantities move.
One methodological note, and it matters: a single train/test split moves by half a percent on noise alone, which is the same size as the effect we're chasing. So we average each setting over five independently generated datasets. If you take one habit from this article, take that one.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
SEEDS = [0, 1, 2, 3, 4]
print(f"{'max_feat':>8} {'single tree':>12} {'rho':>7} {'ensemble':>10}")
for mf in [1, 2, 4, 6, 10, 15, 20]:
singles, rhos, ens = [], [], []
for s in SEEDS:
X, y = make_classification(n_samples=2000, n_features=20,
n_informative=5, n_redundant=5,
random_state=s)
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.3, random_state=s)
m = RandomForestClassifier(n_estimators=200, max_features=mf,
random_state=42, n_jobs=-1).fit(X_tr, y_tr)
P = np.array([t.predict(X_te) for t in m.estimators_])
C = np.corrcoef(P)
rhos.append(np.nanmean(C[np.triu_indices_from(C, k=1)]))
singles.append(np.mean([t.score(X_te, y_te) for t in m.estimators_]))
ens.append(m.score(X_te, y_te))
print(f"{mf:>8} {np.mean(singles):>12.4f} "
f"{np.mean(rhos):>7.3f} {np.mean(ens):>10.4f}")
max_feat single tree rho ensemble
1 0.7266 0.311 0.8993
2 0.7861 0.453 0.9170
4 0.8237 0.554 0.9257
6 0.8371 0.593 0.9303
10 0.8453 0.621 0.9300
15 0.8469 0.633 0.9243
20 0.8480 0.644 0.9230
READING THE SWEEP:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Both middle columns move MONOTONICALLY:
max_features ↑ -> single trees BETTER (0.727 -> 0.848)
max_features ↑ -> rho WORSE (0.311 -> 0.644)
The ensemble does NOT. It peaks in the MIDDLE:
mf=1 0.8993 too blindfolded, trees near-useless
mf=4 0.9257 sqrt lands here
mf=6 0.9303 the peak ✓
mf=10 0.9300 still on the plateau
mf=20 0.9230 no blindfold, trees agree too much
sqrt(20) ≈ 4.47 -> sklearn's default sits on the
RISING EDGE of a broad plateau.
Good, safe, slightly conservative.
There is a genuine optimum and it is interior — that is the claim the theory makes, and the data agrees. Too much diversity and your judges are incompetent; too little and they're clones.
But notice what the honest numbers show that a tidier story would have hidden: the peak is at 6, not at sqrt ≈ 4.5, and the top is a plateau from 4 to 10 rather than a spike. sqrt(p) is a sensible default precisely because it is a little conservative — it sits safely on the diverse side of the plateau, where being wrong costs you least. It is a good starting point, not an answer. On regression the story is different enough that it gets its own post on Friday.
When Random Forest Wins, and When It Doesn't
RANDOM FOREST: THE HONEST ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
REACH FOR IT WHEN:
✓ Tabular data with mixed types
✓ You want a strong baseline in one line
✓ Non-linear interactions you can't specify
✓ You have no time to tune anything
✓ You need free validation (OOB — next article)
✓ Moderate feature count with real redundancy
LOOK ELSEWHERE WHEN:
✗ You need one explainable rule (use one tree)
✗ Data is genuinely linear (use ridge/lasso)
✗ Very high-dimensional sparse text
(linear SVM / NB usually wins)
✗ Extrapolation beyond training range
(trees CANNOT extrapolate. At all.)
✗ You need last-percent accuracy on tabular
(gradient boosting usually edges it out)
✗ Latency budget is tight (500 trees is 500 trees)
The extrapolation limitation deserves emphasis because it surprises people. A forest trained on houses priced 100k–500k will never predict 900k — not badly, but at all. Its prediction is always an average of training labels, so it is mathematically incapable of leaving the range it was shown.
The Hyperparameters That Matter
| Parameter | What it controls | Sensible range | Notes |
|---|---|---|---|
n_estimators |
Number of trees | 200–1000 | More is never worse for accuracy, only slower. Not a regularizer. |
max_features |
Features per split |
'sqrt' (clf), 0.3–0.5 (reg) |
The one that actually matters. See tomorrow's note on the regression default. |
max_depth |
Tree depth |
None, or 10–30 |
Usually leave unlimited; the ensemble handles it |
min_samples_leaf |
Min samples in a leaf | 1–5 (clf), 5–20 (reg) | The cheapest real regularizer here |
max_samples |
Bootstrap size |
None (=n) or 0.5–0.8 |
Smaller adds diversity and speed |
class_weight |
Imbalance handling | 'balanced_subsample' |
Better than 'balanced' for forests |
n_jobs |
Parallelism | -1 |
Trees are embarrassingly parallel. Always set it. |
One anti-pattern worth naming: tuning n_estimators as if it were a regularizer. It isn't. Adding trees monotonically reduces variance and then plateaus. Set it as high as your latency budget allows and tune max_features and min_samples_leaf instead.
Quick Reference Card
RANDOM FOREST: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
Bagging + random feature subset at EVERY split.
Two randomness knobs: rows (bootstrap), columns (per split).
WHY THE SECOND KNOB EXISTS:
Var(avg) = ρσ² + (1-ρ)σ²/N
└── more trees can't touch this term.
Only lower correlation can.
THE CORE TRADE:
individual tree accuracy ▼ slightly
tree-to-tree correlation ▼▼▼ a lot
ensemble accuracy ▲
DEFAULTS THAT ARE ACTUALLY GOOD:
n_estimators = 300+
max_features = 'sqrt' (classification)
= 0.3 (regression — NOT the default!)
min_samples_leaf = 1 (clf) / 5 (reg)
n_jobs = -1
WHAT NOT TO DO:
✗ Tune n_estimators as a regularizer
✗ Leave max_features=1.0 on regression
✗ Expect extrapolation
✗ Trust feature_importances_ blindly (tomorrow's post)
RF vs BAGGING:
Bagging = RF with max_features = all
RF beats it whenever features are redundant
(i.e. on essentially all real tabular data)
SKLEARN:
from sklearn.ensemble import (
RandomForestClassifier, RandomForestRegressor)
Key Takeaways
Random Forest = bagging + per-split feature subsampling — two knobs of randomness, not one, and the second is the one that earns its keep.
The subset is redrawn at every node, not once per tree. Per-tree subsampling is a different, weaker algorithm.
Correlation sets a hard floor on variance — survives no matter how many trees you add. Diversity is the only lever that touches it.
Individually worse trees make a collectively better forest. We measured it: single-tree accuracy fell from 0.848 to 0.824 while the ensemble rose from 0.9230 to 0.9257.
max_featuresis the parameter that matters. It is the dial that trades competence for independence, and ensemble accuracy peaks on a broad interior plateau — around 6 of 20 features here, withsqrtsitting safely on its diverse edge.Average over several seeds before believing any tuning result. A single split wobbles by as much as the effect you're measuring. Most "this setting is better" claims on the internet are noise.
n_estimatorsis not a regularizer. More trees never overfit; they just cost time. Set it high and stop thinking about it.Trees cannot extrapolate — every prediction is an average of training labels, so a forest can never leave the range it was shown.
Bagging is the special case where you turn the second knob off.
max_features=Nonerecovers plain bagged trees, and on redundant real-world features it is measurably worse.
The One-Sentence Summary
Random Forest works because Elder Booth was right: a committee's reliability depends less on how good each member is than on how differently they are wrong, so deliberately blindfolding every tree to a random slice of the evidence at every single decision — making each one measurably worse at its job — breaks the correlation that would otherwise put a hard floor under the ensemble's variance, and buys more accuracy than any amount of individual excellence could.
What's Next?
Now that you understand where a forest's strength comes from, you're ready for:
- Feature Importance — the forest will happily tell you which features matter, and it will happily lie to you. Tomorrow we find out why.
- Out-of-Bag Error — roughly 36.8% of rows sit out every tree, which means you have a free validation set you're probably not using.
- Extra Trees — what happens when you add a third source of randomness and stop looking for the best split entirely.
- Gradient Boosting — the other way to combine trees, which reduces bias instead of variance.
Follow me for the next article in the Random Forests Deep Dive series!
Let's Connect!
If the blindfold paradox made Random Forest click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
What's the highest n_estimators you've ever actually needed? I have never once seen a real gain past 500, and I have watched a lot of people pay for 2000. 🌲
The blindfold paradox shows up far outside machine learning. The best engineering teams I have managed were not the ones where everyone was strongest — they were the ones where people were wrong in different directions, and said so out loud. Excellence that all points the same way is just one opinion wearing twelve hats.
Share this with someone who thinks more trees is always the answer. Tell them to check their correlation instead.
Top comments (0)