The One-Line Summary: Every Random Forest decision from this series compressed onto one printable page — the three randomness knobs and what each one costs, starting hyperparameter ranges split by classification versus regression, which importance method to trust and when, the RF vs Bagging vs Extra Trees table, and a symptom-to-fix list for the eight ways a forest goes wrong in production.
The One Line That Explains Everything Else
THE VARIANCE OF AN AVERAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Var(average) = rho*s2 + (1-rho)*s2/N
| |
| +- shrinks to 0 with N
+--------------- NEVER shrinks
More trees cannot fix correlation.
Only diversity can fix correlation.
But diversity is bought with per-tree accuracy,
and on regression that bill comes due.
The Three Randomness Knobs
KNOB WHAT IT RANDOMISES MEASURED rho
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. bootstrap which ROWS each tree 0.640
sees (with replacement)
2. max_features which COLUMNS a split 0.571
may consider (sqrt)
3. random splits which THRESHOLD gets 0.515
tried (Extra Trees)
Each rung lowers correlation and lowers
per-tree accuracy. The ensemble peaks in the
middle, not at either end.
Hyperparameters, With Starting Ranges
PARAMETER CLASSIFICATION REGRESSION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
n_estimators 300+ 300+
(not a regulariser - more is
never worse, only slower)
max_features 'sqrt' (default) 1.0 (default)
try 0.3-0.5 try 0.5-0.7
NEVER 1.0 blindly NEVER 'sqrt'
max_depth None None
min_samples_leaf 1-5 5-20
max_samples None, or 0.5-0.8 to add diversity
class_weight 'balanced_subsample' n/a
n_jobs -1 -1
random_state always set it always set it
The single highest-value line in that table: max_features is the only knob that moves correlation, and its right value differs by problem type. Everything else is comparatively minor.
Which Importance Method?
CHOOSING AN IMPORTANCE METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Do you need a number you would defend
in a review?
|
+- No, just a rough sense
| -> feature_importances_ (MDI), but know
| it is a TRAINING statistic and it
| rewards high-cardinality columns
|
+- Yes
|
+- Are features correlated?
|
+- No -> permutation importance
| on HELD-OUT rows,
| n_repeats >= 10
|
+- Yes -> permutation on held-out rows,
then report the GROUP, not
the winner. Correlated columns
split the credit and the
ranking flips on noise.
MEASURED: a column of random IDs scored 0.5602
on MDI against 0.2722 for a genuinely predictive
binary flag. Pure noise went 0.18% -> 57.17%
importance on cardinality alone.
RF vs Bagging vs Extra Trees
BAGGING RF EXTRA TREES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
rows bootstrapped yes yes optional
columns per split ALL subset subset
threshold best best DRAWN
sorting needed yes yes no
speed slow medium fastest
measured rho 0.640 0.571 0.515
single-tree acc 0.8342 0.8204 0.8034
ensemble acc 0.8967 0.8983 0.9050
Bagging = RF with max_features off.
Extra Trees = RF with one more knob turned.
Out-of-Bag: Free Validation, Three Traps
OOB IN ONE BLOCK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
(1 - 1/n)^n -> 1/e = 36.8% of rows are out-of-bag
for any given tree. Measured 0.3679 at n=10,000.
Score each row with only the trees that never
saw it: validation for the price of ONE fit.
MEASURED: OOB 0.9021 | 5-fold 0.9036 | test 0.8983
~5x less wall clock than 5-fold
WHERE IT LIES:
x too few trees B=5 -> OOB 0.7829 for a
0.8750 model. Use B >= 100.
x grouped rows 5 visits/patient -> OOB
1.0000 vs GroupKFold 0.8620
x heavy imbalance 0.9710 accuracy hiding
0.1944 recall
Eight Ways a Forest Goes Wrong
SYMPTOM -> LIKELY CAUSE / FIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Great CV, bad production -> leakage, or grouped
rows split randomly.
Use GroupKFold.
Adding trees changes nothing -> you are at the rho
floor. Lower
max_features.
Importances differ every run -> random_state unset,
or correlated
columns tied.
An ID column ranks top -> MDI cardinality bias.
Switch to permutation
on held-out rows.
oob_score_ = 1.0 -> non-independent rows.
It is not that good.
High accuracy, useless model -> imbalance. Read
recall / PR-AUC.
Predictions never exceed a -> trees cannot
range you have seen extrapolate. Ever.
Regressor slower than expected -> max_features=1.0
default. Try 0.5.
Key Takeaways
Correlation is the ceiling. Everything else in this table is a way of buying diversity, and every purchase has a price.
max_featuresis the knob that matters — and its correct value differs between classification and regression. Do not copy one to the other.n_estimatorsis not a regulariser. Set it high, stop thinking about it.MDI importance is a training statistic that rewards cardinality. Permutation on held-out rows is the number you defend.
OOB is nearly free and usually accurate — until rows are grouped, trees are few, or classes are skewed.
Trees cannot extrapolate. No amount of tuning changes this.
The One-Sentence Summary
A Random Forest is three randomness knobs stacked on top of bagged trees, and tuning one is really a single decision repeated: how much per-tree accuracy will you trade for tree-to-tree independence — a trade that pays off handsomely on classification, has to be made carefully on regression, and is invisible in every metric except the one number, correlation, that no library reports to you.
What's Next?
- The series recap — the one number that ties all six posts together.
- Gradient Boosting — the other way to combine trees, attacking bias instead of variance.
- XGBoost — the engineering that made boosting win competitions.
Follow me for the next article in the Random Forests Deep Dive series!
Let's Connect!
Print this one. It is the page I actually keep open while tuning.
Questions? Ask in the comments — I read and respond to every one.
What belongs on this page that I left off? The symptom-to-fix table is the part I add to most often. 🌲
Every line in this cheat sheet was a measurement before it was a rule. The ones I inherited from blog posts rather than experiments are exactly the ones that turned out to be wrong.
Top comments (0)