The One-Line Summary: Grid-searching
learning_rateagainstn_estimatorswastes most of your compute, because the boosting model that comes out depends almost entirely on their product — measured across seven learning rates, the optimallr × n_estimatorsheld at 4.4 on classification and converged to about 3.0 on regression, so the right move is to fix the rate as low as your patience allows and let early stopping pick the count.
The Grid You Should Stop Searching
This is the search space almost everyone writes first:
{"learning_rate": [0.3, 0.1, 0.03, 0.01],
"n_estimators": [100, 300, 1000, 3000]}
Sixteen combinations. Most of them are the same model fitted at different costs, and yesterday's shrinkage table already hinted at why: the peak moved to 1, 7, 27, and 95 trees as the rate fell 1.0, 0.3, 0.1, 0.03. Those products are 1.0, 2.1, 2.7, 2.9.
So I ran it properly.
The Measurement
600 rows, 40 features, 3 informative, noise 60 — a deliberately hard regression. For each rate, fit 2000 trees and record where held-out R2 actually peaked.
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)
print(" lr peak trees peak R2 lr*trees")
for lr in (1.0, 0.5, 0.3, 0.1, 0.05, 0.03, 0.01):
m = GradientBoostingRegressor(n_estimators=2000, learning_rate=lr,
max_depth=3, random_state=0).fit(A, a)
st = np.array([r2_score(b, p) for p in m.staged_predict(B)])
k = int(st.argmax()) + 1
print(f" {lr:<6} {k:>8} {st[k-1]:>8.4f} {lr*k:>8.2f}")
lr peak trees peak R2 lr*trees
1.0 1 0.1860 1.00
0.5 2 0.2377 1.00
0.3 7 0.2838 2.10
0.1 27 0.2717 2.70
0.05 48 0.2726 2.40
0.03 95 0.2795 2.85
0.01 309 0.2795 3.09
The tree count moves by a factor of 309. The product moves from 2.10 to 3.09 once you are at or below lr=0.3 — and it is converging, not drifting.
Classification is tidier still. 1500 rows, 15% label noise, log loss on held-out data:
lr peak trees peak logloss lr*trees
1.0 2 0.4231 2.00
0.3 15 0.3864 4.50
0.1 44 0.3842 4.40
0.03 147 0.3840 4.41
4.50, 4.40, 4.41. That is one number, measured three times.
The Direct Test
If the product is what matters, four models with the same product should agree.
lr x trees = 3.0 fit time test R2
lr=0.3 n=10 0.029s 0.2549
lr=0.1 n=30 0.083s 0.2703
lr=0.03 n=100 0.273s 0.2786
lr=0.01 n=300 0.830s 0.2790
They very nearly do — and the gaps say something useful. Dropping from 0.03 to 0.01 bought 0.0004 R2 for 3x the fit time. Dropping from 0.3 to 0.1 bought 0.0154 for 2.9x. The returns are real but they die fast, which is exactly why 0.05–0.1 is the range you see in every serious codebase.
What To Do Instead
Stop searching the rate against the count. Pick the rate from your time budget, then let the count be discovered:
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
X, y = make_regression(n_samples=600, n_features=40, n_informative=3,
noise=60.0, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)
m = GradientBoostingRegressor(
learning_rate=0.05,
n_estimators=5000, # a ceiling, not a target
max_depth=3,
n_iter_no_change=30, # stop when held-out loss stalls
validation_fraction=0.15,
random_state=0).fit(A, a)
print(m.n_estimators_) # how many it actually used
Spend the search budget on max_depth, min_samples_leaf and subsample instead. Those change what the model can express. The rate mostly changes how long you wait to get there.
One caveat, because I measured this too. Early stopping is not free:
lr=0.1 stopped at 44 trees test R2 0.2158
lr=0.03 stopped at 76 trees test R2 0.2597
Both stopped short of the true peak (27 trees / 0.2717 and 95 / 0.2795). With only 420 training rows, a 15% validation slice is 63 rows and it is noisy enough to call "stalled" early. On small data, raise n_iter_no_change, raise validation_fraction, or pick the count with proper cross-validation. On tens of thousands of rows this stops being a problem.
Key Takeaways
lr × n_estimatorsis approximately invariant. Measured at 4.4 across three rates on classification and converging to ~3.0 on regression, while the tree count changed 300-fold.The rule breaks at high rates. At
lr=1.0and0.5the product collapsed to 1.0 and the peak R2 was 0.186 and 0.238 versus 0.28 lower down. Very large steps overshoot the good solutions entirely — there is no count that rescues them.Returns below
lr=0.03are negligible. 0.0004 R2 for 3x the compute. Somewhere around 0.05 is the honest default.A rate × count grid is mostly duplicate work. Fix the rate, put a high ceiling on the count, and early-stop.
Verify early stopping on small data. It stopped at 44 trees when the peak was 27, and at 76 when the peak was 95 — a tiny validation slice cuts both ways.
The One-Sentence Summary
Because each boosting round adds learning_rate × tree to a running sum, halving the rate and doubling the rounds lands in nearly the same place — so learning_rate and n_estimators are two dials wired to one shaft, and the only thing you gain by turning them independently is a longer grid search.
What's Next?
- The boosting family cheat sheet — tomorrow. Every knob from this series, plus a flowchart for choosing between AdaBoost, GBM, XGBoost, LightGBM and CatBoost.
- XGBoost — second-order gradients and regularisation written into the objective.
- LightGBM — histogram binning and leaf-wise growth.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If you have a rate × count grid running right now, kill it and put the compute on max_depth.
Questions? Ask in the comments — I read and respond to every one.
What is the most expensive redundant grid search you have shipped? Mine ran 96 configurations to discover something a single staged-prediction plot would have shown in one fit. 🗿
Boosting is a sum, and the shape of a sum tells you which of its parameters are really independent. Most of my tuning mistakes have come from treating a formula's arguments as separate knobs when the formula had already welded them together.
Top comments (0)