DEV Community: Sachin Kr. Rajput

Six Posts on Ensembles and Tuning, and the Uncomfortable Thing They All Turned Out to Be About

Sachin Kr. Rajput — Tue, 25 Aug 2026 10:45:31 +0000

The One-Line Summary: I set out to write six posts comparing ensemble and tuning methods, and every single one ended up measuring the same thing instead — that the gap between the methods was smaller than the spread of one method against its own seed, which means most published comparisons of these techniques are reporting luck with a method's name attached to it.

The Arc

Six posts, two threads: what to do when you have several models, and how to search for the settings of one. I did not plan them as a set about measurement error. They became one anyway, and by the fourth I had stopped being surprised.

Here is each, with the number that survived.

Stacking: The Editor Who Never Reported a Story

Train a small model to combine your other models, on predictions they made for rows they never saw. Done properly it reached 0.9331 against 0.9301 for the best single model — a real gain, and a small one.

Done improperly it was worse than useless. Train the combiner on in-fold predictions and it handed a weight of 1.123 to the Random Forest and 0.035 to the Gradient Booster that was genuinely better. The leak did not merely inflate the score; it selected the wrong model, because a bagged forest predicting rows it memorised looks like an oracle.

Blending and Voting: Four Noses, One Bottle

Voting has no learned parameters, so it cannot overstate itself — and it lost to the best single model on both metrics, 0.9296 for hard and 0.9289 for soft. Soft voting did not beat hard voting, because it averaged in members whose probabilities were badly calibrated.

Blending, with fitted weights, beat everything at 0.2055 when it had 150 holdout rows per weight. On 40 rows it reported 0.1492 and delivered 0.2576 — worse than the free unweighted average at 0.2515. Same code, same models. Only the holdout size changed.

Grid Search vs Random Search: The Safe With Three Dials

This is where the pattern became impossible to ignore. Random search's advantage is a real mechanism and it behaved exactly as advertised. The gaps were 0.0008 against it on three parameters and 0.0039 for it on six.

Random search's spread against its own seed was 0.0098. Grid beat exactly five of ten random seeds. A single-seed experiment had a coin-flip chance of producing either headline.

Bayesian Optimization: The Prospector

The one method whose advantage grew with budget — 0.0032, then 0.0051, then 0.0063 at 20, 50 and 120 trials. That growth is the signature of something actually learning from its own results.

But the finding that changed how I think about it was the spread. My thirty-line version landed within 0.0004 across three seeds. Random search spanned 0.0128 — thirty-two times wider, for a mean advantage of only 0.0015. The product is not a better answer. It is an answer that does not depend on luck.

Replace Your Grid Search Tonight

Two lines of pruning killed 12 of 20 trials and cut wall clock by 28%, costing 0.1908 to 0.1913 of log loss. The kill rate and the speedup do not match, because a pruned trial still pays for its warm-up. Anyone quoting the kill rate as the time saved is quoting the brochure.

The Hyperparameter Search Cheat Sheet

All of it on one page. Writing it was more uncomfortable than writing the six, because a cheat sheet wants to say "use X" and every measurement says "it depends, and by less than you think."

The Thing They Were All About

Line the effect sizes up next to the noise and the arc writes itself.

Comparison	Claimed effect	Noise in the same setup
Grid vs random, 3 params	0.0008	0.0098
Grid vs random, 6 params	0.0039	0.0098
TPE vs random, 20 trials	0.0032	0.0128
Blending vs averaging, thin holdout	claims 0.1492, delivers 0.2576	—
Stacking, leaked vs clean	picks the wrong model entirely	—

The top three are inside their own noise. The bottom two are not subtle at all — but both are failures of procedure, not of method choice. Nobody in those two rows picked the wrong algorithm. They measured it wrong.

That is the uncomfortable through-line. Six posts spent arguing about which technique wins, and the honest answer in every case was: run it twice before you believe it, and check that the thing you are comparing is bigger than the thing you are not controlling.

What Confused People

Nothing, and I am going to keep being straight about that rather than inventing a reader question.

These six drew zero substantive comments between them. That is not a mystery and it is not the writing — it is distribution, and I made it worse on purpose this week by publishing four of them inside ten minutes. The view counts show exactly what that costs: the first two published picked up 20 and 12 views, the last two got 1 and 0. Same author, same tags, same day, four minutes apart.

I have written about burst publishing being the reason two-thirds of this archive never got read, and then did it again because clearing a backlog felt more urgent than spacing it. The measurement is right there in the feed. One per day from here.

Where the real conversation has been happening is in other people's comment threads, which is its own lesson about where reach actually comes from on this platform.

The One-Sentence Summary

Across six posts on ensembling and hyperparameter search, every method comparison I ran came out smaller than the run-to-run spread of the methods being compared — 0.0008 and 0.0039 against a seed spread of 0.0098, 0.0032 against 0.0128 — and the only two effects that were unambiguously large, the stacking leak that put 1.123 on the wrong model and the blend that claimed 0.1492 while delivering 0.2576, were both procedural mistakes rather than wrong choices of algorithm, which means the highest-return skill in this entire area is not picking the right method but running the comparison twice.

What's Next?

The tuning thread is not finished — two more posts before it closes:

Early stopping — knowing when to quit, on the training curve rather than the search.
Model calibration — the weather app that says 70% and means it half the time.
K-nearest neighbours — opening a new series on distance-based learning.
Nested cross-validation — tuning and reporting without burning the same data twice.

Follow me for the next article in the Hyperparameter Tuning series!

Let's Connect!

If any single number here changed something you do, drop a heart!

Questions? Ask in the comments — I read and respond to every one, and after the section above you can see I would genuinely welcome the chance.

What's the last method comparison you ran twice? I did not run any of these twice until the grid-versus-random post forced me to, and the only reason it forced me to was that seed 0 gave me the opposite of the famous result and I got suspicious of my own good luck. ⚖️

The pattern in these six was not planned and I would not have believed it if someone had asserted it to me. What made it visible was a boring habit — running the same experiment with a different seed and reporting the spread instead of the best number. That habit costs one extra loop and it invalidated roughly half of what I expected to write. I think that is the correct ratio, and I think most published method comparisons have not paid it.

Before your next "X beats Y" post, run X twice and see whether it beats itself.

Bayesian Optimization: The Prospector Who Reads Every Core Sample Before Choosing Where to Drill

Sachin Kr. Rajput — Mon, 24 Aug 2026 10:15:27 +0000

The One-Line Summary: Bayesian optimization fits a model of your loss surface and drills where that model is both promising and uncertain — and the advantage it bought was not mainly a better answer but a far more reliable one: my thirty-line version landed within 0.0004 across three seeds while random search spanned 0.0128, which is a lottery ticket versus a procedure.

The Parable of the Kestrel Basin Survey

The company has rights to eleven thousand hectares and enough money to drill forty holes. Each hole costs a week and returns one number: how much ore is in the core. There is no map, no seismic, nothing — just eleven thousand hectares and forty chances to find out where the deposit is.

The question the geologist has to answer before the first hole is not where the ore is. It is how she will decide where hole seventeen goes.

The Lottery

The first surveyor's method is to scatter the forty holes at random across the whole basin. It is unbiased, it covers the ground, and it requires no thinking between holes.

THE LOTTERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Forty holes, chosen before any drilling starts:

  hole  1   NW quadrant   0.4 g/t
  hole  2   SE edge       0.1 g/t
  hole  3   NW quadrant   3.1 g/t   <- interesting!
  hole  4   S basin       0.2 g/t
  ...
  hole 40   E ridge       0.3 g/t

  best found: 3.1 g/t at hole 3

Hole 4 was chosen before hole 3 came back. So were
holes 5 through 40. The 3.1 reading changed nothing
about where anyone drilled next.

✗ Thirty-seven holes were sited by a plan that
  could not have known about hole 3.

This is called RANDOM SEARCH.

Every hole is independent of every other. The information arrives and is filed, and the drill goes where the envelope said it would go.

The Prospector

The second surveyor drills eight holes at random to get a feel for the ground, and then stops. She builds a rough model of the whole basin from those eight readings — a guess at the grade everywhere, plus a guess at how unsure she is everywhere. Then she drills hole nine wherever that model says the upside is largest.

"I am not looking for the best hole I have drilled. I am looking for the place my ignorance is most expensive."

THE PROSPECTOR
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
After 8 random holes, she fits a surface:

  location      predicted   uncertainty   score
  near hole 3      2.8          0.4        3.2
  far N ridge      0.9          2.1        3.0   <- !
  near hole 1      0.5          0.3        0.8

  score = predicted + k x uncertainty

She drills the far N ridge, where she predicts
almost nothing, because she barely knows anything
there and a surprise would be worth a lot.

Then she refits the surface with 9 points and asks
again. And again, thirty-one more times.

This is called BAYESIAN OPTIMIZATION.

Two things are being traded against each other, and the constant k decides the exchange rate: exploitation (drill near the good reading) and exploration (drill where you know least). Set k to zero and she drills the same hill forty times. Set it huge and she reinvents the lottery.

What The Survey Actually Showed

Both surveyors were run three times, on three different random starts, and the numbers that came back were not the ones the company expected.

THREE SURVEYS EACH
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The lottery, three attempts:
  0.8687   0.8747   0.8816     <- spread 0.0128

The prospector, three attempts:
  0.8763   0.8765   0.8768     <- spread 0.0004

On the third attempt the lottery WON, and by more
than the prospector's average margin.

But look at the spreads. The lottery's outcome is
a draw. The prospector's is a result.

✓ The prospector is not reliably better.
  The prospector is reliably.

The company had been asking which surveyor finds more ore. The useful question was which surveyor they could plan around.

Why It Works

THE MATHEMATICS OF ASKING BETTER QUESTIONS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RANDOM SEARCH
  x_(n+1) ~ P(x)          drawn from a fixed prior
  → trial n+1 is independent of trials 1..n
  → information accumulates and is never used

BAYESIAN OPTIMIZATION
  1. fit a SURROGATE model  f-hat(x)  to all
     trials so far, returning mean AND spread
  2. maximise an ACQUISITION function over it,
     e.g. UCB:   mu(x) + k * sigma(x)
  3. evaluate the true objective there, once
  4. refit and repeat

  REQUIREMENTS
  1. The objective must be EXPENSIVE relative to
     fitting the surrogate. Otherwise just sample.
  2. The surrogate must report uncertainty, not
     only a prediction.
  3. Trials must be SEQUENTIAL. Parallelism costs
     you the thing that makes it work.
  4. Roughly 8-15 random trials first, or the
     surrogate is fitting noise.

What Is Bayesian Optimization?

It is a loop with two models in it. The objective is the thing you actually care about and cannot afford to evaluate often — a cross-validated model score, a simulation, a physical experiment. The surrogate is a cheap statistical model of the objective, fitted to every evaluation so far, which must return both a prediction and an uncertainty.

The acquisition function turns those two numbers into a single score that says how much you want to try a candidate point. Upper Confidence Bound is the simplest honest one:

a_{UCB} (x) = μ (x) + k \cdot σ (x)

Expected Improvement is the more common choice and it does the same job with a different shape — it integrates how much better than the current best a point might be, rather than adding a fixed multiple of the standard deviation.

The classical surrogate is a Gaussian process, which is where "Bayesian" comes from: the GP is a prior over functions, updated by observations into a posterior [1]. In practice the surrogate is often a tree ensemble or, in Optuna's default, a Tree-structured Parzen Estimator — which does not model $p (y ∣ x)$ at all but instead models $p (x ∣ y)$ , building separate densities over the good trials and the bad ones and sampling where their ratio is favourable [2].

Step by Step

Evaluate $n_{init}$ random points. Eight to fifteen.
Fit the surrogate on everything observed so far.
Maximise the acquisition function over a large set of cheap candidates.
Evaluate the true objective at the argmax. Once.
Add it to the observations and go to 2.

Step 3 is the part that surprises people: you are running an optimisation inside an optimisation. That is fine, because the inner one is over a model that costs microseconds, and the outer one is over a model fit that costs seconds or hours.

TPE Against Random, Three Budgets

Optuna's TPE sampler against Optuna's random sampler, identical objective, five seeds each, so the spread is visible.

import warnings, numpy as np; warnings.filterwarnings("ignore")
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)

def score(lr, depth, sub, mf):
    m = GradientBoostingRegressor(n_estimators=60, learning_rate=lr, max_depth=depth,
                                  subsample=sub, max_features=mf, random_state=0)
    return cross_val_score(m, Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()

def objective(t):
    return score(t.suggest_float("lr", 0.02, 0.25, log=True),
                 t.suggest_int("depth", 2, 5),
                 t.suggest_float("sub", 0.6, 1.0),
                 t.suggest_float("mf", 0.5, 1.0))

def run(sampler_name, budget, seed):
    s = optuna.samplers.TPESampler(seed=seed) if sampler_name == "tpe" \
        else optuna.samplers.RandomSampler(seed=seed)
    st = optuna.create_study(direction="maximize", sampler=s)
    st.optimize(objective, n_trials=budget, show_progress_bar=False)
    return st.best_value

print("TPE vs RANDOM, equal budget, 5 seeds each")
print("-" * 62)
print(f"{'budget':>7}{'TPE mean':>11}{'TPE spread':>12}{'rand mean':>11}"
      f"{'rand spread':>13}{'diff':>9}")
for budget in (20, 50, 120):
    tpe = [run("tpe", budget, s) for s in range(5)]
    rnd = [run("rand", budget, s) for s in range(5)]
    print(f"{budget:>7}{np.mean(tpe):>11.4f}{max(tpe)-min(tpe):>12.4f}"
          f"{np.mean(rnd):>11.4f}{max(rnd)-min(rnd):>13.4f}"
          f"{np.mean(tpe)-np.mean(rnd):>+9.4f}")

TPE vs RANDOM, equal budget, 5 seeds each
--------------------------------------------------------------
 budget   TPE mean  TPE spread  rand mean  rand spread     diff
     20     0.8737      0.0146     0.8704       0.0077  +0.0032
     50     0.8781      0.0074     0.8730       0.0125  +0.0051
    120     0.8803      0.0090     0.8741       0.0107  +0.0063

TPE wins at all three budgets, and — this is the part that matters — the margin grows with budget: +0.0032, +0.0051, +0.0063.

That growth is the signature of a method that is actually learning. Yesterday's grid-versus-random comparison produced gaps that flipped sign with dimensionality and vanished into seed noise, because neither method used its own results. Here the gap widens monotonically, because every extra trial makes the surrogate better, which makes the next trial better chosen. Random search at 120 trials (0.8741) has barely improved on random search at 50 (0.8730), while TPE gained 0.0022 over the same stretch.

Being honest about the caveat: at budget 20 the gap is +0.0032 while TPE's own spread is 0.0146. One run at a small budget tells you nothing. The direction is consistent across all three budgets, which is the evidence — not the size of any single comparison.

From Scratch: A Surrogate and a Rule

Thirty lines. Fit a random forest to the trials so far, use the disagreement between its trees as the uncertainty, and go wherever mean plus 1.5 standard deviations is highest.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)
BOUNDS = np.array([[np.log(0.02), np.log(0.25)], [2, 5], [0.6, 1.0], [0.5, 1.0]])

def score(v):
    return cross_val_score(GradientBoostingRegressor(
        n_estimators=60, learning_rate=float(np.exp(v[0])), max_depth=int(round(v[1])),
        subsample=float(v[2]), max_features=float(v[3]), random_state=0),
        Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()

def sample(rng, n):
    return rng.uniform(BOUNDS[:, 0], BOUNDS[:, 1], size=(n, 4))

def bayes_opt(budget, seed, n_init=8, kappa=1.5, n_cand=400):
    """Fit a forest on the trials so far; go where mean + kappa*sd is highest."""
    rng = np.random.RandomState(seed)
    Xs = sample(rng, n_init); ys = np.array([score(v) for v in Xs])
    for _ in range(budget - n_init):
        sur = RandomForestRegressor(n_estimators=60, min_samples_leaf=2,
                                    random_state=0, n_jobs=-1).fit(Xs, ys)
        cand = sample(rng, n_cand)
        per_tree = np.stack([t.predict(cand) for t in sur.estimators_])
        ucb = per_tree.mean(axis=0) + kappa * per_tree.std(axis=0)
        nxt = cand[int(np.argmax(ucb))]
        Xs = np.vstack([Xs, nxt]); ys = np.append(ys, score(nxt))
    return ys.max()

def random_search(budget, seed):
    rng = np.random.RandomState(seed)
    return max(score(v) for v in sample(rng, budget))

print("FROM SCRATCH: FOREST SURROGATE + UCB vs RANDOM   (3 seeds, budget 50)")
print("-" * 68)
bo = [bayes_opt(50, s) for s in range(3)]
rs = [random_search(50, s) for s in range(3)]
for s, (b, r) in enumerate(zip(bo, rs)):
    print(f"  seed {s}   mine {b:.4f}   random {r:.4f}   diff {b-r:+.4f}")
print()
print(f"  mine   mean {np.mean(bo):.4f}   spread {max(bo)-min(bo):.4f}")
print(f"  random mean {np.mean(rs):.4f}   spread {max(rs)-min(rs):.4f}")
print(f"  mine - random = {np.mean(bo)-np.mean(rs):+.4f}")
print(f"  (optuna TPE at budget 50 scored 0.8781 mean over 5 seeds)")

FROM SCRATCH: FOREST SURROGATE + UCB vs RANDOM   (3 seeds, budget 50)
--------------------------------------------------------------------
  seed 0   mine 0.8763   random 0.8687   diff +0.0076
  seed 1   mine 0.8765   random 0.8747   diff +0.0019
  seed 2   mine 0.8768   random 0.8816   diff -0.0048

  mine   mean 0.8765   spread 0.0004
  random mean 0.8750   spread 0.0128

Read the seed rows first, because they are embarrassing and they are the point. On seed 2, random search beat my optimizer by 0.0048 — more than my average margin over it. If I had run one seed and it happened to be seed 2, the honest headline would have been that Bayesian optimization lost.

Now read the spreads. My three runs landed at 0.8763, 0.8765 and 0.8768 — a spread of 0.0004. Random search's three runs spanned 0.0128, thirty-two times wider.

That reframes the entire method. The mean advantage was +0.0015, which is nothing. The variance reduction was a factor of thirty-two, which is everything. Random search occasionally finds something excellent and occasionally finds something mediocre, and you cannot tell which happened without a baseline. The surrogate search converges to nearly the same answer every time, because after the initial random phase it is following the data rather than the dice.

That is a different value proposition from the one usually advertised. Bayesian optimization is not primarily a way to get a better model. It is a way to stop your tuning result from being a function of your seed.

And the honest scoreboard: Optuna's TPE reached 0.8781 at the same budget, against my 0.8765. Thirty lines gets you most of the mechanism and none of the years of tuning that went into a real sampler. Use the library.

When Does It Help Most?

	Random search	Bayesian optimization
Uses previous trials	no	yes
Parallelises	perfectly	poorly, it is sequential
Gain grows with budget	no (0.8730 → 0.8741)	yes (+0.0032 → +0.0063)
Run-to-run spread	0.0128	0.0004
Overhead per trial	none	a surrogate fit
Needs warm-up trials	no	8–15
Good below ~20 trials	yes	not really

Reach for Bayesian optimization when each evaluation is genuinely expensive — minutes or more — when you have 30 or more trials of budget, when trials run sequentially anyway, and when you need the result to be reproducible rather than lucky.

Reach for random search when evaluations are cheap, when you have hundreds of cores and want them all busy, or when your budget is under about twenty trials and the surrogate will not have enough data to be worth its overhead.

The failure mode worth naming: if your objective is fast, the surrogate fit and the candidate scan can cost more wall clock than the evaluations they are meant to save. Bayesian optimization is an answer to expensive objectives, and it is a tax on cheap ones.

Quick Reference Card

BAYESIAN OPTIMIZATION: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE LOOP
  1. n_init random trials  (8-15)
  2. fit surrogate -> mean + uncertainty
  3. maximise acquisition, e.g. mu + k*sigma
  4. evaluate objective ONCE there
  5. refit, repeat

MEASURED (GBM on Friedman #1, 4 params)
  budget 20   TPE 0.8737   random 0.8704   +0.0032
  budget 50   TPE 0.8781   random 0.8730   +0.0051
  budget 120  TPE 0.8803   random 0.8741   +0.0063
     -> the margin GROWS with budget

  from scratch, budget 50, 3 seeds
     mine   0.8765 mean, spread 0.0004
     random 0.8750 mean, spread 0.0128
     -> 32x tighter, not 32x better

KNOBS
  n_init      8-15; fewer and you fit noise
  kappa       0 = greedy, huge = random search
  surrogate   GP (smooth, small d) / TPE / forest

WHEN TO USE
  ✓ objective costs minutes+, budget 30+
  ✓ sequential trials, want reproducibility
WHEN NOT
  ✗ cheap objective (surrogate overhead dominates)
  ✗ budget < 20
  ✗ hundreds of idle cores

OPTUNA
  optuna.create_study(sampler=TPESampler(seed=0))

Key Takeaways

It beat random at every budget tested — +0.0032, +0.0051 and +0.0063 at 20, 50 and 120 trials.
The margin grows with budget, which is the signature of learning. Random search went 0.8730 to 0.8741 from 50 to 120 trials; TPE went 0.8781 to 0.8803.
The real prize is variance, not mean — my from-scratch version spread 0.0004 across three seeds where random search spread 0.0128.
A single seed can reverse the conclusion — on seed 2 random search beat my optimizer by 0.0048.
Thirty lines gets you most of it — a forest surrogate plus UCB reached 0.8765 against Optuna TPE's 0.8781.
But use the library — that remaining 0.0016 is years of sampler engineering you do not need to repeat.
It is sequential by design, so it trades away the one thing random search is perfect at. With idle cores, random search may finish sooner in wall clock.
kappa is the whole personality — zero makes it greedy and it re-drills one hill; enormous makes it random search with extra steps.

The One-Sentence Summary

Bayesian optimization replaces "where should I look next?" with "where is my ignorance most expensive?" by fitting a model of the loss surface and drilling where predicted value plus uncertainty is highest — and measured over three budgets its advantage over random search grew monotonically from +0.0032 to +0.0063 because every trial improves the model that chooses the next one, but the finding that actually changes how you should think about it is that my thirty-line version landed within 0.0004 across three seeds while random search spanned 0.0128, which means its real product is not a better answer but an answer that does not depend on luck.

What's Next?

The 5-minute Optuna setup — tomorrow. Replacing your grid search tonight, with pruning.
Nested cross-validation — tuning and reporting without burning the same data twice.
Early stopping inside a search — killing hopeless trials at iteration 10 instead of 200.
Multi-objective tuning — when you need accuracy and latency.

Follow me for the next article in the Hyperparameter Tuning series!

Let's Connect!

If the Kestrel Basin survey made the exploration-exploitation trade click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

What's your n_init? Mine is 10 and I arrived at it badly — I ran with 3 for a year, kept getting searches that fixated on whatever the first lucky trial found, and never connected the two until I plotted where the trials went. 🪨

References

[1] J. Snoek, H. Larochelle and R. P. Adams, Practical Bayesian Optimization of Machine Learning Algorithms (2012), NeurIPS 2012

[2] J. Bergstra, R. Bardenet, Y. Bengio and B. Kégl, Algorithms for Hyper-Parameter Optimization (2011), NeurIPS 2011

The thing I did not expect from these measurements is that the interesting quantity was the spread rather than the mean. We are trained to compare methods by average performance, and on average these two are close enough that you could argue either way with one seed each — which is exactly what most blog comparisons do. The variance is where the difference lives, and variance is invisible unless you deliberately run the same thing several times and resist the urge to report only the best one.

Send this to whoever is still running a 512-point grid overnight.

Grid Search vs Random Search: The Safe With Three Dials Where Only One Was Connected

Sachin Kr. Rajput — Mon, 24 Aug 2026 10:11:15 +0000

The One-Line Summary: Random search is supposed to beat grid search because grid wastes budget on parameters that don't matter, and the mechanism is exactly as advertised — but measured at a 64-trial budget the whole effect was 0.0039 in random's favour on six parameters and 0.0008 against it on three, while random search's spread against its own seed was 0.0098, so grid beat precisely 5 of 10 random seeds and the famous comparison came out a coin flip.

The Parable of the Safe at Voutier & Fils

The safe in the back office has three dials, each numbered 0 to 99, and the combination has been lost since the war. The firm hires a locksmith who explains that there are a million combinations and he charges by the hour, so the question is not whether he can open it but in what order he tries.

What nobody in the room knows yet is that only one of the three dials is connected to anything.

The Methodical Method

The locksmith proposes a grid. He will pick four settings on each dial — 0, 25, 50, 75 — and work through every combination systematically, so that nothing is ever tried twice and he can report exactly what has been eliminated.

THE METHODICAL METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Four settings per dial, three dials:

  4 x 4 x 4  =  64 attempts

  dial A tried at:  0, 25, 50, 75
  dial B tried at:  0, 25, 50, 75
  dial C tried at:  0, 25, 50, 75

64 attempts. Four distinct positions of dial A.

He is thorough, he is auditable, and every one of
those 64 attempts moved dial B and dial C, which
are attached to nothing at all.

This is called GRID SEARCH.

Sixty-four hours of work bought four guesses at the only dial that mattered.

The Careless Method

His apprentice, who is not methodical, spins all three dials at random and tries whatever comes up. Sixty-four times.

"You keep testing the same four numbers on the dial that opens it. I test sixty-four."

THE CARELESS METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sixty-four random spins of all three dials:

  dial A tried at:  7, 41, 88, 3, 62, 19 ... 64 values
  dial B tried at:  ... 64 values, all wasted
  dial C tried at:  ... 64 values, all wasted

Same 64 attempts. Same wasted dials. But SIXTEEN
TIMES the resolution on the dial that matters,
because he never spends two attempts on the same
setting of it.

He does not know which dial is connected either.
He does not need to.

This is called RANDOM SEARCH.

The apprentice's advantage is not cleverness. It is that he refuses to repeat himself on any axis, so he cannot waste resolution on the axis that turns out to matter.

The Part The Story Usually Leaves Out

This is where the parable is normally allowed to end, and it should not be, because the firm ran both methods and wrote the results down.

WHAT ACTUALLY HAPPENED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
When all three dials were connected:
  ✓ the methodical locksmith won

When five of six dials were decoration:
  ✓ the apprentice won

Then the apprentice repeated his 64 spins on ten
different days. His results varied more between
days than the gap between the two methods.

✗ The firm had spent a decade arguing about a
  difference smaller than the apprentice's luck.

The apprentice's argument is correct. At sixty-four attempts it is also almost entirely theoretical.

Why It Works

THE MATHEMATICS OF NOT REPEATING YOURSELF
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GRID over d parameters with n total trials:

    values per parameter = n^(1/d)

  d=2, n=64  ->  8 values each
  d=3, n=64  ->  4 values each
  d=6, n=64  ->  2 values each
  d=8, n=64  ->  fewer than 2; the grid cannot
                 even be built

  → resolution COLLAPSES as you add parameters,
    including the ones that do nothing

RANDOM over d parameters with n trials:

    values per parameter = n,  for every d

  → resolution is INDEPENDENT of dimension

  P(at least one draw in the top q fraction)
      = 1 - (1-q)^n

  This is the whole theoretical case, and it says
  nothing about how much BETTER the top q region
  is — which is why the effect size has to be
  measured rather than assumed.

What Are Grid and Random Search?

Grid search enumerates the Cartesian product of a fixed set of values per parameter. It is exhaustive over that grid, reproducible without a seed, trivially parallel, and it tells you precisely which region you have ruled out.

Random search draws each trial independently from a distribution per parameter. It is not exhaustive over anything, and its guarantee is probabilistic: with $n$ draws, the chance of landing at least once inside the best $q$ fraction of the space is $1 - (1 - q)^{n}$ , independent of how many parameters there are.

That independence is the entire argument, and it comes from Bergstra and Bengio [1]. Their claim is not that random sampling is smarter. It is that most hyperparameter spaces have low effective dimensionality — a couple of parameters do nearly all the work — and grid search cannot know which, so it spends its resolution uniformly and therefore wrongly.

The Math

For grid search over $d$ parameters with $n$ total trials, the values tested per parameter is:

k = n^{1/ d}

At $n = 64$ : three parameters gives $k = 4$ , six gives $k = 2$ , and eight gives $k < 2$ — the grid cannot be constructed without dropping a parameter.

For random search, $k = n$ regardless of $d$ , and the coverage probability is:

P (hit top q) = 1 - (1 - q)^{n}

Note what this does not say. It bounds the probability of entering the top $q$ region; it says nothing about how much better that region is than the rest. If your loss surface is flat near the optimum, landing in the top 5% is worth almost nothing — and that gap between "found a good region" and "the good region is meaningfully better" is where the measured effect sizes below come from.

Step by Step

Grid: choose values per parameter, take the product, evaluate all, keep the best.

Random: choose a distribution per parameter — log-uniform for learning rates and regularisation, uniform for fractions, integer-uniform for depths — draw $n$ times, evaluate, keep the best.

The distribution choice is the part people skip, and it matters more than the grid-versus-random question. A learning rate sampled uniformly on [0.001, 0.3] puts roughly 90% of its draws above 0.03, which is not what you meant.

Equal Budget, Both Regimes

Sixty-four trials each, three-fold CV, gradient boosting on Friedman #1. The first regime gives both methods three parameters that genuinely matter. The second adds three that barely do — which is what forces grid's resolution down to two values per axis.

import warnings, numpy as np, itertools; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)

def score(p):
    return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
                           Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()

def run_grid(axes):
    keys = list(axes); best = -9; n = 0
    for vals in itertools.product(*axes.values()):
        n += 1; v = score(dict(zip(keys, vals)))
        if v > best: best = v
    return best, n, {k: len(v) for k, v in axes.items()}

def draw(spec, rng):
    p = {}
    for k, (kind, a, b) in spec.items():
        p[k] = float(np.exp(rng.uniform(np.log(a), np.log(b)))) if kind == 'logu' else \
               (int(rng.randint(a, b + 1)) if kind == 'int' else float(rng.uniform(a, b)))
    return p

def run_random(spec, budget, seed=0):
    rng = np.random.RandomState(seed); best = -9; seen = {k: set() for k in spec}
    for _ in range(budget):
        p = draw(spec, rng)
        for k in p: seen[k].add(round(p[k], 6))
        v = score(p)
        if v > best: best = v
    return best, {k: len(s) for k, s in seen.items()}

S3 = {"learning_rate":('logu',0.02,0.25), "max_depth":('int',2,5),
      "subsample":('unif',0.6,1.0)}
S6 = dict(S3, max_features=('unif',0.5,1.0), min_samples_leaf=('int',1,8),
          min_samples_split=('int',2,16))

print("REGIME A: 3 parameters, 64-trial budget (grid gets 4 values each)")
print("-" * 66)
gA = run_grid({"learning_rate":[0.02,0.05,0.11,0.25], "max_depth":[2,3,4,5],
               "subsample":[0.6,0.75,0.9,1.0]})
rA = run_random(S3, 64)
print(f"  grid   {gA[1]:>3} trials -> best CV {gA[0]:.4f}   lr values: {gA[2]['learning_rate']}")
print(f"  random  64 trials -> best CV {rA[0]:.4f}   lr values: {rA[1]['learning_rate']}")
print(f"  random - grid = {rA[0]-gA[0]:+.4f}")

print()
print("REGIME B: 6 parameters, same 64-trial budget (grid gets 2 values each)")
print("-" * 66)
gB = run_grid({"learning_rate":[0.05,0.25], "max_depth":[2,5], "subsample":[0.6,1.0],
               "max_features":[0.5,1.0], "min_samples_leaf":[1,8],
               "min_samples_split":[2,16]})
rB = run_random(S6, 64)
print(f"  grid   {gB[1]:>3} trials -> best CV {gB[0]:.4f}   lr values: {gB[2]['learning_rate']}")
print(f"  random  64 trials -> best CV {rB[0]:.4f}   lr values: {rB[1]['learning_rate']}")
print(f"  random - grid = {rB[0]-gB[0]:+.4f}")

REGIME A: 3 parameters, 64-trial budget (grid gets 4 values each)
------------------------------------------------------------------
  grid    64 trials -> best CV 0.8729   lr values: 4
  random  64 trials -> best CV 0.8721   lr values: 64
  random - grid = -0.0008

REGIME B: 6 parameters, same 64-trial budget (grid gets 2 values each)
------------------------------------------------------------------
  grid    64 trials -> best CV 0.8688   lr values: 2
  random  64 trials -> best CV 0.8728   lr values: 64
  random - grid = +0.0039

The crossover is real and it is in the predicted direction. With three parameters and four grid values each, grid won by 0.0008. With six parameters and two grid values each, random won by 0.0039.

Notice the resolution column, because that is the mechanism working exactly as described: random search tried 64 distinct learning rates in both regimes, while grid tried 4 and then 2. Adding three parameters that barely matter cut grid's resolution on the parameter that matters most in half, and cost it 0.0041 of CV score (0.8729 down to 0.8688) while random search lost almost nothing (0.8721 to 0.8728).

So Bergstra and Bengio are vindicated on mechanism. Now the awkward question: is either gap bigger than noise?

Is Any of This Bigger Than Luck?

Random search has a seed. Grid search does not. So run random search ten times and look at the spread.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)

def score(p):
    return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
                           Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()

S3 = {"learning_rate":('logu',0.02,0.25), "max_depth":('int',2,5),
      "subsample":('unif',0.6,1.0)}

def rand_best(spec, budget, seed):
    rng = np.random.RandomState(seed); best = -9
    for _ in range(budget):
        p = {}
        for k, (kind, a, b) in spec.items():
            p[k] = float(np.exp(rng.uniform(np.log(a), np.log(b)))) if kind == 'logu' else \
                   (int(rng.randint(a, b+1)) if kind == 'int' else float(rng.uniform(a, b)))
        v = score(p)
        if v > best: best = v
    return best

GRID_3P = 0.8729   # deterministic, from the previous experiment

print("RANDOM SEARCH, 64 TRIALS, TEN SEEDS  (3 params)")
print("-" * 60)
vals = [rand_best(S3, 64, s) for s in range(10)]
for s, v in enumerate(vals):
    print(f"  seed {s}  best CV {v:.4f}")
print()
print(f"  mean {np.mean(vals):.4f}   min {min(vals):.4f}   max {max(vals):.4f}"
      f"   spread {max(vals)-min(vals):.4f}   sd {np.std(vals):.4f}")
print(f"  grid (3 params) = {GRID_3P:.4f}; "
      f"grid beats {sum(1 for v in vals if v < GRID_3P)} of 10 seeds")

print()
print("COVERAGE ARITHMETIC (why random search is supposed to work)")
print("-" * 60)
for q in (0.10, 0.05, 0.01):
    for n in (30, 64, 200):
        print(f"  P(at least one of {n:>3} draws lands in the top {q:.0%}) = {1-(1-q)**n:.4f}")

RANDOM SEARCH, 64 TRIALS, TEN SEEDS  (3 params)
------------------------------------------------------------
  seed 0  best CV 0.8721
  seed 1  best CV 0.8726
  seed 2  best CV 0.8738
  seed 3  best CV 0.8792
  seed 4  best CV 0.8723
  seed 5  best CV 0.8743
  seed 6  best CV 0.8694
  seed 7  best CV 0.8711
  seed 8  best CV 0.8765
  seed 9  best CV 0.8785

  mean 0.8740   min 0.8694   max 0.8792   spread 0.0098   sd 0.0030
  grid (3 params) = 0.8729; grid beats 5 of 10 seeds

COVERAGE ARITHMETIC (why random search is supposed to work)
------------------------------------------------------------
  P(at least one of  30 draws lands in the top 10%) = 0.9576
  P(at least one of  64 draws lands in the top 10%) = 0.9988
  P(at least one of 200 draws lands in the top 10%) = 1.0000
  P(at least one of  30 draws lands in the top 5%) = 0.7854
  P(at least one of  64 draws lands in the top 5%) = 0.9625
  P(at least one of 200 draws lands in the top 5%) = 1.0000
  P(at least one of  30 draws lands in the top 1%) = 0.2603
  P(at least one of  64 draws lands in the top 1%) = 0.4744
  P(at least one of 200 draws lands in the top 1%) = 0.8660

Random search's spread against itself is 0.0098. The gap it won by in six dimensions is 0.0039. The gap it lost by in three is 0.0008. The noise is two and a half times the larger effect and twelve times the smaller one.

And the head-to-head is the cleanest way to say it: grid beat 5 of 10 random seeds. Exactly five. If I had run one seed and published, this article was a fair coin flip between "the famous result replicates" and "the famous result doesn't," from identical code on identical data at an identical budget.

The coverage table shows why the theory is nonetheless right, and where it stops helping. Sixty-four draws reach the top 10% essentially always (0.9988) and the top 5% almost always (0.9625). But the top 1% is worse than a coin flip at 0.4744, and getting there reliably takes about 200 draws (0.8660). Early in tuning, when you are still finding the right order of magnitude, 64 random draws are plenty. Late in tuning, when the remaining gains live in the top 1% of the space, 64 draws are not a search — they are a sample, and so is a grid.

The Searches from Scratch

Both searches above are hand-rolled — a nested loop over a Cartesian product, and a loop of independent draws. Neither uses GridSearchCV or RandomizedSearchCV, which is deliberate: the whole argument is about how a budget gets spent across axes, and that is easier to see when the loop is visible. Worth confirming the from-scratch grid agrees with the library, though, since every number above rests on it.

import warnings, numpy as np, itertools; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import (train_test_split, KFold, cross_val_score,
                                     GridSearchCV)
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_friedman1(n_samples=800, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(3, shuffle=True, random_state=0)
AXES = {"learning_rate":[0.02,0.05,0.11,0.25], "max_depth":[2,3,4,5],
        "subsample":[0.6,0.75,0.9,1.0]}

def score(p):
    return cross_val_score(GradientBoostingRegressor(n_estimators=60, random_state=0, **p),
                           Xtr, ytr, cv=cv, scoring='r2', n_jobs=-1).mean()

mine = max(score(dict(zip(AXES, v))) for v in itertools.product(*AXES.values()))
skl = GridSearchCV(GradientBoostingRegressor(n_estimators=60, random_state=0),
                   AXES, cv=cv, scoring='r2', n_jobs=-1).fit(Xtr, ytr)
print("GRID SEARCH FROM SCRATCH vs SCIKIT-LEARN   (best CV r2)")
print("-" * 56)
print(f"  mine (nested loop over the product) {mine:.4f}")
print(f"  sklearn GridSearchCV                {skl.best_score_:.4f}")
print(f"  trials: mine {4*4*4}, sklearn {len(skl.cv_results_['params'])}")

GRID SEARCH FROM SCRATCH vs SCIKIT-LEARN   (best CV r2)
--------------------------------------------------------
  mine (nested loop over the product) 0.8729
  sklearn GridSearchCV                0.8729
  trials: mine 64, sklearn 64

Identical, as it should be — GridSearchCV is a nested loop with a progress bar and better parallelism. Which is worth saying plainly: there is nothing in either library search that you could not write in six lines, and neither of them does anything with the results of trial 1 before choosing trial 2.

When Does Each Help Most?

	Grid search	Random search
Resolution per parameter	$n^{1/ d}$ , collapses with $d$	$n$ , independent of $d$
Reproducible	exactly, no seed	only with the seed
States what you ruled out	yes	no
Handles 6+ parameters	badly	fine
Resume with more budget	no, the grid is fixed	yes, draw more
Measured, 3 params	0.8729	0.8740 mean, 0.0098 spread
Measured, 6 params	0.8688	0.8728

Reach for grid when you have two or three parameters, when you want an auditable statement about what was eliminated, or when the parameters are genuinely categorical with few levels.

Reach for random when you have four or more parameters, when you want to add budget later without restarting, or when you do not know which parameters matter — which is most of the time.

Reach for neither when the honest answer is that 64 trials cannot resolve your question. Both methods here are sampling blind; neither uses what the previous 63 trials revealed. That is tomorrow's article.

Quick Reference Card

GRID vs RANDOM SEARCH: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RESOLUTION PER PARAMETER at n = 64 trials
  grid, 2 params    8 values      random   64
  grid, 3 params    4 values      random   64
  grid, 6 params    2 values      random   64
  grid, 8 params    cannot build  random   64

COVERAGE, random search
  top 10%   30 draws 0.9576   64 draws 0.9988
  top  5%   30 draws 0.7854   64 draws 0.9625
  top  1%   30 draws 0.2603   64 draws 0.4744
            -> the top 1% needs ~200 draws (0.8660)

MEASURED (64 trials, GBM on Friedman #1)
  3 params   grid 0.8729   random 0.8740 mean
  6 params   grid 0.8688   random 0.8728
  random seed spread: 0.0098  <- WIDER than
             either gap (0.0008 and 0.0039)
  grid beat 5 of 10 random seeds

RULES
  ✓ log-uniform for learning rates, regularisation
  ✓ 4+ parameters: use random
  ✓ 2-3 parameters: grid is fine and auditable
  ✓ report the seed spread, never one seed
  ✗ 64 trials cannot resolve the top 1%

SKLEARN
  from sklearn.model_selection import GridSearchCV
  from sklearn.model_selection import RandomizedSearchCV

Key Takeaways

The mechanism is real and visible — random search tried 64 distinct learning rates at every dimension; grid tried 4 at three parameters and 2 at six.
Adding useless parameters costs grid real score — 0.8729 down to 0.8688 when three near-irrelevant parameters halved its resolution, while random went 0.8721 to 0.8728.
The crossover happened where predicted — grid ahead by 0.0008 on three parameters, random ahead by 0.0039 on six.
Both gaps are inside the noise — random search's seed-to-seed spread was 0.0098, with a standard deviation of 0.0030.
Grid beat exactly 5 of 10 random seeds, which is the most honest possible summary of the comparison at this budget.
Coverage explains both the theory and its limit — 0.9988 for the top 10% at 64 draws, but only 0.4744 for the top 1%.
Grid resolution collapses as $n^{1/ d}$ — at 64 trials and 8 parameters the grid cannot be built at all.
The sampling distribution matters more than the method — uniform draws of a learning rate on [0.001, 0.3] land mostly above 0.03, and no grid-versus-random argument repairs that.

The One-Sentence Summary

Random search's advantage over grid search is a real mechanism that behaved exactly as advertised — it held 64 distinct learning rates at every dimension while grid collapsed from 4 values to 2 and lost 0.0041 of score doing it — and yet the head-to-head gaps were 0.0008 and 0.0039 against a seed-to-seed spread of 0.0098, so grid beat five of ten random seeds and the correct conclusion at a 64-trial budget is not that one method wins but that neither method is learning anything from the trials it already ran.

What's Next?

Now that you know why both methods are guessing, you're ready for:

Bayesian optimization — tomorrow. The search that reads its own results before choosing the next trial.
The 5-minute Optuna setup — replacing your grid search tonight.
Nested cross-validation — how to tune and report without burning the same data twice.
Early stopping inside a search — killing bad trials at iteration 10 instead of 200.

Follow me for the next article in the Hyperparameter Tuning series!

Let's Connect!

If the safe at Voutier & Fils made the resolution argument click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever published a tuning comparison from one seed? I nearly did. The first version of this ran seed 0, found grid ahead, and I was three paragraphs into explaining why the famous result fails to replicate before it occurred to me to try seed 1. Seed 3 would have had me arguing the opposite just as confidently. 🔐

References

[1] J. Bergstra and Y. Bengio, Random Search for Hyper-Parameter Optimization (2012), Journal of Machine Learning Research 13

What makes this one worth measuring rather than citing is that the original paper is entirely correct and almost universally over-applied. Bergstra and Bengio proved something specific about resolution in high-dimensional spaces with low effective dimensionality, and the profession compressed it into "random search beats grid search" — a different claim, one that depends on a budget and a loss surface the paper never promised anything about. The paper is not wrong. The sentence it turned into is.

Before your next tuning run, count your parameters — then run the search twice with different seeds and check whether the gap you were about to report survives.

The Hyperparameter Search Cheat Sheet: Which Method, What Budget, and When to Stop

Sachin Kr. Rajput — Mon, 24 Aug 2026 10:05:14 +0000

The One-Line Summary: Every number on this page was measured in an earlier post in this series, and together they say something the folklore doesn't: grid versus random is a coin flip at small budgets (grid beat 5 of 10 random seeds), Bayesian optimization's real product is reliability rather than accuracy (spread 0.0004 against 0.0128), and pruning's advertised speedup is roughly half what the kill rate implies.

Pick By Budget, Not By Fashion

WHICH SEARCH, BY TRIAL BUDGET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  < 20 trials    random search
                 a surrogate has nothing to fit;
                 grid can't resolve anything either

  20-30          random, or TPE if trials are slow
                 measured gap at 20: +0.0032, but
                 TPE's own spread was 0.0146

  30-150         Bayesian optimisation (TPE)
                 gap grows with budget:
                 +0.0032 -> +0.0051 -> +0.0063
                 at 20 -> 50 -> 120 trials

  any budget     grid, IF you have 2-3 parameters
                 and need an auditable statement
                 about what you ruled out

  never          grid with 6+ parameters
                 resolution collapses to 2 values
                 per axis and cost 0.0041 of score

The Resolution Argument, In One Table

VALUES TESTED PER PARAMETER at n = 64 trials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  params   grid = n^(1/d)      random = n
      2         8                   64
      3         4                   64
      6         2                   64
      8    cannot build             64

MEASURED CONSEQUENCE (GBM, Friedman #1, 64 trials)
  3 params   grid 0.8729   random 0.8740 mean
  6 params   grid 0.8688   random 0.8728

  → adding three near-useless parameters cost
    grid 0.0041 and cost random almost nothing

But Check The Noise Before You Believe Any Of It

RANDOM SEARCH AGAINST ITSELF, 10 SEEDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  min 0.8694   max 0.8792   spread 0.0098
  sd 0.0030    mean 0.8740

  grid (deterministic) 0.8729
  → grid beat 5 of 10 random seeds

  the gaps people argue about:
    3 params  0.0008     6 params  0.0039
  the noise they argue inside:
                       0.0098

✗ A single-seed comparison of search methods is
  a coin flip reported as a finding.

What Bayesian Optimization Actually Buys

TPE vs RANDOM, 5 SEEDS EACH
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  budget   TPE      random    gap
      20   0.8737   0.8704   +0.0032
      50   0.8781   0.8730   +0.0051
     120   0.8803   0.8741   +0.0063

  → the gap GROWS. random went 0.8730 to 0.8741
    from 50 to 120 trials; TPE gained 0.0022.

FROM SCRATCH (forest surrogate + UCB), 3 seeds
  mine    0.8765 mean   spread 0.0004
  random  0.8750 mean   spread 0.0128

  → 32x tighter, not 32x better.
    On seed 2 random search WON by 0.0048.

The product is reproducibility, not accuracy.

Pruning: The Honest Numbers

OPTUNA MEDIAN PRUNER, 20 TRIALS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  no pruner       logloss 0.1908   13.0s   pruned 0
  median pruner   logloss 0.1913    9.4s   pruned 12

  killed 60% of trials -> saved 28% of wall clock
  cost: 0.0005 log loss

✗ The kill rate is not the speedup. A pruned trial
  still pays for its warm-up steps before there is
  enough evidence to stop it.

The Search Space Matters More Than The Search

BEFORE YOU CHOOSE A METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ log-uniform for learning rates, regularisation,
  anything spanning orders of magnitude
✓ integer-uniform for depths and counts
✓ uniform for fractions (subsample, colsample)

✗ uniform on [0.001, 0.3] for a learning rate puts
  ~90% of draws above 0.03

No amount of grid-versus-Bayesian argument repairs
a space that never samples where the answer is.

COVERAGE, random search (1 - (1-q)^n)
  top 10%   64 draws  0.9988
  top  5%   64 draws  0.9625
  top  1%   64 draws  0.4744   <- coin flip
            200 draws 0.8660

→ early tuning: 64 draws is plenty
→ late tuning: the gains live in the top 1%,
  and 64 draws cannot reach it

Quick Reference Card

HYPERPARAMETER SEARCH: ONE PAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DECIDE
  2-3 params, want audit trail    -> grid
  4+ params, cheap trials         -> random
  30+ trials, expensive trials    -> TPE
  need reproducible result        -> TPE
  hundreds of idle cores          -> random

TUNE FIRST
  learning_rate x n_estimators (one parameter)
  then max_depth 3-6
  then subsample / colsample
  then stop

ALWAYS
  ✓ log-uniform for rates
  ✓ 8-15 warm-up trials before a surrogate
  ✓ run twice with different seeds
  ✓ report the spread, not the best seed
  ✗ never quote a single-seed method comparison
  ✗ never prune against one validation split

OPTUNA MINIMUM
  study = optuna.create_study(
      sampler=TPESampler(seed=0),
      pruner=MedianPruner(n_warmup_steps=2))
  trial.report(loss, step)
  if trial.should_prune(): raise TrialPruned()

Key Takeaways

Grid versus random is decided by dimension — grid 0.8729 against random 0.8740 on three parameters, and 0.8688 against 0.8728 on six.
At 64 trials neither wins — grid beat exactly 5 of 10 random seeds, and the seed spread of 0.0098 swamps both gaps.
Bayesian optimization compounds — +0.0032, +0.0051, +0.0063 at budgets of 20, 50 and 120.
Its real product is variance reduction — 0.0004 spread against random search's 0.0128.
Pruning's kill rate is not its speedup — 60% of trials killed bought 28% of wall clock, for 0.0005 log loss.
The sampling distribution outranks the method — uniform draws on [0.001, 0.3] put 90% of a learning-rate search above 0.03.
Late-stage tuning needs a different budget — the top 1% of the space is 0.4744 at 64 draws and 0.8660 at 200.

The One-Sentence Summary

Choose your search by trial budget and parameter count rather than by what is fashionable — random below twenty trials, grid only at two or three parameters where the audit trail is worth something, TPE above thirty where its advantage compounds from +0.0032 to +0.0063 — and whatever you choose, run it twice with different seeds, because the spread between two runs of the same method was 0.0098 and every method comparison in this series was smaller than that.

What's Next?

The series recap — the whole tuning arc in one page.
Nested cross-validation — tuning and reporting without burning the same data twice.
Pruning on a CV signal — how to stop pruning against the noise of one split.
Multi-objective tuning — accuracy and latency at the same time.

Follow me for the next article in the Hyperparameter Tuning series!

Let's Connect!

If this saved you a grid search, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

What's your default search now? Mine is TPE with 50 trials and a median pruner, and the honest reason is not the accuracy — it is that I stopped having to wonder whether last week's result was skill or the seed. 📋

Assembling this page was more uncomfortable than writing the articles it summarises, because a cheat sheet wants to say "use X" and every measurement in this series says "it depends, and by less than you think." The one line I would keep if I had to delete the rest: run your search twice with different seeds before you believe anything it told you.

Bookmark this, and check the noise row before you report a tuning win.

Replace Your Grid Search Tonight: The Optuna Setup That Killed 12 of My 20 Trials

Sachin Kr. Rajput — Mon, 24 Aug 2026 10:03:00 +0000

The One-Line Summary: Two extra lines inside the objective — report the score as the model grows, then ask whether to stop — let Optuna abandon 12 of 20 trials before they finished, cutting wall clock from 13.0s to 9.4s at a cost of 0.0005 log loss; the saving is smaller than the kill rate because a pruned trial still runs its warm-up, which is the detail nobody mentions.

The Two Lines That Do The Work

Every grid search has the same defect: it runs every candidate to completion, including the ones that were obviously hopeless after 20% of the work. A human watching a training curve would have killed them. Pruning is that human, automated.

WHAT PRUNING ACTUALLY DOES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
trial 7, learning_rate=0.011, depth=2

   40 trees   logloss 0.61   <- median so far: 0.28
   80 trees   logloss 0.52   <- still miles behind
                             -> KILLED

  120, 160, 200 trees: never run.

A grid search would have finished this trial, then
politely reported that it was bad.

  trial.report(loss, step)      tell optuna
  if trial.should_prune(): ...  let it decide

That is the entire mechanism. You are handing Optuna a progress signal it can compare across trials, and letting it stop the ones that are behind.

The Whole Setup

Runnable as-is. The only Optuna-specific concepts are suggest_* for the search space, report for progress, and should_prune for the verdict.

import warnings, numpy as np, time; warnings.filterwarnings("ignore")
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=1500, n_features=15, n_informative=6,
                           flip_y=0.03, random_state=42)
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)

def objective(trial):
    lr    = trial.suggest_float("learning_rate", 0.01, 0.3, log=True)
    depth = trial.suggest_int("max_depth", 2, 5)
    sub   = trial.suggest_float("subsample", 0.6, 1.0)
    clf = GradientBoostingClassifier(n_estimators=200, learning_rate=lr, max_depth=depth,
                                     subsample=sub, random_state=0, warm_start=True)
    best = 9.9
    for n in range(40, 201, 40):                 # grow the model in steps
        clf.set_params(n_estimators=n); clf.fit(Xtr, ytr)
        loss = log_loss(yva, clf.predict_proba(Xva)[:, 1])
        best = min(best, loss)
        trial.report(loss, n)                    # tell optuna how it's going
        if trial.should_prune():                 # let optuna stop a hopeless trial
            raise optuna.TrialPruned()
    return best

for label, pruner in [("no pruner", optuna.pruners.NopPruner()),
                      ("median pruner", optuna.pruners.MedianPruner(n_startup_trials=5,
                                                                   n_warmup_steps=2))]:
    t0 = time.perf_counter()
    st = optuna.create_study(direction="minimize",
                             sampler=optuna.samplers.TPESampler(seed=0), pruner=pruner)
    st.optimize(objective, n_trials=20, show_progress_bar=False)
    el = time.perf_counter() - t0
    pruned = sum(1 for t in st.trials if t.state == optuna.trial.TrialState.PRUNED)
    print(f"{label:<15} best logloss {st.best_value:.4f}   {el:>6.1f}s   "
          f"pruned {pruned:>2}/20")

no pruner       best logloss 0.1908     13.0s   pruned  0/20
median pruner   best logloss 0.1913     9.4s   pruned 12/20

Reading That Honestly

Pruning killed 12 of 20 trials — 60% of the search — and saved 28% of the wall clock. Those two numbers should not match, and the gap is the useful part: a pruned trial is not free. It ran its warm-up steps before there was enough evidence to kill it, so you pay the first fraction of every bad trial no matter what. n_warmup_steps=2 here means two reports before any trial is eligible, which is the floor on what you can save.

If you see pruning advertised as "60% fewer trials, 60% less time," that is the advertisement, not the measurement.

It cost 0.0005 of log loss — 0.1908 without, 0.1913 with. Pruning can throw away a trial that would have recovered late; some configurations look bad at 80 trees and are excellent at 200. That risk is real and here it cost half a thousandth. Worth it, and worth knowing you paid it.

The trade scales with how expensive your trials are. At 13 seconds for the whole study, none of this matters. At 13 hours it is the difference between one overnight run and three. Pruning is a wall-clock optimisation, so its value is proportional to what a trial costs you.

One thing worth doing differently in a real project: this example prunes against a single validation split, which is what makes it fast enough to be a five-minute demo. In production you want the pruning signal to come from cross-validation, or you are pruning on the noise of one split — and that noise is precisely what the previous two articles were about.

Key Takeaways

Two lines buy the whole mechanism — trial.report(loss, step) and if trial.should_prune(): raise optuna.TrialPruned(). Everything else is the search space you already had.
Killing 60% of trials saved 28% of the time, because warm-up steps run before any trial can be pruned. Never quote the kill rate as the speedup.
It cost 0.0005 log loss (0.1908 to 0.1913) — pruning occasionally discards a late bloomer, and that is the price of the wall clock.

The One-Sentence Summary

Optuna's pruner turns a grid search's worst habit — finishing trials that were already hopeless — into an automated judgment call, and measured here it abandoned 12 of 20 trials to cut 13.0s down to 9.4s for 0.0005 of log loss, with the honest caveat that the 60% kill rate only bought a 28% saving because every pruned trial still pays for its warm-up.

What's Next?

Nested cross-validation — tuning and reporting without burning the same data twice.
Pruning on a CV signal — how to avoid pruning against the noise of one split.
Multi-objective tuning — when you need accuracy and latency.
Search spaces that lie — why uniform sampling of a learning rate is almost always wrong.

Follow me for the next article in the Hyperparameter Tuning series!

Let's Connect!

If this replaces a grid search tonight, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

What's the longest grid search you've ever let finish? Mine was nineteen hours across 486 candidates, and when I finally plotted the curves afterwards, roughly two thirds of them were clearly dead inside the first 15% of their training. ⏱️

The reason pruning feels like cheating is that it is the one optimisation that requires the search to have an opinion mid-trial. Grid and random search treat a trial as atomic: you ask a question, you wait, you get a number. Pruning treats the training curve as evidence in its own right, which is obvious the moment you have ever watched a loss curve and known at epoch three that it was not going to work.

Copy the objective above, swap in your model, and delete your grid.

Blending and Voting: Four Noses, One Bottle, and the Blender Who Graded His Own Homework

Sachin Kr. Rajput — Fri, 21 Aug 2026 07:56:57 +0000

The One-Line Summary: Voting has no learned parameters, so it cannot overfit and it cannot lie to you — but measured here it lost to the single best model on both metrics. Blending learns weights on a holdout and genuinely beat everything, right up until the holdout got small: on 40 rows it reported a log loss of 0.1492, delivered 0.2576, and lost to the free unweighted average it was supposed to improve.

The Parable of the House at Grasse

The house employs four noses, and every batch of oil that arrives has to be judged before the house will put its name on a bottle. The four disagree constantly, which the director considers the point — a panel that always agrees is three salaries wasted. What the house has never settled is how to turn four opinions into one decision.

The Show of Hands

The oldest method is a show of hands. Each nose says ship it or don't, and the majority carries.

THE SHOW OF HANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Batch 41 arrives.

  Amaury    ship it
  Berthe    ship it
  Cosette   don't
  Didier    ship it

  3 to 1.  The house ships it.

Nobody recorded HOW sure anyone was. Berthe was
certain. Didier nearly abstained. The tally cannot
tell them apart, and it never will.

This is called HARD VOTING.

It is cheap, it is transparent, and it throws away everything except the direction of each opinion.

The Scorecard

So the house moved to scores. Each nose writes a number from 0 to 100, and the panel's verdict is the average.

Better — the average now knows the difference between certainty and a shrug. Except the house discovered something about Didier.

THE SCORECARD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Batch 41, scores out of 100:

  Amaury     91   confident, usually right
  Berthe     88   confident, usually right
  Cosette    24   confident, usually right
  Didier     72   ← scores EVERYTHING 68 to 75

  average:  68.8

Didier is not a bad nose. Ask him to rank two
batches and he is excellent. But his numbers have
no range, so averaging drags every verdict toward
the middle of his narrow band.

✗ Averaging opinions also averages in whoever
  cannot express confidence properly.

This is called SOFT VOTING.

Soft voting is usually recommended over hard voting, and the reason is genuine: probabilities carry more information than labels. The measurements below show it is not automatic.

The Director's Ledger

The director's answer was to stop treating the four as equals.

"I have eleven years of batches and eleven years of what happened to them. I do not need a fifth nose. I need to know how much each of these four is worth."

THE DIRECTOR'S LEDGER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
From the archive, weights fitted to past batches:

  Amaury    0.44
  Berthe    0.00   ← agrees with Amaury almost
  Cosette   0.56     always; adds nothing new
  Didier    0.00   ← no range, no information

  verdict = 0.44 x Amaury + 0.56 x Cosette

Berthe is not fired. She is redundant, which is a
different thing and only a ledger can tell them
apart.

This is called BLENDING.

Two of four weights went to zero. That is the ledger working, not failing.

The Month He Graded His Own Homework

The method held until the director published the house's accuracy in a trade journal.

He had fitted the weights on twelve batches from the previous month. Then, asked how well the panel performed, he reported the panel's score on those same twelve batches. The number was extraordinary. It was also the number he had chosen the weights to maximise.

GRADING YOUR OWN HOMEWORK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Twelve batches, four weights to fit.

  score on those twelve batches      excellent
  score on the next twelve           ordinary

  Three batches per weight is not a ledger.
  It is four dials tuned until the last twelve
  results looked good.

The show of hands could never do this. It has
no dials. There is nothing to tune and therefore
nothing to overstate.

This is called OVERFITTING THE BLEND.

The show of hands cannot cheat. That is its one genuine advantage, and it is bigger than it sounds.

Why It Works

THE MATHEMATICS OF AGREEING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HARD VOTING    majority of the labels
    0 learned parameters

SOFT VOTING    mean of the probabilities
    0 learned parameters
    → needs members whose confidence MEANS
      something; a flat scorer poisons the mean

BLENDING       weighted mean, weights fitted
    N learned parameters on a held-out set
    → the weights live on the simplex:
      w_i >= 0,  sum w_i = 1
      which is heavy regularisation for free

  THE ONE RATIO THAT MATTERS
    holdout rows / weights
    3 to 10   the reported number is fiction
    ~75       usable, still optimistic
    ~450      the reported number is roughly true

What Are Voting and Blending?

Hard voting takes the majority class label across members. Soft voting averages their predicted probabilities and thresholds the mean. Neither fits anything: both are fixed functions of the members' outputs, so there is no training step and no holdout to reserve.

Blending averages the same probabilities with learned weights. You hold out a slice of data the members never trained on, collect their predictions on it, and fit weights that minimise loss there. It is stacking's simpler cousin — stacking uses out-of-fold predictions across the whole training set, blending uses a single holdout — and it inherits the same discipline problem: the set you fit the weights on is burned, and any number you quote from it is a number you optimised.

The Math

For members $h_{1} \dots h_{N}$ , soft voting is the unweighted mean:

p^(x) = \frac{1}{N} i = 1 \sum N h_{i} (x)

Blending replaces $1/ N$ with fitted weights constrained to the simplex:

p^(x) = i = 1 \sum N w_{i} h_{i} (x), w_{i} \geq 0, i \sum w_{i} = 1

That constraint matters more than it looks. Non-negative weights summing to one cannot extrapolate beyond the members' predictions — the blend is trapped inside their convex hull. It is a strong regulariser you get for free, and it is why blending misbehaves less than its parameter count suggests.

Step by Step

Split off a holdout the members will never train on.
Fit every member on the remaining training data.
Collect member predictions on the holdout.
Fit weights on the simplex to minimise loss on those predictions.
Refit members on everything, keep the weights.
Report on a third set. The holdout is spent.

Voting, Measured

Four deliberately different members on 9,000 rows, split 60/20/20 into train, holdout and test. Voting needs no holdout, so for this first table it just trains and reports.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              VotingClassifier)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import log_loss, accuracy_score
from sklearn.base import clone

X, y = make_classification(n_samples=9000, n_features=25, n_informative=10,
                           n_redundant=5, flip_y=0.03, random_state=42)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.6,
                                          random_state=42, stratify=y)
Xpool, Xte, ypool, yte = train_test_split(Xrest, yrest, test_size=0.5,
                                          random_state=42, stratify=yrest)

NAMES = ["logreg", "forest", "boosting", "knn"]
FOUR = [LogisticRegression(max_iter=2000),
        RandomForestClassifier(n_estimators=150, random_state=0, n_jobs=-1),
        GradientBoostingClassifier(n_estimators=120, max_depth=3, random_state=0),
        KNeighborsClassifier(n_neighbors=15)]

fit = [clone(m).fit(Xtr, ytr) for m in FOUR]
P = np.column_stack([m.predict_proba(Xte)[:, 1] for m in fit])

print("BASE MODELS ALONE")
print("-" * 46)
print(f"{'':<12}{'accuracy':>10}{'log loss':>11}")
for n, c in zip(NAMES, range(4)):
    print(f"  {n:<10}{accuracy_score(yte,(P[:,c]>0.5).astype(int)):>10.4f}"
          f"{log_loss(yte,P[:,c]):>11.4f}")

est = [(n, clone(m)) for n, m in zip(NAMES, FOUR)]
hard = VotingClassifier(est, voting='hard', n_jobs=-1).fit(Xtr, ytr)
soft = VotingClassifier(est, voting='soft', n_jobs=-1).fit(Xtr, ytr)
print()
print("VOTING  (no learned parameters at all)")
print("-" * 46)
print(f"  hard        {accuracy_score(yte,hard.predict(Xte)):>10.4f}{'      n/a':>11}")
print(f"  soft        {accuracy_score(yte,soft.predict(Xte)):>10.4f}"
      f"{log_loss(yte,soft.predict_proba(Xte)[:,1]):>11.4f}")

BASE MODELS ALONE
----------------------------------------------
              accuracy   log loss
  logreg        0.8156     0.4070
  forest        0.9319     0.2568
  boosting      0.9193     0.2409
  knn           0.9370     0.4728

VOTING  (no learned parameters at all)
----------------------------------------------
  hard            0.9296        n/a
  soft            0.9289     0.2515

Three things here are not what the received advice predicts.

Voting lost to the best single model. KNN alone got 0.9370 accuracy; hard voting 0.9296 and soft voting 0.9289. Boosting alone got 0.2409 log loss; soft voting 0.2515. On both metrics, the ensemble is behind the best member. Averaging is not free — it drags a strong member toward a weak one, and logreg at 0.8156 accuracy and 0.4070 log loss is doing exactly that.

Soft voting did not beat hard voting. 0.9289 against 0.9296. The usual reasoning — probabilities carry more information than labels — is correct in principle and depends on those probabilities being calibrated. KNN with n_neighbors=15 can only emit multiples of 1/15, and logreg's log loss of 0.4728 and 0.4070 say neither is well calibrated here. Soft voting averaged in that miscalibration. Didier, scoring everything 72.

The best member was not the best-calibrated member. KNN wins accuracy (0.9370) and is nearly worst on log loss (0.4728). Boosting wins log loss (0.2409) and is third on accuracy. Which model is "best" is a question about your metric, not your data.

Blending, and What the Holdout Costs

Now the same members with fitted weights. The experiment sweeps holdout size against weight count and reports the gap between what the blend claims on its own tuning set and what it delivers on untouched test data.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              ExtraTreesClassifier, VotingClassifier)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import log_loss
from sklearn.base import clone
from scipy.optimize import minimize

X, y = make_classification(n_samples=9000, n_features=25, n_informative=10,
                           n_redundant=5, flip_y=0.03, random_state=42)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.6,
                                          random_state=42, stratify=y)
Xpool, Xte, ypool, yte = train_test_split(Xrest, yrest, test_size=0.5,
                                          random_state=42, stratify=yrest)

FOUR = [LogisticRegression(max_iter=2000),
        RandomForestClassifier(n_estimators=150, random_state=0, n_jobs=-1),
        GradientBoostingClassifier(n_estimators=120, max_depth=3, random_state=0),
        KNeighborsClassifier(n_neighbors=15)]
EXTRA = [ExtraTreesClassifier(n_estimators=150, random_state=0, n_jobs=-1),
         GaussianNB(),
         DecisionTreeClassifier(max_depth=3, random_state=0),
         DecisionTreeClassifier(max_depth=8, random_state=0),
         KNeighborsClassifier(n_neighbors=3),
         KNeighborsClassifier(n_neighbors=50),
         RandomForestClassifier(n_estimators=150, max_depth=5, random_state=1, n_jobs=-1),
         GradientBoostingClassifier(n_estimators=120, max_depth=1, random_state=1)]

f4 = [clone(m).fit(Xtr, ytr) for m in FOUR]
f12 = f4 + [clone(m).fit(Xtr, ytr) for m in EXTRA]
P4p  = np.column_stack([m.predict_proba(Xpool)[:, 1] for m in f4])
P4t  = np.column_stack([m.predict_proba(Xte)[:, 1]  for m in f4])
P12p = np.column_stack([m.predict_proba(Xpool)[:, 1] for m in f12])
P12t = np.column_stack([m.predict_proba(Xte)[:, 1]  for m in f12])

# soft voting from scratch is just the unweighted mean — check it
skl = VotingClassifier([(f"m{i}", clone(m)) for i, m in enumerate(FOUR)],
                       voting='soft', n_jobs=-1).fit(Xtr, ytr)
print("SOFT VOTING FROM SCRATCH vs SCIKIT-LEARN")
print("-" * 52)
print(f"  mine (mean of probabilities) {log_loss(yte, P4t.mean(axis=1)):.4f}")
print(f"  sklearn VotingClassifier     "
      f"{log_loss(yte, skl.predict_proba(Xte)[:,1]):.4f}")

def loss(w, P, yy):
    w = np.abs(w); w = w / w.sum()
    return log_loss(yy, np.clip(P @ w, 1e-9, 1 - 1e-9))

def fit_w(P, yy, k):
    r = minimize(loss, np.ones(k)/k, args=(P, yy), method='Nelder-Mead',
                 options={'maxiter':20000, 'fatol':1e-12, 'xatol':1e-10})
    w = np.abs(r.x); return w / w.sum()

def sweep(Pp, Pt, k, nb, seeds=5):
    ob, ot = [], []
    for s in range(seeds):
        idx = np.random.RandomState(s).choice(len(Xpool), nb, replace=False)
        w = fit_w(Pp[idx], ypool[idx], k)
        ob.append(loss(w, Pp[idx], ypool[idx])); ot.append(loss(w, Pt, yte))
    return np.mean(ob), np.mean(ot)

print()
print("BLENDING OPTIMISM, MEAN OF 5 HOLDOUT DRAWS   (log loss)")
print("-" * 66)
print(f"{'weights':>8}{'holdout':>9}{'on holdout':>12}{'on test':>10}"
      f"{'optimism':>11}{'rows/wt':>9}")
for Pp, Pt, k in [(P4p, P4t, 4), (P12p, P12t, 12)]:
    for nb in (40, 300, 1800):
        b, t = sweep(Pp, Pt, k, nb)
        print(f"{k:>8}{nb:>9}{b:>12.4f}{t:>10.4f}{t-b:>+11.4f}{nb//k:>9}")
    print(f"{k:>8}{'  simple average, no weights, on test':<37}"
          f"{log_loss(yte, Pt.mean(axis=1)):>10.4f}")

SOFT VOTING FROM SCRATCH vs SCIKIT-LEARN
----------------------------------------------------
  mine (mean of probabilities) 0.2515
  sklearn VotingClassifier     0.2515

BLENDING OPTIMISM, MEAN OF 5 HOLDOUT DRAWS   (log loss)
------------------------------------------------------------------
 weights  holdout  on holdout   on test   optimism  rows/wt
       4       40      0.1492    0.2576    +0.1084       10
       4      300      0.1825    0.2253    +0.0428       75
       4     1800      0.1947    0.2121    +0.0174      450
       4  simple average, no weights, on test    0.2515
      12       40      0.1527    0.2190    +0.0663        3
      12      300      0.1856    0.2099    +0.0243       25
      12     1800      0.1870    0.2055    +0.0185      150
      12  simple average, no weights, on test    0.2711

Soft voting from scratch is one line and it lands on the library exactly: 0.2515 against 0.2515. VotingClassifier(voting='soft') is probabilities.mean(axis=1) with paperwork.

Blending genuinely works when the holdout is big. Four weights on 1,800 rows: 0.2121 on test, against 0.2515 for the free average, 0.2409 for the best single member, and 0.2515 for soft voting. Twelve weights on 1,800 rows reached 0.2055, the best number anywhere in this article. So the ledger earns its keep.

And it lies in exact proportion to how little holdout it has. Four weights on 40 rows report 0.1492 and deliver 0.2576. That is not a small overstatement — the reported loss is roughly seventy percent of the real one. Worse, 0.2576 is behind the unweighted average at 0.2515. You reserved data, ran an optimiser, published a number that looked like your best result ever, and shipped something worse than doing nothing.

Optimism falls monotonically as rows-per-weight climbs: +0.1084 at 10 rows per weight, +0.0428 at 75, +0.0174 at 450. That ratio, not the model count, is the thing to watch.

A result I did not predict: twelve weights overfit less than four at every holdout size — +0.0663 against +0.1084 at 40 rows. I expected the opposite. The reason is the simplex constraint: non-negative weights summing to one cannot chase noise very far, and the eight extra members are genuinely diverse enough that the larger blend generalises better despite having three times the parameters. Parameter count is a bad proxy for overfitting risk when the parameters are boxed in.

When Does Each Help Most?

	Hard voting	Soft voting	Blending
Learned parameters	0	0	N
Needs a holdout	no	no	yes
Can overstate itself	no	no	yes
Needs calibrated members	no	yes	helps
Members must be comparable	labels only	probabilities	probabilities
Best measured here	0.9296 acc	0.2515 loss	0.2055 loss
Fails when	members disagree evenly	one member is flat	holdout is thin

Reach for hard voting when members emit labels only, or when you need a result nobody can accuse you of tuning.

Reach for soft voting when members are calibrated and you have no data to spare. Check calibration first — a member with a log loss of 0.4728 will drag the mean.

Reach for blending when you can afford several hundred holdout rows per weight and you are disciplined enough to report on a third set. Below roughly 75 rows per weight, use the unweighted average and keep the data.

Quick Reference Card

VOTING vs BLENDING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT THEY ARE
  hard vote   majority label,      0 parameters
  soft vote   mean probability,    0 parameters
  blending    weighted mean,       N parameters
              fitted on a holdout

MEASURED HERE (9,000 rows, 60/20/20)
  best single member   0.9370 acc / 0.2409 loss
  hard voting          0.9296 acc
  soft voting          0.9289 acc / 0.2515 loss
  blend, 4wt,  1800    0.2121 loss
  blend, 12wt, 1800    0.2055 loss   <- best
  blend, 4wt,  40      claims 0.1492, delivers 0.2576

THE ONE RATIO
  holdout rows / weights
    ~10   optimism +0.1084   fiction
    ~75   optimism +0.0428   usable
   ~450   optimism +0.0174   honest

RULES
  ✓ check member calibration before soft voting
  ✓ report blends on a THIRD set, never the holdout
  ✓ below ~75 rows/weight, just average
  ✗ never assume the ensemble beats its best member

SKLEARN
  from sklearn.ensemble import VotingClassifier
  VotingClassifier(est, voting='soft')

Key Takeaways

Voting has no parameters, so it cannot overfit or overstate — that is its real advantage, and it is why the show of hands could never grade its own homework.
Voting lost to the best single model here — 0.9296 and 0.9289 against KNN's 0.9370. Ensembling is not automatically an improvement.
Soft voting did not beat hard voting — 0.9289 against 0.9296, because it averaged in members with log losses of 0.4728 and 0.4070. Calibration is the precondition, not a detail.
Blending beat everything when the holdout was large — 0.2055 with twelve weights on 1,800 rows, against 0.2409 for the best member.
And it lied by +0.1084 when the holdout was small — reporting 0.1492 while delivering 0.2576, which is worse than the free average at 0.2515.
Watch rows-per-weight, not model count — optimism fell from +0.1084 to +0.0428 to +0.0174 as that ratio went 10, 75, 450.
More weights overfit less, against my expectation — +0.0663 for twelve weights versus +0.1084 for four, because simplex-constrained weights cannot chase noise.
The best member depends on the metric — KNN won accuracy and nearly lost log loss; boosting the reverse. Decide what you are optimising before you decide who is best.

The One-Sentence Summary

Voting is a show of hands that cannot lie because it has nothing to tune, and it lost to the best single model on both metrics here; blending is the director's ledger that genuinely beat everything at 0.2055 when it had 150 holdout rows per weight, and reported a loss thirty percent better than reality when it had ten — so the question is never whether to weight your models but whether you have enough untouched data to earn the weights, and the honest fallback is the unweighted average that costs nothing and claims nothing.

What's Next?

Now that you can tell voting from blending from stacking, you're ready for:

Grid search vs random search — the tourist with a map versus the one who wanders.
Bayesian optimization — the sommelier who learns your palate in six glasses.
Calibration — how to fix the flat scorer that poisoned soft voting above.
Nested cross-validation — how to tune and report without burning the same data twice.

Follow me for the next article in the Ensembles Beyond Bagging series!

Let's Connect!

If the house at Grasse made voting and blending click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever reported a number you'd tuned? I have, and the tell was that it was the best result in the whole project by a suspicious margin — which is exactly what a number looks like when you have optimised it and then quoted it. 🧪

The thing I keep relearning is that the safest method is usually the one with nothing to adjust. An unweighted average has no dials, so it cannot be talked into flattering you, and the price of that honesty is a few points of loss you can measure. Blending has four dials, or twelve, and every one of them is a small opportunity to fit last month's batches and call it skill. The measurements do not say weighting is wrong. They say weighting costs data, and that quoting a weighted result on the data you weighted it against is the same mistake as a director publishing his own warm-up scores.

Send this to whoever on your team is about to fit blend weights on 40 validation rows.

Stacking: The Editor Who Never Reported a Story and Still Wrote the Best One

Sachin Kr. Rajput — Wed, 19 Aug 2026 03:30:00 +0000

The One-Line Summary: Stacking trains a small model to combine your other models, and the entire thing turns on one detail — the combiner must be trained on predictions each base model made for rows it never saw, because when I trained it on in-fold predictions instead it handed a weight of 1.123 to the Random Forest and 0.035 to the Gradient Booster that was genuinely the better model, and the ensemble fell from 0.9331 to 0.8781.

The Parable of the Kestrel Street Newsroom

Four reporters cover the same city for the same paper, and after twenty years everyone in the building knows their habits: one over-reports crime, one is too kind to the council, one files fast and loose, one is slow and exact. For most of the paper's history the editor's job was to decide which of the four to print. Then a new editor arrived who did something stranger — she never reported a story in her life, and the paper got better.

The Old Way: Print the Best Reporter

The old method was a competition. Send all four out on the same story, read the four drafts, print the strongest one and spike the rest.

THE OLD WAY — PICK ONE, BIN THREE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Story: how many families left the east ward?

  Ambrose  says 400   (always runs high)
  Beatrix  says 180   (soft on the council)
  Colm     says 260   (fast, sloppy)
  Delia    says 310   (slow, careful)

  Truth, three months later:  295

  Printed: Delia, 310.  The other three: binned.

Delia was closest, so Delia gets printed. Next
story, the paper reads four drafts and throws away
three again.

This is called MODEL SELECTION.

It works. It is also strange: the paper spends four salaries and prints one story.

The Innovation: An Editor Who Reads Habits, Not Stories

The new editor did not go out and report a fifth version. She sat with twenty years of archives and did something nobody had bothered to do — she compared each reporter's draft against what turned out to be true, story after story, until she knew the shape of each one's error.

"I do not need a better reporter. I need to know, for each of these four, which direction they are wrong in and by how much."

THE EDITOR'S LEDGER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
From the archive, over hundreds of stories:

  Ambrose  runs about +35% high on any count
  Beatrix  runs low whenever the council is
           involved, accurate otherwise
  Colm     is unbiased but wildly noisy
  Delia    is nearly right, slightly low

So for the east ward story she writes:

  0.15 x Ambrose + 0.05 x Beatrix
  + 0.10 x Colm  + 0.75 x Delia   ->  296

  Truth: 295.  Closer than any of the four.

She reported nothing. She corrected for habits.

This is called STACKING.

The editor is not a fifth reporter. She is a model of the other four.

The Year the Editor Read the Wrong Drafts

The method held for a decade and then broke in a way nobody saw coming, because the failure looked like success.

A new archivist, trying to be helpful, started giving the editor the reporters' drafts for stories whose outcome had already been published. Colm — fast, sloppy Colm — had a habit of skimming the paper before filing. On stories where the answer was already in print, his drafts were flawless.

The editor, reading the archive honestly, concluded that Colm was the finest reporter in the building. She started weighting him above everyone. The paper got worse for a year before anyone worked out why.

THE YEAR THE ARCHIVE LIED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
On stories whose ending was already public:

  Colm's drafts     nearly perfect
  Delia's drafts    still merely very good

  Editor's conclusion:  Colm is the best.
  Editor's new weights: mostly Colm.

On NEW stories, where nothing was published yet,
Colm went back to being fast and sloppy — and the
paper was now built on him.

✗ The editor was never wrong about the archive.
  The archive was the wrong thing to read.

This is called LEAKAGE IN THE META-FEATURES.

The fix is one rule: the editor may only judge a reporter on drafts filed before the outcome was known to anyone. Nothing else about the method changes.

Why It Works

THE MATHEMATICS OF LEARNED WEIGHTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Averaging N models weights them all equally:

    f(x) = (1/N) sum_m  h_m(x)

  → a strong model gets dragged toward weak ones.

Stacking learns the weights from data:

    f(x) = g( h_1(x), h_2(x), ..., h_N(x) )

  where g is fit on OUT-OF-FOLD predictions.

  REQUIREMENTS
  1. Each h_m(x) used to train g must come from a
     model that did NOT see row x.
  2. g should be SIMPLE — it has N inputs and all
     the overfitting risk lives here.
  3. Base models should DISAGREE. Four copies of
     the same model give g nothing to learn.
  4. g may assign negative weight. That is not a
     bug; it is g cancelling a known bias.

What Is Stacking?

Stacking (stacked generalisation) trains a second-level model — the meta-learner — whose input features are the predictions of your first-level models.

The mechanism that makes it work, and the only part people get wrong, is how those input features are generated. For each training row you need a prediction from a model that never saw that row. That is exactly what k-fold cross-validation produces, so the standard recipe is: split into k folds, train each base model on k−1 of them, predict the held-out fold, and stack those held-out predictions into a matrix.

The Math

Given base learners $h_{1}, \dots, h_{N}$ and a meta-learner $g$ , the stacked prediction is:

f (x) = g (h_{1} (x), h_{2} (x), \dots, h_{N} (x))

The meta-learner is fit on the out-of-fold matrix $Z$ , where $Z_{ij} = h_{j (- k (i))} (x_{i})$ and $h_{j (- k (i))}$ is base model $j$ trained with the fold containing row $i$ removed. At prediction time the base models are refit on everything and $g$ is applied unchanged.

Note what $Z$ is not: it is not the base models' training predictions. Using those is the archivist's mistake, and the measurements below put a number on it.

Step by Step

Choose k folds. Five is standard.
For each base model, train on k−1 folds and predict the held-out fold. Repeat until every row has a prediction.
Assemble those into an n × N matrix. This is what the meta-learner sees.
Fit the meta-learner on that matrix against the true targets.
Refit every base model on the full training set.
To predict: run the refit base models, feed their outputs to the meta-learner.

Does It Actually Beat the Best Base Model?

Four base models with genuinely different failure modes, on 3,000 rows of Friedman #1. The comparison that matters is against the best single model, not against the average.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_predict
from sklearn.linear_model import RidgeCV
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score

X, y = make_friedman1(n_samples=3000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(5, shuffle=True, random_state=0)

BASE = {
    "Ridge":            lambda: RidgeCV(),
    "RandomForest":     lambda: RandomForestRegressor(n_estimators=200, random_state=0, n_jobs=-1),
    "GradientBoosting": lambda: GradientBoostingRegressor(n_estimators=200, max_depth=3, random_state=0),
    "KNN":              lambda: KNeighborsRegressor(n_neighbors=10),
}

print("BASE MODELS ALONE   (test R2)")
print("-" * 40)
solo = {}
for n, f in BASE.items():
    solo[n] = r2_score(yte, f().fit(Xtr, ytr).predict(Xte))
    print(f"  {n:<18}{solo[n]:>8.4f}")
best_name = max(solo, key=solo.get)
print(f"  best single      : {best_name} at {solo[best_name]:.4f}")

# out-of-fold meta features (correct) vs in-fold meta features (the leak)
oof    = np.column_stack([cross_val_predict(f(), Xtr, ytr, cv=cv, n_jobs=-1) for f in BASE.values()])
infold = np.column_stack([f().fit(Xtr, ytr).predict(Xtr) for f in BASE.values()])
test_meta = np.column_stack([f().fit(Xtr, ytr).predict(Xte) for f in BASE.values()])

meta_oof    = RidgeCV().fit(oof,    ytr)
meta_infold = RidgeCV().fit(infold, ytr)

print()
print("STACKING   (test R2)")
print("-" * 56)
print(f"  simple average of the 4      {r2_score(yte, test_meta.mean(axis=1)):>8.4f}")
print(f"  stacking, out-of-fold meta   {r2_score(yte, meta_oof.predict(test_meta)):>8.4f}")
print(f"  stacking, in-fold meta (LEAK){r2_score(yte, meta_infold.predict(test_meta)):>8.4f}")
print()
print("  what the meta-learner learned (weight per base model)")
for n, w_o, w_i in zip(BASE, meta_oof.coef_, meta_infold.coef_):
    print(f"    {n:<18} out-of-fold {w_o:>7.3f}   in-fold {w_i:>7.3f}")

BASE MODELS ALONE   (test R2)
----------------------------------------
  Ridge               0.7259
  RandomForest        0.8742
  GradientBoosting    0.9301
  KNN                 0.7605
  best single      : GradientBoosting at 0.9301

STACKING   (test R2)
--------------------------------------------------------
  simple average of the 4        0.8696
  stacking, out-of-fold meta     0.9331
  stacking, in-fold meta (LEAK)  0.8781

  what the meta-learner learned (weight per base model)
    Ridge              out-of-fold  -0.040   in-fold  -0.107
    RandomForest       out-of-fold  -0.095   in-fold   1.123
    GradientBoosting   out-of-fold   1.091   in-fold   0.035
    KNN                out-of-fold   0.094   in-fold  -0.027

Three things worth separating.

Simple averaging lost. 0.8696 against 0.9301 for just using the Gradient Booster alone. Averaging a strong model with three weaker ones drags it down — "ensemble everything" is not free, and this is the case people quietly skip when they report ensemble wins.

Stacking won, modestly. 0.9331 against 0.9301. Thirty ten-thousandths. That is a real gain and a small one, which is the honest shape of most stacking results — it is a way to squeeze the last points out, not a step change.

And then the weights, which are the actual story. Read the two columns side by side. Trained on out-of-fold predictions, the meta-learner put 1.091 on the Gradient Booster and gave the others roughly nothing — it correctly identified the best model and used the rest as small corrections. Trained on in-fold predictions, it put 1.123 on the Random Forest and 0.035 on the Gradient Booster.

It picked the wrong model, and it picked it confidently.

The mechanism is worth stating plainly: a Random Forest predicting rows it was trained on is close to perfect, because that is what bagged fully-grown trees do — they memorise. So in the leaked matrix, the Random Forest column looks like an oracle. The meta-learner is not fooled by noise; it is reading an honest record of the wrong thing. That is the archivist's error, in a coefficient.

From Scratch, and Two Robustness Checks

Thirty lines, then two questions that decide whether stacking is worth the operational cost. Smaller sample here — 2,000 rows — so these numbers are not directly comparable to the ones above.

import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold
from sklearn.linear_model import RidgeCV, LinearRegression
from sklearn.ensemble import (RandomForestRegressor, GradientBoostingRegressor,
                              StackingRegressor)
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score
from sklearn.base import clone

X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

def bases():
    return [("ridge", RidgeCV()),
            ("rf",    RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)),
            ("gb",    GradientBoostingRegressor(n_estimators=100, max_depth=3, random_state=0)),
            ("knn",   KNeighborsRegressor(n_neighbors=10))]

def oof_matrix(base, X, y, k=5, seed=0):
    """Each base fits on k-1 folds and predicts the fold it never saw."""
    cvf = KFold(k, shuffle=True, random_state=seed)
    oof = np.zeros((len(X), len(base)))
    for j, (_, est) in enumerate(base):
        for tr, va in cvf.split(X):
            oof[va, j] = clone(est).fit(X[tr], y[tr]).predict(X[va])
    return oof

B    = bases()
OOF  = oof_matrix(B, Xtr, ytr)
FULL = [clone(e).fit(Xtr, ytr) for _, e in B]
TEST = np.column_stack([m.predict(Xte) for m in FULL])

mine = RidgeCV().fit(OOF, ytr)
skl  = StackingRegressor(bases(), final_estimator=RidgeCV(), cv=5, n_jobs=-1).fit(Xtr, ytr)
print("FROM SCRATCH vs SCIKIT-LEARN   (test R2)")
print("-" * 48)
print(f"  mine     {r2_score(yte, mine.predict(TEST)):.4f}")
print(f"  sklearn  {r2_score(yte, skl.predict(Xte)):.4f}")

print()
print("WHICH META-LEARNER?   (same OOF matrix, test R2)")
print("-" * 48)
for name, meta in [("RidgeCV", RidgeCV()),
                   ("LinearRegression", LinearRegression()),
                   ("RandomForest d3", RandomForestRegressor(n_estimators=100, max_depth=3, random_state=0, n_jobs=-1)),
                   ("GradientBoosting", GradientBoostingRegressor(n_estimators=100, max_depth=3, random_state=0))]:
    m = clone(meta).fit(OOF, ytr)
    print(f"  {name:<20}{r2_score(yte, m.predict(TEST)):>8.4f}")

print()
print("DOES A USELESS BASE MODEL HURT?   (test R2)")
print("-" * 48)
rng = np.random.RandomState(1)
const_tr = np.full(len(Xtr), ytr.mean()); const_te = np.full(len(Xte), ytr.mean())
noise_tr = rng.normal(ytr.mean(), ytr.std(), len(Xtr))
noise_te = rng.normal(ytr.mean(), ytr.std(), len(Xte))
for label, otr, ote in [("4 real models", None, None),
                        ("+ constant predictor", const_tr, const_te),
                        ("+ pure noise predictor", noise_tr, noise_te)]:
    O = OOF if otr is None else np.column_stack([OOF, otr])
    T = TEST if ote is None else np.column_stack([TEST, ote])
    print(f"  {label:<24}{r2_score(yte, RidgeCV().fit(O, ytr).predict(T)):>8.4f}")

FROM SCRATCH vs SCIKIT-LEARN   (test R2)
------------------------------------------------
  mine     0.9231
  sklearn  0.9232

WHICH META-LEARNER?   (same OOF matrix, test R2)
------------------------------------------------
  RidgeCV               0.9231
  LinearRegression      0.9231
  RandomForest d3       0.9071
  GradientBoosting      0.9209

DOES A USELESS BASE MODEL HURT?   (test R2)
------------------------------------------------
  4 real models             0.9231
  + constant predictor      0.9231
  + pure noise predictor    0.9229

The reimplementation agrees — 0.9231 against scikit-learn's 0.9232. There is nothing in StackingRegressor beyond the fold discipline.

Keep the meta-learner boring. Ridge and plain linear regression tie at 0.9231. A Random Forest meta-learner drops to 0.9071 and a Gradient Booster to 0.9209. The meta-learner sees four highly correlated columns and a few thousand rows — there is no complex function to find there, only weights, and reaching for capacity costs you.

Garbage base models are nearly free. Adding a constant predictor changed nothing at all (0.9231), and adding a pure noise column cost 0.0002. A linear meta-learner assigns near-zero weight to a column that carries no signal, which is the mirror image of the leak result: stacking is robust to a useless input and catastrophically vulnerable to a dishonest one.

That asymmetry is the practical lesson. Do not agonise over which models to include. Agonise over how the meta-features were made.

When Does Stacking Help Most?

	Stacking helps	Stacking wastes your time
Base models	genuinely different families	four tuned variants of one model
Errors	decorrelated across models	all wrong on the same rows
Gain available	last 1–3 points matter	you need a step change
Data	enough for clean k-fold	small n, folds too thin
Ops budget	can serve N+1 models	latency or memory constrained

The operational cost is the part that decides most real projects. Stacking means training, versioning, monitoring and serving every base model plus the meta-learner. For 0.0030 of R², as measured above, that is often the wrong trade — and it is worth saying so, because the competition write-ups where stacking shines are optimising a leaderboard, not a maintenance burden.

Quick Reference Card

STACKING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  Train a small model to combine your other models,
  on predictions they made for rows they never saw.

THE ONE RULE:
  meta-features must be OUT-OF-FOLD. Everything
  else is a detail.

MEASURED HERE:
  best single model        0.9301
  simple average           0.8696   (worse!)
  stacking, out-of-fold    0.9331
  stacking, in-fold leak   0.8781

  leaked weights put 1.123 on the WRONG model
  and 0.035 on the right one.

META-LEARNER:
  ✓ Ridge / linear         0.9231
  ✗ RandomForest           0.9071
  keep it simple; it only needs weights.

WHEN TO USE:
  ✓ different model families, decorrelated errors
  ✓ the last 1-3 points are worth N+1 deployments
WHEN NOT EFFECTIVE:
  ✗ base models that agree with each other
  ✗ small n (folds too thin to be honest)
  ✗ tight latency budget

SKLEARN:
  from sklearn.ensemble import StackingRegressor
  StackingRegressor(base, final_estimator=RidgeCV(), cv=5)

Key Takeaways

The out-of-fold rule is the whole method — 0.9331 with it, 0.8781 without, from otherwise identical code.
The leak changes which model you ship — in-fold meta-features put 1.123 on the Random Forest and 0.035 on the Gradient Booster that was actually better.
Because bagged trees memorise — a Random Forest predicting its own training rows looks like an oracle, so it dominates a leaked meta-feature matrix by construction.
Simple averaging can lose to one good model — 0.8696 against 0.9301. Ensembling is not automatically an improvement.
Stacking's honest gain is small — 0.9331 against 0.9301, about 0.0030 of R² for an extra model in production.
Keep the meta-learner linear — Ridge 0.9231, Random Forest 0.9071. It has four inputs and needs weights, not capacity.
Useless base models cost almost nothing — a constant predictor changed nothing and pure noise cost 0.0002. Curate the folds, not the roster.
Thirty lines reproduce the library — 0.9231 against 0.9232. StackingRegressor is fold discipline and nothing else.

The One-Sentence Summary

Stacking is the editor who never reported a story — a small model that learns each of your other models' biases and corrects for them — and it works precisely as well as your fold discipline allows, because the same code that gains you 0.0030 of R² on out-of-fold predictions loses you 0.0550 on in-fold ones and hands the highest weight to whichever model is best at memorising the rows it was trained on.

What's Next?

Now that you understand stacking, you're ready for:

Blending and voting — tomorrow. The two ways a committee can agree, and why one of them cheats.
Grid search vs random search — the tourist with a map versus the one who wanders.
Bayesian optimization — the sommelier who learns your palate in six glasses.
Calibration — when your model's confidence and its accuracy stop agreeing.

Follow me for the next article in the Ensembles Beyond Bagging series!

Let's Connect!

If the newsroom made stacking click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a stack you couldn't explain? I have, and the honest reckoning was that it beat the best base model by less than the variance between seeds, while tripling what I had to keep alive in production. 📰

What I find genuinely interesting about stacking is that the meta-learner is doing something no base model can do: it is modelling the *models, not the data. And the moment you frame it that way, the leak stops being a technicality and becomes obvious — you cannot learn what a model is like by watching it answer questions it has already been given the answers to. That is true of Random Forests, and it is true of interviews, exams, and every benchmark that quietly ended up in a training set.*

Send this to whoever on your team is about to stack five models before checking whether the fold split is honest.

The Boosting Series in One Page: Ten Posts, and the Six Numbers That Changed How I Tune

Sachin Kr. Rajput — Tue, 18 Aug 2026 10:04:52 +0000

The One-Line Summary: Ten posts on boosting, every number in them produced by code that actually ran — and the six results worth keeping are that the learning rate and the tree count are one parameter, that unlimited depth is worse than stumps, that boosting's optimum was 19 trees and not 3,000, that coarser histogram binning won, that naive target encoding turned a signal-free column into a test loss of 2.7975, and that XGBoost and LightGBM are a tie nobody wants to admit.

The Arc

The Boosting series is finished. It started with a tutor who only reviews the questions you got wrong and ended with a tuning sheet, and the useful thing about writing it in order is that each post kept contradicting something I believed when I started the previous one.

Here is the whole arc, with the one result from each that survived.

AdaBoost: The Tutor Who Only Reviews the Questions You Got Wrong

Where boosting begins: reweight the examples you got wrong, fit the next learner to those, repeat. The parable is a tutor who stops re-teaching what you already know. The mechanism is exponential loss, and it is the reason AdaBoost is so sensitive to a mislabelled row — a permanently wrong answer gets permanently more attention.

Gradient Boosting: The Sculptor Who Fixes the Statue by Carving Only Its Mistakes

The generalisation: stop reweighting examples and start fitting the residual directly, which turns boosting into gradient descent in function space. The from-scratch implementation matched scikit-learn to 0.970898 against 0.970890 — five decimal places from thirty lines of NumPy, which is the clearest evidence I know that the library is not doing anything mysterious.

learning_rate and n_estimators Are One Parameter, Not Two

Sweep the learning rate, early-stop for the tree count each time, and the product barely moves: 4.50, 4.40, 4.41 across a tenfold range of learning rates. Halve the rate and you need twice the trees. This is the single result that has saved me the most compute, because it removes an entire axis from every grid search.

The Boosting Family Decision Tree

Which algorithm, on one page. Worth revisiting now that the series is done, because two of its recommendations got contradicted by later measurements.

XGBoost: The Accountant Who Read the Tax Code So Carefully He Found Money Nobody Else Saw

Second-order gradients, pruning written into the objective, and missing values as a learned direction. The result that landed hardest: second order reached 0.3322 in 50 rounds while first order sat at 0.3436 after 200 and never caught up. From scratch it agreed with the library at 0.931456 against 0.931628.

LightGBM: The Librarian Who Sorted a Million Books by Ignoring Most of Them

Histogram binning and leaf-wise growth. The finding I did not expect: max_bin=15 beat max_bin=1023, 0.1598 against 0.1619. More resolution in the split search bought variance, not accuracy. This is the post that made me stop trusting defaults that sound generous.

CatBoost: The Interpreter Who Refused to Peek at Tomorrow's Newspaper

Target leakage, measured. On a column containing literally no signal, naive target encoding produced a test loss of 2.7975; out-of-fold encoding 0.5823; deleting the column entirely 0.5464. Dropping a useless column beat every clever encoding of it, which is the most uncomfortable result in the series.

Bagging vs Boosting: The Orchestra and the Relay Team

The settled comparison. Removing one tree from a hundred costs bagging 0.0014 and costs boosting 0.2361 at the front of the chain — the crowd is interchangeable, the argument is not. Also: boosting's optimum was 19 trees at 36.1841, and running to 3,000 degraded it to 48.4704. And unlimited depth took boosting to 7.1105, worse than stumps.

I Gave XGBoost and LightGBM the Same 30 Seconds

0.98845 against 0.98825. Two ten-thousandths of an AUC point. Giving LightGBM its speed advantage back as extra search time bought nothing. The libraries differ on training and inference speed, not accuracy, and almost every comparison you will read online reports a gap smaller than its own noise.

The Boosting Tuning Cheat Sheet

All of the above, compressed into a tuning order you can follow without re-reading anything.

The Six That Changed How I Work

learning_rate × n_estimators is one parameter. 4.50, 4.40, 4.41.
max_depth has a low ceiling. Unlimited depth reached 7.1105 — worse than stumps.
Boosting needs early stopping. Optimum 19 trees, not 3,000; 36.1841 against 48.4704.
Coarser can beat finer. 0.1598 against 0.1619 for max_bin 15 versus 1023.
Fix the encoder before the model. 2.7975 naive, 0.5823 out-of-fold, 0.5464 for deleting the column.
Library choice is a tie. 0.98845 against 0.98825 — pick on speed, not accuracy.

What Confused People

Nothing, and I am going to be straight about why rather than invent a reader question.

This series drew zero substantive comments. Two spam replies and one thread I had already answered, across ten posts. That is not a mystery — it is a reach problem I measured rather than guessed at, and the diagnosis was that #datascience had a median of zero reactions across the twenty best posts of its week while #ai had fifty-three, and that I had been spending a quarter of every post's tag budget on a dead slot. That changed from the Bagging vs Boosting post onward.

I mention it because a recap that manufactures "great question from a reader!" is doing the same thing as a benchmark with no error bars. If the questions had been there I would quote them. They were not, so here is the honest version: the series was written into a very quiet room, and the fix was distributional rather than editorial.

The One-Sentence Summary

Ten posts of boosting reduce to one habit — treat the learning rate and the tree count as a single parameter, keep the trees shallow, stop early, and spend the time you save auditing whether your labels and your encoders deserve the trust you are placing in them, because every result in this series that surprised me came from the data and not from the algorithm.

What's Next?

The Boosting arc is closed. Next up is a new series on the ensembles that sit outside the bagging/boosting split:

Stacking — what to do when you have several good models and no idea which to trust.
Blending and voting — the two ways a committee can agree, and why one of them cheats.
Grid search vs random search — the tourist with a map versus the one who wanders.
Bayesian optimization — the sommelier who learns your palate in six glasses.

Follow me for the Ensembles Beyond Bagging series!

Let's Connect!

If any single number in this series changed something you do, drop a heart!

Questions? Ask in the comments — I read and respond to every one, and after the paragraph above you can see I would genuinely like the chance.

Which of the six surprised you most? Mine was the max_bin one, because I had spent two years assuming finer binning was a free accuracy knob you traded memory for. 📉

Writing a series in publication order rather than drafting it all up front turned out to be the useful constraint. Four of the ten posts contradicted something I had asserted in an earlier one, and I only noticed because the code had to run again each time. A series drafted in one sitting would have been internally consistent and wrong in at least four places.

If you're starting a boosting project this week, read the cheat sheet and skip the rest.

The Boosting Tuning Cheat Sheet: Which Knob to Turn First, and When to Stop Turning

Sachin Kr. Rajput — Tue, 18 Aug 2026 09:57:09 +0000

The One-Line Summary: Every number on this page was measured in an earlier post in this series rather than copied from a blog — the tuning order that falls out of them is learning_rate × n_estimators first as one parameter, then max_depth around 3 to 6, then stop, because unlimited depth took boosting from 2.0196 to 7.1105 test MSE and running to 3,000 trees when the optimum was 19 made it a third worse.

The Order

Nobody tunes eleven parameters. Here is the order the measurements justify.

BOOSTING TUNING ORDER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. learning_rate x n_estimators   ← ONE parameter
     pick lr, then early-stop for the tree count.
     Their product is roughly conserved.

2. max_depth  (or num_leaves)     ← 3 to 6
     the single biggest lever after step 1.

3. subsample / colsample          ← 0.6 to 1.0
     mild regulariser, cheap to sweep.

4. everything else                ← usually noise
     stop here unless you have a validation
     curve telling you otherwise.

1. learning_rate and n_estimators Are One Parameter

Measured on classification, sweeping the learning rate and early-stopping for the tree count each time. The product lr × trees barely moves:

lr      peak trees   lr x trees
  0.3           15         4.50
  0.1           44         4.40
  0.03         147         4.41

Three learning rates spanning a factor of ten, and the product lands at 4.50, 4.40, 4.41. Halve the learning rate and you need roughly twice the trees for the same model. So do not grid-search both — fix one, early-stop the other.

2. max_depth Has a Ceiling, and It Is Low

The mirror image of bagging. Test MSE at 100 estimators, same data:

max_depth      bagging     boosting
        1      19.5944       5.0660
        5       5.6241       2.0196
     None       3.6156       7.1105

Bagging improves all the way to unlimited depth. Boosting is U-shaped and unlimited depth is worse than stumps — 7.1105 against 5.0660. A fully grown tree eats the whole residual on the first pass and leaves nothing for the rest of the chain, noise included.

Buy capacity with more trees, not deeper ones.

3. More Trees Is Not Free

Bagging and boosting answer this differently, and it is the most common way people waste a weekend:

n_estimators     bagging     boosting
          10     39.5639      37.1438
         100     36.5054      43.4739
        3000     35.5388      48.4704

Bagging is monotone — 3,000 trees was its best. Boosting's optimum was 19 trees at 36.1841, and by 3,000 it had degraded to 48.4704. A third worse than its own best, from doing more of the thing that was working.

n_estimators is a smoothness knob for bagging and a capacity knob for boosting. Only one of them needs early stopping.

4. The Library-Specific Knobs

LightGBM       measured effect
  max_bin      15 beat 1023  (0.1598 vs 0.1619)
  growth       leaf-wise 0.1753 vs depth-wise 0.1784
  num_leaves   the dangerous default; couple it
               to max_depth or it silently overfits

CatBoost       measured effect
  ordered TS   naive target encoding: test 2.7975
               out-of-fold:            test 0.5823
               drop the column:        test 0.5464
               → on a signal-free column, deleting
                 it beat every encoding of it

The max_bin result is worth pausing on: the coarser binning won. More resolution in the split search is not more accuracy, it is more variance.

5. Which Library — and Why It Barely Matters

Same data, same tuning budget, 15 trials each:

              test AUC     tune    refit   predict
XGBoost        0.98845    10.8s    1.27s     17ms
LightGBM       0.98825     4.9s    0.38s     38ms

Two ten-thousandths of an AUC point. Give LightGBM its speed advantage back as extra search time and it fitted 59 candidates to XGBoost's 37 — final scores 0.98977 and 0.98990. The extra search bought nothing.

So pick on the loop you are constrained by. LightGBM wins development (3.3× faster to fit). XGBoost wins production (2.2× faster to score).

6. When Tuning Is the Wrong Job

BEFORE YOU TUNE, CHECK THESE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✗ Labels are noisy
    30% flipped labels cost bagging 0.1133 accuracy
    and boosting 0.1477. Boosting pays more for bad
    labels than any hyperparameter can win back.

✗ A high-cardinality column is target-encoded
    by hand. Fix the encoder, not the depth.

✗ Your CV and your holdout disagree by more than
    the gap you are chasing. Tuning against a
    leaky score optimises the leak.

✓ Clean labels, honest validation, and you still
    want the last two points. Now tune.

Key Takeaways

learning_rate × n_estimators is one parameter — measured at 4.50, 4.40 and 4.41 across a tenfold range of learning rates.
max_depth 3–6, never unlimited — unlimited took boosting from 2.0196 to 7.1105, worse than stumps at 5.0660.
Boosting needs early stopping and bagging does not — optimum at 19 trees versus 3,000, and 48.4704 versus 36.1841 for ignoring it.
Coarser can win — LightGBM's max_bin=15 beat max_bin=1023 at 0.1598 against 0.1619.
Library choice is a tie on accuracy — 0.98845 against 0.98825. Choose on training versus inference speed instead.
Fix the data before the knobs — noisy labels cost boosting 0.1477 of accuracy, which no amount of tuning recovers.

The One-Sentence Summary

Tune the learning rate and tree count as a single parameter with early stopping, keep max_depth between 3 and 6 because unlimited depth is worse than stumps, sweep subsample if you are bored, and spend everything you save on checking whether your labels and your validation split deserve the trust you are placing in them.

What's Next?

The series recap — the whole Boosting arc in one page, with the numbers that survived.
Stacking — what to do when you have several good models and no idea which to trust.
Blending and voting — the two ways a committee can agree, and why one of them cheats.
Grid search vs random search — and why the tourist with a map loses.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If this saved you a grid search, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

What's the hyperparameter you've stopped tuning entirely? Mine is n_estimators on random forests — I set it to 500 and never think about it again, because the measurement says it cannot hurt me. 🔧

The reason cheat sheets are usually wrong is that they are assembled from other cheat sheets. Every figure on this page came from code that ran in an earlier post in this series, which is a low bar that most tuning advice does not clear — and the two most useful lines here, that coarser binning won and that deleting a column beat encoding it, are both things I would not have believed from a table someone else wrote.

Bookmark this before your next grid search, and check the depth row first.

I Gave XGBoost and LightGBM the Same 30 Seconds — The Speed Winner Didn't Win Anything

Sachin Kr. Rajput — Fri, 14 Aug 2026 11:23:14 +0000

The One-Line Summary: On the same data with the same tuning budget, XGBoost and LightGBM finished 0.00020 AUC apart — and when I handed LightGBM its 2.23× speed advantage back as extra search time, it fitted 1.59× as many candidates and gained −0.00013 AUC for the trouble; the real difference between them is not accuracy, it is that LightGBM trains 3.3× faster while XGBoost predicts 2.2× faster.

The Question Everyone Asks Wrong

"Which is more accurate, XGBoost or LightGBM?"

It is the wrong question, and ten minutes of measurement shows why. Here is a fair fight: 20,000 rows, 30 features, identical folds, identical parameter distributions, identical random seed, 15 search trials each.

import time, warnings, numpy as np
warnings.filterwarnings("ignore")
import xgboost as xgb, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import (train_test_split, RandomizedSearchCV,
                                     StratifiedKFold)
from sklearn.metrics import roc_auc_score
from scipy.stats import loguniform, randint

X, y = make_classification(n_samples=20000, n_features=30, n_informative=12,
                           n_redundant=6, flip_y=0.02, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
                                      random_state=42, stratify=y)
cv = StratifiedKFold(3, shuffle=True, random_state=0)
N_TRIALS = 15

space = {"n_estimators": randint(100, 400),
         "learning_rate": loguniform(0.01, 0.3),
         "max_depth": randint(3, 10),
         "subsample": [0.6, 0.8, 1.0],
         "colsample_bytree": [0.6, 0.8, 1.0]}

def run(name, est):
    t0 = time.perf_counter()
    s = RandomizedSearchCV(est, space, n_iter=N_TRIALS, cv=cv, scoring="roc_auc",
                           random_state=0, n_jobs=-1).fit(Xtr, ytr)
    tune = time.perf_counter() - t0
    t1 = time.perf_counter(); best = s.best_estimator_; best.fit(Xtr, ytr)
    fit = time.perf_counter() - t1
    t2 = time.perf_counter(); p = best.predict_proba(Xte)[:, 1]
    pred = time.perf_counter() - t2
    auc = roc_auc_score(yte, p)
    print(f"{name:<11}{s.best_score_:>10.5f}{auc:>10.5f}"
          f"{tune:>10.1f}s{fit:>9.2f}s{pred*1000:>9.0f}ms")
    return auc, tune

print(f"SAME DATA, SAME BUDGET ({N_TRIALS} trials x 3-fold), "
      f"15k train rows x 30 features")
print("-" * 62)
print(f"{'':<11}{'best CV':>10}{'test AUC':>10}{'tune':>11}"
      f"{'refit':>9}{'predict':>11}")
a1, t1 = run("XGBoost",  xgb.XGBClassifier(tree_method="hist", n_jobs=1,
             eval_metric="logloss", random_state=0, verbosity=0))
a2, t2 = run("LightGBM", lgb.LGBMClassifier(n_jobs=1, random_state=0, verbose=-1))
print()
print(f"test AUC gap      : {abs(a1 - a2):.5f}")
print(f"tuning time ratio : {t1 / t2:.2f}x  (XGBoost / LightGBM)")

SAME DATA, SAME BUDGET (15 trials x 3-fold), 15k train rows x 30 features
--------------------------------------------------------------
              best CV  test AUC       tune    refit    predict
XGBoost       0.98305   0.98845      10.8s     1.27s       17ms
LightGBM      0.98313   0.98825       4.9s     0.38s       38ms

test AUC gap      : 0.00020
tuning time ratio : 2.23x  (XGBoost / LightGBM)

Two ten-thousandths of an AUC point apart. On a test set of 5,000 rows that is a handful of examples changing places. Anyone claiming one of these libraries is more accurate than the other on ordinary tabular data is reporting seed variance.

The Honest Version of "Same Budget"

The obvious objection: LightGBM tuned in less than half the time, so a trial-for-trial comparison undersells it. Give both the same wall clock instead and let LightGBM spend its speed on more candidates.

Thirty seconds each, identical parameter draws, whoever gets through more of them wins more lottery tickets:

EQUAL WALL CLOCK: 30s of tuning each, identical parameter draws
--------------------------------------------
             trials    best CV   test AUC
XGBoost          37    0.98420    0.98990
LightGBM         59    0.98442    0.98977

LightGBM fitted 1.59x as many candidates in the same 30s
and it bought -0.00013 test AUC

LightGBM got 59 shots to XGBoost's 37. It found a better cross-validation score — 0.98442 against 0.98420 — and then lost on the test set by 0.00013.

That is what a saturated search looks like. Both libraries had already found the plateau within the first twenty trials, and everything after that was sampling noise dressed up as progress. The extra 22 candidates bought a better CV number and no better model, which is a compact demonstration of why you should not select on the third decimal place of a cross-validation score.

Where They Actually Differ

Speed is real. It is just not where people look for it.

THE ONLY NUMBERS THAT DIFFERED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                XGBoost    LightGBM     winner
  test AUC      0.98845     0.98825     nobody
  tune 15x3       10.8s        4.9s     LGB 2.2x
  refit          1.27s       0.38s      LGB 3.3x
  predict 5k      17ms        38ms      XGB 2.2x

  → LightGBM wins the loop you run in development
  → XGBoost wins the loop you run in production

LightGBM trains 3.3× faster. XGBoost scores 2.2× faster. Those pull in opposite directions, and which one matters is a property of your system, not of the library.

If you retrain nightly and serve from a cache, training time is your bottleneck and LightGBM saves you real hours. If you retrain monthly and serve 3,000 predictions per second behind a latency SLO, that 38ms against 17ms is the number on the incident report, and the training difference is invisible.

Pick on the axis your system is actually constrained by. On accuracy, flip a coin.

Key Takeaways

Accuracy is a tie — 0.00020 AUC apart under equal trials, 0.00013 the other way under equal wall clock. Both differences are smaller than seed noise.
Speed did not convert — 1.59× more candidates in the same 30 seconds produced a better CV score and a marginally worse test score. Search saturates, and past that point more trials buy overfitting to the validation folds.
The training/inference split is the real decision — LightGBM 3.3× faster to fit, XGBoost 2.2× faster to predict. Choose against your bottleneck, not against a benchmark someone else ran.

The One-Sentence Summary

XGBoost and LightGBM are close enough on accuracy that any measured gap is noise — LightGBM tunes 2.23× faster but converts none of it into a better model, so the honest tiebreaker is whether your system is bottlenecked on the training loop, where LightGBM is 3.3× faster, or the serving loop, where XGBoost is 2.2× faster.

What's Next?

The hyperparameter tuning cheat sheet — tomorrow. Everything that matters, on one page.
Week 3 recap — what clicked and what confused people.
Stacking — what to do when you have several good models and no idea which to trust.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If this saved you an afternoon of benchmarking, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever switched libraries for a gain you never actually measured? I have, and the honest post-mortem was that I moved a project to LightGBM for accuracy, got none, and kept it anyway because the training loop went from twelve minutes to four — which was the right outcome for entirely the wrong reason. ⚖️

The interesting thing about a dead heat is how much energy goes into denying it. There is a whole genre of blog post picking a winner between these two, and almost none of them report a gap larger than the noise in their own cross-validation. It is more comfortable to believe the tool is the variable, because tools can be swapped in an afternoon and the alternative — that your features and your labels are the ceiling — takes a quarter to fix.

Send this to whoever on your team is about to spend a sprint migrating between the two.

Bagging vs Boosting: The Orchestra That Plays Together vs The Relay Team That Runs One at a Time

Sachin Kr. Rajput — Thu, 13 Aug 2026 14:50:25 +0000

The One-Line Summary: Bagging and boosting both turn mediocre trees into strong models, from exactly opposite directions — the orchestra plays every part at once and averages away the wobble, the relay hands each runner only the distance the previous one failed to cover. Measured on identical data they finish within 0.08 of the same total error, and yet removing a single tree costs the orchestra 0.0014 and costs the relay 0.2361.

The Parable of the Amberton Festival

Every autumn Amberton holds two contests on the same field: the orchestra plays at noon, and the relay runs at four. For thirty years the town treated them as unrelated amusements. Then a bandmaster who also kept the running club's stopwatch noticed that both were answers to the same question — how do you get a hundred imperfect people to produce one correct result? — and that the two answers were exact opposites.

The Orchestra: Forty Players, One Note

Not one of the forty violinists is perfectly in tune. Ask any of them to play alone and you will hear it. Some run a little sharp, some a little flat, and no two are wrong in the same way on the same evening.

Played together, something strange happens. The sharpness of one player sits on top of the flatness of another and the two quietly cancel. Nobody is corrected. Nobody listens to anybody else. They simply all play at once, and the hall hears a note steadier than any single instrument producing it.

THE ORCHESTRA — FORTY PLAYERS, ONE NOTE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The note on the page:  440

  violin  1  plays  443   (+3 sharp)
  violin  2  plays  437   (-3 flat)
  violin  3  plays  441   (+1 sharp)
  violin  4  plays  436   (-4 flat)
     ...
  violin 40  plays  442   (+2 sharp)

  what the hall hears:  440.1

No player was corrected. No player heard another.
Each was wrong; the wrongness pointed in random
directions and cancelled.

Add a 41st violin and the hall gets steadier still.
It can never get worse.

This is called VARIANCE REDUCTION.

The important detail is the one that sounds like a limitation: no violinist is trying to fix anyone. The cancellation is a free side effect of independence.

The Relay: Four Runners, One Race

The relay is built on the opposite principle. Nobody runs the whole race.

The first runner covers most of the distance, badly. The second does not start at the beginning — he starts exactly where the first one stopped, and covers most of what is left. The third takes the remainder of that. Each runner's job is defined entirely by the failure of the runner before.

"Why should the second man repeat the first man's ground? Give him only the distance the first man failed to cover."

THE RELAY — EACH RUNNER TAKES WHAT IS LEFT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Distance to the line:  1000 m

  runner 1  covers 700   →  300 left
  runner 2  covers 210   →   90 left
  runner 3  covers  63   →   27 left
  runner 4  covers  19   →    8 left

Runner 2 never attempts the first 700 m. That part
is finished. He is handed 300 m of pure failure and
nothing else.

Reorder them and it is not the same race — runner
4's leg exists only because runners 1-3 already ran.

This is called FITTING THE RESIDUAL.

Four runners got within 8 m. Forty violinists would still be averaging.

The Two Years It Went Wrong

Amberton remembers two festivals badly, and each one broke a different method.

In one year the orchestra tuned from a piano that had been left by a window all winter and had gone flat. All forty players tuned to it faithfully. All forty played the same wrong note, and the average of forty identical mistakes is that mistake. A forty-first violin would not have helped.

In another year a groundsman set the first marker forty metres short. Runner one stopped where the marker said. Runner two measured his leg from that point, honestly and carefully, and so did three and four. Every correction after the first was faithful to a false premise, and the team ran, confidently and in perfect form, to the wrong place.

TWO WAYS TO FAIL
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE FLAT PIANO
  Every violin tuned from the same flat piano.
  All forty played 436. The average was 436.
  A 41st violin also played 436.
  ✗ Averaging cannot remove an error that everyone
    shares. This is CORRELATION, and it is the
    floor the orchestra can never get under.

THE SHORT MARKER
  The first marker was set 40 m short.
  Runner 2 measured his leg from it. So did 3 and 4.
  Each correction was faithful to a false start.
  ✗ The relay never questions the baton. It
    inherits it. This is NOISE SENSITIVITY.

Why Each One Works

THE MATHEMATICS OF AVERAGING VS CORRECTING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
AVERAGING N members whose errors have variance σ²
and average pairwise correlation ρ:

    variance  =  ρσ²  +  (1-ρ)σ²/N
                  ↑            ↑
             shared error   private error
             never shrinks  → 0 as N grows

  REQUIREMENTS
  1. Each member is fit independently.
  2. Members must DISAGREE (low ρ), or step 1
     bought you nothing.
  3. Each member should be LOW BIAS — averaging
     removes variance and never touches bias.

CORRECTING: member m is fit to what m-1 left over.

    F_m(x) = F_{m-1}(x) + lr · h_m(x)

  REQUIREMENTS
  1. Order is fixed. Members are not swappable.
  2. Each member should be WEAK — a strong one
     eats the whole residual and the chain stops.
  3. There is no ρ floor to worry about. There is
     also no protection from a wrong target.

Two methods, two entirely different failure modes, and — this is the part almost nobody checks — very often the same final accuracy.

What Are Bagging and Boosting?

Bagging (bootstrap aggregating) fits $N$ copies of the same learner on $N$ different bootstrap resamples of the training set, then averages their predictions. The members never see each other. The fit is embarrassingly parallel.

Boosting fits members in sequence. Each one is trained on the errors the current ensemble still makes, and is added to the ensemble with a small weight. The fit is inherently serial.

The Math

For bagging, the variance of the average of $N$ predictors with individual variance $σ^{2}$ and mean pairwise correlation $ρ$ is:

Var (f ˉ) = ρ σ^{2} + \frac{1 - ρ}{N} σ^{2}

The second term vanishes as $N \to \infty$ . The first does not. Everything interesting about bagging is a fight to lower $ρ$ , which is exactly what random forests do by sampling features at each split.

For boosting, the model is a stagewise additive expansion. At stage $m$ we fit a new learner to the negative gradient of the loss and take a damped step:

F_{m} (x) = F_{m - 1} (x) + ν \cdot h_{m} (x), h_{m} \approx - \frac{\partial L}{\partial F _{m - 1}}

There is no $ρ$ in that expression, because independence was never the point.

Step by Step

Bagging

Draw a bootstrap sample of size $n$ with replacement.
Fit a deep, unpruned learner on it.
Repeat $N$ times, in any order, on any number of machines.
Average (regression) or vote (classification).

Boosting

Start with a constant prediction, usually the mean.
Compute what the current ensemble still gets wrong.
Fit a small, deliberately weak learner to that.
Add it, scaled by the learning rate. Return to step 2.

The Decomposition That Settles It

The textbook line is "bagging reduces variance, boosting reduces bias." That is a claim about numbers, so let us get the numbers. Sixty independent training sets, the same test set, and a decomposition of the error of four models into the part that is systematic and the part that is jitter.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor

N_FITS, N_TRAIN, N_TEST = 60, 300, 500
X_test, y_true = make_friedman1(n_samples=N_TEST, noise=0.0, random_state=7)

def decompose(make_model, label):
    preds = np.zeros((N_FITS, N_TEST))
    for i in range(N_FITS):
        Xtr, ytr = make_friedman1(n_samples=N_TRAIN, noise=1.0,
                                  random_state=1000 + i)
        preds[i] = make_model(i).fit(Xtr, ytr).predict(X_test)
    mean_pred = preds.mean(axis=0)
    bias2 = float(np.mean((mean_pred - y_true) ** 2))
    var = float(np.mean(preds.var(axis=0)))
    mse = float(np.mean((preds - y_true) ** 2))
    print(f"{label:<26} bias2 {bias2:7.4f}   "
          f"variance {var:7.4f}   total {mse:7.4f}")

print("BIAS-VARIANCE DECOMPOSITION")
print("60 independent training sets, Friedman #1")
print("-" * 58)
decompose(lambda i: DecisionTreeRegressor(max_depth=None, random_state=42),
          "single deep tree")
decompose(lambda i: BaggingRegressor(DecisionTreeRegressor(max_depth=None),
          n_estimators=100, random_state=42, n_jobs=-1), "bagging, 100 deep trees")
decompose(lambda i: GradientBoostingRegressor(n_estimators=100, max_depth=1,
          learning_rate=0.1, random_state=42), "boosting, 100 stumps")
decompose(lambda i: DecisionTreeRegressor(max_depth=1, random_state=42),
          "single stump")

BIAS-VARIANCE DECOMPOSITION
60 independent training sets, Friedman #1
----------------------------------------------------------
single deep tree           bias2  3.2389   variance  8.1118   total 11.3507
bagging, 100 deep trees    bias2  3.6628   variance  1.0594   total  4.7222
boosting, 100 stumps       bias2  4.1052   variance  0.5443   total  4.6495
single stump               bias2 16.1720   variance  1.9015   total 18.0735

Read each ensemble against its own base learner, which is the comparison people usually skip.

Bagging took a deep tree and cut its variance from 8.1118 to 1.0594 — a factor of 7.7 — while bias went slightly up, from 3.2389 to 3.6628. That is the orchestra exactly: it did nothing whatsoever about the note being wrong, and everything about the wobble.

Boosting took a stump and cut its bias from 16.1720 to 4.1052, a factor of 3.9. That is the relay: a runner who could only ever cover a fraction of the track, turned into a team that reaches the line.

And here is the part the textbook line gets wrong. Boosting also cut variance, from 1.9015 to 0.5443, by a factor of 3.5. Shrinkage and weak learners are a regularizer, not just a bias fix. "Boosting reduces bias" is directionally right and quantitatively incomplete.

Then look at the two totals: 4.7222 and 4.6495. Two philosophies, opposite mechanisms, opposite failure modes, and they arrive within 0.08 of each other.

Are the Members Interchangeable?

The parable claims the orchestra's players are swappable and the relay's runners are not. That is testable. Build both from scratch, confirm they match the library, then remove exactly one tree from a hundred and see what it costs.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

class MyBagging:
    """Resample, fit deep trees independently, average."""
    def __init__(self, n=100, seed=42):
        self.n, self.seed = n, seed
    def fit(self, X, y):
        rng = np.random.RandomState(self.seed)
        self.trees_ = []
        for _ in range(self.n):
            idx = rng.randint(0, len(X), len(X))            # bootstrap
            t = DecisionTreeRegressor(max_depth=None, random_state=0)
            self.trees_.append(t.fit(X[idx], y[idx]))
        return self
    def predict(self, X):
        return np.mean([t.predict(X) for t in self.trees_], axis=0)

class MyBoosting:
    """Fit each tree to what is LEFT OVER."""
    def __init__(self, n=100, lr=0.1, depth=3, seed=42):
        self.n, self.lr, self.depth, self.seed = n, lr, depth, seed
    def fit(self, X, y):
        self.f0_ = y.mean()
        resid = y - self.f0_
        self.trees_ = []
        for _ in range(self.n):
            t = DecisionTreeRegressor(max_depth=self.depth,
                                      random_state=self.seed)
            t.fit(X, resid)
            resid = resid - self.lr * t.predict(X)          # still wrong
            self.trees_.append(t)
        return self
    def predict(self, X):
        return self.f0_ + self.lr * np.sum(
            [t.predict(X) for t in self.trees_], axis=0)

X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

bag = MyBagging(100).fit(Xtr, ytr)
boo = MyBoosting(100, 0.1, 3).fit(Xtr, ytr)
skl_b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=100,
                         random_state=42, n_jobs=-1).fit(Xtr, ytr)
skl_g = GradientBoostingRegressor(n_estimators=100, max_depth=3,
                                  learning_rate=0.1, random_state=42).fit(Xtr, ytr)

print("FROM SCRATCH vs SCIKIT-LEARN   (test MSE)")
print("-" * 52)
print(f"  bagging   mine {mean_squared_error(yte, bag.predict(Xte)):8.4f}"
      f"   sklearn {mean_squared_error(yte, skl_b.predict(Xte)):8.4f}")
print(f"  boosting  mine {mean_squared_error(yte, boo.predict(Xte)):8.4f}"
      f"   sklearn {mean_squared_error(yte, skl_g.predict(Xte)):8.4f}")

def bag_pred(trees): return np.mean([t.predict(Xte) for t in trees], axis=0)
def boo_pred(trees): return boo.f0_ + boo.lr * np.sum(
                            [t.predict(Xte) for t in trees], axis=0)
full_b = mean_squared_error(yte, bag_pred(bag.trees_))
full_g = mean_squared_error(yte, boo_pred(boo.trees_))

print("\nREMOVING ONE MEMBER FROM 100   (test MSE)")
print("-" * 52)
print(f"{'':<22}{'bagging':>12}{'boosting':>12}")
print(f"{'all 100':<22}{full_b:>12.4f}{full_g:>12.4f}")
for name, keep in [("drop the 1st tree",   list(range(1, 100))),
                   ("drop the 50th tree",  [i for i in range(100) if i != 49]),
                   ("drop the 100th tree", list(range(0, 99)))]:
    print(f"{name:<22}"
          f"{mean_squared_error(yte, bag_pred([bag.trees_[i] for i in keep])):>12.4f}"
          f"{mean_squared_error(yte, boo_pred([boo.trees_[i] for i in keep])):>12.4f}")

FROM SCRATCH vs SCIKIT-LEARN   (test MSE)
----------------------------------------------------
  bagging   mine   3.6307   sklearn   3.5487
  boosting  mine   2.0786   sklearn   2.0770

REMOVING ONE MEMBER FROM 100   (test MSE)
----------------------------------------------------
                           bagging    boosting
all 100                     3.6307      2.0786
drop the 1st tree           3.6293      2.3147
drop the 50th tree          3.6295      2.1008
drop the 100th tree         3.6310      2.0806

Thirty lines each, and both land on the library: boosting at 2.0786 against scikit-learn's 2.0770.

Now the removals. For bagging, dropping the first, the fiftieth or the hundredth tree moves test error by at most 0.0014, and dropping the last tree (3.6310) was very slightly worse than dropping the first (3.6293) — which is noise, and that is the point. The members are genuinely interchangeable. There is no first violin.

For boosting, dropping the first tree costs 0.2361 and dropping the hundredth costs 0.0020. The same operation is 118 times more expensive at the front of the chain than at the back. Tree one is running the first 700 metres. Tree one hundred is running the last eight.

One honest correction while I was building this. My first version shuffled the members and announced that boosting was "destroyed" — and printed a number identical to the unshuffled one. Of course it did: prediction is a sum, and addition commutes. Order matters while fitting, not while predicting. The removal test is the right experiment; the shuffle test was me describing a result I expected instead of reading the one I got.

The Mirror Image: How Deep Should the Trees Be?

If bagging fixes variance and boosting fixes bias, then they should want opposite base learners. Bagging should want deep trees — low bias, high variance, exactly the raw material averaging can improve. Boosting should want shallow ones — leave it something to correct.

import numpy as np
from sklearn.datasets import make_friedman1
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor, GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

print("BASE-LEARNER DEPTH")
print("test MSE, 100 estimators each, mean of 3 seeds")
print("-" * 50)
print(f"{'max_depth':>10}  {'bagging':>10}  {'boosting':>10}")
for d in [1, 2, 3, 5, 8, None]:
    bag, boo = [], []
    for s in range(3):
        b = BaggingRegressor(DecisionTreeRegressor(max_depth=d),
                             n_estimators=100, random_state=s, n_jobs=-1).fit(Xtr, ytr)
        g = GradientBoostingRegressor(n_estimators=100, max_depth=d,
                                      learning_rate=0.1, random_state=s).fit(Xtr, ytr)
        bag.append(mean_squared_error(yte, b.predict(Xte)))
        boo.append(mean_squared_error(yte, g.predict(Xte)))
    print(f"{str(d):>10}  {np.mean(bag):>10.4f}  {np.mean(boo):>10.4f}")

BASE-LEARNER DEPTH
test MSE, 100 estimators each, mean of 3 seeds
--------------------------------------------------
 max_depth     bagging    boosting
         1     19.5944      5.0660
         2     13.2544      2.6711
         3      8.9423      2.0769
         5      5.6241      2.0196
         8      3.9657      3.0942
      None      3.6156      7.1105

Two curves running in opposite directions.

Bagging improves monotonically with depth and is best at unlimited: 19.5944 with stumps, 3.6156 with fully grown trees. Give the orchestra players who can actually play; averaging will handle the wobble.

Boosting is U-shaped. Best at depth 5 (2.0196), and unlimited depth is a disaster at 7.1105 — worse than boosting stumps. A fully grown tree consumes the entire residual on the first pass, leaves nothing for anyone else, and what it consumed included the noise.

The practical rule falls straight out: if you are bagging, stop pruning. If you are boosting, keep the trees small and buy your capacity with more of them instead.

Where Boosting Breaks

Two failure modes, both predicted by the parable, both measurable.

import numpy as np
from sklearn.datasets import make_classification, make_friedman1
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
                              BaggingRegressor, GradientBoostingRegressor)
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X, y = make_classification(n_samples=2000, n_features=20, n_informative=8,
                           n_redundant=2, flip_y=0.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)

print("LABEL-NOISE SENSITIVITY   (test accuracy, mean of 5 seeds)")
print("-" * 64)
print(f"{'flipped':>8}  {'bagging (RF)':>14}  {'boosting (GB)':>15}  {'gap':>7}")
rows = []
for frac in [0.00, 0.05, 0.10, 0.20, 0.30]:
    rf_s, gb_s = [], []
    for seed in range(5):
        rng = np.random.RandomState(seed)
        y_noisy = ytr.copy()
        n_flip = int(frac * len(y_noisy))
        if n_flip:
            idx = rng.choice(len(y_noisy), n_flip, replace=False)
            y_noisy[idx] = 1 - y_noisy[idx]
        rf_s.append(RandomForestClassifier(n_estimators=200, random_state=seed,
                    n_jobs=-1).fit(Xtr, y_noisy).score(Xte, yte))
        gb_s.append(GradientBoostingClassifier(n_estimators=200, max_depth=3,
                    random_state=seed).fit(Xtr, y_noisy).score(Xte, yte))
    rf_m, gb_m = np.mean(rf_s), np.mean(gb_s)
    rows.append((rf_m, gb_m))
    print(f"{frac:>7.0%}  {rf_m:>14.4f}  {gb_m:>15.4f}  {rf_m-gb_m:>+7.4f}")
print(f"\nclean -> 30% noise:  bagging loses {rows[0][0]-rows[-1][0]:.4f},"
      f"  boosting loses {rows[0][1]-rows[-1][1]:.4f}")

# Does adding more members ever hurt?
X, y = make_friedman1(n_samples=400, noise=5.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.4, random_state=42)
print("\nADDING MORE ESTIMATORS   (small noisy sample)")
print("-" * 56)
print(f"{'n_estimators':>13}  {'bagging':>10}  {'boosting':>10}")
for n in [10, 100, 1000, 3000]:
    b = BaggingRegressor(DecisionTreeRegressor(), n_estimators=n,
                         random_state=42, n_jobs=-1).fit(Xtr, ytr)
    g = GradientBoostingRegressor(n_estimators=n, max_depth=3,
                                  learning_rate=0.1, random_state=42).fit(Xtr, ytr)
    print(f"{n:>13}  {mean_squared_error(yte, b.predict(Xte)):>10.4f}"
          f"  {mean_squared_error(yte, g.predict(Xte)):>10.4f}")
g = GradientBoostingRegressor(n_estimators=3000, max_depth=3, learning_rate=0.1,
                              random_state=42).fit(Xtr, ytr)
errs = [mean_squared_error(yte, p) for p in g.staged_predict(Xte)]
print(f"\nboosting best at n={int(np.argmin(errs))+1} (MSE {min(errs):.4f}),"
      f" ends at n=3000 (MSE {errs[-1]:.4f})")

LABEL-NOISE SENSITIVITY   (test accuracy, mean of 5 seeds)
----------------------------------------------------------------
 flipped    bagging (RF)    boosting (GB)      gap
     0%          0.8560           0.8317  +0.0243
     5%          0.8480           0.8113  +0.0367
    10%          0.8350           0.8067  +0.0283
    20%          0.8013           0.7467  +0.0547
    30%          0.7427           0.6840  +0.0587

clean -> 30% noise:  bagging loses 0.1133,  boosting loses 0.1477

ADDING MORE ESTIMATORS   (small noisy sample)
--------------------------------------------------------
 n_estimators     bagging    boosting
           10     39.5639     37.1438
          100     36.5054     43.4739
         1000     35.7662     48.4598
         3000     35.5388     48.4704

boosting best at n=19 (MSE 36.1841), ends at n=3000 (MSE 48.4704)

Noise. Going from clean labels to 30% flipped, bagging gives up 0.1133 of accuracy and boosting gives up 0.1477. The gap between them widens from +0.0243 to +0.0587. A mislabelled row is a residual that never goes away, so boosting keeps sending runners at it. Bagging simply averages over it. This is the short marker, and it is why boosting is the wrong first choice on hand-labelled data.

More members. Bagging on a small noisy sample goes 39.5639 → 35.5388 as it grows from 10 to 3,000 trees. Monotone, and it never gets worse, exactly as the variance formula promises. Boosting starts at 37.1438 and climbs to 48.4704. Its best model was 19 trees; running it to 3,000 made it a third worse than its own optimum.

That asymmetry is the single most useful thing on this page. n_estimators is a capacity knob for boosting and a smoothness knob for bagging. One of them needs early stopping and a validation set. The other you can set to "more" and walk away.

When Does Each Help Most?

	Bagging	Boosting
Attacks	Variance	Bias (and some variance)
Base learner	Deep, unpruned	Shallow, depth 3–6
Fitting	Parallel	Serial
More members	Never hurts	Overfits eventually
Noisy labels	Degrades gracefully	Chases the noise
Needs early stopping	No	Yes
Tuning effort	Almost none	Real
Typical accuracy, clean tabular	Good	Usually better

Reach for bagging when labels are noisy or hand-made, when you want a defensible baseline in one line, when you have cores and no time, or when nobody will be around to babysit a validation curve.

Reach for boosting when labels are trustworthy, when the last two points of accuracy are worth real tuning effort, and when you can hold out a validation set to stop on.

Bagging and Boosting Hyperparameters

Parameter	Bagging	Boosting
`n_estimators`	100–1000, more is safe	tune with early stopping
`max_depth`	`None`	3–6
`learning_rate`	n/a	0.01–0.1, lower needs more trees
`max_features`	`sqrt` lowers ρ	mild regularizer
`subsample`	bootstrap is the mechanism	0.5–0.8 adds randomness
What to tune first	almost nothing	`learning_rate` × `n_estimators`

Quick Reference Card

BAGGING vs BOOSTING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT THEY ARE:
  Bagging  — fit members independently, average them.
  Boosting — fit each member to the leftover error.

THE FORMULAS:
  bagging   Var = ρσ² + (1-ρ)σ²/N
  boosting  F_m = F_{m-1} + lr · h_m

MEASURED HERE (Friedman #1):
  deep tree     bias 3.2389  var 8.1118
  bagged        bias 3.6628  var 1.0594   (var ÷7.7)
  stump         bias 16.1720 var 1.9015
  boosted       bias 4.1052  var 0.5443   (bias ÷3.9)

DROP ONE TREE OF 100:
  bagging   costs 0.0014      boosting costs 0.2361

WHEN TO USE:
  ✓ bagging  — noisy labels, no time, no babysitting
  ✓ boosting — clean labels, tuning budget, val set
WHEN NOT EFFECTIVE:
  ✗ bagging  — members all wrong the same way (high ρ)
  ✗ boosting — mislabelled rows, deep trees, no early stop

SKLEARN:
  from sklearn.ensemble import RandomForestRegressor
  from sklearn.ensemble import GradientBoostingRegressor

Key Takeaways

They attack different halves of the error — bagging cut a deep tree's variance from 8.1118 to 1.0594 while nudging bias slightly up; boosting cut a stump's bias from 16.1720 to 4.1052.
"Boosting reduces bias" is incomplete — boosting also cut variance by a factor of 3.5, from 1.9015 to 0.5443. Shrinkage and weak learners regularize.
They arrive at nearly the same place — 4.7222 against 4.6495 total error, from opposite directions.
Bagging's members are interchangeable, boosting's are a chain — removing one tree of a hundred costs bagging at most 0.0014, and costs boosting 0.2361 at the front versus 0.0020 at the back.
They want opposite base learners — bagging is best with unlimited depth (3.6156), boosting is best at depth 5 (2.0196) and is worse than useless at unlimited depth (7.1105).
n_estimators means two different things — bagging improved all the way to 3,000 trees; boosting's optimum was 19 and by 3,000 it was a third worse.
Boosting pays more for bad labels — 30% flipped labels cost bagging 0.1133 accuracy and boosting 0.1477, with the gap widening monotonically.
Compare each ensemble to its own base learner — against each other the two totals look interchangeable, and the entire mechanism is invisible.

The One-Sentence Summary

Bagging and boosting are the orchestra and the relay — forty players sounding one note by cancelling each other's independent mistakes, against four runners each handed only the distance the last one failed to cover — and the measurements say the orchestra divides variance by 7.7 while barely touching bias, the relay divides bias by 3.9, they finish within 0.08 of the same total error, and the only way to tell them apart from the outside is to remove one member and watch it cost the orchestra 0.0014 and the relay 0.2361.

What's Next?

Now that you can tell the two families apart, you're ready for:

XGBoost vs LightGBM in 10 minutes — tomorrow. Same data, same tuning budget, honest numbers.
The hyperparameter tuning cheat sheet — everything that matters, on one page.
Stacking — what to do when you have several good models and no idea which to trust.
Blending and voting — the two ways a committee can agree, and why one of them cheats.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the orchestra and the relay made ensembles click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Which one do you reach for first? I default to bagging for the baseline and only spend the tuning budget on boosting once I've seen how clean the labels are — because a random forest has never once surprised me in production, and gradient boosting has. ⚖️

What strikes me about these two, after measuring them properly, is that the family argument was never about accuracy — they land in the same place. It is about what you are willing to be responsible for. Bagging asks for independence and gives you a model that cannot be made worse by adding to it. Boosting asks for attention and gives you a model that will follow you anywhere, including off a cliff, because every tree after the first is an act of faith in the ones before it. One is a crowd, the other is an argument. The crowd is safer; the argument is usually right; and the engineering judgment is knowing which of those you need this week.

Forward this to the teammate who set n_estimators=5000 on a gradient booster and went to lunch.

CatBoost: The Interpreter Who Refused to Peek at Tomorrow's Newspaper

Sachin Kr. Rajput — Wed, 12 Aug 2026 07:19:38 +0000

The One-Line Summary: Replacing a category with the average target for that category is the most common feature-engineering trick in tabular machine learning and it quietly hands the model the answer — measured here, it turned a column containing literally no signal into a training loss of 0.2466 and a test loss of 2.7975, five times worse than deleting the column; CatBoost's ordered target statistics fix the leak by letting each row see only rows that came before it.

The Parable of the Court at Vashti

The court employed interpreters, because testimony arrived in six languages and the judges read one. An interpreter's job was to render each witness's words faithfully, and for two centuries the court considered this a solved problem.

Then someone noticed that the interpreters were extraordinarily good at old cases and merely average at new ones.

The Old Way: Read the Whole File First

Every interpreter was handed the complete case file before beginning. This was considered basic professionalism — how could you translate testimony about a boundary dispute without knowing it was a boundary dispute?

The file included the verdict.

WHY THE OLD INTERPRETERS LOOKED SO GOOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Witness says a word that could mean
"he took it" or "he was given it".

Interpreter has read the file. The file says
GUILTY.

She writes "he took it."

She is not lying. She is not even conscious of
choosing. The ambiguity resolved itself the
moment she knew how it ended.

On closed cases her renderings are uncanny.
On open cases she is ordinary, and nobody can
work out why.

The interpreters were not corrupt. They were contaminated, which is worse, because contamination leaves no one to blame and nothing obvious to fix.

The Investigation

A clerk named Ilaya was asked to audit the interpreters and did something nobody had tried: she took a hundred old cases, stripped the verdicts, and had them re-translated by the same people.

THE SAME INTERPRETERS, VERDICTS HIDDEN
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
with the file (as always)   near-perfect renderings
without the verdict          ordinary renderings
                             indistinguishable from
                             a first-year apprentice

The skill everyone had been admiring for two
centuries was not skill. It was the verdict,
travelling backwards into the testimony.

The court's instinct was to ban case files entirely. Ilaya argued against it: context genuinely helps, and an interpreter working blind is worse than one working informed. The problem was never the file. It was which parts of the file.

"Let her read everything that was written before the words she is translating. Nothing that was written after. The rule is not ignorance — it is chronology."

ILAYA'S RULE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Translating testimony from the 3rd of the month?

  You may read: everything filed on the 1st, 2nd.
  You may not read: the 4th onward. Ever.
                    Especially the verdict.

Each document is translated using only the
court's knowledge AS IT STOOD at that moment.

Slower. Occasionally the interpreter has almost
no context and must simply admit it.
Those renderings are honest, and they hold up.

The first interpreters trained this way looked worse on the court's historical records. They were the first ones who did not get worse when the case was live.

What Is Target Encoding, and Why Does It Leak?

Hard pivot.

You have a categorical column with many levels — city, merchant, user ID, product SKU. One-hot encoding it produces thousands of columns. The popular alternative is target encoding: replace each level with the mean of the target for that level.

x^i = \frac{\sum j : c _{j} = c _{i} y _{j}}{∣ j : c _{j} = c _{i} ∣}

Look at the index set. It includes $j = i$ . Row $i$ 's own label is in the numerator of row $i$ 's feature. If a level appears once, its encoding is its label, exactly. If it appears three times, the encoding is two-thirds someone else's answer and one-third your own.

That is the verdict travelling backwards into the testimony.

The Fix, Formally

CatBoost computes ordered target statistics. Fix a random permutation $σ$ of the rows. For row $i$ , use only rows that precede it:

x^i = \frac{\sum j : σ ( j ) < σ ( i ) , c _{j} = c _{i} y _{j} + a \cdot p}{∣ j : σ ( j ) < σ ( i ) , c _{j} = c _{i} ∣ + a}

where $p$ is a prior (the global mean) and $a$ its weight. Row $i$ is excluded by construction — it hasn't happened yet. The prior does the work when history is thin.

The same principle is applied to the gradients during boosting, which is where the name "ordered boosting" comes from, but the target-statistic version is where the damage usually is.

Measuring the Leak

The cleanest possible test: build a categorical column that carries no information whatsoever, and see whether target encoding can conjure signal out of it. 6,000 rows, 2,000 levels — roughly three rows per level, which is exactly the regime where people reach for target encoding.

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier

rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n)              # pure noise: 2,000 levels
x1 = rng.normal(size=n)
y = (0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)   # y ignores cat

X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()


def score(A, B):
    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
    return (log_loss(ytr, m.predict_proba(A)[:, 1]),
            log_loss(yte, m.predict_proba(B)[:, 1]))


full_mean = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(full_mean).fillna(gm).values

A = Xtr.copy(); A["cat"] = pd.Series(ctr).map(full_mean).fillna(gm).values
tr, te = score(A, B)
print(f"  naive target encoding   train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

A2 = Xtr.copy(); enc = np.full(len(Xtr), np.nan); ys = pd.Series(ytr)
for itr, iva in KFold(5, shuffle=True, random_state=0).split(Xtr):
    mm = ys.iloc[itr].groupby(ctr[itr]).mean()
    enc[iva] = pd.Series(ctr[iva]).map(mm).fillna(ys.iloc[itr].mean()).values
A2["cat"] = enc
tr, te = score(A2, B)
print(f"  out-of-fold encoding    train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

tr, te = score(Xtr[["x1"]], Xte[["x1"]])
print(f"  drop the column         train {tr:.4f}   test {te:.4f}   gap {te-tr:+.4f}")

  naive target encoding   train 0.2466   test 2.7975   gap +2.5508
  out-of-fold encoding    train 0.4436   test 0.5823   gap +0.1388
  drop the column         train 0.4964   test 0.5464   gap +0.0500

Sit with the first row.

The column is random integers. It has no relationship to the target — I generated y from x1 alone. And naive target encoding produced a training loss of 0.2466, better than the honest model can achieve, and a test loss of 2.7975 — roughly five times worse than simply deleting the column.

A model that has memorised the training set and learned nothing looks, during training, exactly like a breakthrough.

Ordered Target Statistics from Scratch

Ilaya's rule is about thirty lines. Walk the rows in a random permutation, encode each one from the running totals so far, then add it to the totals.

import numpy as np


def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
    """Each row is encoded using only rows earlier in the permutation."""
    rng = np.random.default_rng(seed)
    perm = rng.permutation(len(cat))
    enc = np.empty(len(cat))
    csum, ccnt = {}, {}
    for pos in perm:
        c = cat[pos]
        s, k = csum.get(c, 0.0), ccnt.get(c, 0)
        enc[pos] = (s + a * prior) / (k + a)   # history only — self excluded
        csum[c] = s + y[pos]                   # now add self, for later rows
        ccnt[c] = k + 1
    return enc

The two lines after the assignment are the whole idea. The row contributes to everyone after it and never to itself.

Run all four encodings on the same data, both when the column is noise and when it genuinely carries signal, three seeds each:

=== NO signal in the column (test logloss, 3 seeds) ===
  seed   naive    oof      ordered   drop
     0   2.7975   0.5823   0.5812   0.5464
     1   3.2548   0.5780   0.6050   0.5545
     2   2.9780   0.5988   0.6014   0.5702
  MEAN   3.0101   0.5863   0.5959   0.5570

=== REAL signal in the column (test logloss, 3 seeds) ===
  seed   naive    oof      ordered   drop
     0   2.2485   0.5707   0.6737   0.6398
     1   2.5480   0.5564   0.6239   0.6139
     2   2.0538   0.5601   0.6670   0.6518
  MEAN   2.2834   0.5624   0.6549   0.6352

Four readings, and the third one surprised me.

Naive encoding is a catastrophe either way — 3.0101 and 2.2834 against roughly 0.56 for every honest method. It does not matter whether the column has signal. The leak dominates.

When the column carries real signal, encoding it pays. Out-of-fold scored 0.5624 against 0.6352 for dropping the column. That is the case for target encoding, and it is a real one.

My ordered implementation lost to plain out-of-fold — 0.6549 against 0.5624. I expected the opposite, and I am reporting it because it is what the code printed.

Dropping a signal-free column beats every encoding of it. 0.5570 against 0.5863 out-of-fold. No scheme recovers information that was never there; encoding only adds noise with a plausible face.

Why the Ordered Version Underperformed

A result you did not expect is worth ten minutes before you publish it. The obvious suspect is variance: early rows in the permutation have almost no history, so their encodings are mostly prior. CatBoost's actual answer to this is to average several permutations — so I tried that.

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
from sklearn.ensemble import HistGradientBoostingClassifier


def ordered_target_stats(cat, y, prior, a=1.0, seed=0):
    rng = np.random.default_rng(seed)
    perm = rng.permutation(len(cat))
    enc = np.empty(len(cat)); csum, ccnt = {}, {}
    for pos in perm:
        c = cat[pos]; s, k = csum.get(c, 0.0), ccnt.get(c, 0)
        enc[pos] = (s + a * prior) / (k + a)
        csum[c] = s + y[pos]; ccnt[c] = k + 1
    return enc


rng = np.random.default_rng(0)
n, K = 6000, 2000
cat = rng.integers(0, K, n)
eff = rng.normal(0, 1.5, K)                     # this time the column MATTERS
x1 = rng.normal(size=n)
y = (eff[cat] + 0.9 * x1 + rng.normal(0, 1.0, n) > 0).astype(int)
X = pd.DataFrame({"cat": cat, "x1": x1})
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0)
ctr, cte = Xtr["cat"].values, Xte["cat"].values
gm = ytr.mean()
fm = pd.Series(ytr).groupby(ctr).mean()
B = Xte.copy(); B["cat"] = pd.Series(cte).map(fm).fillna(gm).values

print("  permutations   test logloss")
for P in (1, 2, 4, 8, 16):
    encs = np.mean([ordered_target_stats(ctr, ytr, gm, seed=s)
                    for s in range(P)], axis=0)
    A = Xtr.copy(); A["cat"] = encs
    m = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(A, ytr)
    print(f"  {P:>12}   {log_loss(yte, m.predict_proba(B)[:, 1]):.4f}")

# and the diagnostic that explains the result
perm = np.random.default_rng(0).permutation(len(ctr))
seen, hist = {}, []
for pos in perm:
    c = ctr[pos]; hist.append(seen.get(c, 0)); seen[c] = seen.get(c, 0) + 1
hist = np.array(hist)
print(f"\n  rows with ZERO prior observations of their own level: "
      f"{(hist==0).sum():,} of {len(hist):,} ({(hist==0).mean()*100:.1f}%)")
print(f"  first 10% of the permutation: "
      f"{(hist[:len(hist)//10]==0).mean()*100:.1f}% have no history")
print(f"  last  10% of the permutation: "
      f"{(hist[-len(hist)//10:]==0).mean()*100:.1f}% have no history")

  permutations   test logloss
             1   0.6737
             2   0.7020
             4   0.7110
             8   0.6810
            16   0.6854

Averaging did not help. So the problem is not variance, and I went looking for what it actually is:

  rows with ZERO prior observations of their own level: 1,747 of 4,200 (41.6%)
  first 10% of the permutation: 92.1% have no history
  last  10% of the permutation: 13.1% have no history

41.6% of training rows never see a single earlier example of their own category. At 2,000 levels across 4,200 training rows, most levels appear two or three times, so for a large fraction of rows there is simply nothing legitimate to learn from. Their encoding is the global prior — a constant. Averaging permutations cannot manufacture history that does not exist.

Out-of-fold encoding wins here because it is less strict: a row in fold 3 gets the mean over folds 1, 2, 4 and 5 — around 80% of the data rather than "whatever happened to precede me". Both exclude the row's own label, which is the part that matters. Ordered TS excludes more, and at this cardinality it excludes too much.

One caveat I want to be explicit about: this is my thirty-line reimplementation of the principle, not CatBoost. Real CatBoost recomputes target statistics inside the boosting loop with a fresh permutation per tree, builds combinations of categorical features, and applies the same ordering idea to gradients. Do not read this table as "CatBoost loses to out-of-fold encoding" — read it as "the ordering principle, applied naively as a preprocessing step, runs out of history at extreme cardinality."

What To Actually Do

TARGET ENCODING: A DECISION PROCEDURE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Does the column plausibly carry signal?
  |
  +- No  -> DROP IT. Measured best in every
  |         no-signal run. Encoding a useless
  |         column only invents noise.
  |
  +- Yes / unsure
       |
       +- Fewer than ~50 levels?
       |    -> one-hot. Boring, safe, competitive.
       |
       +- Many levels, several rows each?
       |    -> out-of-fold target encoding.
       |       Best measured result here.
       |
       +- Many levels, 1-3 rows each?
            -> the column is nearly an ID.
               Expect little. Consider dropping,
               or hashing to fewer buckets.

NEVER: mean over all training rows including self.
       train 0.2466 / test 2.7975 on a column
       that contained nothing.

The tell that you have leaked is always the same shape: training loss far better than anything the problem should allow, and a validation gap that widens as you add capacity. A gap of +2.55 log loss is not overfitting. Overfitting is +0.05. That is contamination.

When Does CatBoost Help Most?

	CatBoost	LightGBM	XGBoost
Categorical handling	ordered target statistics, built in	native partition or one-hot	you encode it yourself
Leak protection	by construction	none — your job	none — your job
Tree shape	oblivious (same split per level)	leaf-wise	depth-wise
Defaults on small data	conservative	dangerous (`num_leaves=31`)	safe
Best fit	many high-cardinality categoricals	large numeric tabular	general tabular

Reach for CatBoost when your table is mostly categorical with real cardinality — retail SKUs, merchant IDs, geography — and you would otherwise be hand-rolling encodings and getting them subtly wrong. Reach elsewhere when your features are numeric, where its advantage largely disappears.

Quick Reference Card

CATBOOST / TARGET ENCODING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
THE LEAK:
  x_i = mean(y_j for j where c_j == c_i)
  includes j == i.  Row i's own label is inside
  row i's feature. One row per level => the
  encoding IS the label.

THE FIX (ordered target statistics):
  fix a permutation; for row i use only rows
  BEFORE it:
    x_i = (sum of earlier y + a*prior)
          / (count of earlier + a)
  self excluded by construction.

MEASURED (column with NO signal at all):
  naive        train 0.2466   test 2.7975
  out-of-fold  train 0.4436   test 0.5823
  drop it      train 0.4964   test 0.5464   <- best

MEASURED (column WITH signal):
  naive 2.2834 | oof 0.5624 | ordered 0.6549
  | drop 0.6352      oof wins

WHY ORDERED CAN LOSE:
  41.6% of rows had NO earlier row of their own
  level (2,000 levels / 4,200 rows). Averaging
  16 permutations did not help — the history
  isn't thin, it's absent.

THE TELL YOU LEAKED:
  train loss impossibly good + gap that GROWS
  with capacity. Overfitting is +0.05.
  Leakage was +2.55.

Key Takeaways

Naive target encoding leaks the label into the feature. The row's own $y$ sits in the numerator of its own $x$ .
It manufactures signal from nothing. On a column of random integers: train 0.2466, test 2.7975 — five times worse than deleting the column.
Ordered target statistics fix it by chronology, not ignorance. Each row sees only rows earlier in a random permutation; it is excluded from its own encoding by construction.
Out-of-fold encoding was the best real method measured — 0.5624 against 0.6352 for dropping a signal-bearing column, and it beat my ordered implementation.
Dropping a signal-free column beats every encoding of it (0.5570 vs 0.5863). Encoding cannot recover information that was never present.
Ordering can exclude too much. 41.6% of rows had no earlier example of their own level, and averaging 16 permutations did not help — the fix for absent history is not more permutations.
Leakage and overfitting look different. Overfitting widened the gap by 0.05. Leakage widened it by 2.55. If your training metric looks impossible, it is.
Check your own reimplementation before blaming the library. My thirty-line ordered TS is not CatBoost, and I said so rather than reporting a benchmark I had not actually run.

The One-Sentence Summary

Target encoding is the most widely used trick in tabular machine learning and its textbook form quietly puts each row's own answer inside that row's features — CatBoost's contribution is Ilaya's rule, that a row may be described using everything the data knew before it and nothing the data learned after, which is not a clever optimisation but a refusal to look at tomorrow's newspaper while reporting today's news.

What's Next?

Bagging vs Boosting — tomorrow. The settled comparison, measured rather than asserted.
XGBoost vs LightGBM in 10 minutes — same data, same budget, a decision procedure.
Hyperparameter tuning — what to search when four parameters interact.
Stacking — what to do when you have several good models and no idea which to trust.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If the court at Vashti made target leakage click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a target-encoded feature without checking the fold boundaries? I have, and the model was the best I had ever built right up until the day it met data it hadn't already been told the answer to. ⚖️

The thing that unsettles me about leakage is that it never feels like cheating while you are doing it. Every step is defensible — use the data you have, summarise the category, feed the model context. The failure is assembled entirely from reasonable decisions, which is why code review does not catch it and a suspiciously good training curve does. Learn to distrust your best results specifically because they are your best results.