The One-Line Summary: Stacking trains a small model to combine your other models, and the entire thing turns on one detail — the combiner must be trained on predictions each base model made for rows it never saw, because when I trained it on in-fold predictions instead it handed a weight of 1.123 to the Random Forest and 0.035 to the Gradient Booster that was genuinely the better model, and the ensemble fell from 0.9331 to 0.8781.
The Parable of the Kestrel Street Newsroom
Four reporters cover the same city for the same paper, and after twenty years everyone in the building knows their habits: one over-reports crime, one is too kind to the council, one files fast and loose, one is slow and exact. For most of the paper's history the editor's job was to decide which of the four to print. Then a new editor arrived who did something stranger — she never reported a story in her life, and the paper got better.
The Old Way: Print the Best Reporter
The old method was a competition. Send all four out on the same story, read the four drafts, print the strongest one and spike the rest.
THE OLD WAY — PICK ONE, BIN THREE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Story: how many families left the east ward?
Ambrose says 400 (always runs high)
Beatrix says 180 (soft on the council)
Colm says 260 (fast, sloppy)
Delia says 310 (slow, careful)
Truth, three months later: 295
Printed: Delia, 310. The other three: binned.
Delia was closest, so Delia gets printed. Next
story, the paper reads four drafts and throws away
three again.
This is called MODEL SELECTION.
It works. It is also strange: the paper spends four salaries and prints one story.
The Innovation: An Editor Who Reads Habits, Not Stories
The new editor did not go out and report a fifth version. She sat with twenty years of archives and did something nobody had bothered to do — she compared each reporter's draft against what turned out to be true, story after story, until she knew the shape of each one's error.
"I do not need a better reporter. I need to know, for each of these four, which direction they are wrong in and by how much."
THE EDITOR'S LEDGER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
From the archive, over hundreds of stories:
Ambrose runs about +35% high on any count
Beatrix runs low whenever the council is
involved, accurate otherwise
Colm is unbiased but wildly noisy
Delia is nearly right, slightly low
So for the east ward story she writes:
0.15 x Ambrose + 0.05 x Beatrix
+ 0.10 x Colm + 0.75 x Delia -> 296
Truth: 295. Closer than any of the four.
She reported nothing. She corrected for habits.
This is called STACKING.
The editor is not a fifth reporter. She is a model of the other four.
The Year the Editor Read the Wrong Drafts
The method held for a decade and then broke in a way nobody saw coming, because the failure looked like success.
A new archivist, trying to be helpful, started giving the editor the reporters' drafts for stories whose outcome had already been published. Colm — fast, sloppy Colm — had a habit of skimming the paper before filing. On stories where the answer was already in print, his drafts were flawless.
The editor, reading the archive honestly, concluded that Colm was the finest reporter in the building. She started weighting him above everyone. The paper got worse for a year before anyone worked out why.
THE YEAR THE ARCHIVE LIED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
On stories whose ending was already public:
Colm's drafts nearly perfect
Delia's drafts still merely very good
Editor's conclusion: Colm is the best.
Editor's new weights: mostly Colm.
On NEW stories, where nothing was published yet,
Colm went back to being fast and sloppy — and the
paper was now built on him.
✗ The editor was never wrong about the archive.
The archive was the wrong thing to read.
This is called LEAKAGE IN THE META-FEATURES.
The fix is one rule: the editor may only judge a reporter on drafts filed before the outcome was known to anyone. Nothing else about the method changes.
Why It Works
THE MATHEMATICS OF LEARNED WEIGHTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Averaging N models weights them all equally:
f(x) = (1/N) sum_m h_m(x)
→ a strong model gets dragged toward weak ones.
Stacking learns the weights from data:
f(x) = g( h_1(x), h_2(x), ..., h_N(x) )
where g is fit on OUT-OF-FOLD predictions.
REQUIREMENTS
1. Each h_m(x) used to train g must come from a
model that did NOT see row x.
2. g should be SIMPLE — it has N inputs and all
the overfitting risk lives here.
3. Base models should DISAGREE. Four copies of
the same model give g nothing to learn.
4. g may assign negative weight. That is not a
bug; it is g cancelling a known bias.
What Is Stacking?
Stacking (stacked generalisation) trains a second-level model — the meta-learner — whose input features are the predictions of your first-level models.
The mechanism that makes it work, and the only part people get wrong, is how those input features are generated. For each training row you need a prediction from a model that never saw that row. That is exactly what k-fold cross-validation produces, so the standard recipe is: split into k folds, train each base model on k−1 of them, predict the held-out fold, and stack those held-out predictions into a matrix.
The Math
Given base learners and a meta-learner , the stacked prediction is:
The meta-learner is fit on the out-of-fold matrix , where and is base model trained with the fold containing row removed. At prediction time the base models are refit on everything and is applied unchanged.
Note what is not: it is not the base models' training predictions. Using those is the archivist's mistake, and the measurements below put a number on it.
Step by Step
- Choose k folds. Five is standard.
- For each base model, train on k−1 folds and predict the held-out fold. Repeat until every row has a prediction.
- Assemble those into an n × N matrix. This is what the meta-learner sees.
- Fit the meta-learner on that matrix against the true targets.
- Refit every base model on the full training set.
- To predict: run the refit base models, feed their outputs to the meta-learner.
Does It Actually Beat the Best Base Model?
Four base models with genuinely different failure modes, on 3,000 rows of Friedman #1. The comparison that matters is against the best single model, not against the average.
import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold, cross_val_predict
from sklearn.linear_model import RidgeCV
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score
X, y = make_friedman1(n_samples=3000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
cv = KFold(5, shuffle=True, random_state=0)
BASE = {
"Ridge": lambda: RidgeCV(),
"RandomForest": lambda: RandomForestRegressor(n_estimators=200, random_state=0, n_jobs=-1),
"GradientBoosting": lambda: GradientBoostingRegressor(n_estimators=200, max_depth=3, random_state=0),
"KNN": lambda: KNeighborsRegressor(n_neighbors=10),
}
print("BASE MODELS ALONE (test R2)")
print("-" * 40)
solo = {}
for n, f in BASE.items():
solo[n] = r2_score(yte, f().fit(Xtr, ytr).predict(Xte))
print(f" {n:<18}{solo[n]:>8.4f}")
best_name = max(solo, key=solo.get)
print(f" best single : {best_name} at {solo[best_name]:.4f}")
# out-of-fold meta features (correct) vs in-fold meta features (the leak)
oof = np.column_stack([cross_val_predict(f(), Xtr, ytr, cv=cv, n_jobs=-1) for f in BASE.values()])
infold = np.column_stack([f().fit(Xtr, ytr).predict(Xtr) for f in BASE.values()])
test_meta = np.column_stack([f().fit(Xtr, ytr).predict(Xte) for f in BASE.values()])
meta_oof = RidgeCV().fit(oof, ytr)
meta_infold = RidgeCV().fit(infold, ytr)
print()
print("STACKING (test R2)")
print("-" * 56)
print(f" simple average of the 4 {r2_score(yte, test_meta.mean(axis=1)):>8.4f}")
print(f" stacking, out-of-fold meta {r2_score(yte, meta_oof.predict(test_meta)):>8.4f}")
print(f" stacking, in-fold meta (LEAK){r2_score(yte, meta_infold.predict(test_meta)):>8.4f}")
print()
print(" what the meta-learner learned (weight per base model)")
for n, w_o, w_i in zip(BASE, meta_oof.coef_, meta_infold.coef_):
print(f" {n:<18} out-of-fold {w_o:>7.3f} in-fold {w_i:>7.3f}")
BASE MODELS ALONE (test R2)
----------------------------------------
Ridge 0.7259
RandomForest 0.8742
GradientBoosting 0.9301
KNN 0.7605
best single : GradientBoosting at 0.9301
STACKING (test R2)
--------------------------------------------------------
simple average of the 4 0.8696
stacking, out-of-fold meta 0.9331
stacking, in-fold meta (LEAK) 0.8781
what the meta-learner learned (weight per base model)
Ridge out-of-fold -0.040 in-fold -0.107
RandomForest out-of-fold -0.095 in-fold 1.123
GradientBoosting out-of-fold 1.091 in-fold 0.035
KNN out-of-fold 0.094 in-fold -0.027
Three things worth separating.
Simple averaging lost. 0.8696 against 0.9301 for just using the Gradient Booster alone. Averaging a strong model with three weaker ones drags it down — "ensemble everything" is not free, and this is the case people quietly skip when they report ensemble wins.
Stacking won, modestly. 0.9331 against 0.9301. Thirty ten-thousandths. That is a real gain and a small one, which is the honest shape of most stacking results — it is a way to squeeze the last points out, not a step change.
And then the weights, which are the actual story. Read the two columns side by side. Trained on out-of-fold predictions, the meta-learner put 1.091 on the Gradient Booster and gave the others roughly nothing — it correctly identified the best model and used the rest as small corrections. Trained on in-fold predictions, it put 1.123 on the Random Forest and 0.035 on the Gradient Booster.
It picked the wrong model, and it picked it confidently.
The mechanism is worth stating plainly: a Random Forest predicting rows it was trained on is close to perfect, because that is what bagged fully-grown trees do — they memorise. So in the leaked matrix, the Random Forest column looks like an oracle. The meta-learner is not fooled by noise; it is reading an honest record of the wrong thing. That is the archivist's error, in a coefficient.
From Scratch, and Two Robustness Checks
Thirty lines, then two questions that decide whether stacking is worth the operational cost. Smaller sample here — 2,000 rows — so these numbers are not directly comparable to the ones above.
import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_friedman1
from sklearn.model_selection import train_test_split, KFold
from sklearn.linear_model import RidgeCV, LinearRegression
from sklearn.ensemble import (RandomForestRegressor, GradientBoostingRegressor,
StackingRegressor)
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score
from sklearn.base import clone
X, y = make_friedman1(n_samples=2000, noise=1.0, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=42)
def bases():
return [("ridge", RidgeCV()),
("rf", RandomForestRegressor(n_estimators=100, random_state=0, n_jobs=-1)),
("gb", GradientBoostingRegressor(n_estimators=100, max_depth=3, random_state=0)),
("knn", KNeighborsRegressor(n_neighbors=10))]
def oof_matrix(base, X, y, k=5, seed=0):
"""Each base fits on k-1 folds and predicts the fold it never saw."""
cvf = KFold(k, shuffle=True, random_state=seed)
oof = np.zeros((len(X), len(base)))
for j, (_, est) in enumerate(base):
for tr, va in cvf.split(X):
oof[va, j] = clone(est).fit(X[tr], y[tr]).predict(X[va])
return oof
B = bases()
OOF = oof_matrix(B, Xtr, ytr)
FULL = [clone(e).fit(Xtr, ytr) for _, e in B]
TEST = np.column_stack([m.predict(Xte) for m in FULL])
mine = RidgeCV().fit(OOF, ytr)
skl = StackingRegressor(bases(), final_estimator=RidgeCV(), cv=5, n_jobs=-1).fit(Xtr, ytr)
print("FROM SCRATCH vs SCIKIT-LEARN (test R2)")
print("-" * 48)
print(f" mine {r2_score(yte, mine.predict(TEST)):.4f}")
print(f" sklearn {r2_score(yte, skl.predict(Xte)):.4f}")
print()
print("WHICH META-LEARNER? (same OOF matrix, test R2)")
print("-" * 48)
for name, meta in [("RidgeCV", RidgeCV()),
("LinearRegression", LinearRegression()),
("RandomForest d3", RandomForestRegressor(n_estimators=100, max_depth=3, random_state=0, n_jobs=-1)),
("GradientBoosting", GradientBoostingRegressor(n_estimators=100, max_depth=3, random_state=0))]:
m = clone(meta).fit(OOF, ytr)
print(f" {name:<20}{r2_score(yte, m.predict(TEST)):>8.4f}")
print()
print("DOES A USELESS BASE MODEL HURT? (test R2)")
print("-" * 48)
rng = np.random.RandomState(1)
const_tr = np.full(len(Xtr), ytr.mean()); const_te = np.full(len(Xte), ytr.mean())
noise_tr = rng.normal(ytr.mean(), ytr.std(), len(Xtr))
noise_te = rng.normal(ytr.mean(), ytr.std(), len(Xte))
for label, otr, ote in [("4 real models", None, None),
("+ constant predictor", const_tr, const_te),
("+ pure noise predictor", noise_tr, noise_te)]:
O = OOF if otr is None else np.column_stack([OOF, otr])
T = TEST if ote is None else np.column_stack([TEST, ote])
print(f" {label:<24}{r2_score(yte, RidgeCV().fit(O, ytr).predict(T)):>8.4f}")
FROM SCRATCH vs SCIKIT-LEARN (test R2)
------------------------------------------------
mine 0.9231
sklearn 0.9232
WHICH META-LEARNER? (same OOF matrix, test R2)
------------------------------------------------
RidgeCV 0.9231
LinearRegression 0.9231
RandomForest d3 0.9071
GradientBoosting 0.9209
DOES A USELESS BASE MODEL HURT? (test R2)
------------------------------------------------
4 real models 0.9231
+ constant predictor 0.9231
+ pure noise predictor 0.9229
The reimplementation agrees — 0.9231 against scikit-learn's 0.9232. There is nothing in StackingRegressor beyond the fold discipline.
Keep the meta-learner boring. Ridge and plain linear regression tie at 0.9231. A Random Forest meta-learner drops to 0.9071 and a Gradient Booster to 0.9209. The meta-learner sees four highly correlated columns and a few thousand rows — there is no complex function to find there, only weights, and reaching for capacity costs you.
Garbage base models are nearly free. Adding a constant predictor changed nothing at all (0.9231), and adding a pure noise column cost 0.0002. A linear meta-learner assigns near-zero weight to a column that carries no signal, which is the mirror image of the leak result: stacking is robust to a useless input and catastrophically vulnerable to a dishonest one.
That asymmetry is the practical lesson. Do not agonise over which models to include. Agonise over how the meta-features were made.
When Does Stacking Help Most?
| Stacking helps | Stacking wastes your time | |
|---|---|---|
| Base models | genuinely different families | four tuned variants of one model |
| Errors | decorrelated across models | all wrong on the same rows |
| Gain available | last 1–3 points matter | you need a step change |
| Data | enough for clean k-fold | small n, folds too thin |
| Ops budget | can serve N+1 models | latency or memory constrained |
The operational cost is the part that decides most real projects. Stacking means training, versioning, monitoring and serving every base model plus the meta-learner. For 0.0030 of R², as measured above, that is often the wrong trade — and it is worth saying so, because the competition write-ups where stacking shines are optimising a leaderboard, not a maintenance burden.
Quick Reference Card
STACKING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
Train a small model to combine your other models,
on predictions they made for rows they never saw.
THE ONE RULE:
meta-features must be OUT-OF-FOLD. Everything
else is a detail.
MEASURED HERE:
best single model 0.9301
simple average 0.8696 (worse!)
stacking, out-of-fold 0.9331
stacking, in-fold leak 0.8781
leaked weights put 1.123 on the WRONG model
and 0.035 on the right one.
META-LEARNER:
✓ Ridge / linear 0.9231
✗ RandomForest 0.9071
keep it simple; it only needs weights.
WHEN TO USE:
✓ different model families, decorrelated errors
✓ the last 1-3 points are worth N+1 deployments
WHEN NOT EFFECTIVE:
✗ base models that agree with each other
✗ small n (folds too thin to be honest)
✗ tight latency budget
SKLEARN:
from sklearn.ensemble import StackingRegressor
StackingRegressor(base, final_estimator=RidgeCV(), cv=5)
Key Takeaways
The out-of-fold rule is the whole method — 0.9331 with it, 0.8781 without, from otherwise identical code.
The leak changes which model you ship — in-fold meta-features put 1.123 on the Random Forest and 0.035 on the Gradient Booster that was actually better.
Because bagged trees memorise — a Random Forest predicting its own training rows looks like an oracle, so it dominates a leaked meta-feature matrix by construction.
Simple averaging can lose to one good model — 0.8696 against 0.9301. Ensembling is not automatically an improvement.
Stacking's honest gain is small — 0.9331 against 0.9301, about 0.0030 of R² for an extra model in production.
Keep the meta-learner linear — Ridge 0.9231, Random Forest 0.9071. It has four inputs and needs weights, not capacity.
Useless base models cost almost nothing — a constant predictor changed nothing and pure noise cost 0.0002. Curate the folds, not the roster.
Thirty lines reproduce the library — 0.9231 against 0.9232.
StackingRegressoris fold discipline and nothing else.
The One-Sentence Summary
Stacking is the editor who never reported a story — a small model that learns each of your other models' biases and corrects for them — and it works precisely as well as your fold discipline allows, because the same code that gains you 0.0030 of R² on out-of-fold predictions loses you 0.0550 on in-fold ones and hands the highest weight to whichever model is best at memorising the rows it was trained on.
What's Next?
Now that you understand stacking, you're ready for:
- Blending and voting — tomorrow. The two ways a committee can agree, and why one of them cheats.
- Grid search vs random search — the tourist with a map versus the one who wanders.
- Bayesian optimization — the sommelier who learns your palate in six glasses.
- Calibration — when your model's confidence and its accuracy stop agreeing.
Follow me for the next article in the Ensembles Beyond Bagging series!
Let's Connect!
If the newsroom made stacking click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
Have you ever shipped a stack you couldn't explain? I have, and the honest reckoning was that it beat the best base model by less than the variance between seeds, while tripling what I had to keep alive in production. 📰
What I find genuinely interesting about stacking is that the meta-learner is doing something no base model can do: it is modelling the *models, not the data. And the moment you frame it that way, the leak stops being a technicality and becomes obvious — you cannot learn what a model is like by watching it answer questions it has already been given the answers to. That is true of Random Forests, and it is true of interviews, exams, and every benchmark that quietly ended up in a training set.*
Send this to whoever on your team is about to stack five models before checking whether the fold split is honest.
Top comments (0)