You have three models. One is a logistic regression, one a k-NN, one a shallow decision tree. Each gets your problem partly right. The obvious move is to average their predictions — a soft vote. But averaging treats all three as equally trustworthy everywhere, which is almost never true. Stacking asks a better question: what if a fourth model learned how to combine the first three?
That fourth model is the whole idea, and it is why stacking sits on top of so many winning solutions.
The setup
Stacking (Wolpert called it "stacked generalization" back in 1992) has two levels.
Level 0 is your base models. You want them diverse — a linear model, a distance-based model, a tree, a gradient booster. The more differently they fail, the better.
Level 1 is the meta-model. Its input features are not the original data. They are the predictions of the base models. Its job is to map "what the bases said" to "the right answer." It learns, from data, how much to trust each base — and it can even learn to trust one base only when two others disagree.
A plain average vote is the special case where the meta-model is frozen into an equal-weight mean. Stacking unfreezes it.
Why diversity is the whole game
If two base models are near-identical, their predictions are redundant columns and the meta-model gains nothing. The payoff comes from uncorrelated errors. When a straight line fails on a curved boundary, a local k-NN might still be right. When k-NN gets fooled by a noisy point, an axis-aligned tree does not care. Combine learners that break in different regions and the union of "where I'm right" can cover the whole space.
A quick sanity check before you build: compute the correlation matrix of your base predictions. Low off-diagonal correlation means real diversity. High correlation means drop one and add something genuinely different.
The out-of-fold trick (do not skip this)
Watch out for the trap that quietly ruins naive stacking.
To train the meta-model you need base predictions to use as features. The tempting shortcut is to train each base on all your data, then have it predict that same data. Do not. A model scoring its own training rows looks unrealistically good — it has partly memorized them. The meta-model then learns to blindly trust bases that only look perfect, and the whole thing collapses on new data.
The fix is out-of-fold (OOF) prediction. Split the training data into k folds. For each fold, train the bases on the other k−1 folds and predict the held-out fold. Rotate through all folds. Now every training row has base predictions produced by models that never saw it — exactly the honest, generalization-level predictions the meta-model will face in production. Stack those, train the meta on them.
In the interactive demo I built, the leaky in-sample version brags about 99% accuracy on its own training data, then quietly drops to the mid-90s on a held-out test set. The OOF version reports that mid-90s number up front — before you deploy, when it still matters.
At predict time it gets simpler
Training and inference are asymmetric. During training you needed OOF predictions, so the bases were refit k times on partial data. For the deployed model you refit each base once on the full training set — you want the strongest bases at inference. To score a new point: run it through each full-data base, collect their probabilities into a little vector, feed that to the trained meta-model, done.
scikit-learn's StackingClassifier handles both jobs for you. Its cv argument is exactly the k folds; final_estimator is your meta-model.
from sklearn.ensemble import StackingClassifier
from sklearn.linear_model import LogisticRegression
stack = StackingClassifier(
estimators=[("lr", lr), ("knn", knn), ("tree", tree)],
final_estimator=LogisticRegression(), # keep the meta simple
cv=5, # out-of-fold folds
stack_method="predict_proba",
)
Keep the meta-model humble. Its input is just a few columns per row; a plain logistic regression learns a weighted blend (negative weights allowed, so a weak base becomes a corrector) and rarely overfits. Complex metas usually overfit that tiny feature set.
When not to bother
Stacking is not free. Each base is refit k+1 times, and you now maintain a zoo of models plus a meta. Inference is heavier. If one tuned gradient booster already dominates, or your bases are correlated, a stack buys almost nothing over a cheap average vote. It shines when several genuinely different strong models exist and the last fraction of a percent is worth the operational weight — which is exactly the situation on a Kaggle leaderboard, and rarely the situation in a latency-bound service.
Bagging cuts variance. Boosting cuts bias. Voting averages. Stacking learns the combination.
Play with the base toggles, fold count, and the leakage demo here:
https://dev48v.infy.uk/ml/day33-stacking.html
Top comments (0)