The One-Line Summary: Voting has no learned parameters, so it cannot overfit and it cannot lie to you — but measured here it lost to the single best model on both metrics. Blending learns weights on a holdout and genuinely beat everything, right up until the holdout got small: on 40 rows it reported a log loss of 0.1492, delivered 0.2576, and lost to the free unweighted average it was supposed to improve.
The Parable of the House at Grasse
The house employs four noses, and every batch of oil that arrives has to be judged before the house will put its name on a bottle. The four disagree constantly, which the director considers the point — a panel that always agrees is three salaries wasted. What the house has never settled is how to turn four opinions into one decision.
The Show of Hands
The oldest method is a show of hands. Each nose says ship it or don't, and the majority carries.
THE SHOW OF HANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Batch 41 arrives.
Amaury ship it
Berthe ship it
Cosette don't
Didier ship it
3 to 1. The house ships it.
Nobody recorded HOW sure anyone was. Berthe was
certain. Didier nearly abstained. The tally cannot
tell them apart, and it never will.
This is called HARD VOTING.
It is cheap, it is transparent, and it throws away everything except the direction of each opinion.
The Scorecard
So the house moved to scores. Each nose writes a number from 0 to 100, and the panel's verdict is the average.
Better — the average now knows the difference between certainty and a shrug. Except the house discovered something about Didier.
THE SCORECARD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Batch 41, scores out of 100:
Amaury 91 confident, usually right
Berthe 88 confident, usually right
Cosette 24 confident, usually right
Didier 72 ← scores EVERYTHING 68 to 75
average: 68.8
Didier is not a bad nose. Ask him to rank two
batches and he is excellent. But his numbers have
no range, so averaging drags every verdict toward
the middle of his narrow band.
✗ Averaging opinions also averages in whoever
cannot express confidence properly.
This is called SOFT VOTING.
Soft voting is usually recommended over hard voting, and the reason is genuine: probabilities carry more information than labels. The measurements below show it is not automatic.
The Director's Ledger
The director's answer was to stop treating the four as equals.
"I have eleven years of batches and eleven years of what happened to them. I do not need a fifth nose. I need to know how much each of these four is worth."
THE DIRECTOR'S LEDGER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
From the archive, weights fitted to past batches:
Amaury 0.44
Berthe 0.00 ← agrees with Amaury almost
Cosette 0.56 always; adds nothing new
Didier 0.00 ← no range, no information
verdict = 0.44 x Amaury + 0.56 x Cosette
Berthe is not fired. She is redundant, which is a
different thing and only a ledger can tell them
apart.
This is called BLENDING.
Two of four weights went to zero. That is the ledger working, not failing.
The Month He Graded His Own Homework
The method held until the director published the house's accuracy in a trade journal.
He had fitted the weights on twelve batches from the previous month. Then, asked how well the panel performed, he reported the panel's score on those same twelve batches. The number was extraordinary. It was also the number he had chosen the weights to maximise.
GRADING YOUR OWN HOMEWORK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Twelve batches, four weights to fit.
score on those twelve batches excellent
score on the next twelve ordinary
Three batches per weight is not a ledger.
It is four dials tuned until the last twelve
results looked good.
The show of hands could never do this. It has
no dials. There is nothing to tune and therefore
nothing to overstate.
This is called OVERFITTING THE BLEND.
The show of hands cannot cheat. That is its one genuine advantage, and it is bigger than it sounds.
Why It Works
THE MATHEMATICS OF AGREEING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HARD VOTING majority of the labels
0 learned parameters
SOFT VOTING mean of the probabilities
0 learned parameters
→ needs members whose confidence MEANS
something; a flat scorer poisons the mean
BLENDING weighted mean, weights fitted
N learned parameters on a held-out set
→ the weights live on the simplex:
w_i >= 0, sum w_i = 1
which is heavy regularisation for free
THE ONE RATIO THAT MATTERS
holdout rows / weights
3 to 10 the reported number is fiction
~75 usable, still optimistic
~450 the reported number is roughly true
What Are Voting and Blending?
Hard voting takes the majority class label across members. Soft voting averages their predicted probabilities and thresholds the mean. Neither fits anything: both are fixed functions of the members' outputs, so there is no training step and no holdout to reserve.
Blending averages the same probabilities with learned weights. You hold out a slice of data the members never trained on, collect their predictions on it, and fit weights that minimise loss there. It is stacking's simpler cousin — stacking uses out-of-fold predictions across the whole training set, blending uses a single holdout — and it inherits the same discipline problem: the set you fit the weights on is burned, and any number you quote from it is a number you optimised.
The Math
For members , soft voting is the unweighted mean:
Blending replaces with fitted weights constrained to the simplex:
That constraint matters more than it looks. Non-negative weights summing to one cannot extrapolate beyond the members' predictions — the blend is trapped inside their convex hull. It is a strong regulariser you get for free, and it is why blending misbehaves less than its parameter count suggests.
Step by Step
- Split off a holdout the members will never train on.
- Fit every member on the remaining training data.
- Collect member predictions on the holdout.
- Fit weights on the simplex to minimise loss on those predictions.
- Refit members on everything, keep the weights.
- Report on a third set. The holdout is spent.
Voting, Measured
Four deliberately different members on 9,000 rows, split 60/20/20 into train, holdout and test. Voting needs no holdout, so for this first table it just trains and reports.
import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
VotingClassifier)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import log_loss, accuracy_score
from sklearn.base import clone
X, y = make_classification(n_samples=9000, n_features=25, n_informative=10,
n_redundant=5, flip_y=0.03, random_state=42)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.6,
random_state=42, stratify=y)
Xpool, Xte, ypool, yte = train_test_split(Xrest, yrest, test_size=0.5,
random_state=42, stratify=yrest)
NAMES = ["logreg", "forest", "boosting", "knn"]
FOUR = [LogisticRegression(max_iter=2000),
RandomForestClassifier(n_estimators=150, random_state=0, n_jobs=-1),
GradientBoostingClassifier(n_estimators=120, max_depth=3, random_state=0),
KNeighborsClassifier(n_neighbors=15)]
fit = [clone(m).fit(Xtr, ytr) for m in FOUR]
P = np.column_stack([m.predict_proba(Xte)[:, 1] for m in fit])
print("BASE MODELS ALONE")
print("-" * 46)
print(f"{'':<12}{'accuracy':>10}{'log loss':>11}")
for n, c in zip(NAMES, range(4)):
print(f" {n:<10}{accuracy_score(yte,(P[:,c]>0.5).astype(int)):>10.4f}"
f"{log_loss(yte,P[:,c]):>11.4f}")
est = [(n, clone(m)) for n, m in zip(NAMES, FOUR)]
hard = VotingClassifier(est, voting='hard', n_jobs=-1).fit(Xtr, ytr)
soft = VotingClassifier(est, voting='soft', n_jobs=-1).fit(Xtr, ytr)
print()
print("VOTING (no learned parameters at all)")
print("-" * 46)
print(f" hard {accuracy_score(yte,hard.predict(Xte)):>10.4f}{' n/a':>11}")
print(f" soft {accuracy_score(yte,soft.predict(Xte)):>10.4f}"
f"{log_loss(yte,soft.predict_proba(Xte)[:,1]):>11.4f}")
BASE MODELS ALONE
----------------------------------------------
accuracy log loss
logreg 0.8156 0.4070
forest 0.9319 0.2568
boosting 0.9193 0.2409
knn 0.9370 0.4728
VOTING (no learned parameters at all)
----------------------------------------------
hard 0.9296 n/a
soft 0.9289 0.2515
Three things here are not what the received advice predicts.
Voting lost to the best single model. KNN alone got 0.9370 accuracy; hard voting 0.9296 and soft voting 0.9289. Boosting alone got 0.2409 log loss; soft voting 0.2515. On both metrics, the ensemble is behind the best member. Averaging is not free — it drags a strong member toward a weak one, and logreg at 0.8156 accuracy and 0.4070 log loss is doing exactly that.
Soft voting did not beat hard voting. 0.9289 against 0.9296. The usual reasoning — probabilities carry more information than labels — is correct in principle and depends on those probabilities being calibrated. KNN with n_neighbors=15 can only emit multiples of 1/15, and logreg's log loss of 0.4728 and 0.4070 say neither is well calibrated here. Soft voting averaged in that miscalibration. Didier, scoring everything 72.
The best member was not the best-calibrated member. KNN wins accuracy (0.9370) and is nearly worst on log loss (0.4728). Boosting wins log loss (0.2409) and is third on accuracy. Which model is "best" is a question about your metric, not your data.
Blending, and What the Holdout Costs
Now the same members with fitted weights. The experiment sweeps holdout size against weight count and reports the gap between what the blend claims on its own tuning set and what it delivers on untouched test data.
import warnings, numpy as np; warnings.filterwarnings("ignore")
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (RandomForestClassifier, GradientBoostingClassifier,
ExtraTreesClassifier, VotingClassifier)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import log_loss
from sklearn.base import clone
from scipy.optimize import minimize
X, y = make_classification(n_samples=9000, n_features=25, n_informative=10,
n_redundant=5, flip_y=0.03, random_state=42)
Xtr, Xrest, ytr, yrest = train_test_split(X, y, test_size=0.6,
random_state=42, stratify=y)
Xpool, Xte, ypool, yte = train_test_split(Xrest, yrest, test_size=0.5,
random_state=42, stratify=yrest)
FOUR = [LogisticRegression(max_iter=2000),
RandomForestClassifier(n_estimators=150, random_state=0, n_jobs=-1),
GradientBoostingClassifier(n_estimators=120, max_depth=3, random_state=0),
KNeighborsClassifier(n_neighbors=15)]
EXTRA = [ExtraTreesClassifier(n_estimators=150, random_state=0, n_jobs=-1),
GaussianNB(),
DecisionTreeClassifier(max_depth=3, random_state=0),
DecisionTreeClassifier(max_depth=8, random_state=0),
KNeighborsClassifier(n_neighbors=3),
KNeighborsClassifier(n_neighbors=50),
RandomForestClassifier(n_estimators=150, max_depth=5, random_state=1, n_jobs=-1),
GradientBoostingClassifier(n_estimators=120, max_depth=1, random_state=1)]
f4 = [clone(m).fit(Xtr, ytr) for m in FOUR]
f12 = f4 + [clone(m).fit(Xtr, ytr) for m in EXTRA]
P4p = np.column_stack([m.predict_proba(Xpool)[:, 1] for m in f4])
P4t = np.column_stack([m.predict_proba(Xte)[:, 1] for m in f4])
P12p = np.column_stack([m.predict_proba(Xpool)[:, 1] for m in f12])
P12t = np.column_stack([m.predict_proba(Xte)[:, 1] for m in f12])
# soft voting from scratch is just the unweighted mean — check it
skl = VotingClassifier([(f"m{i}", clone(m)) for i, m in enumerate(FOUR)],
voting='soft', n_jobs=-1).fit(Xtr, ytr)
print("SOFT VOTING FROM SCRATCH vs SCIKIT-LEARN")
print("-" * 52)
print(f" mine (mean of probabilities) {log_loss(yte, P4t.mean(axis=1)):.4f}")
print(f" sklearn VotingClassifier "
f"{log_loss(yte, skl.predict_proba(Xte)[:,1]):.4f}")
def loss(w, P, yy):
w = np.abs(w); w = w / w.sum()
return log_loss(yy, np.clip(P @ w, 1e-9, 1 - 1e-9))
def fit_w(P, yy, k):
r = minimize(loss, np.ones(k)/k, args=(P, yy), method='Nelder-Mead',
options={'maxiter':20000, 'fatol':1e-12, 'xatol':1e-10})
w = np.abs(r.x); return w / w.sum()
def sweep(Pp, Pt, k, nb, seeds=5):
ob, ot = [], []
for s in range(seeds):
idx = np.random.RandomState(s).choice(len(Xpool), nb, replace=False)
w = fit_w(Pp[idx], ypool[idx], k)
ob.append(loss(w, Pp[idx], ypool[idx])); ot.append(loss(w, Pt, yte))
return np.mean(ob), np.mean(ot)
print()
print("BLENDING OPTIMISM, MEAN OF 5 HOLDOUT DRAWS (log loss)")
print("-" * 66)
print(f"{'weights':>8}{'holdout':>9}{'on holdout':>12}{'on test':>10}"
f"{'optimism':>11}{'rows/wt':>9}")
for Pp, Pt, k in [(P4p, P4t, 4), (P12p, P12t, 12)]:
for nb in (40, 300, 1800):
b, t = sweep(Pp, Pt, k, nb)
print(f"{k:>8}{nb:>9}{b:>12.4f}{t:>10.4f}{t-b:>+11.4f}{nb//k:>9}")
print(f"{k:>8}{' simple average, no weights, on test':<37}"
f"{log_loss(yte, Pt.mean(axis=1)):>10.4f}")
SOFT VOTING FROM SCRATCH vs SCIKIT-LEARN
----------------------------------------------------
mine (mean of probabilities) 0.2515
sklearn VotingClassifier 0.2515
BLENDING OPTIMISM, MEAN OF 5 HOLDOUT DRAWS (log loss)
------------------------------------------------------------------
weights holdout on holdout on test optimism rows/wt
4 40 0.1492 0.2576 +0.1084 10
4 300 0.1825 0.2253 +0.0428 75
4 1800 0.1947 0.2121 +0.0174 450
4 simple average, no weights, on test 0.2515
12 40 0.1527 0.2190 +0.0663 3
12 300 0.1856 0.2099 +0.0243 25
12 1800 0.1870 0.2055 +0.0185 150
12 simple average, no weights, on test 0.2711
Soft voting from scratch is one line and it lands on the library exactly: 0.2515 against 0.2515. VotingClassifier(voting='soft') is probabilities.mean(axis=1) with paperwork.
Blending genuinely works when the holdout is big. Four weights on 1,800 rows: 0.2121 on test, against 0.2515 for the free average, 0.2409 for the best single member, and 0.2515 for soft voting. Twelve weights on 1,800 rows reached 0.2055, the best number anywhere in this article. So the ledger earns its keep.
And it lies in exact proportion to how little holdout it has. Four weights on 40 rows report 0.1492 and deliver 0.2576. That is not a small overstatement — the reported loss is roughly seventy percent of the real one. Worse, 0.2576 is behind the unweighted average at 0.2515. You reserved data, ran an optimiser, published a number that looked like your best result ever, and shipped something worse than doing nothing.
Optimism falls monotonically as rows-per-weight climbs: +0.1084 at 10 rows per weight, +0.0428 at 75, +0.0174 at 450. That ratio, not the model count, is the thing to watch.
A result I did not predict: twelve weights overfit less than four at every holdout size — +0.0663 against +0.1084 at 40 rows. I expected the opposite. The reason is the simplex constraint: non-negative weights summing to one cannot chase noise very far, and the eight extra members are genuinely diverse enough that the larger blend generalises better despite having three times the parameters. Parameter count is a bad proxy for overfitting risk when the parameters are boxed in.
When Does Each Help Most?
| Hard voting | Soft voting | Blending | |
|---|---|---|---|
| Learned parameters | 0 | 0 | N |
| Needs a holdout | no | no | yes |
| Can overstate itself | no | no | yes |
| Needs calibrated members | no | yes | helps |
| Members must be comparable | labels only | probabilities | probabilities |
| Best measured here | 0.9296 acc | 0.2515 loss | 0.2055 loss |
| Fails when | members disagree evenly | one member is flat | holdout is thin |
Reach for hard voting when members emit labels only, or when you need a result nobody can accuse you of tuning.
Reach for soft voting when members are calibrated and you have no data to spare. Check calibration first — a member with a log loss of 0.4728 will drag the mean.
Reach for blending when you can afford several hundred holdout rows per weight and you are disciplined enough to report on a third set. Below roughly 75 rows per weight, use the unweighted average and keep the data.
Quick Reference Card
VOTING vs BLENDING: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT THEY ARE
hard vote majority label, 0 parameters
soft vote mean probability, 0 parameters
blending weighted mean, N parameters
fitted on a holdout
MEASURED HERE (9,000 rows, 60/20/20)
best single member 0.9370 acc / 0.2409 loss
hard voting 0.9296 acc
soft voting 0.9289 acc / 0.2515 loss
blend, 4wt, 1800 0.2121 loss
blend, 12wt, 1800 0.2055 loss <- best
blend, 4wt, 40 claims 0.1492, delivers 0.2576
THE ONE RATIO
holdout rows / weights
~10 optimism +0.1084 fiction
~75 optimism +0.0428 usable
~450 optimism +0.0174 honest
RULES
✓ check member calibration before soft voting
✓ report blends on a THIRD set, never the holdout
✓ below ~75 rows/weight, just average
✗ never assume the ensemble beats its best member
SKLEARN
from sklearn.ensemble import VotingClassifier
VotingClassifier(est, voting='soft')
Key Takeaways
Voting has no parameters, so it cannot overfit or overstate — that is its real advantage, and it is why the show of hands could never grade its own homework.
Voting lost to the best single model here — 0.9296 and 0.9289 against KNN's 0.9370. Ensembling is not automatically an improvement.
Soft voting did not beat hard voting — 0.9289 against 0.9296, because it averaged in members with log losses of 0.4728 and 0.4070. Calibration is the precondition, not a detail.
Blending beat everything when the holdout was large — 0.2055 with twelve weights on 1,800 rows, against 0.2409 for the best member.
And it lied by +0.1084 when the holdout was small — reporting 0.1492 while delivering 0.2576, which is worse than the free average at 0.2515.
Watch rows-per-weight, not model count — optimism fell from +0.1084 to +0.0428 to +0.0174 as that ratio went 10, 75, 450.
More weights overfit less, against my expectation — +0.0663 for twelve weights versus +0.1084 for four, because simplex-constrained weights cannot chase noise.
The best member depends on the metric — KNN won accuracy and nearly lost log loss; boosting the reverse. Decide what you are optimising before you decide who is best.
The One-Sentence Summary
Voting is a show of hands that cannot lie because it has nothing to tune, and it lost to the best single model on both metrics here; blending is the director's ledger that genuinely beat everything at 0.2055 when it had 150 holdout rows per weight, and reported a loss thirty percent better than reality when it had ten — so the question is never whether to weight your models but whether you have enough untouched data to earn the weights, and the honest fallback is the unweighted average that costs nothing and claims nothing.
What's Next?
Now that you can tell voting from blending from stacking, you're ready for:
- Grid search vs random search — the tourist with a map versus the one who wanders.
- Bayesian optimization — the sommelier who learns your palate in six glasses.
- Calibration — how to fix the flat scorer that poisoned soft voting above.
- Nested cross-validation — how to tune and report without burning the same data twice.
Follow me for the next article in the Ensembles Beyond Bagging series!
Let's Connect!
If the house at Grasse made voting and blending click, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
Have you ever reported a number you'd tuned? I have, and the tell was that it was the best result in the whole project by a suspicious margin — which is exactly what a number looks like when you have optimised it and then quoted it. 🧪
The thing I keep relearning is that the safest method is usually the one with nothing to adjust. An unweighted average has no dials, so it cannot be talked into flattering you, and the price of that honesty is a few points of loss you can measure. Blending has four dials, or twelve, and every one of them is a small opportunity to fit last month's batches and call it skill. The measurements do not say weighting is wrong. They say weighting costs data, and that quoting a weighted result on the data you weighted it against is the same mistake as a director publishing his own warm-up scores.
Send this to whoever on your team is about to fit blend weights on 40 validation rows.
Top comments (0)