The One-Line Summary: I set out to write six posts comparing ensemble and tuning methods, and every single one ended up measuring the same thing instead — that the gap between the methods was smaller than the spread of one method against its own seed, which means most published comparisons of these techniques are reporting luck with a method's name attached to it.
The Arc
Six posts, two threads: what to do when you have several models, and how to search for the settings of one. I did not plan them as a set about measurement error. They became one anyway, and by the fourth I had stopped being surprised.
Here is each, with the number that survived.
Stacking: The Editor Who Never Reported a Story
Train a small model to combine your other models, on predictions they made for rows they never saw. Done properly it reached 0.9331 against 0.9301 for the best single model — a real gain, and a small one.
Done improperly it was worse than useless. Train the combiner on in-fold predictions and it handed a weight of 1.123 to the Random Forest and 0.035 to the Gradient Booster that was genuinely better. The leak did not merely inflate the score; it selected the wrong model, because a bagged forest predicting rows it memorised looks like an oracle.
Blending and Voting: Four Noses, One Bottle
Voting has no learned parameters, so it cannot overstate itself — and it lost to the best single model on both metrics, 0.9296 for hard and 0.9289 for soft. Soft voting did not beat hard voting, because it averaged in members whose probabilities were badly calibrated.
Blending, with fitted weights, beat everything at 0.2055 when it had 150 holdout rows per weight. On 40 rows it reported 0.1492 and delivered 0.2576 — worse than the free unweighted average at 0.2515. Same code, same models. Only the holdout size changed.
Grid Search vs Random Search: The Safe With Three Dials
This is where the pattern became impossible to ignore. Random search's advantage is a real mechanism and it behaved exactly as advertised. The gaps were 0.0008 against it on three parameters and 0.0039 for it on six.
Random search's spread against its own seed was 0.0098. Grid beat exactly five of ten random seeds. A single-seed experiment had a coin-flip chance of producing either headline.
Bayesian Optimization: The Prospector
The one method whose advantage grew with budget — 0.0032, then 0.0051, then 0.0063 at 20, 50 and 120 trials. That growth is the signature of something actually learning from its own results.
But the finding that changed how I think about it was the spread. My thirty-line version landed within 0.0004 across three seeds. Random search spanned 0.0128 — thirty-two times wider, for a mean advantage of only 0.0015. The product is not a better answer. It is an answer that does not depend on luck.
Replace Your Grid Search Tonight
Two lines of pruning killed 12 of 20 trials and cut wall clock by 28%, costing 0.1908 to 0.1913 of log loss. The kill rate and the speedup do not match, because a pruned trial still pays for its warm-up. Anyone quoting the kill rate as the time saved is quoting the brochure.
The Hyperparameter Search Cheat Sheet
All of it on one page. Writing it was more uncomfortable than writing the six, because a cheat sheet wants to say "use X" and every measurement says "it depends, and by less than you think."
The Thing They Were All About
Line the effect sizes up next to the noise and the arc writes itself.
| Comparison | Claimed effect | Noise in the same setup |
|---|---|---|
| Grid vs random, 3 params | 0.0008 | 0.0098 |
| Grid vs random, 6 params | 0.0039 | 0.0098 |
| TPE vs random, 20 trials | 0.0032 | 0.0128 |
| Blending vs averaging, thin holdout | claims 0.1492, delivers 0.2576 | — |
| Stacking, leaked vs clean | picks the wrong model entirely | — |
The top three are inside their own noise. The bottom two are not subtle at all — but both are failures of procedure, not of method choice. Nobody in those two rows picked the wrong algorithm. They measured it wrong.
That is the uncomfortable through-line. Six posts spent arguing about which technique wins, and the honest answer in every case was: run it twice before you believe it, and check that the thing you are comparing is bigger than the thing you are not controlling.
What Confused People
Nothing, and I am going to keep being straight about that rather than inventing a reader question.
These six drew zero substantive comments between them. That is not a mystery and it is not the writing — it is distribution, and I made it worse on purpose this week by publishing four of them inside ten minutes. The view counts show exactly what that costs: the first two published picked up 20 and 12 views, the last two got 1 and 0. Same author, same tags, same day, four minutes apart.
I have written about burst publishing being the reason two-thirds of this archive never got read, and then did it again because clearing a backlog felt more urgent than spacing it. The measurement is right there in the feed. One per day from here.
Where the real conversation has been happening is in other people's comment threads, which is its own lesson about where reach actually comes from on this platform.
The One-Sentence Summary
Across six posts on ensembling and hyperparameter search, every method comparison I ran came out smaller than the run-to-run spread of the methods being compared — 0.0008 and 0.0039 against a seed spread of 0.0098, 0.0032 against 0.0128 — and the only two effects that were unambiguously large, the stacking leak that put 1.123 on the wrong model and the blend that claimed 0.1492 while delivering 0.2576, were both procedural mistakes rather than wrong choices of algorithm, which means the highest-return skill in this entire area is not picking the right method but running the comparison twice.
What's Next?
The tuning thread is not finished — two more posts before it closes:
- Early stopping — knowing when to quit, on the training curve rather than the search.
- Model calibration — the weather app that says 70% and means it half the time.
- K-nearest neighbours — opening a new series on distance-based learning.
- Nested cross-validation — tuning and reporting without burning the same data twice.
Follow me for the next article in the Hyperparameter Tuning series!
Let's Connect!
If any single number here changed something you do, drop a heart!
Questions? Ask in the comments — I read and respond to every one, and after the section above you can see I would genuinely welcome the chance.
What's the last method comparison you ran twice? I did not run any of these twice until the grid-versus-random post forced me to, and the only reason it forced me to was that seed 0 gave me the opposite of the famous result and I got suspicious of my own good luck. ⚖️
The pattern in these six was not planned and I would not have believed it if someone had asserted it to me. What made it visible was a boring habit — running the same experiment with a different seed and reporting the spread instead of the best number. That habit costs one extra loop and it invalidated roughly half of what I expected to write. I think that is the correct ratio, and I think most published method comparisons have not paid it.
Before your next "X beats Y" post, run X twice and see whether it beats itself.
Top comments (0)