The One-Line Summary: Every number on this page was measured in an earlier post in this series, and together they say something the folklore doesn't: grid versus random is a coin flip at small budgets (grid beat 5 of 10 random seeds), Bayesian optimization's real product is reliability rather than accuracy (spread 0.0004 against 0.0128), and pruning's advertised speedup is roughly half what the kill rate implies.
Pick By Budget, Not By Fashion
WHICH SEARCH, BY TRIAL BUDGET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
< 20 trials random search
a surrogate has nothing to fit;
grid can't resolve anything either
20-30 random, or TPE if trials are slow
measured gap at 20: +0.0032, but
TPE's own spread was 0.0146
30-150 Bayesian optimisation (TPE)
gap grows with budget:
+0.0032 -> +0.0051 -> +0.0063
at 20 -> 50 -> 120 trials
any budget grid, IF you have 2-3 parameters
and need an auditable statement
about what you ruled out
never grid with 6+ parameters
resolution collapses to 2 values
per axis and cost 0.0041 of score
The Resolution Argument, In One Table
VALUES TESTED PER PARAMETER at n = 64 trials
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
params grid = n^(1/d) random = n
2 8 64
3 4 64
6 2 64
8 cannot build 64
MEASURED CONSEQUENCE (GBM, Friedman #1, 64 trials)
3 params grid 0.8729 random 0.8740 mean
6 params grid 0.8688 random 0.8728
→ adding three near-useless parameters cost
grid 0.0041 and cost random almost nothing
But Check The Noise Before You Believe Any Of It
RANDOM SEARCH AGAINST ITSELF, 10 SEEDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
min 0.8694 max 0.8792 spread 0.0098
sd 0.0030 mean 0.8740
grid (deterministic) 0.8729
→ grid beat 5 of 10 random seeds
the gaps people argue about:
3 params 0.0008 6 params 0.0039
the noise they argue inside:
0.0098
✗ A single-seed comparison of search methods is
a coin flip reported as a finding.
What Bayesian Optimization Actually Buys
TPE vs RANDOM, 5 SEEDS EACH
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
budget TPE random gap
20 0.8737 0.8704 +0.0032
50 0.8781 0.8730 +0.0051
120 0.8803 0.8741 +0.0063
→ the gap GROWS. random went 0.8730 to 0.8741
from 50 to 120 trials; TPE gained 0.0022.
FROM SCRATCH (forest surrogate + UCB), 3 seeds
mine 0.8765 mean spread 0.0004
random 0.8750 mean spread 0.0128
→ 32x tighter, not 32x better.
On seed 2 random search WON by 0.0048.
The product is reproducibility, not accuracy.
Pruning: The Honest Numbers
OPTUNA MEDIAN PRUNER, 20 TRIALS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
no pruner logloss 0.1908 13.0s pruned 0
median pruner logloss 0.1913 9.4s pruned 12
killed 60% of trials -> saved 28% of wall clock
cost: 0.0005 log loss
✗ The kill rate is not the speedup. A pruned trial
still pays for its warm-up steps before there is
enough evidence to stop it.
The Search Space Matters More Than The Search
BEFORE YOU CHOOSE A METHOD
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✓ log-uniform for learning rates, regularisation,
anything spanning orders of magnitude
✓ integer-uniform for depths and counts
✓ uniform for fractions (subsample, colsample)
✗ uniform on [0.001, 0.3] for a learning rate puts
~90% of draws above 0.03
No amount of grid-versus-Bayesian argument repairs
a space that never samples where the answer is.
COVERAGE, random search (1 - (1-q)^n)
top 10% 64 draws 0.9988
top 5% 64 draws 0.9625
top 1% 64 draws 0.4744 <- coin flip
200 draws 0.8660
→ early tuning: 64 draws is plenty
→ late tuning: the gains live in the top 1%,
and 64 draws cannot reach it
Quick Reference Card
HYPERPARAMETER SEARCH: ONE PAGE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
DECIDE
2-3 params, want audit trail -> grid
4+ params, cheap trials -> random
30+ trials, expensive trials -> TPE
need reproducible result -> TPE
hundreds of idle cores -> random
TUNE FIRST
learning_rate x n_estimators (one parameter)
then max_depth 3-6
then subsample / colsample
then stop
ALWAYS
✓ log-uniform for rates
✓ 8-15 warm-up trials before a surrogate
✓ run twice with different seeds
✓ report the spread, not the best seed
✗ never quote a single-seed method comparison
✗ never prune against one validation split
OPTUNA MINIMUM
study = optuna.create_study(
sampler=TPESampler(seed=0),
pruner=MedianPruner(n_warmup_steps=2))
trial.report(loss, step)
if trial.should_prune(): raise TrialPruned()
Key Takeaways
Grid versus random is decided by dimension — grid 0.8729 against random 0.8740 on three parameters, and 0.8688 against 0.8728 on six.
At 64 trials neither wins — grid beat exactly 5 of 10 random seeds, and the seed spread of 0.0098 swamps both gaps.
Bayesian optimization compounds — +0.0032, +0.0051, +0.0063 at budgets of 20, 50 and 120.
Its real product is variance reduction — 0.0004 spread against random search's 0.0128.
Pruning's kill rate is not its speedup — 60% of trials killed bought 28% of wall clock, for 0.0005 log loss.
The sampling distribution outranks the method — uniform draws on [0.001, 0.3] put 90% of a learning-rate search above 0.03.
Late-stage tuning needs a different budget — the top 1% of the space is 0.4744 at 64 draws and 0.8660 at 200.
The One-Sentence Summary
Choose your search by trial budget and parameter count rather than by what is fashionable — random below twenty trials, grid only at two or three parameters where the audit trail is worth something, TPE above thirty where its advantage compounds from +0.0032 to +0.0063 — and whatever you choose, run it twice with different seeds, because the spread between two runs of the same method was 0.0098 and every method comparison in this series was smaller than that.
What's Next?
- The series recap — the whole tuning arc in one page.
- Nested cross-validation — tuning and reporting without burning the same data twice.
- Pruning on a CV signal — how to stop pruning against the noise of one split.
- Multi-objective tuning — accuracy and latency at the same time.
Follow me for the next article in the Hyperparameter Tuning series!
Let's Connect!
If this saved you a grid search, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
What's your default search now? Mine is TPE with 50 trials and a median pruner, and the honest reason is not the accuracy — it is that I stopped having to wonder whether last week's result was skill or the seed. 📋
Assembling this page was more uncomfortable than writing the articles it summarises, because a cheat sheet wants to say "use X" and every measurement in this series says "it depends, and by less than you think." The one line I would keep if I had to delete the rest: run your search twice with different seeds before you believe anything it told you.
Bookmark this, and check the noise row before you report a tuning win.
Top comments (0)