The One-Line Summary: Two extra lines inside the objective — report the score as the model grows, then ask whether to stop — let Optuna abandon 12 of 20 trials before they finished, cutting wall clock from 13.0s to 9.4s at a cost of 0.0005 log loss; the saving is smaller than the kill rate because a pruned trial still runs its warm-up, which is the detail nobody mentions.
The Two Lines That Do The Work
Every grid search has the same defect: it runs every candidate to completion, including the ones that were obviously hopeless after 20% of the work. A human watching a training curve would have killed them. Pruning is that human, automated.
WHAT PRUNING ACTUALLY DOES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
trial 7, learning_rate=0.011, depth=2
40 trees logloss 0.61 <- median so far: 0.28
80 trees logloss 0.52 <- still miles behind
-> KILLED
120, 160, 200 trees: never run.
A grid search would have finished this trial, then
politely reported that it was bad.
trial.report(loss, step) tell optuna
if trial.should_prune(): ... let it decide
That is the entire mechanism. You are handing Optuna a progress signal it can compare across trials, and letting it stop the ones that are behind.
The Whole Setup
Runnable as-is. The only Optuna-specific concepts are suggest_* for the search space, report for progress, and should_prune for the verdict.
import warnings, numpy as np, time; warnings.filterwarnings("ignore")
import optuna; optuna.logging.set_verbosity(optuna.logging.WARNING)
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import log_loss
X, y = make_classification(n_samples=1500, n_features=15, n_informative=6,
flip_y=0.03, random_state=42)
Xtr, Xva, ytr, yva = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)
def objective(trial):
lr = trial.suggest_float("learning_rate", 0.01, 0.3, log=True)
depth = trial.suggest_int("max_depth", 2, 5)
sub = trial.suggest_float("subsample", 0.6, 1.0)
clf = GradientBoostingClassifier(n_estimators=200, learning_rate=lr, max_depth=depth,
subsample=sub, random_state=0, warm_start=True)
best = 9.9
for n in range(40, 201, 40): # grow the model in steps
clf.set_params(n_estimators=n); clf.fit(Xtr, ytr)
loss = log_loss(yva, clf.predict_proba(Xva)[:, 1])
best = min(best, loss)
trial.report(loss, n) # tell optuna how it's going
if trial.should_prune(): # let optuna stop a hopeless trial
raise optuna.TrialPruned()
return best
for label, pruner in [("no pruner", optuna.pruners.NopPruner()),
("median pruner", optuna.pruners.MedianPruner(n_startup_trials=5,
n_warmup_steps=2))]:
t0 = time.perf_counter()
st = optuna.create_study(direction="minimize",
sampler=optuna.samplers.TPESampler(seed=0), pruner=pruner)
st.optimize(objective, n_trials=20, show_progress_bar=False)
el = time.perf_counter() - t0
pruned = sum(1 for t in st.trials if t.state == optuna.trial.TrialState.PRUNED)
print(f"{label:<15} best logloss {st.best_value:.4f} {el:>6.1f}s "
f"pruned {pruned:>2}/20")
no pruner best logloss 0.1908 13.0s pruned 0/20
median pruner best logloss 0.1913 9.4s pruned 12/20
Reading That Honestly
Pruning killed 12 of 20 trials — 60% of the search — and saved 28% of the wall clock. Those two numbers should not match, and the gap is the useful part: a pruned trial is not free. It ran its warm-up steps before there was enough evidence to kill it, so you pay the first fraction of every bad trial no matter what. n_warmup_steps=2 here means two reports before any trial is eligible, which is the floor on what you can save.
If you see pruning advertised as "60% fewer trials, 60% less time," that is the advertisement, not the measurement.
It cost 0.0005 of log loss — 0.1908 without, 0.1913 with. Pruning can throw away a trial that would have recovered late; some configurations look bad at 80 trees and are excellent at 200. That risk is real and here it cost half a thousandth. Worth it, and worth knowing you paid it.
The trade scales with how expensive your trials are. At 13 seconds for the whole study, none of this matters. At 13 hours it is the difference between one overnight run and three. Pruning is a wall-clock optimisation, so its value is proportional to what a trial costs you.
One thing worth doing differently in a real project: this example prunes against a single validation split, which is what makes it fast enough to be a five-minute demo. In production you want the pruning signal to come from cross-validation, or you are pruning on the noise of one split — and that noise is precisely what the previous two articles were about.
Key Takeaways
Two lines buy the whole mechanism —
trial.report(loss, step)andif trial.should_prune(): raise optuna.TrialPruned(). Everything else is the search space you already had.Killing 60% of trials saved 28% of the time, because warm-up steps run before any trial can be pruned. Never quote the kill rate as the speedup.
It cost 0.0005 log loss (0.1908 to 0.1913) — pruning occasionally discards a late bloomer, and that is the price of the wall clock.
The One-Sentence Summary
Optuna's pruner turns a grid search's worst habit — finishing trials that were already hopeless — into an automated judgment call, and measured here it abandoned 12 of 20 trials to cut 13.0s down to 9.4s for 0.0005 of log loss, with the honest caveat that the 60% kill rate only bought a 28% saving because every pruned trial still pays for its warm-up.
What's Next?
- Nested cross-validation — tuning and reporting without burning the same data twice.
- Pruning on a CV signal — how to avoid pruning against the noise of one split.
- Multi-objective tuning — when you need accuracy and latency.
- Search spaces that lie — why uniform sampling of a learning rate is almost always wrong.
Follow me for the next article in the Hyperparameter Tuning series!
Let's Connect!
If this replaces a grid search tonight, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
What's the longest grid search you've ever let finish? Mine was nineteen hours across 486 candidates, and when I finally plotted the curves afterwards, roughly two thirds of them were clearly dead inside the first 15% of their training. ⏱️
The reason pruning feels like cheating is that it is the one optimisation that requires the search to have an opinion mid-trial. Grid and random search treat a trial as atomic: you ask a question, you wait, you get a number. Pruning treats the training curve as evidence in its own right, which is obvious the moment you have ever watched a loss curve and known at epoch three that it was not going to work.
Copy the objective above, swap in your model, and delete your grid.
Top comments (0)