DEV Community

Sachin Kr. Rajput
Sachin Kr. Rajput

Posted on

The Boosting Tuning Cheat Sheet: Which Knob to Turn First, and When to Stop Turning

#machinelearning #ai #python #programming

The One-Line Summary: Every number on this page was measured in an earlier post in this series rather than copied from a blog — the tuning order that falls out of them is learning_rate × n_estimators first as one parameter, then max_depth around 3 to 6, then stop, because unlimited depth took boosting from 2.0196 to 7.1105 test MSE and running to 3,000 trees when the optimum was 19 made it a third worse.

The Order

Nobody tunes eleven parameters. Here is the order the measurements justify.

BOOSTING TUNING ORDER
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. learning_rate x n_estimators   ← ONE parameter
     pick lr, then early-stop for the tree count.
     Their product is roughly conserved.

2. max_depth  (or num_leaves)     ← 3 to 6
     the single biggest lever after step 1.

3. subsample / colsample          ← 0.6 to 1.0
     mild regulariser, cheap to sweep.

4. everything else                ← usually noise
     stop here unless you have a validation
     curve telling you otherwise.
Enter fullscreen mode Exit fullscreen mode

1. learning_rate and n_estimators Are One Parameter

Measured on classification, sweeping the learning rate and early-stopping for the tree count each time. The product lr × trees barely moves:

lr      peak trees   lr x trees
  0.3           15         4.50
  0.1           44         4.40
  0.03         147         4.41
Enter fullscreen mode Exit fullscreen mode

Three learning rates spanning a factor of ten, and the product lands at 4.50, 4.40, 4.41. Halve the learning rate and you need roughly twice the trees for the same model. So do not grid-search both — fix one, early-stop the other.

2. max_depth Has a Ceiling, and It Is Low

The mirror image of bagging. Test MSE at 100 estimators, same data:

max_depth      bagging     boosting
        1      19.5944       5.0660
        5       5.6241       2.0196
     None       3.6156       7.1105
Enter fullscreen mode Exit fullscreen mode

Bagging improves all the way to unlimited depth. Boosting is U-shaped and unlimited depth is worse than stumps — 7.1105 against 5.0660. A fully grown tree eats the whole residual on the first pass and leaves nothing for the rest of the chain, noise included.

Buy capacity with more trees, not deeper ones.

3. More Trees Is Not Free

Bagging and boosting answer this differently, and it is the most common way people waste a weekend:

n_estimators     bagging     boosting
          10     39.5639      37.1438
         100     36.5054      43.4739
        3000     35.5388      48.4704
Enter fullscreen mode Exit fullscreen mode

Bagging is monotone — 3,000 trees was its best. Boosting's optimum was 19 trees at 36.1841, and by 3,000 it had degraded to 48.4704. A third worse than its own best, from doing more of the thing that was working.

n_estimators is a smoothness knob for bagging and a capacity knob for boosting. Only one of them needs early stopping.

4. The Library-Specific Knobs 

LightGBM       measured effect
  max_bin      15 beat 1023  (0.1598 vs 0.1619)
  growth       leaf-wise 0.1753 vs depth-wise 0.1784
  num_leaves   the dangerous default; couple it
               to max_depth or it silently overfits

CatBoost       measured effect
  ordered TS   naive target encoding: test 2.7975
               out-of-fold:            test 0.5823
               drop the column:        test 0.5464
               → on a signal-free column, deleting
                 it beat every encoding of it
Enter fullscreen mode Exit fullscreen mode

The max_bin result is worth pausing on: the coarser binning won. More resolution in the split search is not more accuracy, it is more variance.

5. Which Library — and Why It Barely Matters

Same data, same tuning budget, 15 trials each:

              test AUC     tune    refit   predict
XGBoost        0.98845    10.8s    1.27s     17ms
LightGBM       0.98825     4.9s    0.38s     38ms
Enter fullscreen mode Exit fullscreen mode

Two ten-thousandths of an AUC point. Give LightGBM its speed advantage back as extra search time and it fitted 59 candidates to XGBoost's 37 — final scores 0.98977 and 0.98990. The extra search bought nothing.

So pick on the loop you are constrained by. LightGBM wins development (3.3× faster to fit). XGBoost wins production (2.2× faster to score).

6. When Tuning Is the Wrong Job 

BEFORE YOU TUNE, CHECK THESE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✗ Labels are noisy
    30% flipped labels cost bagging 0.1133 accuracy
    and boosting 0.1477. Boosting pays more for bad
    labels than any hyperparameter can win back.

✗ A high-cardinality column is target-encoded
    by hand. Fix the encoder, not the depth.

✗ Your CV and your holdout disagree by more than
    the gap you are chasing. Tuning against a
    leaky score optimises the leak.

✓ Clean labels, honest validation, and you still
    want the last two points. Now tune.
Enter fullscreen mode Exit fullscreen mode

Key Takeaways

  1. learning_rate × n_estimators is one parameter — measured at 4.50, 4.40 and 4.41 across a tenfold range of learning rates.

  2. max_depth 3–6, never unlimited — unlimited took boosting from 2.0196 to 7.1105, worse than stumps at 5.0660.

  3. Boosting needs early stopping and bagging does not — optimum at 19 trees versus 3,000, and 48.4704 versus 36.1841 for ignoring it.

  4. Coarser can win — LightGBM's max_bin=15 beat max_bin=1023 at 0.1598 against 0.1619.

  5. Library choice is a tie on accuracy — 0.98845 against 0.98825. Choose on training versus inference speed instead.

  6. Fix the data before the knobs — noisy labels cost boosting 0.1477 of accuracy, which no amount of tuning recovers.

The One-Sentence Summary

Tune the learning rate and tree count as a single parameter with early stopping, keep max_depth between 3 and 6 because unlimited depth is worse than stumps, sweep subsample if you are bored, and spend everything you save on checking whether your labels and your validation split deserve the trust you are placing in them.

What's Next?

  1. The series recap — the whole Boosting arc in one page, with the numbers that survived.
  2. Stacking — what to do when you have several good models and no idea which to trust.
  3. Blending and voting — the two ways a committee can agree, and why one of them cheats.
  4. Grid search vs random search — and why the tourist with a map loses.

Follow me for the next article in the Boosting: The Complete Guide series!

Let's Connect!

If this saved you a grid search, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

What's the hyperparameter you've stopped tuning entirely? Mine is n_estimators on random forests — I set it to 500 and never think about it again, because the measurement says it cannot hurt me. 🔧

The reason cheat sheets are usually wrong is that they are assembled from other cheat sheets. Every figure on this page came from code that ran in an earlier post in this series, which is a low bar that most tuning advice does not clear — and the two most useful lines here, that coarser binning won and that deleting a column beat encoding it, are both things I would not have believed from a table someone else wrote.

Bookmark this before your next grid search, and check the depth row first.

Top comments (0)