DEV Community

Cover image for LightGBM: The Librarian Who Sorted a Million Books by Ignoring Most of Them
Sachin Kr. Rajput
Sachin Kr. Rajput

Posted on

LightGBM: The Librarian Who Sorted a Million Books by Ignoring Most of Them

The One-Line Summary: LightGBM keeps XGBoost's objective exactly and changes only how the tree is searched and grown — it bins every column into a few hundred buckets so a split search costs the same on a million rows as on a thousand, and it grows the single most promising leaf instead of every leaf at that level, which buys accuracy at equal budget and hands you num_leaves=31, the most dangerous default in tabular machine learning.


The Parable of the Athenaeum at Vashti

The Athenaeum held a million volumes and one impossible job: every winter, divide the collection into a catalogue so that a reader arriving with a vague question could be sent down the right corridor in four turns.

Building that catalogue took the entire winter, and it was always late.


The Old Way: Compare Everything to Everything

To decide where to divide the collection by, say, thickness, the junior librarians did the obvious thing. They lined up every volume, and for each gap between two neighbouring books they asked: if I cut here, how much cleaner are the two halves?

FINDING ONE CUT, THE HONEST WAY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
A million volumes, sorted by thickness.

  book 1 | book 2 | book 3 | ... | book 1,000,000
        ^        ^        ^              ^
      cut?     cut?     cut?           cut?

999,999 possible cuts. For ONE attribute.
The Athenaeum catalogued forty attributes.

Every winter. Every branch of the catalogue.
Enter fullscreen mode Exit fullscreen mode

It was not stupid. It was exact. It was also the reason the catalogue was always late, and the reason nobody had ever tried to catalogue the Athenaeum's ten-million-volume annex.


The First Idea: Shelves, Not Books

Kavya was hired to speed up the winter, and she began by asking a question that sounded like giving up.

"Does the cut have to fall between two particular books? Or does it only have to fall in roughly the right place?"

Her proposal: before cataloguing, put every volume on one of 255 numbered shelves, thinnest to thickest. Then stop looking at books entirely. You can only cut between shelves — and there are 254 of those.

FINDING ONE CUT, KAVYA'S WAY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Shelve first: 255 shelves, thinnest to thickest.

  shelf 1 | shelf 2 | ... | shelf 255
         ^         ^              ^
       cut?      cut?           cut?

254 possible cuts. Always 254.
A million volumes: 254.
Ten million volumes: still 254.

Shelving costs one pass through the collection,
ONCE. Every branch after that is nearly free.
Enter fullscreen mode Exit fullscreen mode

The council objected that this was obviously worse — you have thrown away the exact thickness of every book. Kavya agreed it was approximate and asked them to compare the finished catalogues.

They could not tell which was which.


The Second Idea: Stop Being Fair to Branches

The old catalogue was built in tidy layers. Split the whole collection, then split both halves, then all four quarters, and so on — every branch expanded to the same depth before anyone moved deeper.

Kavya watched a junior spend a morning subdividing a corridor that was already pure poetry, purely because its sibling corridor needed the work.

"Why are we being fair to the corridors? Expand whichever one is the most confusing. Only that one. Then look again."

TWO WAYS TO SPEND TEN DIVISIONS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
BY LAYERS          every corridor at depth 1,
                   then every corridor at depth 2.
                   Tidy. Symmetrical. Wastes effort
                   on corridors already clean.

MOST-CONFUSING     always split whichever single
FIRST              corridor is the biggest mess.
                   Lopsided. Ugly on paper.
                   Every division earns its keep.

Same ten divisions. The second catalogue sends
more readers down the right corridor.
Enter fullscreen mode Exit fullscreen mode

It worked, and it produced catalogues that looked deranged — one corridor sixteen turns deep, its neighbour a single turn — but performed better.


The Trap Kavya Nearly Walked Into

The lopsided method has an appetite. Give it a generous budget of divisions and a small collection, and it will happily carve a private corridor for every individual book.

THE SAME METHOD, A SMALL COLLECTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2,000 volumes, budget of 31 corridors:

  the catalogue answers every question about
  THESE 2,000 volumes perfectly.

  it is useless on the next donation.

2,000 volumes, budget of 7 corridors:

  slightly worse on the shelf you have.
  TWICE AS GOOD on what arrives tomorrow.

The budget is not a detail. On a small collection
it is the single most important number, and the
default is set for a large one.
Enter fullscreen mode Exit fullscreen mode

What Is LightGBM?

Hard pivot. LightGBM does not change the objective from day 13 at all — same second-order expansion, same w=G/(H+λ)w^* = -G/(H+\lambda) , same gain formula. It changes how the split is searched and which leaf grows next.

Histogram Binning

Before training, each feature is discretised into at most max_bin buckets (default 255). Split finding then scans buckets, not rows.

Exact split search costs O(nrows×nfeatures)O(n_{\text{rows}} \times n_{\text{features}}) per node. Histogram search costs O(nbins×nfeatures)O(n_{\text{bins}} \times n_{\text{features}}) , and nbinsn_{\text{bins}} is a constant you chose.

There is a second trick that matters as much. Once you have the histogram of a node and the histogram of one child, the other child is free:

hist(sibling)=hist(parent)hist(child) \text{hist}(\text{sibling}) = \text{hist}(\text{parent}) - \text{hist}(\text{child})

So LightGBM builds the histogram for whichever child has fewer rows and subtracts to get the other. Half the histogram work disappears.

import numpy as np

print("  n rows     exact (unique vals)   histogram (max_bin=255)   ratio")
for n in (1_000, 10_000, 100_000, 1_000_000):
    rng = np.random.default_rng(0)
    exact = len(np.unique(rng.normal(size=n))) - 1
    print(f"  {n:>9,}  {exact:>16,}   {254:>18,}   {exact/254:>8.0f}x")
Enter fullscreen mode Exit fullscreen mode
  n rows     exact (unique vals)   histogram (max_bin=255)   ratio
      1,000               999                  254          4x
     10,000             9,999                  254         39x
    100,000            99,999                  254        394x
  1,000,000           999,999                  254       3937x
Enter fullscreen mode Exit fullscreen mode

The saving is not constant — it grows with your data. That is the whole reason the technique exists.

Leaf-Wise Growth

XGBoost's default grows depth-wise: expand every node at the current level, then move down. LightGBM grows leaf-wise: of all current leaves, split the one whose split gain is highest, then look again.

For a fixed number of leaves, leaf-wise reaches a lower training loss, because it never spends a leaf on a split that was not the best available. The cost is that trees become deeply unbalanced, and max_depth no longer bounds anything unless you set it.

The Parameter That Actually Controls Capacity

In XGBoost you reach for max_depth. In LightGBM max_depth defaults to -1 — unlimited — and capacity is set by num_leaves. A depth- dd balanced tree has 2d2^d leaves, so num_leaves=31 is roughly "depth 5" — but only roughly, and only if the tree were balanced, which it is not.


Does Binning Cost Accuracy?

The objection to binning is obvious: you threw away resolution. Measured, across four bin counts:

import time, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=40000, n_features=30, n_informative=10,
                           flip_y=0.05, random_state=3)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=3)

print("  max_bin   fit time   test logloss")
for mb in (15, 63, 255, 1023):
    t0 = time.perf_counter()
    m = lgb.LGBMClassifier(n_estimators=300, learning_rate=0.1, max_bin=mb,
                           num_leaves=31, random_state=0,
                           verbose=-1).fit(Xtr, ytr)
    dt = time.perf_counter() - t0
    p = m.predict_proba(Xte)[:, 1]
    print(f"  {mb:>7}   {dt:>7.2f}s   {log_loss(yte, p):.4f}")
Enter fullscreen mode Exit fullscreen mode
  max_bin   fit time   test logloss
       15      0.32s   0.1598
       63      0.39s   0.1599
      255      0.39s   0.1603
     1023      0.61s   0.1619
Enter fullscreen mode Exit fullscreen mode

Fifteen bins was the best of the four, and nearly twice as fast as 1023. Not "almost as good" — better.

That is not a fluke of this dataset, it is what regularisation looks like. Coarse bins refuse to split on distinctions too fine to be real, which is exactly the behaviour you want on noisy data. If you have been raising max_bin hoping for accuracy, you have been buying overfitting and paying for it in wall clock.


Leaf-Wise vs Depth-Wise, Same Budget

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=20000, n_features=25, n_informative=8,
                           flip_y=0.05, random_state=7)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=7)

for name, kw in (("leaf-wise ", dict(num_leaves=32)),
                 ("depth-wise", dict(num_leaves=32, max_depth=5))):
    m = lgb.LGBMClassifier(n_estimators=200, learning_rate=0.1,
                           random_state=0, verbose=-1, **kw).fit(Xtr, ytr)
    p = m.predict_proba(Xte)[:, 1]
    print(f"  {name}   32 leaves   {log_loss(yte, p):.4f}")
Enter fullscreen mode Exit fullscreen mode
  leaf-wise    32 leaves   0.1753
  depth-wise   32 leaves   0.1784
Enter fullscreen mode Exit fullscreen mode

Same leaf budget, same rounds, same rate. Leaf-wise wins — modestly, but it wins for a principled reason rather than by accident.

Now look at what those trees are actually shaped like:

import numpy as np, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=25, n_informative=8,
                           flip_y=0.05, random_state=7)
Xtr, _, ytr, _ = train_test_split(X, y, test_size=0.25, random_state=7)

for nl in (8, 31, 128):
    m = lgb.LGBMClassifier(n_estimators=50, num_leaves=nl, learning_rate=0.1,
                           random_state=0, verbose=-1).fit(Xtr, ytr)
    d = m.booster_.trees_to_dataframe().groupby("tree_index")["node_depth"].max()
    print(f"  num_leaves={nl:>4}  actual max depth {d.max():>3}  "
          f"mean {d.mean():>5.1f}   balanced would be "
          f"{int(np.ceil(np.log2(nl)))}")
Enter fullscreen mode Exit fullscreen mode
  num_leaves=   8  actual max depth   8  mean   5.8   balanced would be 3
  num_leaves=  31  actual max depth  17  mean  10.5   balanced would be 5
  num_leaves= 128  actual max depth  32  mean  16.2   balanced would be 7
Enter fullscreen mode Exit fullscreen mode

At the default num_leaves=31, trees reach depth 17 where a balanced tree would be depth 5. At 128 leaves they reach depth 32 against a balanced 7.

A path 32 nodes deep is 32 stacked conditions — an interaction of 32 features describing, in all likelihood, a handful of training rows. This is the mechanism behind the next result.


The Most Dangerous Default in Tabular ML

2,000 rows. 10% label noise. Everything at defaults except num_leaves.

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=2000, n_features=25, n_informative=6,
                           flip_y=0.10, random_state=11)
A, B, a, b = train_test_split(X, y, test_size=0.3, random_state=11)

print("  num_leaves   train logloss   test logloss    gap")
for nl in (7, 31, 127, 511):
    m = lgb.LGBMClassifier(n_estimators=300, num_leaves=nl, learning_rate=0.1,
                           random_state=0, verbose=-1).fit(A, a)
    tr = log_loss(a, m.predict_proba(A)[:, 1])
    te = log_loss(b, m.predict_proba(B)[:, 1])
    print(f"  {nl:>10}   {tr:>13.4f}   {te:>11.4f}   {te-tr:>+6.4f}")
Enter fullscreen mode Exit fullscreen mode
  num_leaves   train logloss   test logloss    gap
           7          0.0386        0.3351   +0.2965
          31          0.0000        0.6793   +0.6793
         127          0.0000        0.7011   +0.7010
         511          0.0000        0.7011   +0.7010
Enter fullscreen mode Exit fullscreen mode

Read the middle row twice. num_leaves=31 is the shipped default, and on 2,000 rows it produced a training loss of 0.0000 — perfect memorisation, including the 10% of labels that are deliberately wrong — and a test loss of 0.6793.

Setting num_leaves=7 cut test loss to 0.3351. The default is twice as bad as a two-second fix.

This is not a bug and LightGBM is not badly designed. The default is tuned for the datasets LightGBM was built for — hundreds of thousands to millions of rows, where 31 leaves is modest. On a 2,000-row table it is enormous. The failure mode is silent: you get a model, it trains fine, the training metrics look spectacular.

The rule I use: if you have fewer than ~10,000 rows, set num_leaves yourself before you do anything else. Start near 7–15 and raise it only if held-out loss improves. Also raise min_child_samples (default 20) — on small data it is the second brake.


Histogram Split Finding from Scratch

Both ideas fit in forty lines. This implements exact split search and histogram split search side by side on the same data, using the gain formula from day 13 — and then demonstrates the sibling-subtraction trick.

The data has genuine structure at x=0.7x = 0.7 , so we can check that binning finds the real boundary rather than merely a fast one.

import numpy as np, time

rng = np.random.default_rng(21)
n = 20000
X = rng.normal(size=n)
g = np.where(X > 0.7, 1.0, -0.3) + rng.normal(0, 0.5, n)  # structure at 0.7
h = np.ones(n)
LAM = 1.0


def score(G, H):
    return G * G / (H + LAM)


def best_split_exact(x, g, h):
    """Scan every gap between adjacent sorted values. O(n) candidates."""
    o = np.argsort(x, kind="stable")
    xs, cg, ch = x[o], np.cumsum(g[o]), np.cumsum(h[o])
    G, H = cg[-1], ch[-1]
    parent, best = score(G, H), (-np.inf, None)
    for i in range(len(xs) - 1):
        if xs[i] == xs[i + 1]:
            continue
        gain = 0.5 * (score(cg[i], ch[i])
                      + score(G - cg[i], H - ch[i]) - parent)
        if gain > best[0]:
            best = (gain, (xs[i] + xs[i + 1]) / 2)
    return best


def bin_feature(x, max_bin=255):
    edges = np.unique(np.quantile(x, np.linspace(0, 1, max_bin + 1)[1:-1]))
    return np.searchsorted(edges, x, side="right"), edges


def best_split_hist(x, g, h, max_bin=255):
    """Accumulate gradients into bins, then scan bins. O(max_bin) candidates."""
    b, edges = bin_feature(x, max_bin)
    nb = len(edges) + 1
    Gh = np.bincount(b, weights=g, minlength=nb)      # the histogram
    Hh = np.bincount(b, weights=h, minlength=nb)
    cg, ch = np.cumsum(Gh), np.cumsum(Hh)
    G, H = cg[-1], ch[-1]
    parent, best = score(G, H), (-np.inf, None)
    for i in range(nb - 1):
        if ch[i] == 0 or ch[i] == H:
            continue
        gain = 0.5 * (score(cg[i], ch[i])
                      + score(G - cg[i], H - ch[i]) - parent)
        if gain > best[0]:
            best = (gain, edges[i])
    return best, (Gh, Hh)


t0 = time.perf_counter(); ge, te = best_split_exact(X, g, h)
de = time.perf_counter() - t0
t0 = time.perf_counter(); (gh, th), (Gp, _) = best_split_hist(X, g, h)
dh = time.perf_counter() - t0

print(f"  exact      gain {ge:.4f}  threshold {te:+.4f}   {de*1000:>6.1f} ms")
print(f"  histogram  gain {gh:.4f}  threshold {th:+.4f}   {dh*1000:>6.1f} ms")
print(f"  gain retained: {gh/ge*100:.2f}%   speedup: {de/dh:.0f}x")

# the sibling histogram is free
b, _ = bin_feature(X)
mask = X < th
Gl = np.bincount(b[mask], weights=g[mask], minlength=len(Gp))
direct = np.bincount(b[~mask], weights=g[~mask], minlength=len(Gp))
print(f"\n  right child built directly : sum {direct.sum():+.4f}")
print(f"  right child by subtraction : sum {(Gp - Gl).sum():+.4f}")
print(f"  max difference over {len(Gp)} bins: {np.abs(direct - (Gp - Gl)).max():.2e}")
Enter fullscreen mode Exit fullscreen mode
  exact      gain 3134.1460  threshold +0.6997     15.0 ms
  histogram  gain 3103.6669  threshold +0.7053      2.0 ms
  gain retained: 99.03%   speedup: 7x

  right child built directly : sum +4865.4350
  right child by subtraction : sum +4865.4350
  max difference over 255 bins: 0.00e+00
Enter fullscreen mode Exit fullscreen mode

The exact search puts the boundary at 0.6997, the histogram at 0.7053 — both within 0.006 of the true structure at 0.7. Histogram search kept 99.03% of the available gain for a seventh of the work, and that ratio only improves as rows grow, because the exact search costs more while the histogram cost stays fixed.

The subtraction check is exact to 0.00e+00 across all 255 bins. That is not an approximation — a histogram is a sum, and sums subtract.


Is It Actually Faster Than XGBoost?

This is the claim LightGBM is famous for, so it deserves a current measurement rather than a citation from 2017. 160,000 rows, 50 features, 300 rounds, both libraries using histograms:

import time, lightgbm as lgb, xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

X, y = make_classification(n_samples=200_000, n_features=50, n_informative=15,
                           flip_y=0.05, random_state=5)
A, B, a, b = train_test_split(X, y, test_size=0.2, random_state=5)

t0 = time.perf_counter()
m1 = lgb.LGBMClassifier(n_estimators=300, learning_rate=0.1, num_leaves=31,
                        random_state=0, verbose=-1, n_jobs=-1).fit(A, a)
d1 = time.perf_counter() - t0

t0 = time.perf_counter()
m2 = xgb.XGBClassifier(n_estimators=300, learning_rate=0.1, max_depth=5,
                       tree_method="hist", random_state=0, n_jobs=-1).fit(A, a)
d2 = time.perf_counter() - t0

print(f"  LightGBM   {d1:>6.2f}s   {log_loss(b, m1.predict_proba(B)[:, 1]):.4f}")
print(f"  XGBoost    {d2:>6.2f}s   {log_loss(b, m2.predict_proba(B)[:, 1]):.4f}")
Enter fullscreen mode Exit fullscreen mode
  LightGBM     1.68s   0.1685
  XGBoost      1.71s   0.1769
Enter fullscreen mode Exit fullscreen mode

They are the same speed. 1.68 seconds against 1.71.

The famous speed gap is gone, and the reason is simple: XGBoost adopted histogram binning. tree_method="hist" has been the default since XGBoost 2.0. LightGBM's 2017 advantage was real, and its competitor copied it.

What survives is the accuracy difference from leaf-wise growth: 0.1685 against 0.1769 here. That is the honest reason to reach for LightGBM today — not speed.


Categorical Features: The Claim and the Measurement

LightGBM advertises native categorical handling — hand it a category column and it partitions the levels directly instead of making you one-hot encode. It sounds strictly better. I measured it against the alternatives on a 300-level column with genuine signal, five seeds:

  seed   native   one-hot   raw int
     0   0.2665   0.2442   0.3657
     1   0.2801   0.2566   0.3553
     2   0.2616   0.2479   0.3561
     3   0.2664   0.2514   0.3564
     4   0.2612   0.2432   0.3459
  MEAN   0.2672   0.2487   0.3559

  one-hot beat native in 5 of 5 seeds
Enter fullscreen mode Exit fullscreen mode

Three things.

One-hot won, consistently. Not by much — 0.0185 log loss — but in every seed. Native categorical splitting sorts levels by gradient statistics and cuts the sorted list, which is powerful and also a very flexible thing to let a model do with 300 levels and finite data. It can overfit the level ordering.

Raw integers were a catastrophe: 0.3559 against 0.2487. Passing a category as an integer tells the model that city 7 lies between city 6 and city 8, which is nonsense, and the model dutifully finds "cities 5 through 12" splits that mean nothing. If you take one thing from this section: never leave a categorical as a bare integer column.

Native is still worth it above a few thousand levels, where one-hot stops fitting in memory. Below that, measure — don't assume the fancy option wins.


When Does LightGBM Help Most?

LightGBM XGBoost Random Forest
Split search histogram histogram (default now) exact
Tree growth leaf-wise depth-wise depth-wise
Capacity knob num_leaves max_depth none needed
Small-data default dangerous safe safe
Categoricals native one-hot or encode one-hot or encode
Speed at 160k rows 1.68s 1.71s n/a
Accuracy, equal budget best of the three close behind on tabular

Reach for it on large tabular data, on wide sparse frames, and when you are willing to set num_leaves deliberately. Reach for XGBoost when you want defaults that behave on small data. Reach for a Random Forest when you want an answer without making any decisions at all.


Hyperparameters

Parameter What it does Typical range Notes
num_leaves capacity — the one that matters 7–255 31 default is too big under ~10k rows
min_child_samples rows required in a leaf 20–200 second brake on small data
learning_rate fraction of each tree added 0.01–0.1 set low, early-stop
n_estimators rounds ceiling, not target callbacks=[early_stopping(50)]
max_depth hard depth cap -1 (off) a guard rail, not the capacity knob
max_bin histogram resolution 63–255 lower is regularising and faster
feature_fraction columns per tree 0.6–1.0 0.8
bagging_fraction + bagging_freq rows per round 0.8 / 1 both needed; fraction alone does nothing
boosting_type gbdt / goss / dart gbdt GOSS is opt-in, not the default

Two footnotes worth having. bagging_fraction is silently ignored unless you also set bagging_freq — a classic wasted afternoon. And GOSS, one of the two techniques the LightGBM paper is named after, is not on by default; boosting_type defaults to plain gbdt.


Quick Reference Card

LIGHTGBM: CHEAT SHEET
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WHAT IT IS:
  XGBoost's objective, searched and grown
  differently. Same w* = -G/(H+lambda), same gain.

TWO CHANGES:
  1. HISTOGRAM BINNING
     features -> max_bin buckets before training
     split cost O(bins) not O(rows)
     sibling hist = parent hist - child hist
  2. LEAF-WISE GROWTH
     split the best leaf, not the whole level
     better at equal budget, wildly unbalanced

THE DEFAULT THAT BITES:
  num_leaves = 31, max_depth = -1
  on 2,000 rows: train 0.0000  test 0.6793
  num_leaves = 7: test 0.3351   <- half the loss

MEASURED:
  max_bin 15 beat 1023 (0.1598 vs 0.1619), 2x faster
  leaf-wise 0.1753 vs depth-wise 0.1784 @ 32 leaves
  vs XGBoost hist: 1.68s vs 1.71s  (speed gap GONE)
  hist split search kept 99.03% of gain, 7x faster
  categoricals: one-hot 0.2487 < native 0.2672
                raw integer 0.3559  <- never do this

WHEN TO USE:
  + large tabular data
  + you will set num_leaves yourself
  + wide/sparse frames
WHEN NOT EFFECTIVE:
  - a few thousand rows on defaults
  - you want safe defaults (use XGBoost)
  - images, audio, raw text

SKLEARN-COMPATIBLE:
  from lightgbm import LGBMClassifier, LGBMRegressor
Enter fullscreen mode Exit fullscreen mode

Key Takeaways

  1. Binning makes split search independent of row count. 999,999 candidate cuts become 254, and the saving grows with your data — 3,937x at a million rows.

  2. Coarser bins can be more accurate, not just faster. max_bin=15 beat max_bin=1023 (0.1598 vs 0.1619) at half the fit time. Raising max_bin for accuracy is backwards.

  3. The sibling histogram is free. Build the smaller child, subtract from the parent. Half the histogram work vanishes.

  4. Leaf-wise beats depth-wise at equal budget — 0.1753 against 0.1784 with 32 leaves each — because no leaf is spent on a split that was not the best available.

  5. num_leaves, not max_depth, is the capacity knob. max_depth defaults to unlimited, and leaf-wise trees at 31 leaves reached depth 17 where a balanced tree would be 5.

  6. The default num_leaves=31 is dangerous under ~10,000 rows. On 2,000 rows it memorised the training set perfectly (0.0000) and doubled test loss versus num_leaves=7 (0.6793 vs 0.3351).

  7. The speed advantage over XGBoost is gone. 1.68s vs 1.71s. XGBoost adopted histograms; hist is its default now. Choose LightGBM for leaf-wise accuracy, not for speed.

  8. Never pass a categorical as a bare integer. 0.3559 against 0.2487 one-hot. And measure native categorical rather than assuming it wins — one-hot beat it in 5 of 5 seeds here.


The One-Sentence Summary

LightGBM is Kavya's two realisations applied to gradient boosting: you do not need to consider every possible cut, only a few hundred well-placed ones, so shelve the data once and search shelves forever after; and you do not owe every branch equal attention, so always grow the most confusing leaf — which together make training scale with your patience rather than your row count, at the price of one shipped default that will quietly memorise any small dataset you hand it.


What's Next?

Now that you know why LightGBM is fast and where it will hurt you, you're ready for:

  1. CatBoost — tomorrow. Ordered boosting, and the target-encoding leak that XGBoost and LightGBM both quietly live with.
  2. Bagging vs Boosting — the settled comparison, measured rather than asserted.
  3. XGBoost vs LightGBM in 10 minutes — same data, same budget, a decision procedure.
  4. Hyperparameter tuning — what to search when four parameters interact.

Follow me for the next article in the Boosting: The Complete Guide series!


Let's Connect!

If the Athenaeum made histogram binning click, drop a heart!

Questions? Ask in the comments — I read and respond to every one.

Have you ever shipped a LightGBM model on a few thousand rows without touching num_leaves? I have, and the training curve looked wonderful right up until it met production. 📚


What I like about LightGBM is that both of its ideas are refusals. It refuses to consider every split, and it refuses to treat every branch as equally deserving. Most performance work looks like that — not a cleverer way to do the work, but a defensible argument for not doing most of it. The hard part is knowing which corners are safe to cut, and the only way anyone ever finds out is by cutting one and measuring what fell off.

Top comments (0)