The One-Line Summary: On the same data with the same tuning budget, XGBoost and LightGBM finished 0.00020 AUC apart — and when I handed LightGBM its 2.23× speed advantage back as extra search time, it fitted 1.59× as many candidates and gained −0.00013 AUC for the trouble; the real difference between them is not accuracy, it is that LightGBM trains 3.3× faster while XGBoost predicts 2.2× faster.
The Question Everyone Asks Wrong
"Which is more accurate, XGBoost or LightGBM?"
It is the wrong question, and ten minutes of measurement shows why. Here is a fair fight: 20,000 rows, 30 features, identical folds, identical parameter distributions, identical random seed, 15 search trials each.
import time, warnings, numpy as np
warnings.filterwarnings("ignore")
import xgboost as xgb, lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import (train_test_split, RandomizedSearchCV,
StratifiedKFold)
from sklearn.metrics import roc_auc_score
from scipy.stats import loguniform, randint
X, y = make_classification(n_samples=20000, n_features=30, n_informative=12,
n_redundant=6, flip_y=0.02, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25,
random_state=42, stratify=y)
cv = StratifiedKFold(3, shuffle=True, random_state=0)
N_TRIALS = 15
space = {"n_estimators": randint(100, 400),
"learning_rate": loguniform(0.01, 0.3),
"max_depth": randint(3, 10),
"subsample": [0.6, 0.8, 1.0],
"colsample_bytree": [0.6, 0.8, 1.0]}
def run(name, est):
t0 = time.perf_counter()
s = RandomizedSearchCV(est, space, n_iter=N_TRIALS, cv=cv, scoring="roc_auc",
random_state=0, n_jobs=-1).fit(Xtr, ytr)
tune = time.perf_counter() - t0
t1 = time.perf_counter(); best = s.best_estimator_; best.fit(Xtr, ytr)
fit = time.perf_counter() - t1
t2 = time.perf_counter(); p = best.predict_proba(Xte)[:, 1]
pred = time.perf_counter() - t2
auc = roc_auc_score(yte, p)
print(f"{name:<11}{s.best_score_:>10.5f}{auc:>10.5f}"
f"{tune:>10.1f}s{fit:>9.2f}s{pred*1000:>9.0f}ms")
return auc, tune
print(f"SAME DATA, SAME BUDGET ({N_TRIALS} trials x 3-fold), "
f"15k train rows x 30 features")
print("-" * 62)
print(f"{'':<11}{'best CV':>10}{'test AUC':>10}{'tune':>11}"
f"{'refit':>9}{'predict':>11}")
a1, t1 = run("XGBoost", xgb.XGBClassifier(tree_method="hist", n_jobs=1,
eval_metric="logloss", random_state=0, verbosity=0))
a2, t2 = run("LightGBM", lgb.LGBMClassifier(n_jobs=1, random_state=0, verbose=-1))
print()
print(f"test AUC gap : {abs(a1 - a2):.5f}")
print(f"tuning time ratio : {t1 / t2:.2f}x (XGBoost / LightGBM)")
SAME DATA, SAME BUDGET (15 trials x 3-fold), 15k train rows x 30 features
--------------------------------------------------------------
best CV test AUC tune refit predict
XGBoost 0.98305 0.98845 10.8s 1.27s 17ms
LightGBM 0.98313 0.98825 4.9s 0.38s 38ms
test AUC gap : 0.00020
tuning time ratio : 2.23x (XGBoost / LightGBM)
Two ten-thousandths of an AUC point apart. On a test set of 5,000 rows that is a handful of examples changing places. Anyone claiming one of these libraries is more accurate than the other on ordinary tabular data is reporting seed variance.
The Honest Version of "Same Budget"
The obvious objection: LightGBM tuned in less than half the time, so a trial-for-trial comparison undersells it. Give both the same wall clock instead and let LightGBM spend its speed on more candidates.
Thirty seconds each, identical parameter draws, whoever gets through more of them wins more lottery tickets:
EQUAL WALL CLOCK: 30s of tuning each, identical parameter draws
--------------------------------------------
trials best CV test AUC
XGBoost 37 0.98420 0.98990
LightGBM 59 0.98442 0.98977
LightGBM fitted 1.59x as many candidates in the same 30s
and it bought -0.00013 test AUC
LightGBM got 59 shots to XGBoost's 37. It found a better cross-validation score — 0.98442 against 0.98420 — and then lost on the test set by 0.00013.
That is what a saturated search looks like. Both libraries had already found the plateau within the first twenty trials, and everything after that was sampling noise dressed up as progress. The extra 22 candidates bought a better CV number and no better model, which is a compact demonstration of why you should not select on the third decimal place of a cross-validation score.
Where They Actually Differ
Speed is real. It is just not where people look for it.
THE ONLY NUMBERS THAT DIFFERED
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
XGBoost LightGBM winner
test AUC 0.98845 0.98825 nobody
tune 15x3 10.8s 4.9s LGB 2.2x
refit 1.27s 0.38s LGB 3.3x
predict 5k 17ms 38ms XGB 2.2x
→ LightGBM wins the loop you run in development
→ XGBoost wins the loop you run in production
LightGBM trains 3.3× faster. XGBoost scores 2.2× faster. Those pull in opposite directions, and which one matters is a property of your system, not of the library.
If you retrain nightly and serve from a cache, training time is your bottleneck and LightGBM saves you real hours. If you retrain monthly and serve 3,000 predictions per second behind a latency SLO, that 38ms against 17ms is the number on the incident report, and the training difference is invisible.
Pick on the axis your system is actually constrained by. On accuracy, flip a coin.
Key Takeaways
Accuracy is a tie — 0.00020 AUC apart under equal trials, 0.00013 the other way under equal wall clock. Both differences are smaller than seed noise.
Speed did not convert — 1.59× more candidates in the same 30 seconds produced a better CV score and a marginally worse test score. Search saturates, and past that point more trials buy overfitting to the validation folds.
The training/inference split is the real decision — LightGBM 3.3× faster to fit, XGBoost 2.2× faster to predict. Choose against your bottleneck, not against a benchmark someone else ran.
The One-Sentence Summary
XGBoost and LightGBM are close enough on accuracy that any measured gap is noise — LightGBM tunes 2.23× faster but converts none of it into a better model, so the honest tiebreaker is whether your system is bottlenecked on the training loop, where LightGBM is 3.3× faster, or the serving loop, where XGBoost is 2.2× faster.
What's Next?
- The hyperparameter tuning cheat sheet — tomorrow. Everything that matters, on one page.
- Week 3 recap — what clicked and what confused people.
- Stacking — what to do when you have several good models and no idea which to trust.
Follow me for the next article in the Boosting: The Complete Guide series!
Let's Connect!
If this saved you an afternoon of benchmarking, drop a heart!
Questions? Ask in the comments — I read and respond to every one.
Have you ever switched libraries for a gain you never actually measured? I have, and the honest post-mortem was that I moved a project to LightGBM for accuracy, got none, and kept it anyway because the training loop went from twelve minutes to four — which was the right outcome for entirely the wrong reason. ⚖️
The interesting thing about a dead heat is how much energy goes into denying it. There is a whole genre of blog post picking a winner between these two, and almost none of them report a gap larger than the noise in their own cross-validation. It is more comfortable to believe the tool is the variable, because tools can be swapped in an afternoon and the alternative — that your features and your labels are the ceiling — takes a quarter to fix.
Send this to whoever on your team is about to spend a sprint migrating between the two.
Top comments (1)
Useful post. Caching at the right layer beats every micro optimization. Which layer did you land on here?