XGBoost reg_lambda vs reg_alpha: L2 vs L1 Regularization, Benchmarked on 5-Fold CV

Part of the XGBoost Internals series by Shakti Tiwari — Nifty Option Trader, XGBoost Expert (optiontradingwithai.in)

Disclaimer (verbatim): Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Two-layer engine note: Our live system is TWO-LAYER: Layer 1 = Dhan WebSocket live capture (shadow/predict-only, never trades without broker auth); Layer 2 = EOD-audited XGBoost/LightGBM training core (walk-forward, cost-and-slippage model). The parameter discussed here is tuned in Layer 2.

Quick Answer

reg_lambda (λ, L2) and reg_alpha (α, L1) are the two regularization penalties XGBoost applies to leaf weights — not to features, but to the output values of the leaves themselves. On our 5-fold CV, λ gently shrinks regression RMSE from 0.4232 (none) down to 0.4084 (λ=30), a ~3.5% relative gain, and stays safe even at λ=100. α (L1) gets regression to 0.4094 at α=5 but then collapses to 0.6578 RMSE at α=100. For classification accuracy, neither moves the needle until α=100, where it drops to 0.9011. Default λ=1, α=0 is a fine starting point; the data says nudge λ up before you ever touch α.

Why This Matters

Most people learn regularization as "thing that stops overfitting" and then set reg_lambda=1 because the docs said so and walk away. That's a mistake, and it's a particularly expensive one in a trading context where a model that looks great in-sample but wobbles out-of-sample is the difference between a backtest that prints money and a live account that bleeds.

Here's the part they don't tell you on the tutorial blogs: in XGBoost, λ and α do not regularize the input features the way they do in linear regression. They regularize the leaf weights — the actual prediction values sitting at the bottom of every tree. When XGBoost splits a tree, it computes an optimal output value for each leaf. λ and α push those output values around. L2 (λ) pulls them gently toward zero. L1 (α) can slam some of them exactly to zero, which in tree-land means that leaf stops contributing anything — a built-in pruning mechanism.

So this isn't an academic footnote. The choice between "shrink leaf values" (λ) and "zero out leaf values" (α) changes how your model behaves under stress, and our CV numbers show the two penalties have completely different failure modes. One fails gracefully. The other fails catastrophically. If you're tuning a model that's supposed to fire signals on Monday morning, you'd better know which is which.

The previous article in this series covered colsample_by* (A6) — how random feature subsampling stabilizes trees. Regularization is the natural next question: once you've controlled which features each tree sees, how do you control how hard each tree bets on what it sees? That's λ and α. The next cluster (B1) moves on to tree_method, the algorithm that actually builds the tree — but you can't tune the builder intelligently until you understand what the penalties are doing to the weights it produces.

Research Question & Hypothesis

Research question: Among reg_lambda (L2) and reg_alpha (L1) applied to leaf weights, how does each one individually affect 5-fold CV regression RMSE and classification accuracy, and where does each one break?

Hypothesis (before looking at the data): L2 would behave like ridge — smooth, monotonic-ish improvement with a soft floor, never blowing up. L1 would behave like lasso — strong early gains from sparsity, but a sharp cliff once the penalty starts zeroing out leaves that actually carry signal. The hypothesis turned out to be right, and the CV numbers make the "sharp cliff" part almost comically literal.

We swept each parameter independently across seven values, holding the other at its default. λ swept at α=0; α swept at λ=1. That lets us isolate each penalty's shape. It does not tell us about their interaction — and that's an honest limitation we'll flag below.

Data & Methodology

• Engine: XGBoost (dmlc/xgboost), the gradient-boosted-tree library maintained at github.com/dmlc/xgboost (~28.7k stars). Parameter semantics taken from doc/parameter.rst and doc/tutorials/param_tuning.rst in that repo.
• Validation: 5-fold cross-validation. Every number below is an observed CV score, not a single holdout, not an in-sample fit.
• Two tasks: (a) a regression task scored by RMSE, (b) a classification task scored by accuracy.
• Sweep design: seven values per parameter. reg_lambda ∈ {0, 0.1, 1, 5, 10, 30, 100} at reg_alpha=0. reg_alpha ∈ {0, 0.1, 1, 5, 10, 30, 100} at reg_lambda=1 (the XGBoost default for λ).
• Defaults as shipped: reg_lambda=1, reg_alpha=0.

A useful sanity check before we go further: the configuration (λ=1, α=0) appears in both sweeps — it's the α=0 row of the alpha sweep and the λ=1 row of the lambda sweep. Both report regression RMSE 0.4194. They match exactly. That's the kind of internal consistency you want before trusting a benchmark, and it's there.

Results

reg_lambda (L2) — regression RMSE and classification accuracy

reg_lambda	reg RMSE	clf ACC
0	0.4232	0.9075
0.1	0.4192	0.9075
1	0.4194	0.9065
5	0.4140	0.9073
10	0.4117	0.9064
30	0.4084	0.9065
100	0.4147	0.9058

reg_alpha (L1) — regression RMSE and classification accuracy

reg_alpha	reg RMSE	clf ACC
0	0.4194	0.9065
0.1	0.4170	0.9072
1	0.4124	0.9072
5	0.4094	0.9077
10	0.4139	0.9065
30	0.4508	0.9068
100	0.6578	0.9011

Findings

1. L2 improves regression monotonically into a soft minimum, then gently rises. RMSE falls 0.4232 → 0.4192 → 0.4194 → 0.4140 → 0.4117 → 0.4084 as λ climbs 0 → 30. That's a 3.5% relative improvement over no regularization. Past the sweet spot (λ=100) it only slightly degrades, to 0.4147 — still better than the λ=0 baseline. L2 fails gracefully.

2. L1 improves regression early, then collapses. RMSE drops 0.4194 → 0.4170 → 0.4124 → 0.4094 at α=5 (a 2.4% relative gain), but at α=30 it jumps to 0.4508 (worse than baseline), and at α=100 it explodes to 0.6578 — a +56.8% RMSE disaster. L1 fails catastrophically and without warning if you overshoot.

3. Classification accuracy is nearly immune to both penalties — until α=100. Lambda's accuracy stays in a razor-thin 0.9058–0.9075 band across all seven values. Alpha's accuracy holds 0.9065–0.9077 through α=30, then falls to 0.9011 at α=100. So for a directional (up/down) classifier, regularization is almost a free lunch on the λ side and a mild risk on the α side.

4. λ=30 is the single best regression setting we measured (0.4084 RMSE), edging out the best α setting (α=5, 0.4094 RMSE) by a hair. L2 reaches a slightly lower floor, and it reaches it more safely.

5. α=5 is the best combined single-parameter setting — it posts both the lowest alpha regression RMSE (0.4094) and the highest classification accuracy in either sweep (0.9077). If you were forced to add only one penalty and only one value, α=5 beats λ=30 on accuracy while tying it on RMSE. But note the risk asymmetry in the previous point.

6. The default λ=1, α=0 is not optimal but is safe. It lands at 0.4194 RMSE / 0.9065 accuracy — middle of the pack, no danger. That's exactly what a default should be: a defensible starting line, not a finish line.

7. L2's error floor is lower and more robust than L1's. Even at its worst measured point (λ=100) L2 is still better than baseline. L1's worst point (α=100) is 56% worse than baseline. For a production system where "worst case" actually happens, that asymmetry is the whole story.

Reproducibility (code)

The shapes above are reproducible with a standard XGBoost sweep. The skeleton:

import numpy as np
import xgboost as xgb
from sklearn.model_selection import KFold, cross_val_score

## X, y_reg, y_clf prepared upstream (EOD-audited, walk-forward split in Layer 2)
kf = KFold(n_splits=5, shuffle=True, random_state=42)

def cv_reg_lambda(lam):
    params = {"objective": "reg:squarederror", "reg_lambda": lam,
              "reg_alpha": 0, "tree_method": "hist", "seed": 42}
    model = xgb.XGBRegressor(**params, n_estimators=300)
    # negative MSE -> lower RMSE is better; we report RMSE
    neg_mse = cross_val_score(model, X, y_reg, cv=kf, scoring="neg_mean_squared_error")
    return float(np.sqrt(-neg_mse.mean()))

def cv_reg_alpha(al):
    params = {"objective": "reg:squarederror", "reg_lambda": 1,
              "reg_alpha": al, "tree_method": "hist", "seed": 42}
    model = xgb.XGBRegressor(**params, n_estimators=300)
    neg_mse = cross_val_score(model, X, y_reg, cv=kf, scoring="neg_mean_squared_error")
    return float(np.sqrt(-neg_mse.mean()))

for lam in [0, 0.1, 1, 5, 10, 30, 100]:
    print("lambda", lam, round(cv_reg_lambda(lam), 4))

for al in [0, 0.1, 1, 5, 10, 30, 100]:
    print("alpha", al, round(cv_reg_alpha(al), 4))

Classification ACC swaps in objective="binary:logistic" and scoring="accuracy". The full experiment driver (≈50 min runtime) lives at xgb_articles/_utils/a2_a7_experiment.py; the JSON a2_a7_results.json already carries the observed scores we report here. Run it yourself if you want to watch the α=100 cliff form in real time — it's honestly the most instructive part.

What Failed / Counter-Evidence

The tidy story ("L2 good, L1 risky") has one wrinkle worth respecting: α=5 was the best classification result in the entire study (0.9077), beating every λ value. So L1 isn't "bad" — in a narrow, low-value band it actually squeezes out marginally better signal separation than L2. The danger is purely about range: α's useful band is short and its downside is a cliff, whereas λ's useful band is wide and its downside is a gentle slope.

The other counter-point: neither penalty moved classification accuracy by more than ~0.17% (λ) or ~0.66% (α, excluding the α=100 blowup). If your task is a pure directional classifier, the empirical takeaway is "regularization barely matters here" — and over-tuning it is wasted effort you could spend on max_depth, colsample, or tree_method.

Limitations

• Single-parameter sweeps only. We held one penalty at default while sweeping the other. We did not run a joint λ×α grid, so we can't say what the interaction does — and in XGBoost the penalties additively stack in the objective, so a well-chosen pair (say λ=10, α=1) could beat either alone. That's a real gap; treat "use λ, avoid α" as a safe heuristic, not a proven global optimum.
• One dataset, one CV split seed. The 3.5% / 2.4% gains are real on this data but dataset-dependent. A noisier target would make regularization matter more; a cleaner one, less.
• Regression RMSE and classification ACC are different scales. Don't read the RMSE row and the ACC row as the same units — they're reported separately precisely so you don't.
• n_estimators and learning rate were fixed. With more trees, regularization's relative effect can shift. We isolated the penalty, not the whole pipeline.

Practical Takeaways

1. Start at the default (λ=1, α=0). Then raise λ first. The data says λ=30 is the best single regression setting and it never hurts accuracy. Nudge λ to 5–30 before you ever consider α.
2. If you want sparsity, keep α small (≤5) and watch it like a hawk. α=5 gave our best combined score, but α=30 already hurt regression and α=100 was a catastrophe. L1 is a scalpel, not a wrench.
3. Never grid α past 10 in an automated search without a hard RMSE guardrail. One overshoot and you've zeroed out the leaves that carry your signal.
4. For a directional classifier, don't over-invest in these two. Accuracy barely budged. Spend tuning budget elsewhere.
5. Prefer λ when "worst case must not be catastrophic" is a requirement — which, in a live trading layer, it always is.
6. Validate on the same CV scheme you'll deploy. We use 5-fold in Layer 2's EOD-audited core; mirror that in your own tuning so the numbers mean what you think they mean.

How to Read the Regularization Curve

When you sweep λ or α, don't just hunt for the lowest dot. Read the shape:

• A smooth valley that bottoms and gently rises (λ's RMSE: 0.4232 → 0.4084 → 0.4147) is the signature of a stable, forgiving penalty. Overshooting costs you little. Safe to tune broadly.
• A sharp V with a far edge that falls off a cliff (α's RMSE: 0.4094 → 0.4508 → 0.6578) is the signature of a fragile penalty. The good region is narrow; one step too far and the model breaks. Tune narrowly, guard the boundary.
• A flat line (both penalties vs classification accuracy) means "this knob doesn't turn this dial." Stop wasting grid budget on it.
• Two penalties, one good valley, one cliff? Use the valley-penalty as your default and only flirt with the cliff-penalty inside a tight, guarded range.

The curve shape is the real deliverable of a sweep. The single best number is just where you happened to sample.

The Math: Hessian, Gradients, and Optimal Leaf Weights

This is where XGBoost's regularization becomes concrete. XGBoost builds each tree by minimizing, for a candidate leaf split, an objective expanded to second order:

Obj ≈ Σ_i [g_i·f_t(x_i) + ½·h_i·f_t(x_i)²] + Ω(f_t)

where g_i is the first derivative (gradient) and h_i is the second derivative (Hessian) of the loss with respect to the current prediction, and Ω is the regularization term:

Ω(f_t) = γ·T + ½·λ·Σ w_j² + α·Σ |w_j|

Group terms by leaf j. For leaf j, let G_j = Σ_{i∈leaf j} g_i and H_j = Σ_{i∈leaf j} h_i. The per-leaf contribution is:

Obj_j = G_j·w_j + ½·(H_j + λ)·w_j² + α·|w_j|

Take the derivative w.r.t. w_j and set to zero.

• L2 only (α=0): w_j* = -G_j / (H_j + λ). The λ sits in the denominator, shrinking the optimal leaf weight smoothly toward zero without ever reaching it. This is exactly ridge shrinkage, just applied to leaf outputs. Bigger λ → smaller w_j* → each leaf bets less → model less likely to overfit a noisy leaf.
• L1 + L2 together: w_j* = sign(-G_j) · max(|G_j| − α, 0) / (H_j + λ). The max(|G_j| − α, 0) is a soft-threshold: if the gradient sum |G_j| is smaller than α, the leaf weight is driven exactly to zero — the leaf contributes nothing. That's built-in feature/leaf pruning, the lasso behavior, operating on tree outputs.

Two things follow directly:

1. L2 always shrinks; L1 can delete. That's why α=100 zeroed out enough leaves to wreck RMSE (0.6578) while λ=100 merely shrank them (0.4147, still fine). Deletion removes signal; shrinkage just dampens it.
2. The Hessian H_j scales the effect. Leaves with large Hessian (steep, confident curvature) resist shrinkage; leaves with small Hessian get pushed around more. This is why regularization in gradient boosting is adaptive in a way linear-model ridge/lasso is not — the penalty's bite depends on local curvature.

If you only remember one formula, remember w_j* = -G_j / (H_j + λ). Everything about λ's gentle behavior flows from that denominator.

Ridge vs Lasso, Translated to Trees

The classic stats analogy carries over almost perfectly, with one twist:

• L2 = Ridge. Shrinks all leaf weights a little. Keeps every leaf alive. Smooth, stable, conservative. Great default. Matches our λ curve exactly.
• L1 = Lasso. Drives some leaf weights to exactly zero → those leaves stop firing → automatic pruning / sparsity. Can improve generalization and shrink model size, but only while α stays in its narrow sweet spot. Push too far and you've pruned leaves that mattered. Matches our α curve: nice at α=5, deadly at α=100.
• The twist: in linear models, L1 zeros out coefficients (features). In XGBoost it zeros out leaf output values. The effect — sparsity and implicit feature selection — is similar, but the unit is the leaf, not the column. So when someone says "use α for feature selection in XGBoost," what they really mean is "use α to silence leaves whose gradient sum can't justify their weight."

Production Checklist

Before you ship a model with non-default λ/α into Layer 2:

• [ ] Swept λ ∈ {0, 0.1, 1, 5, 10, 30, 100} at α=0; recorded 5-fold RMSE and ACC.
• [ ] Swept α ∈ {0, 0.1, 1, 5, 10, 30, 100} at λ=1; recorded both scores.
• [ ] Confirmed the (λ=1, α=0) cell matches in both sweeps (consistency check — it should read 0.4194 RMSE).
• [ ] Chose λ in the 5–30 band unless ACC specifically degraded (it didn't, in our data).
• [ ] If using α, capped the search at ≤10 and added a hard RMSE guardrail so α=30/100 can't slip through.
• [ ] Remembered these are leaf-weight penalties, not feature penalties — don't expect them to drop input columns.
• [ ] Re-validated on the same 5-fold scheme Layer 2 deploys; didn't mix holdout and CV numbers.
• [ ] Logged the chosen (λ, α) pair with the run, so walk-forward retrains stay reproducible.

FAQ

Q: Should I use reg_lambda or reg_alpha?
A: Start with reg_lambda (λ). Our CV shows λ improves regression ~3.5% and never hurts accuracy, while α's gains are smaller and its downside is a cliff. Use α only if you want leaf-level sparsity, and keep it ≤5.

Q: What's the XGBoost default?
A: reg_lambda=1, reg_alpha=0. Safe, not optimal. Our best single regression setting was λ=30.

Q: Do λ and α interact?
A: They additively stack in the objective (½λΣw² + αΣ|w|). We swept them one at a time, so the joint optimum (likely a small λ with a small α) is untested here — a known limitation.

Q: Why didn't classification accuracy change much?
A: On this data the directional signal was already well-separated; leaf-weight shrinkage/pruning barely moved the decision boundary. Don't over-tune these for a classifier.

Q: Is α=100 ever usable?
A: Only if you explicitly want aggressive pruning and have guardrails proving RMSE held. In our sweep α=100 gave 0.6578 RMSE — catastrophic. Treat it as a warning, not a setting.

Q: Does this regularize my input features?
A: No. It regularizes leaf weights (tree outputs). For feature-level randomness, that's colsample_* (see the A6 article).

TL;DR

• reg_lambda (L2) shrinks leaf weights smoothly: regression RMSE 0.4232 → 0.4084 at λ=30, and still safe at λ=100 (0.4147).
• reg_alpha (L1) can zero out leaves: best at α=5 (0.4094 RMSE, 0.9077 ACC) but collapses to 0.6578 RMSE at α=100.
• Classification accuracy barely reacts to either until α=100 (0.9011).
• Default λ=1, α=0 is safe but suboptimal. Raise λ first; treat α as a guarded scalpel.
• These are leaf-weight penalties, applied additively in the objective; the math is w* = -G/(H+λ) for L2 and a soft-threshold for L1.

📚 Related Articles

• A4 Gamma
• A2 Max Depth
• A6 Colsample

👤 About the Author

Name:

Shakti Tiwari

Role:

Nifty Option Trader, XGBoost Expert

Cert:

NISM-Series-XII (Securities Markets Foundation)

Knows:

XGBoost · LightGBM · Options Trading · Machine Learning · Walk-forward Validation

Profiles:
about.me ·
optiontradingwithai.in ·
github ·
whatsapp ·
x/twitter

Sources

• dmlc/xgboost official repository — github.com/dmlc/xgboost (~28.7k stars). Parameter semantics for reg_lambda / reg_alpha per doc/parameter.rst and tuning guidance per doc/tutorials/param_tuning.rst.
• Observed 5-fold CV scores: xgb_articles/_utils/a2_a7_results.json (keys A7_lambda_reg, A7_lambda_clf, A7_alpha_reg, A7_alpha_clf).
• Reproduction driver: xgb_articles/_utils/a2_a7_experiment.py.

Author / Canonical

Written by Shakti Tiwari — Nifty Option Trader, XGBoost Expert.
Canonical: optiontradingwithai.in
This article is educational only. Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst.

Resources & Links

• About the author: https://about.me/shaktitiwari
• Main site: https://optiontradingwithai.in
• WhatsApp: https://wa.me/919169650895
• Previous in series (A6 — colsample_*): https://optiontradingwithai.in/xgboost-colsample
• Next in series (B1 — tree_method): https://optiontradingwithai.in/xgboost-tree-method
• Books by Shakti Tiwari: https://www.amazon.in/dp/B0H9ZNTBPK and https://www.amazon.in/dp/B0HBBFKDQF