XGBoost colsample_bytree / colsample_bylevel / colsample_bynode: The Column-Sampling Trio Explained with 5-Fold CV Numbers

XGBoost colsample_bytree / colsample_bylevel / colsample_bynode: The Column-Sampling Trio

Quick Answer: colsample_bytree, colsample_bylevel, and colsample_bynode are XGBoost's three feature-sampling (column-sampling) knobs. All three default to 1.0 (use every feature). colsample_bytree samples a fraction of columns once per tree; colsample_bylevel re-samples per level (depth) inside the tree; colsample_bynode re-samples per split (node). They multiply together, so the columns actually seen at a split = bytree × bylevel × bynode × total_features. In our 5-fold test, dropping columns helped classification (peak ACC 0.9082 at 0.7) but hurt regression (best RMSE 0.4093 at the full 1.0).

Disclaimer: Shakti Tiwari is NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Why This Matters

If subsample is XGBoost's row-bagging (see the previous article, A5), then the colsample_* trio is its feature-bagging — the exact idea that makes Random Forests work. Where subsample asks "which training rows do I look at this round?", the colsample_* family asks "which features am I even allowed to split on at this moment?"

Most beginners only ever touch colsample_bytree, because that is the one every tutorial mentions. But XGBoost actually gives you three distinct scopes of column sampling, and they behave very differently:

• colsample_bytree — samples once when a brand-new tree begins. Every split in that tree draws from the same reduced feature pool.
• colsample_bylevel — samples again at every depth level. So level 0 (root) might see features {A, C, F}, level 1 sees {B, D, G}, level 2 sees {A, E, H}, and so on. The pool is refreshed as the tree gets deeper.
• colsample_bynode — the most aggressive. It samples fresh at every single node, i.e. at every individual split decision.

This is the single most misunderstood corner of XGBoost's parameter docs, and I have seen production trading models silently mis-tuned because someone set all three to 0.8 and got an effective sampling of 0.8³ = 0.512 without realizing it. We will unpack that multiplicative trap below, because it is the headline lesson of this article.

Two-layer engine note (where this parameter lives in our stack): Our live system is TWO-LAYER: Layer 1 = Dhan WebSocket live capture (shadow/predict-only, never trades without broker auth); Layer 2 = EOD-audited XGBoost/LightGBM training core (walk-forward, cost-and-slippage model). The parameter discussed here — colsample_bytree, and by extension its siblings — is tuned in Layer 2, the EOD-audited training core, never in the live shadow feed.

The Random Forest analogy (so this sticks)

In a Random Forest, each tree is grown on a bootstrap of rows and a random subset of features at every split. That feature randomness is what decorrelates the trees and is the main reason RF generalizes well. XGBoost is a boosting algorithm (trees are sequential, not independent), so naive feature sampling is less central — but it still works as a regularizer. colsample_bytree is the closest analogue to RF's max_features: it gives each tree a different, limited view of the world. colsample_bylevel and colsample_bynode go further than classic RF by re-randomizing deeper inside each tree, which can squeeze out even more diversity (and, if overdone, more noise).

Primary source

All three parameters are documented in the official dmlc/xgboost repository (github.com/dmlc/xgboost, ~28.7k stars) under doc/parameter.rst and the tuning guide doc/tutorials/param_tuning.rst. The key doc quote, paraphrased faithfully: subsampling of columns happens once per tree (colsample_bytree), once per level (colsample_bylevel), and once per node (colsample_bynode), and the three ratios compose multiplicatively. That is the canonical statement we build on.

Research Question / Hypothesis

Hypothesis: Reducing colsample_bytree below 1.0 decorrelates trees and should improve out-of-sample generalization — i.e. lower regression RMSE and higher classification accuracy — up to a point, after which too few features starve the model and performance collapses.

Test design: Sweep colsample_bytree ∈ {0.3, 0.5, 0.7, 0.8, 0.9, 1.0} holding colsample_bylevel = colsample_bynode = 1.0 (so we isolate the bytree effect first). We run both a regression (reg:squarederror) and a classification (binary:logistic) objective, because feature-sampling can affect the two tasks differently — and as you will see, it does.

The two sibling parameters are explained conceptually after the experiment, since the measured numbers come from colsample_bytree alone.

Data & Methodology Box

• Data: Synthetic, deterministic (seed=42). Regression: 20,000 rows × 24 features. Classification: 20,000 rows × 24 features. Signal is moderately dense (multiple features carry real signal), which matters for the interpretation below.
• CV: 5-fold cross-validation, fold seed = 0. Reported numbers are out-of-sample (OOS) on the held-out fold each time.
• Model: gbtree booster. eta=0.1, max_depth=4, subsample=1.0 (we are isolating column sampling, not row sampling), colsample_bylevel=1.0, colsample_bynode=1.0. 500 boosting rounds with early stopping (patience 30) on a validation fold.
• Metrics: Regression → OOS RMSE (lower is better, ↓). Classification → OOS accuracy (higher is better, ↑).
• Reproducible: the exact run lives in xgb_articles/_utils/a2_a7_experiment.py; the cached results are in xgb_articles/_utils/a2_a7_results.json (keys A6_col_reg and A6_col_clf).

Results

Primary Table — colsample_bytree vs OOS (bylevel = bynode = 1.0)

colsample_bytree	Regression OOS RMSE ↓	Classification OOS ACC ↑
0.3	0.6010	0.9057
0.5	0.4288	0.9066
0.7	0.4144	0.9082 (peak)
0.8	0.4194	0.9065
0.9	0.4134	0.9072
1.0 (default)	0.4093 (best)	0.9071

Findings

1. Regression wants ALL the columns. RMSE falls monotonically (with tiny wiggles) as colsample_bytree rises: 0.6010 → 0.4288 → 0.4144 → 0.4194 → 0.4134 → 0.4093 at the default 1.0. Dropping features hurt the regression task at every step. The 0.3 setting is a disaster — 0.6010 is ~47% worse than 0.4093. With a dense signal spread across many features, hiding columns just discards useful information.

2. Classification has a genuine sweet spot at 0.7. Accuracy climbs from 0.9057 (0.3) to a peak of 0.9082 at 0.7, then dips at 0.8 (0.9065), recovers at 0.9 (0.9072), and sits at 0.9071 for the default 1.0. The gain from 1.0 → 0.7 is +0.0011 accuracy, modest but real and consistent with the "feature sampling reduces overfit" theory for a classification boundary.

3. The two tasks disagree — and that is the lesson. Same data family, same depth, same rows; only the objective differs. Regression improves with more features, classification with fewer. This is exactly why you must tune colsample_bytree per objective and per dataset rather than copying a forum's "0.8 works great" value.

4. The default 1.0 is not automatically best, but it is rarely catastrophic. For classification, 1.0 (0.9071) is within 0.0011 of the peak — so leaving it at default costs almost nothing here. For regression, 1.0 is the best. The danger zone is the low end (0.3), not the top.

5. The 0.8 "magic number" myth takes a hit. A lot of blog posts claim "set colsample_bytree=0.8" as a universal regularizer. In our regression run, 0.8 (0.4194) is actually worse than both 0.7 (0.4144) and 0.9 (0.4134), and far worse than 1.0. The number is dataset-specific, not sacred.

6. Starvation is asymmetric and sudden. Dropping from 0.5 to 0.3 in regression blows RMSE up from 0.4288 to 0.6010 — a 40% jump for a 0.2 change in the ratio. Feature starvation has a cliff, not a slope. In production you feel it as a model that suddenly "forgets" how to price.

How to read this curve (so you can do it on your own data)

Plot RMSE and ACC against the colsample_bytree value. You are looking for one of three shapes:

• Monotonic-down (regression here): more features = better; keep it near 1.0, maybe 0.9 if you want a hair of regularization.
• Hump (classification here): rises to a peak then falls — that peak (0.7) is your tuning target.
• U-shape: too few and too many both hurt; the bottom of the U is the sweet spot.

Do not just read the table row-by-row. Read the shape. The shape tells you whether feature sampling is helping at all. If the curve is flat across the whole sweep, colsample_bytree is not your lever — go tune max_depth or subsample instead.

Reproducibility

import xgboost as xgb
from sklearn.model_selection import KFold
import numpy as np

## Load your X (n_samples, 24) and y
## dtrain = xgb.DMatrix(X, label=y)

def cv_colsample(colsample, objective, num_boost=500):
    kf = KFold(n_splits=5, shuffle=True, random_state=0)
    scores = []
    for tr, te in kf.split(X):
        p = dict(
            max_depth=4,
            eta=0.1,
            subsample=1.0,            # isolating COLUMN sampling
            colsample_bytree=colsample,
            colsample_bylevel=1.0,   # siblings held at default
            colsample_bynode=1.0,
            objective=objective,      # 'reg:squarederror' or 'binary:logistic'
            verbosity=0,
        )
        # early-stop on a validation slice of tr
        bst = xgb.train(p, dtrain_sub, num_boost)
        pred = bst.predict(dtest_sub)
        # RMSE for reg, ACC for clf
        scores.append(score(pred, y_te))
    return np.mean(scores)

for c in [0.3, 0.5, 0.7, 0.8, 0.9, 1.0]:
    print(c, cv_colsample(c, 'reg:squarederror'),
             cv_colsample(c, 'binary:logistic'))

To extend the experiment to the siblings, simply vary colsample_bylevel or colsample_bynode instead (or in addition) while holding the others at 1.0, and watch how the effective sampling compounds.

What Failed / Counter-Evidence

• We expected colsample_bytree < 1 to help regression too. Theory said "feature sampling = regularization = better OOS." Reality: regression RMSE got worse as we dropped columns, all the way to the default. The regularization benefit only showed up for classification. This is the most important counter-intuitive result and the reason we report both objectives rather than hiding the regression line.
• The 0.8 folklore did not hold. Several tuning guides treat 0.8 as a safe regularizer. Our numbers show 0.8 is a local dip for classification (0.9065, below the 0.7 peak and even below 0.9) and strictly worse than 1.0 for regression.
• No free lunch from stacking all three. Conceptually, setting bytree=bylevel=bynode=0.8 gives an effective 0.512 — we did not measure this in the sweep, but the regression 0.3 result (effective starvation) is a strong warning that triple-sampling would likely crater both tasks on this dataset.

Limitations

• Synthetic data. The dense-signal synthetic set is a clean teaching environment. Real, sparse financial feature sets (e.g. order-flow fragments, option Greeks) may show a stronger regularization benefit from column sampling, because noise features dominate and hiding them helps.
• Single depth / single eta. We fixed max_depth=4, eta=0.1. At deeper trees, per-node sampling (colsample_bynode) becomes more impactful because there are more splits to randomize. Our sweep isolates bytree only; the siblings are characterized conceptually.
• One random seed family. 5-fold CV with seed 0 is robust for the trend, but a single dataset does not prove the regression-vs-classification split is universal. Treat it as a strong hypothesis to re-test on your own data.

Practical Takeaways (Production Checklist)

1. Isolate first. Tune colsample_bytree with bylevel = bynode = 1.0. Only after you understand the bytree curve should you touch the siblings.
2. Sweep, don't guess. Always run the {0.3, 0.5, 0.7, 0.8, 0.9, 1.0} grid and read the shape (monotonic-down / hump / U). Our regression case proves the default 1.0 can be the winner.
3. Mind the multiplicative trap. Effective sampling = bytree × bylevel × bynode. If you set all three to 0.8 you are at 0.512, not 0.8. Log the product in your tuning tracker.
4. Per-objective tuning. If your pipeline trains both a regressor and a classifier (common in an options stack: a regression for a continuous target, a classifier for a direction call), tune colsample_* separately for each. One size fits none here.
5. Use siblings for stubborn overfit. If colsample_bytree alone cannot buy you enough generalization, escalate to colsample_bylevel (re-randomize per depth) before colsample_bynode (re-randomize per split). bynode is the strongest and noisiest; save it for last.
6. Pair with subsample (A5). Row + column sampling compose into a powerful regularizer combo — but tune them on separate grids so you can attribute gains.
7. This lives in Layer 2. Remember the two-layer note: these are EOD training-core settings, validated by walk-forward with a cost-and-slippage model. The Dhan WebSocket shadow feed only consumes the trained model; it never re-samples features live.

The sibling explainer (bytree done, now bylevel and bynode)

Now that the measured effect of colsample_bytree is clear, here is how the other two fit:

• colsample_bylevel (default 1.0): Re-samples the column pool at each depth level of the tree. Think of bytree as choosing which books are in the library for the whole tree; bylevel re-stocks the shelf every floor you walk down. It adds diversity between shallow and deep splits without starving any single tree as hard as a low bytree would.
• colsample_bynode (default 1.0): The finest grain — a fresh column sample at every split node. This is more random than classic Random Forest (which samples per split too, but RF has no deeper re-sampling). It maximizes tree diversity and is the strongest regularizer of the three, but also the most likely to inject noise, because a split that should have used feature X may never see it.
• Composition: Because each stage samples from what the previous stage already narrowed, the columns actually eligible at a given split = colsample_bytree × colsample_bylevel × colsample_bynode of the original feature count. A tree built with (0.9, 0.9, 0.9) effectively sees 0.729 of features at each split — not 0.9. This is why aggressive triple-sampling can quietly starve a model the way our bytree=0.3 regression run did.

A good mental model: bytree = "which features may this whole tree use?", bylevel = "which features may this depth use?", bynode = "which features may this exact split use?" Each question narrows the candidate set further, and the product is what the split algorithm actually optimizes over.

FAQ

Q: What is the difference between subsample and colsample_bytree?
A: subsample samples rows (training examples) each boosting round — bagging over data. colsample_bytree samples columns (features) for each tree — bagging over inputs. They attack overfitting from two orthogonal directions and compose beautifully together. (See A5 for subsample.)

Q: What are the defaults?
A: All three colsample_* parameters default to 1.0 — i.e. no column sampling, every feature is always eligible. The regularization only kicks in when you drop them below 1.0.

Q: Should I use bytree, bylevel, or bynode?
A: Start with colsample_bytree — it is the coarsest and most interpretable. If you still overfit, try colsample_bylevel next. Reserve colsample_bynode for stubborn cases because it is the noisiest. And never forget they multiply.

Q: Why did regression get worse when I dropped columns?
A: With a dense signal spread across many features, hiding columns just throws away real information — there is no noise feature to "drown out." Classification, by contrast, benefits from the decorrelation because its decision boundary is more sensitive to any single dominant feature dominating every tree. Always check the curve shape for your own objective.

Q: Which layer of our system tunes this?
A: The EOD-audited training core (Layer 2). The Dhan WebSocket shadow feed (Layer 1) only runs inference on the already-trained model.

TL;DR

colsample_bytree, colsample_bylevel, and colsample_bynode are XGBoost's three feature-sampling knobs, all defaulting to 1.0 and multiplying together. In our 5-fold test, colsample_bytree hurt regression (best RMSE 0.4093 at 1.0) but helped classification (peak ACC 0.9082 at 0.7). Sweep it per-objective, watch the curve shape, and respect the multiplicative trap.

📚 Related Articles

• A5 Subsample
• A4 Gamma
• A7 Lambda Alpha

👤 About the Author

Name:

Shakti Tiwari

Role:

Nifty Option Trader, XGBoost Expert

Cert:

NISM-Series-XII (Securities Markets Foundation)

Knows:

XGBoost · LightGBM · Options Trading · Machine Learning · Walk-forward Validation

Profiles:
about.me ·
optiontradingwithai.in ·
github ·
whatsapp ·
x/twitter

Sources

• dmlc/xgboost repository — doc/parameter.rst (colsample_bytree / colsample_bylevel / colsample_bynode definitions; multiplicative composition note). github.com/dmlc/xgboost (~28.7k stars).
• dmlc/xgboost — doc/tutorials/param_tuning.rst (column subsampling as a regularization strategy).
• Experiment source: xgb_articles/_utils/a2_a7_experiment.py; cached results xgb_articles/_utils/a2_a7_results.json (keys A6_col_reg, A6_col_clf).

Author / Canonical Attribution

Shakti Tiwari — Nifty Option Trader, XGBoost Expert. NISM-Series-XII certified; not a SEBI-registered Research Analyst. Content is educational only.

Resources & Links

• About: https://about.me/shaktitiwari
• Site: https://optiontradingwithai.in
• WhatsApp: https://wa.me/919169650895
• Prev: A5 subsample | Next: A7 lambda / alpha
• Books: Option Trading with AI (B0H9ZNTBPK) | The AI Opportunity (B0HBBFKDQF)