📌 Full profile & books: https://optiontradingwithai.in/about | Option Trading with AI (B0H9ZNTBPK) · The AI Opportunity (B0HBBFKDQF)

XGBoost for Options Trading — Feature Engineering That Actually Moves the Model

Educational only — not SEBI-registered investment advice. Past model performance does not guarantee future results.

Most trading-AI posts show you model.fit(X, y) and stop. The model was never your problem. Your features were. Here's what actually moves XGBoost on options data — and why it beats the neural-net hype for this exact problem.

Why XGBoost, not a neural net

The largest study to date — McElfresh et al., "When Do Neural Nets Outperform Boosted Trees on Tabular Data?" (arXiv:2305.02997, NeurIPS 2023) — compared 19 algorithms across 176 datasets. The honest finding: the NN-vs-GBDT gap is often small and dataset-dependent, and a specialized net (TabPFN) actually tops the average on small training sets. But — and this is the part that matters for market data — GBDTs decisively beat neural nets on skewed, heavy-tailed, and irregular feature distributions. That describes options data almost perfectly: fat-tailed returns, sharp thresholds on individual columns. A tree splits one column at a time, drawing clean decision boundaries; a neural net has to learn those thresholds geometrically and usually drowns in the noise.

The geometry argument: real market features (PCR crossing 1.2, depth imbalance flipping sign) are step changes, not smooth manifolds. Trees capture steps natively. Save neural nets for images and text.

One honesty note: the "XGBoost beats neural nets" claim is real but context-bound. Grinsztajn (2022) showed trees still win on tabular data, but on enough data a well-tuned net can close the gap — and the difference is often inside statistical noise (Gu-Kelly-Xiu 2020: NN edge over trees not always significant). For options tabular features specifically, gradient boosting is the pragmatic winner. Don't read it as "DL is useless" — read it as "right tool for this data shape."

Practical proof: a fintech team burned $13K on a tabular neural architecture and their tuned XGBoost caught fraud patterns the net missed entirely (Medium, Dec 2025). Same lesson applies to options.

The feature set that matters

I run XGBoost on NIFTY options. The model sees none of the raw price — only engineered structure:

Group	Features	Why
Order-flow	delta_oi_zscore, depth_imbalance, vol_z	Real buying/selling pressure, not lagging price
Option-chain	pcr, pcr_zscore, max_pain_gap, iv_skew	Where money is positioned before move
Volatility	atr_z, rv_20d, iv_rank	Regime context
Time	minutes_to_expiry, dte_bucket	Decay pressure

The key insight: delta_OI (change in open interest) beats absolute OI. Absolute OI includes stale positions; delta reveals fresh positioning. That single feature separation is what most "AI trading" scripts miss.

Hyperparameters that don't overfit

XGBoost docs + Kaggle tuning guides converge on a stable range:

params = {
    'max_depth': 4,        # 3–6; shallow trees = stable interactions
    'learning_rate': 0.05,  # 0.01–0.1; lower = better generalization
    'subsample': 0.8,       # 0.5–1; <1 prevents overfit
    'colsample_bytree': 0.7,
    'n_estimators': 400,    # more trees when lr is low
    'eval_metric': 'logloss'
}

The overfitting tell (from a controlled depth test): at max_depth=10, training F1 hit 0.21 but test F1 stalled at 0.10 — a widening gap. At max_depth=4, both stayed close. Shallow trees + low LR = the only config that generalizes on noisy market data.

What I got wrong

1. Feeding price as a feature — it lags, the model learned nothing.
2. max_depth=10 — memorized training noise, died live.
3. Absolute OI instead of delta — stale signal.
4. No walk-forward — a single random split lied about performance.

Blueprint over fit

The model scores structure. The blueprint gates the trade: edge must clear a threshold AND drawdown cap must be open. XGBoost finds the edge; discipline keeps you alive.

---

Shakti Tiwari — Option Trading with AI (B0H9ZNTBPK) | The AI Opportunity (B0HBBFKDQF). Building real systems, not screenshots.