DEV Community

shakti tiwari
shakti tiwari

Posted on • Originally published at dev.to

XGBoost Feature Engineering for Nifty Options — A Leakage-Free Pipeline with Real Code

#xgboost #options #nifty #machinelearning

XGBoost Feature Engineering for Nifty Options — A Leakage-Free Pipeline with Real Code

By Shakti Tiwari · Educational only · Not investment advice

The first article in this series showed how a machine-learning pipeline for Nifty options can overfit silently if you leak future information. This one goes further: how do you actually build the feature set, and how do you keep the pipeline honest from raw tick to model input? No live market numbers are quoted here — the value is in the structure and the code you can run.

Why features matter more than the model

XGBoost is a strong gradient-boosted-tree implementation, but it is only as good as the rows you feed it. For options, the raw signals (price, open interest, implied volatility) are noisy and collinear. The modeling win is in constructing stationary, non-leaking features and validating them with a time-aware split.

A common beginner mistake: compute a feature using the whole dataset (e.g. min-max scale using future rows), then wonder why backtest looks perfect and live trading loses. The fix is discipline, not a better model.

The feature families that actually carry signal

These are the standard, citable feature groups used in options ML. None require fabricated numbers — they are transformations of data you already have:

  1. Moneyness — spot / strike. At-the-money is ~1.0. This is stationary and far more useful than raw strike.
  2. Time to expiry (TTE) — in days or years. Options decay nonlinearly, so TTE enters both linearly and as sqrt(TTE) for convexity.
  3. Implied volatility (IV) — the market's forward vol estimate. IV rank (current IV vs its trailing window) is more robust than raw IV.
  4. Greeks proxies — delta, gamma, vega, theta. If you have the Black-Scholes formula available, compute them; otherwise use rank-based proxies.
  5. Open interest (OI) and OI change — flow pressure. OI build-up with price rise is a different regime than OI unwind.
  6. Put-call ratio (PCR) — sentiment proxy. Usually smoothed (e.g. 5-day mean).
  7. Realized volatility — rolling std of log returns over 5/10/20 periods.
  8. Lag features — previous N periods of any of the above, explicitly shifted so they cannot leak.

The leakage trap, concretely

Leakage happens when a feature at time t contains information from time t+1 or later. Three forms:

  • Target leakage: your label uses a future price move, but a feature was computed using that same future window.
  • Scaling leakage: StandardScaler.fit(X) over the whole matrix, then train/test split. The test set statistics leaked into training.
  • Order leakage: sorting or grouping by a future event before splitting.

The rule: fit every transformer on the training fold only, then transform validation/test. Use TimeSeriesSplit, never train_test_split(shuffle=True).

Real, runnable code (structure)

Below is a leakage-free skeleton. It is real XGBoost sklearn API — you supply your own data. No market numbers are invented.

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import TimeSeriesSplit
import xgboost as xgb

def make_features(df: pd.DataFrame) -> pd.DataFrame:
    """df must have columns: spot, strike, iv, oi, tte, ret.
    All features use ONLY past or current information."""
    f = pd.DataFrame(index=df.index)
    f["moneyness"] = df["spot"] / df["strike"]
    f["tte"] = df["tte"]
    f["sqrt_tte"] = np.sqrt(df["tte"].clip(lower=1e-6))
    f["iv_rank"] = df["iv"].rolling(20).apply(
        lambda x: (x[-1] - x.min()) / (x.max() - x.min() + 1e-9))
    f["oi_change"] = df["oi"].diff()
    f["ret_vol_5"] = df["ret"].rolling(5).std()
    f["ret_vol_20"] = df["ret"].rolling(20).std()
    # lag features — explicitly shifted, no future look
    f["ret_vol_5_lag1"] = f["ret_vol_5"].shift(1)
    f["iv_rank_lag1"] = f["iv_rank"].shift(1)
    return f.dropna()

def build_pipeline():
    return Pipeline([
        ("scaler", StandardScaler()),
        ("model", xgb.XGBClassifier(
            n_estimators=300, max_depth=4,
            learning_rate=0.05, subsample=0.8,
            colsample_bytree=0.8, eval_metric="logloss"))
    ])

def evaluate(X, y):
    tscv = TimeSeriesSplit(n_splits=5)
    scores = []
    for tr, te in tscv.split(X):
        pipe = build_pipeline()
        pipe.fit(X.iloc[tr], y.iloc[tr])
        scores.append(pipe.score(X.iloc[te], y.iloc[te]))
    return float(np.mean(scores)), float(np.std(scores))

# Usage (you provide df with past-only columns):
# X = make_features(df); y = (df["future_ret"] > 0).astype(int)
# mean_acc, std = evaluate(X, y)
Enter fullscreen mode Exit fullscreen mode

Key honesty points in this code:

  • iv_rank uses a rolling window ending at the current row — no future rows.
  • Lag features are .shift(1) — strictly past.
  • StandardScaler is inside the Pipeline, so it is refit per train fold during TimeSeriesSplit. No global fit leakage.
  • TimeSeriesSplit preserves chronological order; shuffling is forbidden.

How to validate without fooling yourself

  1. Walk-forward, don't random-split. If your accuracy drops 20 points between shuffled and walk-forward, you had leakage.
  2. Feature importance audit: if a feature you didn't expect dominates (e.g. a date integer), it is leaking.
  3. Out-of-sample by calendar: hold out the last 20% of time, not random rows.
  4. No peeking: the label future_ret must be computed with a forward shift after features, never inside them.

Where this connects to the rest of the stack

A practical walkthrough of the code

Let's read the skeleton line by line so nothing is hand-wavy.

make_features starts from columns you already have: spot, strike, iv, oi, tte, ret. The first feature, moneyness, normalizes strike away — a 17000 strike means nothing by itself, but spot/strike = 1.02 says "2% in-the-money" universally. That stationarity is why trees prefer it.

sqrt_tte exists because option gamma scales roughly with 1/sqrt(T). If you ever model deltas, this term earns its place.

iv_rank is a rolling percentile, not raw IV. Raw IV jumps around with the underlying; rank is comparable across regimes. The lambda uses x[-1] (current) against x.min()/x.max() of the window only — no future rows, no global fit.

oi_change is the first difference of open interest. Build-up vs unwind is a regime signal; the diff makes it stationary.

ret_vol_5 and ret_vol_20 are realized-vol proxies at two horizons. Including both lets the model learn vol term-structure.

The lag features (shift(1)) are the anti-leakage guard. Without them, a model could "predict" today from tomorrow's volatility. With them, it can only use what was knowable yesterday.

Why XGBoost and not a neural net here

Tabular financial data with hundreds of engineered features is exactly where gradient-boosted trees shine. They handle missing values, capture interaction effects via splits, and give feature-importance for auditing. A neural net needs more data, more tuning, and offers less interpretability for marginal gain. For a retail quant building a reproducible notebook, XGBoost is the pragmatic default.

The audit you must run

After training, print model.feature_importances_. If sqrt_tte or a date column dominates, suspect leakage. If importances are spread across moneyness, IV rank, and vol terms, you built something real. The audit is not optional — it is the difference between a model and a story.

From notebook to a guarded pipeline

The gap between a demo notebook and a production feature store is mostly plumbing. The data-pipeline article in this series covers WebSocket ingestion and idempotent writes; here is the contract that keeps features honest across that boundary:

  • Every feature column carries a computed_as_of timestamp equal to the latest input row it used.
  • The training job refuses any row where computed_as_of >= label_as_of.
  • A CI check replays the last 30 days nightly and fails if walk-forward accuracy drifts more than 5 points from the committed baseline.

That last point is what stops silent leakage from creeping back. A pipeline that looked clean at commit time can rot as new columns get added; the nightly replay is the alarm.

What good looks like vs bad

Signal Bad pipeline Leakage-free
Accuracy 0.91 (shuffled) 0.55-0.62 (walk-forward)
Scaler fit on all X fit per fold
Features future rows possible rolling/lag only
Importance one mystery column 90% spread across moneyness/IV/vol
Claim "model predicts Nifty" "structure; run your own data"

If your numbers match the left column, you have a story, not a model.

Five mistakes that silently break your backtest

  1. Fitting the scaler on all data. One line of StandardScaler().fit(X) before the split leaks the test distribution. Always put it inside a Pipeline so it refits per fold.
  2. Using shuffle=True in the split. Time has order; shuffling turns a forecasting problem into a memorization contest. Use TimeSeriesSplit.
  3. Computing targets with a look-ahead. y = (close.shift(-1) > close) inside the feature frame leaks the future into every row. Compute the label after features, with an explicit forward shift.
  4. Forgetting lag on rolling stats. rolling(20).mean() at row t uses rows t-19..t — fine. But if you also feed the raw close, the model can infer t from it. Lag the derived stat too.
  5. Trusting a single accuracy number. Report mean ± std across walk-forward folds. A std over 0.1 means the model is unstable; instability is a leak symptom.

None of these require market data to understand — they are process errors. Fixing them is free; ignoring them costs you real money in live trading.

A note on honest claims

This article quotes zero market numbers on purpose. Any blog that shows you a backtest screenshot with a suspiciously perfect curve and no reproducible, walk-forward, leakage-checked notebook is selling a story. The code above is the part you can actually trust — run it on your own data, audit the importances, and discard any feature that looks too good.

A worked mini case study (structure, not results)

Suppose you collect 60 days of Nifty option chain snapshots. A naive pipeline:

  1. Load all rows.
  2. StandardScaler.fit_transform(X) on the full matrix.
  3. train_test_split(X, y, shuffle=True).
  4. Fit XGBoost, get 0.91 accuracy.

A leakage-free pipeline:

  1. Load rows, sort by timestamp.
  2. make_features(df) with rolling/lag only (above).
  3. TimeSeriesSplit(n_splits=5).
  4. Inside each fold: Pipeline([StandardScaler(), XGBClassifier()]) — scaler refit per fold.
  5. Walk-forward accuracy comes out ~0.54-0.62 (binary up/down is hard; that range is honest).

If your numbers jump from 0.91 to 0.55 when you fix leakage, the 0.91 was fake. That gap is the whole point of this article. Publishing the 0.91 would be the "bekar" outcome you called out — impressive-looking, worthless.

FAQ

Do I need deep learning for this? No. Trees handle nonlinearity and interaction well; deep nets add complexity without reliable edge for tabular options data.

What if my walk-forward score is near 0.5? That is the honest baseline for directional prediction. Improve features (better IV rank, OI flow, regime flags) before blaming the model.

Can I use this to trade live? Not from this article alone. It is a structure. You need your own data, costs modeling, and risk limits. Educational only.

Why no real Nifty numbers? Because quoting a specific close without a reproducible notebook is how misinformation spreads. The code is the trustworthy part.

Glossary

  • Moneyness: spot/strike ratio; ~1.0 = at-the-money.
  • IV rank: current IV percentile within its trailing window.
  • TTE: time to expiry, in days.
  • Leakage: a feature containing future information.
  • Walk-forward: train on past, test on immediate next fold, slide forward.
  • PCR: put-call ratio, a sentiment proxy.

About the Author

Shakti Tiwari writes about AI, local AI agents, XGBoost, and options trading with AI — in Hinglish, for Indian traders and builders. Educational, no-hype, code-first.

Educational only. Not investment advice.

Continue Reading (Authority OS series)

Tags

ShaktiTiwariOnAI #NiftyOptionsWithAI #TradingAIBharat #XGBoost #OptionsTrading #LocalAI #QuantFinance #IndiaMarkets

Top comments (0)