Nifty Options + XGBoost: A Leakage-Free ML Pipeline (No Overfitting, No Promises)
By Shakti Tiwari · Educational only · Not investment advice
Two engineering habits decide whether an ML options project teaches you something or quietly lies to you: (1) your data pipeline must not duplicate or reorder ticks, and (2) your model must never see the future it is trying to predict. This piece connects those two ideas into one reproducible pipeline — and is honest about what it does not give you.
Why this matters for Nifty options specifically
Nifty weekly expiry creates a repeating, high-frequency label surface: every Thursday you get a fresh target (direction, max pain, IV move). That cadence is catnip for leakage. A notebook that "predicts" Thursday's move with 95% accuracy usually just learned to read Thursday's close from a feature computed after Thursday's close. The fix is structural, not parametric.
Step 1 — Trustworthy data (idempotent ingest)
Before any model, your feature store must be correct. A WebSocket reconnect that resends a trailing window will inject duplicates that look like "more samples" but are the same ticks. Use a content-hash + per-stream monotonic sequence-id dedup. The test that matters: accepted_total == distinct_ticks under injected reconnects. If that fails, every number downstream is fiction. A concrete symptom: you train on what looks like 50,000 ticks but 8,000 are duplicates from three reconnects. Your model learns the reconnect pattern, not the market. You will not notice until live.
Step 2 — Strict time ordering (kill leakage)
Define the decision at time t using only data available at t: features (spot, IV, Greeks proxies, volume, time-of-day, rolling stats) all computed ≤ t; label is the forward move t → t+h — the only future you may see, never as a feature. Never shuffle timestamps. Walk-forward: train [0,k], validate [k,k+v], roll, repeat. Common leak patterns to grep for: a feature column named next_close, future_iv, t_plus_1, target_shift; train_test_split(..., shuffle=True) on time-indexed rows; a rolling mean with a future-looking window; the label column included in features. The leakage checker tool scans for exactly these.
Step 3 — Report the worst fold, not the best
A defensible study reports the distribution of out-of-sample performance across folds and highlights the worst-regime fold. If the worst fold is flat or negative, the model is a lesson, not a strategy. Why the worst fold matters: a model that makes money in trending 2023 but loses it all in rangebound 2024 has an average that looks fine and a reality that is brutal. The average hides regime dependence; the worst fold exposes it.
Step 4 — Calibration, not just accuracy
Accuracy on a 90/10 up/down split is meaningless if the base rate is 90/10. Report ROC-AUC and PR-AUC, Brier score for probability calibration, and a confusion matrix per regime fold. A model with 92% accuracy that is just predicting "up" because markets go up is worthless. Calibration tells you whether predicted probabilities mean what you think.
What XGBoost is actually good at here
XGBoost is a gradient-boosted tree model. For tabular market features it tends to beat linear models when the signal is non-linear and interaction-heavy. Its real value in a trading context is three things: forcing feature discipline (you must define inputs explicitly, killing the "I just felt it" trap); walk-forward honesty (a proper backtest with rolling retrain exposes overfitting a single split hides); calibration checks (measure whether predicted probabilities mean what you think).
What it is NOT
It is not a profit button. Past accuracy on Nifty data says nothing about next expiry. A model trained on one regime degrades the moment the regime shifts. This is the single most common failure, and the most hidden.
A worked leakage example
Suppose at time t you build a feature iv_next = df['iv'].shift(-1) (next expiry's IV) and set label = sign(close[t+5] - close[t]). The feature iv_next is computed from data at t+1, which is in the future relative to t. The model trains on iv_next and "predicts" the label with suspicious ease — because both look at the future. This is the classic look-ahead. The fix: every feature must be computable using only data available at t. If you cannot compute it at t in live trading, it is a leak.
An honest experiment design (not a result)
A defensible study: define the decision at time t using only data ≤ t; label using the forward move from t to t+h (the only future you're allowed to see is the label, never as a feature); walk-forward train/validate/roll; report the distribution of out-of-sample performance across folds, not the best fold; show what happens in the worst regime fold. That number is the honest one. If the worst-fold out-of-sample is flat or negative, the model isn't a strategy — it's a lesson. Both are useful; only one is honest.
Checklist before you trust any number
Every feature computable at decision time t using only data ≤ t. No shift(-1), future_, next_ in feature code. Labels computed in a separate, time-ordered pass. Walk-forward, not single shuffle split. Worst-fold out-of-sample reported as a distribution. Leakage checker runs clean on the notebook. Calibration reported, not just accuracy. If any box is unchecked, the accuracy number is fiction.
Limitations of this piece
I have not run a live backtest inside this article; the design above is the method, not a claimed result. Any specific accuracy/return figure would have to come from a reproducible notebook on dated data, and even then it is not a forecast. This is educational. It is not a recommendation to buy or sell anything.
FAQ
Q: Can XGBoost predict Nifty direction reliably? Not reliably. It can improve how you structure the decision and validate honestly, but markets are adversarial and regime-dependent. Q: What's the minimum data I need? Enough clean, time-ordered, deduped ticks to survive walk-forward without the smallest fold being noise. Q: Should I use options price or underlying? Features from the underlying are cleaner; option mids are noisier. Q: How do I know if I leaked? Run the leakage checker; if it flags future_feature or shuffle, you leaked. Q: What metric matters most? Worst-fold out-of-sample, as a distribution.
Next step
If you want the reproducible notebook + leakage checklist, see the XGBoost walk-forward post and the backtesting notes. Build your own model with free tools — no paid course, no black box. Start with clean data, strict ordering, and brutal honesty about the worst fold.
Extended case study: the 95% notebook
A reader sends you a notebook: "XGBoost, 95% accuracy on Nifty options direction!" You run the leakage checker. It flags future_iv and shuffle=True. You remove those. Accuracy drops to 53%. The reader is disappointed. You explain: 53% with honest validation and a real edge over the 50% base rate (minus fees) is a starting point; 95% from a leak is nothing. The checker didn't break their model — it revealed it never existed.
This is the single most valuable habit the pipeline teaches: distrust any number you cannot reproduce from clean, time-ordered data. The leakage checker is five lines of regex; the discipline it enforces is worth more than any model.
Feature engineering that is safe
Safe features: spot return over [t-20, t]; realized volatility up to t; IV rank vs its 30-day range up to t; time-to-expiry; day-of-week (computed from t, not future). Unsafe: anything referencing t+1 or later; rolling windows that include future bars; the label or its proxies; exchange data timestamped after t. When in doubt, ask: "At exactly time t, with only what the screen showed, could I compute this?" If no, it is a leak.
Model selection without overfitting
Start with a constant baseline (always predict the base rate). If XGBoost cannot beat it out-of-sample on the worst fold, stop — there is no edge yet. Then compare to logistic regression; if trees win only on the best fold, the linear model is the honest choice. Use the model that is robust across folds, not the one with the highest average. Average hides the worst fold; the worst fold is where live trading happens.
Practical walkthrough: building the honest study
Step 1 — Load deduped, idempotent data from your clean store. Step 2 — Engineer only decision-time features; grep for shift(-1), future_, next_ and delete any match. Step 3 — Define the label as the forward move t to t+h in a separate pass. Step 4 — Split walk-forward: train [0,k], validate [k,k+v], roll k forward, repeat for the whole history. Step 5 — On each fold record ROC-AUC, PR-AUC, Brier, and the confusion matrix; store them. Step 6 — Report the distribution and specifically the worst fold. Step 7 — Run the leakage checker on the notebook; it must come back clean.
What good looks like: the worst fold's out-of-sample is only slightly below the median, calibration is sane (predicted probabilities track realized frequencies), and removing any single feature does not magically create or destroy the edge. If the worst fold is deeply negative while the best is stellar, you have a regime problem, not a model.
A note on h (horizon)
The label horizon h is a modelling choice with consequences. Too short and you trade noise; too long and the move is dominated by events you cannot feature. For weekly options, h of a few sessions often balances signal and survivability. Test several h values walk-forward; pick the one whose worst fold is most stable, not the one with the best average.
A second case study: feature importance lies too
You build the honest pipeline, worst fold is acceptable, and you celebrate. Then you inspect feature importance and see iv_rank at t dominates. You feel smart. But iv_rank at t is computed from the day's range up to t — and on expiry days the range explodes, so the model is mostly reading "is today expiry" from the range. That is a real signal, but a narrow one. The lesson: importance is not understanding. Validate each important feature's behaviour across regimes; if it only fires on one regime, your model is that regime's proxy, not a general edge. Walk-forward per-regime importance, not just aggregate.
Key takeaways
XGBoost is a tool, not a verdict. Used inside a leakage-free, walk-forward MLOps loop it can sharpen how you decide; used as a black box it sharpens how you lose. The pipeline in this note is the difference. Build the data store first, the labels second, the model third, and the honesty always. The market does not reward the most complex model — it rewards the most reproducible one.
A note on reproducibility vs performance
When two models disagree, prefer the one you can reproduce exactly from a version string. A 2% better average from an unreproducible notebook is worse than a 2% worse number you can rebuild blind. Reproducibility is a feature, not a constraint. Pin it, version it, store it next to the model, and treat any result you cannot rebuild as unverified.
An illustrated example end to end
Picture the Thursday expiry workflow. At 09:15 the collector starts ingesting NIFTY ticks through the idempotent layer; by 09:20 the store has clean, deduped ticks. The feature job computes spot return, realized vol, and IV rank — all using data up to each tick's timestamp. The label job, run separately, marks the forward move to Thursday close. No feature references Thursday's close; the label is computed only after features are frozen. Walk-forward splits the history into twelve rolling windows. On fold three, a rangebound fortnight, the model's ROC-AUC dips to 0.52 — barely above chance. On fold nine, a trending stretch, it reaches 0.61. The reported distribution spans 0.52 to 0.61, median 0.57. That median, with the worst fold shown, is the honest summary. Anyone quoting only fold nine would be selling fold nine. The pipeline shows all twelve.
This is what discipline looks like in practice: not a single impressive number, but a distribution you can defend. The model is not a prophecy; it is a slightly-better-than-chance reader of structure that you must size and hedge regardless. The worst fold is your friend because it tells you the floor before the market does.
Related reading
About the Author
Shakti Tiwari writes about AI, local AI agents, XGBoost, and options trading with AI — in Hinglish, for Indian traders and builders. Educational, no-hype, code-first.
- 🐦 X: https://x.com/shaktitiwari
- 💼 LinkedIn: https://linkedin.com/in/shakti-tiwari-a3b22a38b
- 💻 GitHub: https://github.com/shaktitiwari715-ai
- 📝 DEV.to: https://dev.to/shaktitiwari
- 🌐 Site: https://optiontradingwithai.in
Educational only. Not investment advice.
Continue Reading (Authority OS series)
- WebSocket collector idempotent
- Leakage-free XGBoost pipeline
- Nifty data pipeline
- AI Trading Myth-Buster
- Interactive Tools Hub
Top comments (0)