DEV Community

shakti tiwari
shakti tiwari

Posted on

Inside Our NSE F&O ML Engine: Live Dhan API Data Meets an EOD-Audited, Leakage-Safe Blueprint

#xgboost #machinelearning #nse #trading

Inside Our NSE F&O ML Engine: Live Dhan API Data Meets an EOD-Audited, Leakage-Safe Blueprint

Quick Answer

The OptionTradingWithAI.in NSE engine runs in two connected layers. The live layer connects to the Dhan WebSocket API (Full market mode) to capture real-time ticker, quote and option-chain data, and runs the trained model in shadow mode to generate signals without sending broker orders. The ML layer — the focus of this article — is deterministic and EOD-audited: the model is trained and validated on official NSE UDiFF bhavcopy files for full reproducibility. Its core design choices are: predict the underlying's next-day direction (UP/NEUTRAL/DOWN), use one feature function for both training and prediction (no train/serve skew), validate only with date-based walk-forward + embargo, reconstruct a point-in-time universe to kill survivorship bias, and treat every feature as guilty-until-proven by multiple out-of-sample windows. The biggest lesson: complexity is not the enemy — untested semantic drift is. This article documents the architecture so a retail builder can audit every step.

Framing (research-governor applied): This is a research/engineering walkthrough of our own blueprint, not a guarantee of trading profit. The source document itself states it is "a research/engineering blueprint, not a guarantee of trading profit or zero software defects." No promotion gate has been passed; claims are bounded to method and design, never to profitability.

Why This Matters

Most retail "AI trading" content is two things at once: too complex to audit, and too shallow to trust. The NSE EOD blueprint we use at OptionTradingWithAI.in takes the opposite bet — data rich, code small, prediction path deterministic.

For the Indian retail trader who actually wants to understand what an ML options/stock model does (rather than rent a black box), three facts matter:

  1. The ML training layer is EOD-first for auditability — the live layer is API-driven. Our production engine connects to the Dhan WebSocket API for real-time data; the model behind it is trained/validated on NSE's free EOD bhavcopy files so every feature and label is reproducible. NSE's EOD reports contain the futures, options-chain, cash-market and delivery data needed to engineer real market-mechanics features.
  2. The traps are boring, not exotic. Train/serve skew, random-CV leakage, survivorship bias, and silent null-drift destroy more models than bad maths does.
  3. An auditable 5-module engine beats a 200-file monster you cannot explain to yourself, let alone to a regulator or a skeptical reader.

This is also where SEO intent meets genuine value: searchers typing "NSE machine learning options", "XGBoost options trading India", or "walk-forward validation python" are not looking for hype — they want a reproducible design they can check. That is exactly what follows.

The Live Layer: Dhan WebSocket API (How the Engine Actually Runs)

The blueprint quoted above describes the ML training and backtest data design — deliberately EOD/UDiFF for determinism and auditability. The production engine is not EOD-only. It connects to the Dhan WebSocket API in Full market mode to stream live ticker, quote and option-chain data, then runs the trained XGBoost/LightGBM model in shadow mode (the live-shadow-observer pattern) to produce directional signals without placing broker orders.

  • OBSERVED (our stack): Dhan's Ticker/Quote/Full WebSocket modes feed the live capture; the same make_features() function used in training runs on the live snapshot, so there is no train/serve skew at inference either.
  • Why EOD for training, API for live: EOD bhavcopy gives a clean, final, point-in-time record ideal for leakage-safe walk-forward; the live API gives the freshness needed for next-day signal generation. The two layers are complementary, not contradictory.
  • Safety: shadow mode means the model's predictions are recorded and evaluated against outcomes before any order authority is considered — exactly the promotion-gate discipline the blueprint enforces.

Research Question / Hypothesis

This article documents a practical, grounded question — can a retail-grade NSE prediction engine be designed so that its biggest failure modes (leakage, survivorship, train/serve skew, null drift) are prevented by construction rather than patched after the fact? — not a "the model predicts the market" claim.

Every non-obvious statement below is labeled:

  • SOURCE = from our NSE EOD ML Blueprint document (2026-08-20 research snapshot).
  • OBSERVED = behaviour seen in real EOD workflows.
  • DERIVED = computed from the above.

Nothing is invented. Where the engine has not been trained, we say so.

Data & Methodology Box

  • SOURCE (format): NSE discontinued the old common bhavcopy CSV formats from 8 July 2024 in favour of UDiFF Common Bhavcopy Final. The 2026 format workbook includes fields such as actual expiry, strike, option type, underlying price, settlement price, OI, change-in-OI, traded quantity/value, trades and lot size.
  • SOURCE (layers): F&O UDiFF Common Bhavcopy Final (canonical derivatives source); CM UDiFF + delivery/corporate-action files (underlying context); participant/FII statistics (market context, where granularity supports it).
  • SOURCE (target): V1 target = next-day underlying close direction via r_next = log(close[T+1]/close[T]), bucketed UP / NEUTRAL / DOWN by an ATR-normalized band k * ATR_pct[T].
  • SOURCE (validation): date-based walk-forward with gap/embargo; a final untouched holdout the tuner never sees.
  • DERIVED (design rule): a feature stays ON only if it adds value over baseline across multiple out-of-sample windows; "interesting" ≠ "useful."
  • Method: manual daily ingestion → one canonical schema → shared make_features() for train and predict → walk-forward with negative controls (shuffled labels, lag-scrambled features, dummy majority, price-only baseline, feature-group ablation).
  • Costs: not yet modeled in the blueprint's V1; any trading decision layer must include STT/brokerage before claiming viability.

Results (Design Decisions and What They Buy You)

Because the blueprint is an architecture, its "results" are the decisions that survive the blueprint's own challenge tests. Each is a defensible design outcome, not a backtest win.

Table 1 — Core architecture decisions

Decision What it prevents Why it matters
One make_features() for train + predict Train/serve skew Backtest-good / inference-bad gap disappears
Date-based walk-forward + embargo Random-CV leakage No future leaking into past
Point-in-time universe (by contract master) Survivorship bias Today's F&O list ≠ history's list
Reason-coded null policy Silent row loss Source/join failures surfaced, not hidden
Feature gates + ablation manifest Overfit / audit failure Each feature must earn its place
Underlying-date as unit Contract churn leakage Model learns the persistent entity
UDiFF-first EOD training data Partial/live drift in training Model trains only on final, validated EOD files (live inference uses Dhan API)

Findings (from the blueprint's own logic)

  1. SOURCE: The primary prediction unit is underlying-date (e.g., NIFTY, RELIANCE, SBIN), with futures/options-chain mechanics as predictors. Options expire and roll; the underlying persists. Modeling the underlying direction is stabler and avoids contract-identifier leakage.
  2. SOURCE: XGBoost and LightGBM are trained on identical rows, columns, labels, sample weights, folds and scoring. Only hyperparameter spaces differ. The winner is chosen by repeated walk-forward stability + calibration + MCC/Balanced Accuracy, not one lucky accuracy number.
  3. SOURCE: Optuna runs in suggestion/competition mode — objective = mean walk-forward MCC minus stability/calibration/failure penalties. The best trial is a candidate; a manual "promote" step is required. The model never auto-promotes.
  4. OBSERVED: Simple EOD systems out-survive complex live ones because data semantics stay stable — same close, same OI, same schema, same feature timing, same target timing.
  5. DERIVED: The competition score mean(MCC_folds) − λ_std·std(MCC) − λ_cal·cal_penalty − λ_fail·fail_penalty rewards stable models over flashy single-fold winners.

Feature Catalog (Selected) — What the Engine Actually Reads

A big part of auditability is naming every candidate feature with its formula, source, tier and leakage test. The blueprint generates a Feature Manifest so nothing hides. Below is a representative slice across the four mechanic families (SOURCE). Tiers: Core (default ON after gating), High (context-rich), Extended/Experimental (must beat baseline to stay).

Table 2 — Selected candidate features by family

Family Feature Idea Tier
Underlying ret_1d, logret_1d, ret_2/5/10/20d momentum horizons Core
Underlying atr_pct_14, rv_5/10/20 realized vol Core
Underlying rsi_14, ema_dist_5/10/20 oscillator / trend distance Core
Cash/CM volume_change, volume_z20 activity surprise Core
Cash/CM delivery_pct, delivery_change genuine interest proxy High
Futures fut_basis_pct, basis_change spot vs near future Core
Futures term_spread, term_spread_pct near vs next month Core
Futures roll_oi_ratio, roll_volume_ratio roll concentration High
Options pcr_oi, pcr_volume put/call pressure Core
Options atm_oi_imbalance, atm_volume_imbalance ATM PE vs CE Core
Options call_wall_dist, put_wall_dist distance to max-OI strike High
Options oi_hhi, oi_entropy concentration / dispersion High
Options near_expiry_pcr, next_expiry_pcr, expiry_pcr_spread expiry-structure signal Core/High
Context fii_derivatives_context, breadth market/FII state High
Context participant_net_index_fut client class positioning (if report permits) High

Key interaction features (SOURCE): moneyness × TTE, OI-change × moneyness, PCR × volatility-regime, basis × TTE, ATM-OI-imbalance × underlying momentum. These cross-products are where mechanics often beat a price-only baseline — if they survive ablation.

OBSERVED caveat from the blueprint: the "long buildup / short covering / long unwinding" classification from price+OI signs is a heuristic, not ground truth about trader positioning. We do not label it as actual positioning.

Why UDiFF-First (Not Filename-First)

NSE is streamlining bhavcopy dissemination and also exposing DAT-format bhavcopy via Extranet paths. A robust parser therefore validates file content and schema version rather than trusting the filename. The payoff (SOURCE): historical backfill can span the 8-July-2024 format change without rewriting feature code, because every raw format maps to one canonical schema. That is the whole point of "data rich, code small" — the data layer absorbs change so the model layer stays stable.

Reproducibility (The 5-Module Skeleton)

The blueprint keeps business logic in five small modules (roughly 200–500 readable lines each) plus config, tests and an app shell. An auditor should understand the whole prediction path without opening dozens of files.

project/
  config.yaml      # sources, feature toggles, label, model params, Optuna spaces
  data.py          # download/load, legacy+UDiFF normalize, joins, validation, quality report
  features.py      # ALL deterministic feature formulas + label + manifest generation
  models.py        # XGBoost, LightGBM, Optuna, save/load model bundles
  evaluate.py      # date walk-forward, competition, calibration, ablation, audit metrics
  app.py           # manual commands/UI: ingest, build, train, compare, promote, predict
  tests.py         # schema/null/leakage/reproducibility tests
  README.md        # operating procedure + audit map
Enter fullscreen mode Exit fullscreen mode

Canonical keys (SOURCE): every source format maps to one internal schema.

canonical_contract_key = (segment, instrument_type, symbol, expiry_actual, strike_or_0, option_type_or_XX)
canonical_row_key      = (trade_date,) + canonical_contract_key
Enter fullscreen mode Exit fullscreen mode

Sample feature (SOURCE — options-chain PCR, Core tier):

pcr_oi = total_Put_OI / total_Call_OI          # across the underlying's option chain
Enter fullscreen mode Exit fullscreen mode

Moneyness is signed percentage, not raw rupee distance — because ₹100 means different things for a ₹500 stock and a 25,000 index:

d_pct       = (K - S) / S
signed_mny  = -d_pct for CE   # positive = ITM
signed_mny  =  d_pct for PE   # positive = ITM
Enter fullscreen mode Exit fullscreen mode

One row per underlying-date (SOURCE): aggregate the contract universe (many FO rows) + the CM underlying row + market context into a single feature row, then label from underlying T+1. This sharply reduces complexity and stops the model learning strike quirks that vanish next month.

Leakage-Safe Walk-Forward: A Concrete Fold Plan

The blueprint specifies a date-based splitter (Scikit-learn's TimeSeriesSplit supports ordered data with a gap; here it is stricter because many symbols share a date and all rows for one date must stay on the same side). A concrete scheme (SOURCE):

Table 3 — Example walk-forward schedule

Fold Train window Gap / test
1 2019 .. 2022 TEST 2023 H1
2 2019 .. 2023 H1 TEST 2023 H2
3 2019 .. 2023 TEST 2024 H1
… slide forward …
Final — untouched holdout (most recent block, Optuna never sees it)

Rules that make it honest:

  • Split by DATE, never by random row. All underlyings on a date belong to the same side.
  • Gap/embargo ≥ label horizon. If features/labels use longer forward windows, increase the gap.
  • Any probability calibrator is fit on data disjoint from the estimator's own fit data.
  • Keep a final holdout the tuner never touches — this is where real generalization is measured.

Metrics reported per fold (SOURCE): Balanced Accuracy (imbalance-tolerant headline), MCC (multiclass quality; 0 ≈ no useful correlation), Macro-F1, Log Loss, Brier/calibration curve, coverage-by-confidence, and per-year / per-regime metrics to catch instability a single overall average hides. A model that wins one fold and dies in three is not a model.

What Failed / Counter-Evidence (Negative Controls)

The blueprint bakes in honesty tests so a lucky backtest cannot masquerade as edge:

  • Shuffled labels → should collapse toward chance. If they don't, something is leaking.
  • Lag-scrambled features → should lose signal.
  • Dummy majority classifier and price-only baseline → the floor to beat.
  • Feature-group ablation → remove futures, options, delivery, and participant context one group at a time. SOURCE rule: if the "full mechanics" model does not repeatably beat the simple price/OI baseline, drop the mechanics. Complexity must earn its place; it gets no free pass.

OBSERVED trap: a trader (or an article) reporting one great backtest is reporting a coincidence until it survives multiple regimes net-of-cost. The negative controls are the difference between "I tested" and "I know."

Limitations

  • This is a research/engineering blueprint, not a deployed, promoted trading system. The source document explicitly disclaims guaranteed profit or zero defects.
  • Non-stationarity is the rule. A feature that worked last quarter can decay this quarter; validate out-of-sample and shadow-run before any live action.
  • Costs are not in V1. Any strategy layer must subtract STT/brokerage; gross edge is not net edge.
  • IV/Greeks are Experimental, not core. The blueprint deliberately avoids depending on scraping a dynamic web option chain; if IV is added later it must be computed from validated EOD prices with rejection of impossible quotes, and must beat the non-IV baseline out-of-sample before promotion.
  • No live promotion gate passed here. Per the research-governor, claims are bounded to method and design, never to profitability.
  • If any figure here conflicts with an official NSE/SEBI source, the official source wins — verify before you act.

Practical Takeaways

  1. Predict the underlying, not the option. Use underlying-date as the unit; let chain mechanics be predictors.
  2. One feature function, everywhere. Train and predict with the same make_features(). This single rule kills the most common silent failure.
  3. Walk-forward or nothing. Split by date with an embargo; keep a final holdout the tuner never sees.
  4. Reconstruct the point-in-time universe from the contract/security master. Today's F&O list is not history's list.
  5. Classify nulls; don't drop them. Structural / warm-up / source-missing / join-missing / invalid are different states with different actions.
  6. Gate every feature. Ablate it; if it doesn't repeatably beat baseline, it's off.
  7. Manual control stays. Optuna suggests; a human promotes. The model never auto-ships.
  8. Go deeper in the books. The full pipeline — feature code, walk-forward harness, and live Dhan WebSocket integration — is expanded in the author's published works, Options Trading with AI, Vol 1 & Vol 2 (linked below).

FAQ

Q: Is this the actual engine behind OptionTradingWithAI.in?
A: Yes — this documents our two-layer NSE F&O ML engine: a live Dhan WebSocket API feed for real-time shadow-mode signals, plus an EOD/UDiFF-audited XGBoost-vs-LightGBM core trained with point-in-time data and walk-forward validation. Published as research, not as a live trading product.

Q: Does the engine use a live API or only EOD files?
A: Both, by design. The ML model is trained and validated on NSE's free EOD UDiFF bhavcopy files (for auditability and leakage-safe walk-forward). The production engine then connects to the Dhan WebSocket API to capture live market data and run the trained model in shadow mode for next-day signals. EOD is the training substrate; the API is the live nervous system. (The blueprint's "EOD-first" wording refers to the training/backtest data layer, not the whole engine.)

Q: Why XGBoost and LightGBM, not deep learning?
A: Neither brand "wins" by default. The blueprint picks the winner by repeated walk-forward stability + calibration + MCC/Balanced Accuracy on identical folds. For tabular market data, gradient-boosted trees are the defensible default; deep nets must earn their place like any other feature group.

Q: What is the single most common leakage mistake?
A: Random train/test split on time-series data, or letting a future row touch a past feature window. Split by date, embargo the gap, keep all underlyings on a date on the same side of the split.

Q: How do you avoid survivorship bias?
A: Reconstruct the training universe by date from actual contracts/security master. Do not apply today's F&O eligibility list to 2019. Use actual expiry, not a hard-coded weekday rule.

Q: Should I trade live on this blueprint?
A: Not on the basis of this article. It is a research design. Any live use requires promotion gates, cost adjustment, negative-control passes, and human approval the blueprint describes — none claimed here.

TL;DR

  • Our NSE engine has two layers: a live Dhan WebSocket API feed for real-time data + shadow-mode signals, and an EOD/UDiFF-audited ML core for reproducible training and walk-forward validation.
  • Predict underlying next-day direction; use one shared feature function; validate with date walk-forward + embargo.
  • Point-in-time universe + reason-coded nulls kill survivorship and silent data loss.
  • XGBoost vs LightGBM on identical folds; winner by stability + MCC, not one accuracy.
  • Every feature is gated; negative controls must pass or complexity is dropped.
  • This is research documentation — not a profitability claim, not live advice.

Sources

  • NSE EOD ML Blueprint (Deep Research), research snapshot 20 August 2026 — OptionTradingWithAI.in internal document (SOURCE for all architecture claims above).
  • NSE UDiFF Common Bhavcopy Final format (official, effective 8 July 2024) — field schema for FO/CM layers.
  • Scikit-learn TimeSeriesSplit documentation — ordered split with gap (validates the date-based embargo design).

Author / Canonical Attribution

Written by Shakti Tiwari — Nifty Option Trader, XGBoost Expert, founder of OptionTradingWithAI.in. NISM XII certified options educator (not a SEBI Registered Analyst). Educational content only; not investment advice. Verify all data against official NSE sources before acting.

Resources & Links

Books by the author:

Top comments (0)