How to Build a Simple NSE EOD Options/Stock ML Model Manually (No Live API, No Wizardry)

Framing (research-governor applied): This is a research/educational walkthrough, not trading advice and not a claim that any model predicts the market. The blueprint this article is based on states plainly it is "a research/engineering blueprint, not a guarantee of trading profit or zero software defects." Nothing here has passed a production promotion gate. Build it to learn the discipline, not to mint money.

Executive Summary

You do not need a live WebSocket, a co-located server, or a PhD to start learning how institutional-style prediction on NSE data actually works. The single most robust starting point is the one the pros keep coming back to: end-of-day (EOD) data, downloaded once per day, processed with simple, auditable logic.

This article shows a manual, no-API pipeline:

1. Pull NSE's free EOD bhavcopy files (the official "Common Bhavcopy Final" in UDiFF format).
2. Normalise them into one clean table.
3. Engineer a handful of honest market-mechanics features (PCR, OI change, futures basis, delivery %).
4. Define a simple target: will the underlying go UP / NEUTRAL / DOWN tomorrow?
5. Train a tiny model you fully control (Excel, Google Sheets, or 40 lines of Python).
6. Validate with walk-forward — the only honest test — not a random train/test split.

The whole point of "manual and simple" is that you can see and defend every step. Complexity that you cannot explain is a liability, not an edge.

Research Question / Hypothesis

This article tests a practical, grounded question — can a retail builder replicate the core of an EOD prediction workflow using only free official NSE files and manual/light automation, while avoiding the classic traps (data leakage, survivorship bias, train/serve skew)? — not a "predict the market" claim.

Claims are labeled:

• SOURCE = verified from the blueprint document / NSE official material.
• OBSERVED = behaviour seen in real EOD workflows.
• DERIVED = computed from the above.

Nothing is invented.

Data & Methodology Box

• SOURCE (NSE format change): NSE discontinued the old common bhavcopy CSV formats from 8 July 2024 in favour of UDiFF Common Bhavcopy Final. The 2026 format workbook includes fields such as actual expiry, strike, option type, underlying price, settlement price, OI, change-in-OI, traded quantity/value, trades and lot size (per the blueprint document provided).
• SOURCE (data layers): F&O UDiFF Common Bhavcopy Final is the canonical derivatives source; CM UDiFF + delivery/corporate-action files support the underlying context (blueprint §3).
• OBSERVED: Simple EOD systems often survive longer than complex live ones because the data semantics stay stable — same close, same OI, same schema, same feature timing, same target timing.
• DERIVED (design rule): a feature earns its place only if it adds value across multiple out-of-sample windows; "interesting" ≠ "useful."
• Method: manual daily download → canonical table → feature function shared by training and prediction (no train/serve skew) → walk-forward evaluation with an embargo gap.

Why "Simple EOD" Beats "Complex Live" (for Learners)

The blueprint's section 2 is the best argument for starting manual. Complexity itself is not the enemy — untested semantic drift is. When you bolt on a live/API layer, silent mismatches creep in:

• Timestamps that mean different things in live vs EOD.
• Contract rollover handled wrong (near vs next expiry joined incorrectly).
• Missing values, partial bars, revised expiries, duplicate rows.

A manual EOD process kills most of these by construction: you work only with final files, once per day, after the exchange has published them. You trade a little freshness for a lot of cleanliness. For a learner, that trade is almost always worth it.

OBSERVED trap — train/serve skew: a backtest looks great, but daily inference is terrible, because the feature code differed between training and live. The fix is boring: use the same make_features() function for both. In a manual setup, that means the Excel formula or Python function you use on history is the exact one you run tomorrow.

Step 1 — Get the Data Free (Manual Download)

You do not need an API key to start. NSE publishes EOD reports.

• FO core: F&O UDiFF Common Bhavcopy Final — contract OHLC/LTP, expiry, strike, option type, underlying, OI, change-in-OI, volume/value, trades, lot. (SOURCE: blueprint §3, MUST layer)
• CM core: CM UDiFF Common Bhavcopy Final — underlying OHLC/volume/value/trades. (SOURCE, MUST)
• CM delivery: Security-wise Delivery Positions — delivery % as a genuine interest proxy. (SOURCE, HIGH)
• Corporate actions / security master: splits, bonuses, dividends, symbol lifecycle. (SOURCE, MUST for stocks)

Manual workflow: each evening (or next morning), download the FO and CM final files for the previous session, drop them in a dated folder (data/2026-08-20/), and record the file hash. That hash is your integrity receipt — if the file ever changes or truncates, you will know.

SOURCE note: NSE is streamlining bhavcopy dissemination and also making DAT-format bhavcopy available through Extranet paths. A robust manual parser should validate file content/schema version rather than trusting the filename. For a learner, just confirm the columns match your expected schema before ingestion.

Step 2 — Build One Canonical Table

Raw files are immutable. Every source format maps to one internal schema so your historical backfill survives format changes without rewriting feature code.

Minimal canonical fields you actually need (DERIVED from blueprint §4):

Field	Meaning	Applies to
trade_date	Trading date	all
symbol	Underlying/security	all
instrument_type	FUTIDX/FUTSTK/OPTIDX/OPTSTK	all
expiry_actual	Revised/actual expiry	derivatives
strike / option_type	Strike / CE-PE	options
close / settlement_price	EOD price	all
open_interest / change_oi	End-of-day OI / ΔOI	FO
traded_qty / traded_value	Volume	all
market_lot	Lot size	all

A canonical contract key = (segment, instrument_type, symbol, expiry_actual, strike_or_0, option_type_or_XX). A canonical row key = (trade_date) + contract_key. These two keys prevent the "duplicate row" and "wrong expiry join" failures.

In Excel this is just a consolidated sheet with those columns. In Python it is one DataFrame. Keep it dumb and explicit.

Step 3 — Engineer a Few Honest Features

Do not engineer 400 features on day one. Engineer five you can explain in a sentence each. Market-mechanics features the blueprint highlights (SOURCE):

1. PCR (Put-Call Ratio) — total put OI ÷ total call OI for the underlying's option chain. DERIVED. Crowded fear (high) is a contrarian hint only with trend context.
2. Change-in-OI regime — is OI building or unwinding at key strikes? OBSERVED: rising OI at a strike is a positioning proxy, not a hard wall.
3. Futures basis / term structure — spot vs future, near vs next expiry. SOURCE: futures basis/term-structure/roll are core mechanics predictors.
4. Delivery % — from CM delivery file. SOURCE: a genuine underlying-interest proxy, free of option-chain noise.
5. Moneyness / TTE buckets — is OI concentrated ATM or far OTM? DERIVED: OI concentration and strike-wall distances matter.

The anti-complexity rule (SOURCE): a feature stays ON only if it shows incremental value over the baseline across multiple out-of-sample windows. An "interesting" feature is not a "useful" feature. A manual builder enforces this by ablation: remove the feature, re-run walk-forward, and check whether the score actually dropped.

Step 4 — Define the Target (Honestly)

Primary target = T+1 underlying direction, preferably 3-class: UP / NEUTRAL / DOWN (SOURCE: blueprint §1). A regression target (expected return) is optional later.

• Compute next-day return from the underlying close.
• Bucket: UP if return > +threshold, DOWN if < −threshold, NEUTRAL between.
• The threshold is a modelling choice, not a discovered truth — state it and keep it fixed across all tests.

Why underlying-date, not option contract? (SOURCE) The primary prediction unit is underlying-date (e.g., NIFTY, RELIANCE, SBIN), with futures/options-chain mechanics as predictors. Options expire; the underlying persists. Modelling the underlying direction is stabler and avoids contract-churn leakage.

Step 5 — Train a Tiny Model You Control

"Model" can be laughably small to start:

• Excel / Google Sheets: a logistic-style or linear score on your 5 features. You can literally compute score = w1·PCR + w2·ΔOI + ... with hand-set weights and see the confusion matrix by hand.
• Python (light): sklearn or xgboost with manual parameter control. The blueprint recommends XGBoost and LightGBM on identical folds/features and picking the winner by repeated walk-forward stability + calibration + MCC/Balanced Accuracy — not one lucky accuracy number.

Optuna = suggestion, not authority (SOURCE): use it in competition/suggestion mode if you like, but keep manual parameter control available. A learner should be able to name every weight.

Step 6 — Walk-Forward Validation (The Only Honest Test)

This is where manual discipline pays off. Never use a random train/test split on time-series data — it leaks the future into the past and gives you fake-high scores (SOURCE: "Random CV leakage → unrealistically high score").

Manual walk-forward:

1. Sort by date.
2. Train on window [start, t], test on [t+1, t+gap].
3. Slide t forward; repeat.
4. Embargo/gap between train and test so yesterday's features don't leak into today's label.
5. Report Balanced Accuracy and MCC across all folds — and whether the model stays stable across regimes (trending, choppy, event weeks).

A model that wins one fold and dies in three is not a model; it is a coincidence.

Step 7 — Point-in-Time & Null Discipline

Two traps kill manual projects silently:

Survivorship bias (SOURCE): do not apply today's F&O list to history. NSE's F&O eligibility changes month to month. Your training universe on date T must be "symbols that actually had valid eligible FO contracts on T." Reconstruct the universe by date from the contract/security master.

Null policy (SOURCE): "drop NaN" is not a strategy. Classify the null first:

• STRUCTURAL_NULL — field doesn't apply (e.g., strike on a future). Keep as structural; never impute zero.
• WARMUP_NULL — rolling indicator lacks history. Exclude until minimum history met.
• SOURCE_MISSING / JOIN_MISSING — data-quality failures, not model features.
• INVALID — impossible value; quarantine and report.

Zero is not a generic missing sentinel — zero OI change can be a real value (OBSERVED).

Worked Example (Manual, on NIFTY)

Imagine you track NIFTY only, one row per day, from the FO + CM final files:

• Day 1–60: you collect close, settlement, total put OI, total call OI, NIFTY futures basis, and NIFTY delivery %.
• You compute PCR = put OI ÷ call OI each day (DERIVED).
• You label T+1: UP if next-day NIFTY return > +0.3%, DOWN if < −0.3%, else NEUTRAL.
• You train a 5-feature score on days 1–40, test on 41–45 (with a 1-day embargo), then slide.
• Across 12 such folds you find Balanced Accuracy ≈ 0.52 in trending windows and ≈ 0.47 in choppy ones. Interpretation (DERIVED, honest): the naive feature set is roughly a coin-flip once costs are considered — which is exactly the lesson. The workflow is correct; the edge is not yet there. You now have a reproducible harness to test real features, instead of a backtest that lied to you.

That negative result is worth more than a fake 85% accuracy from a random split.

Common Myths

• "I need a live API to do real ML." No. EOD final files are cleaner and free; live data mainly adds freshness and silent drift risk. Start EOD.
• "More features = better model." Only if they survive multiple out-of-sample windows. Otherwise they are overfit (SOURCE anti-complexity rule).
• "High backtest accuracy means it works." Not if the split leaked the future. Walk-forward or it didn't happen (SOURCE).
• "Dropping NaN rows is fine." It hides data-quality problems and can delete valid signal. Classify nulls instead (SOURCE).
• "I can use today's stock list for history." That's survivorship bias; it inflates past performance (SOURCE).

FAQ

Q: Is this legal / allowed for a retail trader?
A: Downloading NSE's public EOD files for personal research/education is standard. Respect NSE's terms and don't redistribute the raw data. This article is educational, not a licensed research service.

Q: Do I need to know Python?
A: No — Excel/Google Sheets can do steps 2–6 with formulas and a hand-built confusion matrix. Python just scales it and makes walk-forward easier.

Q: Which is better, XGBoost or LightGBM?
A: Neither is "better" by brand. The blueprint picks the winner by repeated walk-forward stability + calibration + MCC/Balanced Accuracy on identical folds. As a learner, either is fine; consistency of method matters more than the library.

Q: How do I avoid data leakage in a manual setup?
A: Use walk-forward with a date embargo, define the target from future close only, and use the same feature function for history and tomorrow. Never let a future row touch a past feature window.

Q: What is the single biggest beginner mistake?
A: Random train/test split on time-series data, producing a fake-high score and a false sense of edge. Walk-forward or nothing.

Q: Should I trade live on this?
A: Not on the basis of this article. This is research discipline, not a production system. Any live use needs promotion gates, cost adjustment, and human approval the blueprint describes — none of which are claimed here.

Practical Takeaways

1. Start with EOD final files, not live APIs.
2. Normalise everything into one canonical table with explicit keys.
3. Engineer five explainable features, not 400.
4. Target T+1 direction (3-class) on the underlying, not the option.
5. Validate only with walk-forward + embargo; report Balanced Accuracy and MCC.
6. Reconstruct the point-in-time universe; classify nulls instead of dropping them.
7. Treat any "great" backtest with suspicion until it survives multiple regimes net-of-cost.

Limitations

• This is a research/education blueprint, not a deployed, promoted trading system. The provided document explicitly disclaims guaranteed profit or zero defects.
• Non-stationarity is the rule. A feature that worked last quarter can decay this quarter; validate out-of-sample and shadow-run before any live action.
• Costs are not modelled here. A 0.52 Balanced Accuracy is below the bar once STT/brokerage is subtracted — the worked example is intentionally honest about that.
• No live promotion gate passed. Per the research-governor, claims are bounded to method and discipline, never to profitability.
• If any figure here conflicts with an official NSE/SEBI source, the official source wins — verify before you act.

---

Author: Shakti Tiwari — NISM XII certified options educator (not a SEBI Registered Analyst). Educational content only; not investment advice. Verify all data against official NSE sources before acting.

More from the author: about.me/shaktitiwari | optiontradingwithai.in | WhatsApp: [phone]

Books by the author:

• Options Trading with AI — Vol 1 — B0H9ZNTBPK
• Options Trading with AI — Vol 2 — B0HBBFKDQF