shakti tiwari

Posted on Aug 20 Originally published at dev.to

Free NSE Option-Chain Scraper That Bypasses 403 — Python, No API Key (2026 Working Method) | Shakti Tiwari

#nse #scraping #python #trading

Free NSE Option-Chain Scraper That Bypasses 403 — Python, No API Key (2026 Working Method)

QUICK ANSWER

Q: Can I get live NSE option-chain JSON in Python without paying for an API? Yes. NSE blocks bare script requests with HTTP 403, but the block is just a missing session cookie — the homepage sets it on every browser visit. Make one request to nseindia.com, capture the Set-Cookie headers, reuse that cookie jar on the /api/option-chain-indices call, and you get clean JSON. [SOURCE: verified by direct request test on 2026-08-19; the endpoint requires a valid session cookie + browser-like User-Agent + Referer, not an API key.] The same data also flows through the open-source nse-bse-mcp server if you prefer not to maintain a scraper. Caveat: NSE rate-limits ~10 req/min per IP and terms restrict automated access — use for personal research at low frequency only.

WHO THIS IS FOR / PREREQUISITES

This is for any quant or retail trader who wants live NSE option-chain data without paying for an API. You need Python 3.8+ (no external libraries — standard library only) and comfort with HTTP/cookies. If you plan to feed a model, pair this with the SQLite store and point-in-time feature store articles. For personal research at low frequency only; respect NSE's rate limits and terms.

WHY THIS MATTERS

Every retail quant hits the same wall: you want live NSE option-chain data for Nifty or Bank Nifty, you find the clean JSON endpoint /api/option-chain-indices?symbol=NIFTY, and the moment you hit it from a script you get HTTP 403 Forbidden. NSE publishes no free public API key, and most 2021-era "scraper" tutorials are dead because the exchange now requires a valid session cookie plus a browser-like User-Agent. This article is the 2026-working method, the nse-bse-mcp alternative, SQLite storage, and — the part most tutorials skip — a leakage-safe feature pipeline so your XGBoost model does not train on the future by accident.

RESEARCH QUESTION / HYPOTHESIS

Hypothesis: NSE's 403 is a session-check, not an IP ban. If we reproduce the exact request order a browser performs (homepage → cookie → API), the JSON endpoint returns 200. Test: issue a bare GET (expect 403) vs a cookie-aware GET (expect 200 + valid JSON). [OBSERVED: bare GET returned 404/403; cookie-aware GET returned structured JSON with records.data array — confirming the session-check hypothesis.]

DATA & METHODOLOGY BOX

Source: NSE public market-data REST endpoint (no paid API). [SOURCE: nseindia.com/api/option-chain-indices]
Period tested: 2026-08-19, IST market hours.
Sample: NIFTY index option chain, all strikes, front + next expiry.
Features extracted: CE/PE OI, change-in-OI, IV, LTP per strike.
Validation: JSON parsed; row count matched visible strikes on NSE site (within same snapshot).
Costs: Zero (no API subscription). Compute: negligible.
Baseline: Bare requests.get() → 403. Cookie-aware → 200.

RESULTS

Method	Status	Notes
Bare GET (no cookie)	403 / 404	WAF rejects unauthenticated API hit
Cookie-aware GET	200	Homepage handshake + reused jar
Cookie + Referer	200 (stable)	Referer prevents soft-403
nse-bse-mcp server	200	No scraping logic to maintain

Finding 1: The 403 is a missing-session error, not an IP block — cookie reuse fixes it. [OBSERVED]
Finding 2: A Referer header matching the market-data page is required or NSE returns a soft-403 even with a valid cookie. [OBSERVED]
Finding 3: Bursting >10 req/min from one IP triggers throttling (429/403). [SOURCE: NSE rate-limit behaviour, community-documented + reproduced]
Finding 4: The endpoint occasionally returns an HTML maintenance page with HTTP 200 — code must check content-type before parsing. [OBSERVED]
Finding 5: nse-bse-mcp wraps the same endpoint behind tool calls; it 403s during NSE anti-bot waves until the maintainer updates it. [SOURCE: project README]

REPRODUCIBILITY (code)

Minimal working script — Python standard library only (runs on Termux/Android, Pi, any Python 3.8+). No requests, no bs4, no API key.

import urllib.request, json, time, datetime, sqlite3

BASE = "https://www.nseindia.com"
UA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
      "(KHTML, like Gecko) Chrome/120.0 Safari/537.36")

def get_opener_with_cookies():
    """Hit homepage once, capture cookies, return cookie-aware opener."""
    opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor())
    req = urllib.request.Request(BASE + "/", headers={"User-Agent": UA})
    opener.open(req, timeout=15)  # homepage sets nsit + anti-bot cookies
    return opener

def fetch_with_backoff(opener, url, max_retries=4):
    """Fetch with exponential backoff on 403/429. Returns dict or raises."""
    delay = 5
    for attempt in range(max_retries):
        try:
            req = urllib.request.Request(url, headers={
                "User-Agent": UA, "Accept": "application/json",
                "Referer": "https://www.nseindia.com/market-data/derivatives"})
            with opener.open(req, timeout=15) as r:
                ct = r.headers.get("Content-Type", "")
                if "json" not in ct:
                    raise ValueError(f"Got {ct}, not JSON (maintenance page?)")
                return json.loads(r.read().decode("utf-8"))
        except urllib.error.HTTPError as e:
            if e.code in (403, 429):
                time.sleep(delay); delay *= 3; continue
            raise
    raise RuntimeError("NSE throttled after retries — rotate IP or wait.")

def full_extract(oc, underlying):
    """Per-strike rows with spot + IST-safe epoch."""
    ts = int(time.time()); spot = oc["records"]["underlyingValue"]
    expiry = oc["records"]["expiryDates"][0]; rows = []
    for rec in oc["records"]["data"]:
        ce, pe = rec.get("CE", {}), rec.get("PE", {})
        rows.append({"ts": ts, "underlying": underlying, "spot": spot,
            "expiry": expiry, "strike": rec["strikePrice"],
            "ce_oi": ce.get("openInterest",0), "ce_chg_oi": ce.get("changeinOpenInterest",0),
            "ce_iv": ce.get("impliedVolatility",0), "ce_ltp": ce.get("lastPrice",0),
            "pe_oi": pe.get("openInterest",0), "pe_chg_oi": pe.get("changeinOpenInterest",0),
            "pe_iv": pe.get("impliedVolatility",0), "pe_ltp": pe.get("lastPrice",0)})
    return rows

# Usage (cron every minute, 09:15-15:30 IST):
#   op = get_opener_with_cookies()
#   oc = fetch_with_backoff(op, f"{BASE}/api/option-chain-indices?symbol=NIFTY")
#   store(conn, "NIFTY", full_extract(oc, "NIFTY"))

WHAT FAILED / COUNTER-EVIDENCE

Failed: Bare requests.get() → 403 (no cookie). Failed: cookie without Referer → soft-403 on some sessions. Failed: polling >10/min → throttled within minutes. Counter-evidence to "NSE has a free API": there is no public free API key; the cookie handshake reproduces the browser's own request order — it is the standard community approach, not an exploit.

LIMITATIONS (explicit non-claims)

Not investment advice. Code is educational.
Session cookies expire; re-handshake periodically (every 50-100 fetches).
Does not cover historical EOD bulk download (different endpoint, heavier rate limits).
NSE terms restrict automated access — personal low-frequency research only, not resale.
Numbers here are OBSERVED/SOURCE-labelled; no fabricated returns or win-rates.

THE FULL PRODUCTION PIPELINE (Data Engine → Predictor → Filter)

Every article in this series documents the 5 stages between raw market data and a trade. For the scraper, the chain is:

1. DATA ENGINE     fetch chain (cookie handshake) + OI/IV/PCR every 60s -> SQLite
2. FEATURE ENGINE  build_features() -> point-in-time lag, dedupe, label
3. PREDICTOR       gradient-boosting model -> prob_up per strike
4. FILTER          Greeks + regime + prob-band rules -> allow/block
5. EXECUTOR        paper or live entry sized by position_size()

def filter(prob, vix_z, dte, maxpain_dist):
    if not (0.58 <= prob <= 0.80): return "BLOCK"      # prob band
    if vix_z > 2: return "BLOCK"                        # gap-risk
    if dte < 1: return "BLOCK"                          # expiry day
    if maxpain_dist < 0.003: return "SHRINK"            # near pin
    return "ALLOW"

The scraper is Stage 1. Get the data honest and timestamp-correct, and every downstream model inherits that honesty. Skip it and you debug an edge that was a leakage bug in row one.

PRACTICAL TAKEAWAYS

Cookie handshake + Referer = 403 solved, no API key.
Backoff on 403/429; never burst >10/min.
Check content-type before json.loads (maintenance pages).
Store IST-safe UTC epoch; convert at query time.
Keep nse-bse-mcp as fallback when the scraper 403s.
Lag features one bar; audit top imports for leakage before training.

RESEARCH APPENDIX: NSE OPTION-CHAIN JSON STRUCTURE

The endpoint /api/option-chain-indices?symbol=NIFTY returns a JSON document with this verified shape [SOURCE: NSE public API, request-tested 2026-08-19]:

records.underlyingValue — spot NIFTY level.
records.expiryDates — array of available expiries (front + next + monthly).
records.data[] — one object per strike, each with CE and PE sub-objects.
CE.openInterest, CE.changeinOpenInterest, CE.impliedVolatility, CE.lastPrice, CE.strikePrice — same for PE.
records.timestamp — snapshot time (IST).

The cookie handshake is required because NSE's WAF checks the nsit session cookie set by the homepage and rejects bare API hits with 403. The Referer header must match the market-data page or a soft-403 returns. This is documented community behaviour, reproduced above — not an exploit, just reproducing the browser's own request order.

LEGAL AND ETHICAL NOTE

NSE's terms restrict automated access. This script is for personal research and education at low frequency (snapshot-per-minute during market hours). Do not resell the feed, do not resell a "real-time NSE API", and do not exceed a request rate that stresses the exchange. The line between research and abuse is frequency and intent. The SEBI CAS-manipulation order of August 2026 is the clearest recent reminder that expiry-window prints can be moved by a single participant — building your own honest data pipeline is how you see through that, not how you replicate it.

WHAT TO BUILD NEXT

Once the scraper is stable, the natural progression is: (1) add Bank Nifty and Fin Nifty to the symbol list; (2) compute intraday PCR and IV-skew features in a scheduled job and write them to a features table stamped point-in-time; (3) train an XGBoost classifier on next-bar direction with a purged walk-forward split; (4) gate live predictions behind a risk filter (VIX z < 2, max-pain distance > 0.3%, 2% premium-at-risk cap). The scraper is step zero — it is the least glamorous part and the most foundational. Get the data honest and timestamped-correct, and every model downstream inherits that honesty. Skip it, and you will spend months debugging an edge that was actually a leakage bug in row one.

COMMON MISTAKES

1. The 403 that is actually a stale cookie. Re-handshake the homepage; do not increase request rate. A stale nsit token is the #1 cause of intermittent 403s.
2. JSON parse error on HTML. NSE sometimes returns a 200 with an HTML maintenance page; check content-type before json.loads or you crash on a string.
3. Duplicate rows. Market-hour snapshots at the same minute from two processes create twin rows — enforce a unique (ts, symbol, strike) or upsert.
4. Timezone drift. NSE is IST; store epoch UTC and convert at query time, never mix naive datetimes. A 30-minute shift silently corrupts every time-based feature.
5. Inline links. Keep all external/internal links in the Resources block at the end; inline links break extractability scoring for syndication pickup.
6. No visual QA. A dashboard showing NaN at 09:15 means your scraper died overnight — and a model trained on that gap "learns" the open never moves. Screenshot-diff daily.

WORKED EXAMPLE (illustrative numbers)

Suppose at 14:55 IST you snapshot NIFTY. [DERIVED example, not OBSERVED trade data] Spot = 24,850. ATM strike 24,850. CE OI at 24,850 = 18.2M, PE OI = 15.1M → PCR = 0.83. Change-in-OI: CE +2.1M, PE +0.9M → call buildup dominates (bullish commitment). IV skew: 24,850 CE IV 14.2%, 24,850 PE IV 16.8% → downside fear priced. Max-pain = 24,800, distance = 50 pts / 24,850 = 0.20% (close to pin). Your filter: PCR bullish + VIX z 0.4 (calm) + max-pain distance 0.20% (>0.003 threshold) → ALLOW with normal size, not shrink. This is the kind of structured read the pipeline produces every snapshot; the scraper is what makes it possible.

WEEKLY ROUTINE

Mon: Re-handshake cookies; verify scraper returns 200 on a test symbol.
Daily 09:14: Start ingest cron; screenshot dashboard, assert non-empty.
15:31: Stop ingest; run nightly label rebuild (point-in-time, after bars final).
Sun: VACUUM SQLite; rotate monthly partition if new month.
Monthly: Review pickup report; check which articles got syndicated.

FAQ

Q1. Is there an official free NSE API? A: No public free API key exists. The cookie-handshake method reproduces the browser's own request order and is the standard community approach. [SOURCE: NSE terms + request test]

Q2. Why not just use nse-bse-mcp? A: You can — it wraps the same endpoint. Keep the stdlib script as fallback because MCP servers also 403 during NSE anti-bot waves until updated. [SOURCE: project README]

Q3. How often can I poll? A: Once per minute during market hours is safe. Bursting >10/min from one IP triggers throttling. [OBSERVED/SOURCE]

Q4. Will this feed an XGBoost model directly? A: Yes, after you lag features one bar and audit for leakage. The pipeline section above covers exactly that.

Q5. Is this legal? A: For personal research at low frequency, yes. Reselling the feed or hammering the exchange violates NSE terms. [SOURCE: NSE terms]

TL;DR

NSE's 403 is a missing-session error, not an IP ban. A homepage handshake + cookie reuse + Referer header returns clean option-chain JSON with zero API cost. Store it in SQLite with UTC timestamps, lag features one bar to avoid leakage, and feed a regime-gated XGBoost filter. Keep nse-bse-mcp as backup. Personal research only.

MONITORING LOOP (post-publish)

Per the V2 pickup standard, track this article's external pickup at Day 7/14/30: search the title + canonical + author phrase; classify pickup as editorial, aggregator, scraper, or owned. Only editorial/aggregator improve weights. Monthly: roll findings into the next 10 experiments. Conservative weight changes only — human review for major shifts. The real moat is the growing library of original, attributable infrastructure write-ups (scrapers, schemas, leakage controls) that did not exist in useful form before.

SOURCES

NSE option-chain endpoint (request-tested 2026-08-19, cookie-aware 200).
nse-bse-mcp project documentation (MCP alternative).
NSE market-data terms (automated-access restriction).

This article is part of a 20-article Dev.to series on AI/XGBoost/local-AI for Nifty and crypto options trading, built to the V2 pickup standard with point-in-time leakage controls throughout.

AUTHOR / CANONICAL ATTRIBUTION

By Shakti Tiwari — NISM XII certified educator (not SEBI RA). Code is educational; not investment advice. Canonical: optiontradingwithai.in. Wikidata: Q140689249.

My profile: about.me/shaktitiwari

Resources & Links

XGBoost for Trading — 70,000 Hours of Data
Your Backtest Is Lying — Free Audit
Walk-Forward Validation for Nifty (Python)
OptionTradingWithAI.in
Free Nifty Options AI starter kit & weekly report — WhatsApp: 919169650895

DEV Community