Nifty Options Data Pipeline: From WebSocket Tick to Clean Feature Store
By Shakti Tiwari · Engineering note · Not investment advice
A trading model is only as honest as the data feeding it. Two bugs destroy more backtests than bad math: duplicated ticks (from reconnects) and leaked labels (from future-looking features). This note connects both into one pipeline you can actually run.
Stage 1 — Idempotent ingest (stop duplicates)
Your WebSocket feed will resend a trailing window on reconnect. Without dedup, those duplicates inflate your sample count and corrupt labels. Use a content-hash + per-stream monotonic sequence-id layer. Test: accepted_total == distinct_ticks under injected reconnects. Concretely, the collector parses each incoming tick into a normalized dict, computes a SHA-256 identity hash from the tick's defining fields, drops the tick if its sequence ≤ the stream's last-seen sequence or if its hash is in the recent-seen set, otherwise persists and updates the seen-set. The result: a stream where every tick is seen exactly once, regardless of how many times the provider resends it.
Stage 2 — Strict time ordering (stop leakage)
Every feature computed at time t must use only data ≤ t. The label is the forward move t→t+h — the only future you may see. Never shuffle timestamps. Walk-forward, never one train/test split. A practical rule: ban any feature whose definition references a future timestamp. Grep your feature code for shift(-1), future_, next_, and target_shift. If found, that feature is leaking.
Stage 3 — Feature store contract
Persist normalized ticks with their content-hash as the primary key. Downstream models read from the store, never the raw socket. This makes every experiment reproducible and every label auditable. Schema sketch: tick_hash TEXT PRIMARY KEY; symbol TEXT, ts INTEGER, price REAL, size REAL, side TEXT, seq INTEGER; features_json TEXT (computed ≤ ts only). Because tick_hash is the key, a reconnect that re-sends a tick is a no-op at the DB level too — defense in depth.
Stage 4 — Walk-forward labels
Labels are computed in a separate, time-ordered pass: for each tick at t, the label is the realized move to t+h. Store labels in a table keyed by tick_hash so they join cleanly to features without ever touching future features.
Stage 5 — Reproducibility
Pin everything: data snapshot hash, feature version, label version, model commit. A result you cannot reproduce is not a result — it is a story. Store the provenance next to the model artifact.
Deep-dive: a minimal end-to-end example
Imagine a Thursday expiry. At 09:15 the feed opens. Ticks stream for NIFTY50 names. Without the pipeline, a 3-second ISP blip at 11:42 resends 1,200 ticks from 11:41:58. Your naive collector appends them. By 15:30 you have 1,200 phantom rows that look like fresh liquidity but are echoes. Your end-of-day feature for "volume in last 5 min" is inflated; your label for "move into expiry" is shifted. With the pipeline: the 1,200 retransmitted ticks hash to keys already in the store. The collector drops all 1,200. Your volume feature is correct. Your label is correct. The model trained that night sees the market, not the echo.
Why this is a governance issue
Because the corruption is silent. There is no error, no exception, no log line saying "you trained on lies." Only the eventual live loss reveals it. Governance means building the check (duplicate ratio alert) and the test (injected reconnect in staging) so the lie is caught before capital is risked. That is the entire point of the Authority OS discipline: verify, don't assume.
Operational monitoring
Duplicate ratio = (received − accepted) / received. Alert on spikes. Label leakage test: a unit test asserting no feature column correlates >0.99 with the label. Reconnect counter: correlates with duplicate bursts. A simple Prometheus counter per stream, scraped every 15s, catches most real-world breakage before it poisons a dataset.
Common mistakes
Computing rolling features with a centered window (leaks both sides) — use trailing only. Forgetting to drop heartbeats/control frames before ingest. Using system time as a feature without realizing the clock drifts vs exchange time. Storing features and labels in the same row, then shuffling for train/test. Running two symbols through one shared last_seq.
What this gives you
Not a profitable model — an honest one. When your worst-regime walk-forward fold is flat or negative, you've learned the truth early instead of after real capital.
FAQ
Q: Why not just dedup at the database? You can, but dedup at ingest is cheaper and catches reordering before storage. A unique constraint is a good second line of defence. Q: How big should the seen-set window be? From your provider's max retransmit window; 100k entries covers days. Q: Does this handle order-book deltas? Yes — normalize the delta into an identity key and hash it. Q: What if my provider sends both trades and quotes? Run two collectors, each with its own last_seq and seen-set. Q: How do I test the whole pipeline? Inject a reconnect + replay in staging before any live training.
Related
Extended case study: the Thursday that looked amazing
You backtest a Nifty options strategy on clean data and get a beautiful equity curve. Then you deploy. First Thursday it loses. You check the pipeline logs: the label for that Thursday was computed using the close at t+5, but a holiday shifted the session, and your label pass silently used a future session's close. The pipeline had no guard against label windows crossing a session boundary. You add a session-boundary check to the label pass: a label may only use data from the same trading session as t. The bug is fixed, the curve is honest, and the strategy is revealed to be marginal — which is the truth you needed before risking capital.
Schema evolution
As you add features, version the schema. A feature added in March must not silently change the meaning of a column computed in January. Store a feature-version hash alongside the data. When you retrain, pin the version. Reproducibility is not glamorous, but it is the difference between "I improved the model" and "I changed the data and called it improvement."
Batch vs streaming
The pipeline works the same in batch (historical replay) and streaming (live). In batch, ingest the whole file through the collector — it still dedups, catching any duplication in the source. In streaming, the collector dedups across reconnects. The invariant is identical: tick_hash is the key, features ≤ ts, labels in a separate pass. Test both paths with the same harness.
Practical walkthrough: assembling the pipeline
Step 1 — Stand up the idempotent collector (see the collector note) and point it at your feed. Step 2 — On each accepted tick, write it to the store keyed by tick_hash with features_json computed using only data up to ts. Step 3 — Run a separate label job that, for each tick, computes the forward move and writes it to a labels table keyed by the same tick_hash. Step 4 — Build a training view that joins features to labels on tick_hash — never by row position, never by shuffled index. Step 5 — Add the duplicate-ratio and leakage-correlation tests to CI. Step 6 — Snapshot the data hash, feature version, and label version alongside every model you train.
What good looks like: a training view you can rebuild exactly from a version string, a duplicate ratio near zero, and a feature-label correlation test that passes. If you cannot rebuild the exact dataset from the version, your experiment is not reproducible and therefore not trustworthy.
Operational runbook
Daily: check duplicate ratio and reconnect count. Weekly: re-run the leakage correlation test on the latest data. On any provider change: re-run the injected-reconnect harness in staging before trusting live data. On model update: bump the feature version and re-snapshot. These few checks turn a fragile pipeline into a dependable one.
Key takeaways
The pipeline is boring on purpose. Idempotent ingest, time-ordered features, separate labels, reproducible snapshots. None of it is clever, all of it is necessary. A backtest built on this foundation means something; one built without it is theatre. The discipline is the product.
A note on scale
At one ticker this fits in a laptop SQLite file. At five hundred tickers you will want partitioned Parquet and a columnar store, but the invariants do not change: tick_hash key, features at or before ts, labels in a separate pass. Scale the storage, not the rules. The rules are what make the data trustworthy at any size.
An illustrated example end to end
Follow one tick through the pipeline. At 10:31:04.250 a NIFTY tick arrives: price 24120.50, size 75, side buy, seq 882314. The collector hashes the identity fields, checks seq against last_seq for NIFTY (882313), accepts, stores the tick under its hash. At 10:31:07 the socket blips; on reconnect the provider resends ticks from 10:31:04.100. The resent 10:31:04.250 tick arrives again with seq 882314. The collector sees seq not greater than last_seq and drops it — zero duplication. Meanwhile the feature job, running on the accepted store, computed at 10:31:04.250 only the IV rank from data up to that timestamp; it never sees the 10:31:07 blip. The label job later marks the forward move. The training view joins them on tick_hash. The result: one tick, one feature row, one label, no echo, no leak. Multiply by millions of ticks and a year of expiries, and this determinism is what makes the backtest mean something.
An illustrated example end to end
Follow one tick through the pipeline. At 10:31:04.250 a NIFTY tick arrives: price 24120.50, size 75, side buy, seq 882314. The collector hashes the identity fields, checks seq against last_seq for NIFTY (882313), accepts, stores the tick under its hash. At 10:31:07 the socket blips; on reconnect the provider resends ticks from 10:31:04.100. The resent tick arrives again with seq 882314. The collector sees seq not greater than last_seq and drops it — zero duplication. Meanwhile the feature job, running on the accepted store, computed at 10:31:04.250 only the IV rank from data up to that timestamp; it never sees the blip. The label job later marks the forward move. The training view joins them on tick_hash. The result: one tick, one feature row, one label, no echo, no leak. Multiply by millions of ticks and a year of expiries, and this determinism is what makes the backtest mean something.
Why engineers skip this and regret it
The pipeline feels like overhead when you just want to "try a model." So you load a CSV, shuffle, fit, and get a great score. Two weeks later live trading loses and you cannot explain why. The missing piece was never the model — it was the discipline of knowing your data. The idempotent collector and the time-ordered label pass take an afternoon to build and save months of confusion. The cost of skipping is not a failed experiment; it is a false successful one, which is far more expensive.
Glossary
Idempotent — a process whose repeated application produces the same result as a single application; here, ingesting the same tick twice stores it once. Content-hash — a fingerprint of a tick's identity fields, used to detect semantic duplicates. Walk-forward — a validation scheme that rolls a training window forward in time, never looking ahead. Feature store — a repository of precomputed features keyed for reuse. Label leakage — a feature that contains information from the future, making the model look better than it is. Tick_hash — the primary key binding a tick to its features and label.
Further reading
The companion notes cover the collector implementation, the leakage-free XGBoost pipeline, and the myth-buster. The governance repository on GitHub holds the leakage checker and backtest scorecard tools referenced throughout. Read the collector note first; every other piece assumes its dedup layer is in place.
Related reading
About the Author
Shakti Tiwari writes about AI, local AI agents, XGBoost, and options trading with AI — in Hinglish, for Indian traders and builders. Educational, no-hype, code-first.
- 🐦 X: https://x.com/shaktitiwari
- 💼 LinkedIn: https://linkedin.com/in/shakti-tiwari-a3b22a38b
- 💻 GitHub: https://github.com/shaktitiwari715-ai
- 📝 DEV.to: https://dev.to/shaktitiwari
- 🌐 Site: https://optiontradingwithai.in
Educational only. Not investment advice.
Continue Reading (Authority OS series)
- WebSocket collector idempotent
- Leakage-free XGBoost pipeline
- Nifty data pipeline
- AI Trading Myth-Buster
- Interactive Tools Hub
Top comments (0)