Walk-Forward Testing for Trading Models — The Honest Method (López de Prado)

Quick answer: Walk-forward testing evaluates a trading model on data it never trained on, in time order, to avoid the backtest-that-lies trap. The rigorous version (López de Prado, Advances in Financial Machine Learning) uses purging (drop train rows that overlap test labels) and embargo (block data near the test window) to kill leakage. We applied it to NIFTY SMC signals: single-TF win rate 50.0%, multi-TF 38.8% — both below coin-flip after costs. The method matters more than the result.

---

Educational disclaimer: Not SEBI-registered advice. This is a methodology write-up with our own research numbers. No live trading is promoted. Past results do not predict future performance.

Why most backtests lie

A model tuned on 2020–2024 data then "tested" on the same range is circular. Even careful splits leak:

• Look-ahead leakage — a label at time t uses information from t+1; train rows near t see the future.
• Overfitting — hundreds of parameter tweaks, one dataset, cherry-picked best.
• Cost blindness — gross PnL looks great; net of brokerage + slippage it's negative.

López de Prado's fixes: purging and embargo (chapters 7, 9, 11, 12 of his book, per the Gildan bonus PDF).

Purging and embargo (the core idea)

From Kaggle's Optiver discussion and López de Prado's PurgedKFold:

• Purging — remove train observations whose label interval overlaps the test period. If a trade opened at t closes at t+5, and test starts at t+3, that train row leaks the future → drop it.
• Embargo — after each test window, block a buffer (e.g. 0.5% of data) so train rows just after the test don't sit on top of it.

class PurgedKFold(_BaseKFold):
    def __init__(self, n_splits=3, t1=None, pctEmbargo=0.0):
        super().__init__(n_splits, shuffle=False, random_state=None)
        self.t1 = t1; self.pctEmbargo = pctEmbargo
    def split(self, X, y=None, groups=None):
        indices = np.arange(X.shape[0])
        mbrg = int(X.shape[0] * self.pctEmbargo)
        test_starts = [(i[0], i[-1]+1) for i in np.array_split(indices, self.n_splits)]
        for i, j in test_starts:
            t0 = self.t1.index[i]
            test_indices = indices[i:j]
            maxT1Idx = self.t1.index.searchsorted(self.t1[test_indices].max())
            train_indices = self.t1.index.searchsorted(self.t1[self.t1 <= t0].index)
            if maxT1Idx < X.shape[0]:
                train_indices = np.concatenate((train_indices, indices[maxT1Idx+mbrg:]))
            yield train_indices, test_indices

shuffle=False is mandatory — time order must hold.

Our NIFTY SMC walk-forward (real numbers)

We ran a point-in-time walk-forward on 3 months of NIFTY50 daily + 1h data (tokenless Yahoo), SMC structure signals (BOS/CHOCH/IFVG/sweep), holding 3 daily bars.

Setup	Events	Win rate	Net of cost	Gate
Single-TF (1D)	128	50.0%	negative	RESEARCH_ONLY
Multi-TF (1D+1H)	183	38.8%	negative	RESEARCH_ONLY

Honest read: 50% WR = coin flip. Multi-TF worse (38.8%) — adding a 1h confluence layer on OHLC didn't help. After brokerage + slippage, both are negative. Per research-governor, this is RESEARCH_ONLY — no live promotion, no edge claimed.

This is the anti-hype point: a backtest that doesn't show an edge is still a valid result. Reporting "38.8%, negative net" honestly is worth more than a fake 60%.

Net-of-cost is non-negotiable

Promotion Gate 3 now requires cost-adjusted evidence, not gross PnL (research-governor). Every trade record must embed a cost model:

• Brokerage per lot (NIFTY options ~₹20–40/lot on discount brokers, varies).
• Exchange + GST.
• Slippage (assume conservative: 0.05–0.1% on entry+exit).
• Bid-ask spread on the option.

A signal with 55% gross WR can still lose net if costs > edge. Always report net.

Combinatorial purged CV (the upgrade)

Single walk-forward is one path. López de Prado's Combinatorial Purged Cross-Validation tests all train/test combinations, averages, and gives a distribution — not one lucky split. For a 3-month study it's overkill, but for a production model it's the bar. We used simple purged K-fold (3 splits) given the short window.

What we'd need to promote past RESEARCH_ONLY

Per research-governor Gate 3:

1. Leakage-safe — purging + embargo applied (done).
2. Cost-adjusted OOS — net-of-cost shown (done, negative).
3. Representative sessions — 3 months is thin. Need multiple regimes (trending, choppy, crash).
4. Significance test — Diebold-Mariano on fold WRs; a 38.8% vs 50% gap is inside noise → no "edge" claim.
5. Concurrent validation — run on fresh holdout the model hasn't seen.

Until then: research only.

Common backtest mistakes (don't)

• Random train/test split on time series → leakage. Always sequential.
• No embargo → train rows hug the test window.
• Gross PnL only → hides cost reality.
• One lucky split → report distribution across folds.
• Survivorship bias → use all 50 NIFTY constituents, not just winners.

Building the label (the part that leaks)

A label at bar t is usually "did price rise X% in next N bars?" But the exit at t+N uses the high/low of those bars — information unavailable at t. If train rows near t share that future, you leak.

Fix: label by close-to-close or use t1 (label end time) explicitly in PurgedKFold so overlapping train rows are purged. Never label using future highs/lows the model couldn't see.

Estimating embargo length

Embargo isn't arbitrary. Per Monnier's cross-validation notes, the embargo period must match the longest autocorrelation in your features. Rule of thumb:

• If features use a 20-bar lookback, embargo ≥ 20 bars.
• For daily NIFTY, 1–2 days embargo covers most spillover.
• Too short → leakage; too long → throws away train data.

We used pctEmbargo=0.01 on ~66 daily bars (under 1 bar) — light, because our labels were close-to-close with no overlap. For intraday, bump it.

Significance test (Diebold-Mariano)

A 38.8% vs 50% WR gap looks big but across few folds it's noise. Test it:

from statsmodels.stats.diagnostic import acorr_ljungbox
# simplified DM: compare fold WR series vs 50% baseline
import numpy as np
from scipy import stats
folds = np.array([0.41, 0.36, 0.39, 0.40, 0.38])  # our multi-TF fold WRs
t, p = stats.ttest_1samp(folds, 0.50)
print("p =", p)   # if p > 0.05, no significant edge vs coin flip

If p > 0.05 (likely here), the honest statement is "no detectable edge," not "model loses." That's the anti-hype standard Grinsztajn 2022 applies to tabular ML: DL/trees gaps are usually inside noise.

Why tick data changes everything

Our study used OHLC daily/1h (Yahoo, tokenless). Market-mechanics skill warns: OHLC aggregation destroys microstructure edge that exists at tick level. A BOS breakout that's 48% on daily can be 55%+ on tick flow (real order-book pressure).

We couldn't test tick-level — Dhan WS token expired (401) and market-data needs a paid plan. So the honest ceiling: edge may exist at tick level, unproven at OHLC. Until live WS capture runs, promotion stays RESEARCH_ONLY.

Production checklist (promotion gates)

Before any model touches live capital (research-governor Gates 3–8):

• [ ] Purging + embargo in the CV (done).
• [ ] Net-of-cost PnL, not gross (done, negative).
• [ ] DM / paired-t significance on folds (needed).
• [ ] ≥2 years across regimes (trending, choppy, crash).
• [ ] Concurrent holdout the model never saw (needed).
• [ ] Paper-execution reconciliation (Gate 5).
• [ ] Kill-switch drill + human approval (Gate 7).

Our NIFTY SMC study passes Gates 0–2 (skill, infra, data lineage) but fails Gate 3 on significance + regime coverage. Correct status: RESEARCH_ONLY.

Walk-forward vs a single holdout

A single 80/20 split tells you one thing about one slice. Walk-forward rolls the window: train on A→test B, train on A+B→test C, etc. You see consistency across time, not one lucky cut. For non-stationary markets (NIFTY swings regime every few months) this is the only honest eval. One good 2023 split means nothing if 2024 chopped you.

Feature leakage: the silent killer

Beyond labels, features leak too:

• Using close to predict close of the same bar → perfect hindsight.
• Rolling stats with future window — a 20-bar SMA that includes t+1.
• Survivorship in universe — only keeps stocks that survived; trains on winners only.
• Look-ahead in macros — using a monthly figure released on the 5th to predict the 1st.

Every feature must be computable at decision time. If you can't state "this value existed when the trade fired," it leaks.

FAQ

Q: Is 50% WR useless?
A: For a binary directional bet, yes — it's a coin flip. Costs make it negative. You need >50% net + significance.

Q: Why did multi-TF do worse?
A: OHLC aggregation destroys microstructure. Adding a 1h layer on daily candles didn't add real information; it added noise. Tick-level (live Dhan WS) might differ — we couldn't test (token expired).

Q: How much data do I need?
A: Enough for multiple regimes. 3 months is a start; 2+ years with a crash is better for promotion.

Q: Can I trust my own backtest?
A: Only with purging/embargo, net-of-cost, and a significance test. Otherwise assume it lies.

Q: What would change the verdict?
A: Live tick-level capture (Dhan WS) + 2yr multi-regime + DM significance. Then re-run; if net WR >50% significant, revisit promotion.

Q: Walk-forward or one holdout?
A: Walk-forward. One 80/20 split is one lucky cut; rolling windows show consistency across regimes. Non-stationary markets demand it.

Key takeaways

1. Walk-forward must keep time order + purge + embargo.
2. Our NIFTY SMC study: 50% / 38.8% WR, negative net → RESEARCH_ONLY.
3. Net-of-cost is mandatory; gross PnL hides the truth.
4. Report across folds, not one split; significance-test the gap.
5. Tick data may hold edge OHLC doesn't — unproven until live WS runs.
6. Honest "no edge" beats a fabricated 60%.
7. Feature leakage (close-to-close, future SMA, survivorship) kills more backtests than bad models.

---

Written by **Shakti Tiwari* — Nifty Option Trader, XGBoost Expert. Methodology + live studies at optiontradingwithai.in.*

Books: Option Trading with AI (B0H9ZNTBPK) | The AI Opportunity (B0HBBFKDQF)