Three-Layer Strategy Validation and the Blind Spot That Bit Me
My validation stack had three layers, and all three were green. DSR came out at 1.000. Coin-block Monte Carlo showed a 0% ruin rate across hundreds of randomized pool combinations. Split-half stability was positive in both halves of the trade sequence. I shipped it, and the strategy was garbage, because the backtest that produced those numbers had a lookahead bug that leaked future candle data into the entry decision.
The uncomfortable lesson: all three layers evaluate the statistical properties of one trade sequence. They assume that sequence was generated honestly. None of them audits how trades were generated. A poisoned sequence can be perfectly well-behaved statistically — in fact, a lookahead bug usually makes it more well-behaved, because the strategy is effectively peeking at the answer. High DSR, zero ruin, clean split-half — that's exactly what a leak looks like.
This post is the four-layer design I run now, with the code signatures and a table of what each layer actually catches. The fourth layer is the one that would have saved me.
Layer 1: Deflated Sharpe Ratio
The DSR (Bailey & López de Prado) adjusts an observed Sharpe ratio for the number of trials you ran, and for the non-normality of returns. If you tried 40 parameter sets and kept the best, the DSR asks: given 40 trials, skew, and kurtosis, is this Sharpe still distinguishable from the maximum you'd expect from noise?
import numpy as np
from scipy.stats import norm
def deflated_sharpe_ratio(
returns: np.ndarray,
n_trials: int,
sr_variance: float | None = None,
periods_per_year: int = 252,
) -> float:
"""
Returns the probability that the true Sharpe exceeds zero,
after deflating for n_trials and return non-normality.
returns: per-period (e.g. daily) strategy returns, chronological
n_trials: number of configurations you actually evaluated
sr_variance: variance of Sharpe across your trials; if None,
use the estimator below (conservative for small n_trials)
"""
r = np.asarray(returns, dtype=float)
r = r[np.isfinite(r)]
n = r.size
if n < 30 or n_trials < 1:
return 0.0
mu, sd = r.mean(), r.std(ddof=1)
if sd == 0:
return 0.0
sr = mu / sd # per-period Sharpe
g1 = ((r - mu) ** 3).mean() / sd**3
g2 = ((r - mu) ** 4).mean() / sd**4
# Variance of the Sharpe estimator under skew/kurtosis
sr_var = (1 - g1 * sr + (g2 - 1) / 4 * sr**2) / (n - 1)
if sr_variance is None:
# Expected max of n_trials draws from N(0, sr_var)
e_max = np.sqrt(sr_var) * (
(1 - np.euler_gamma) * norm.ppf(1 - 1 / n_trials)
+ np.euler_gamma * norm.ppf(1 - 1 / (n_trials * np.e))
)
else:
e_max = np.sqrt(sr_variance) * norm.ppf(1 - 1 / n_trials)
z = (sr - e_max) / np.sqrt(sr_var)
return float(norm.cdf(z))
Two things matter operationally. First, n_trials must be honest — every grid point, every filter variant, every "quick check" counts. Second, the DSR is only as good as the return series you feed it. It has no opinion on whether that series is real.
Layer 2: Coin-Block Monte Carlo
I don't bootstrap individual trades. Trades cluster — volatility regimes, correlated entries, streaks. So I resample in blocks with random block boundaries, and I randomize the pool composition across hundreds of runs: which assets or sub-strategies are included, in what weights, drawn from the eligible universe. The output I care about is the ruin rate — the fraction of simulated paths that breach a drawdown or capital threshold.
def coin_block_monte_carlo(
trade_pnls: np.ndarray,
n_sims: int = 500,
mean_block: int = 10,
ruin_dd: float = 0.25,
initial_capital: float = 1.0,
rng: np.random.Generator | None = None,
) -> dict:
"""
Resample the trade sequence in random-length blocks and measure
the fraction of paths that breach ruin_dd.
trade_pnls: per-trade PnL as a fraction of capital, chronological
mean_block: average block length (geometric block lengths)
ruin_dd: drawdown threshold as fraction of peak equity
"""
rng = rng or np.random.default_rng(7)
pnl = np.asarray(trade_pnls, dtype=float)
n = pnl.size
ruins, finals = 0, np.empty(n_sims)
for s in range(n_sims):
path, i = [], 0
while len(path) < n:
blk = max(1, int(rng.geometric(1.0 / mean_block)))
start = rng.integers(0, n)
idx = (start + np.arange(blk)) % n # wrap-around blocks
path.extend(pnl[idx])
i += blk
eq = initial_capital + np.cumsum(np.array(path[:n]))
peak = np.maximum.accumulate(np.concatenate([[initial_capital], eq]))
dd = 1.0 - eq / peak[1:]
if dd.max() >= ruin_dd:
ruins += 1
finals[s] = eq[-1]
return {
"ruin_rate": ruins / n_sims,
"median_final": float(np.median(finals)),
"p05_final": float(np.percentile(finals, 5)),
}
This layer answers: given this sequence's dependence structure, how fragile is the equity path? It does not answer whether the sequence could have been produced by the strategy in the first place. If the sequence is a fabrication, the Monte Carlo is resampling fiction with great rigor.
Layer 3: Split-Half Stability
The simplest and most brutal check: split the trade sequence in two, require positive expectation in both halves. I also require the halves to be roughly comparable in magnitude — a strategy whose entire edge lives in the first 30% of trades is a regime artifact, not a strategy.
def split_half_stability(
trade_pnls: np.ndarray,
min_trades_per_half: int = 30,
max_decay_ratio: float = 0.5,
) -> dict:
"""
Require positive mean in both halves and that the second half
retains at least max_decay_ratio of the first half's mean.
trade_pnls: per-trade PnL, chronological
"""
pnl = np.asarray(trade_pnls, dtype=float)
n = pnl.size
if n < 2 * min_trades_per_half:
return {"pass": False, "reason": "insufficient trades"}
h1, h2 = pnl[: n // 2], pnl[n // 2 :]
m1, m2 = h1.mean(), h2.mean()
ok = m1 > 0 and m2 > 0 and m2 >= max_decay_ratio * m1
return {
"pass": bool(ok),
"mean_h1": float(m1),
"mean_h2": float(m2),
"decay_ratio": float(m2 / m1) if m1 > 0 else float("nan"),
}
Split-half catches overfitting to a specific time window, edge decay, and single-regime dependence. It cannot catch a leak, because a leak inflates both halves. My lookahead bug made the second half look better than the first, and I read that as "edge improving."
The Blind Spot
Here's the thing all three layers share: they take trade_pnls as an axiom. Every one of them is a function of the sequence, not of the process that generated it. The DSR deflates for how many sequences you tried, not for whether each sequence was built from legal information. The Monte Carlo reshuffles the sequence. Split-half cuts it. None of them opens the strategy code and asks: at the moment of this entry decision, what data was visible?
My bug was mundane. The feature pipeline computed a rolling statistic with center=True on a window that extended past the decision timestamp, then joined features to signals on bar index. The backtest engine was happy to serve it. Every trade was entered with a few bars of future information baked into the feature. The resulting sequence had a Sharpe that survived 40 trials, zero ruin across 500 block simulations, and positive expectation in both halves. DSR = 1.000. Ruin = 0%. Both halves positive. All three green, all three meaningless.
The failure mode is structural: statistical validation is orthogonal to implementation correctness. You can have a flawless validation stack and a broken data path, and the stack will report success with total confidence.
Layer 4: Replay Testing
The fix is to stop trusting the backtest's trade sequence and instead force two independent code paths to agree on the same history. Replay testing drives the production trading code — the same signal generator, the same risk module, the same order logic that runs live — over real historical candles, bar by bar, with the data feed truncated at each decision timestamp. Then it diffs the resulting trades against the backtest's trades. Any divergence is a bug in one of the two, and the diff tells you where.
from dataclasses import dataclass
from typing import Callable, Iterable
@dataclass
class Candle:
ts: int
open: float
high: float
low: float
close: float
volume: float
@dataclass
class Trade:
ts: int
side: str
qty: float
price: float
def replay_production(
candles: Iterable[Candle],
on_candle: Callable[[list[Candle]], list[Trade]],
warmup: int = 200,
) -> list[Trade]:
"""
Drive production code over history with a strictly causal feed.
on_candle receives ONLY candles visible at the current timestamp.
Returns the trade list the production code would have produced.
"""
history: list[Candle] = []
trades: list[Trade] = []
for c in candles:
history.append(c)
if len(history) <= warmup:
continue
# The engine must never see beyond `history` here.
trades.extend(on_candle(history))
return trades
def compare_trade_streams(
backtest: list[Trade],
replay: list[Trade],
ts_tolerance: int = 0,
qty_rel_tol: float = 1e-6,
) -> dict:
"""
Diff two trade streams. Any mismatch is a validation failure,
regardless of what layers 1-3 said.
"""
if len(backtest) != len(replay):
return {"match": False, "reason": "trade count differs",
"backtest_n": len(backtest), "replay_n": len(replay)}
mismatches = []
for i, (b, r) in enumerate(zip(backtest, replay)):
if abs(b.ts - r.ts) > ts_tolerance or b.side != r.side:
mismatches.append({"i": i, "kind": "timing_or_side",
"backtest": b, "replay": r})
continue
rel = abs(b.qty - r.qty) / max(abs(b.qty), 1e-12)
if rel > qty_rel_tol:
mismatches.append({"i": i, "kind": "qty",
"backtest": b, "replay": r})
return {"match": not mismatches, "mismatches": mismatches[:20],
"n_mismatches": len(mismatches)}
The critical property is causality enforcement. The replay harness hands the production code a list that ends at the current bar — there is no future data in the object, so a center=True bug becomes a crash or an obviously wrong feature, not a silent edge. If the production code needs a warmup window, it gets only the past. If it tries to index forward, it raises.
Two practical notes from running this:
- Diff at the trade level, not the signal level. Signals can diverge harmlessly due to float ordering. Trades — timestamp, side, quantity — must match. A single-trade mismatch is worth a full investigation.
- Run replay on the same candles the backtest used, byte for byte. If the backtest reads from a research store and replay reads from a production feed, you're testing data plumbing, not strategy logic. Pin one dataset.
When I added this layer, the lookahead bug surfaced in minutes. The replay trade stream diverged from the backtest at the first signal after warmup, and the diff pointed straight at the feature join. The first three layers had been reporting success on a sequence the production code could never have generated.
What Each Layer Catches
| Layer | Catches | Does not catch |
|---|---|---|
| DSR | Multiple-testing inflation, non-normal Sharpe, selection bias across trials | Lookahead, data leakage, execution mismatches, regime fragility |
| Coin-block Monte Carlo | Path fragility, drawdown/ruin risk under dependence, pool-composition sensitivity | Whether the trade sequence is achievable by production code |
| Split-half | Edge decay, single-regime dependence, time-window overfitting | Leaks (they inflate both halves), implementation divergence |
| Replay | Lookahead, engine/production divergence, data-path bugs, timestamp handling | Overfitting, multiple testing, ruin risk, regime dependence |
The layers are complementary, and the order matters less than the coverage. Layers 1–3 audit the statistics of the sequence. Layer 4 audits the provenance of the sequence. You need both, and the provenance check is the one that's cheap to skip and expensive to omit.
The general principle: every validation layer should be tested against a deliberately broken strategy. Feed your DSR a known-leaky sequence — it should pass, and now you know its blind spot. Feed your replay harness a strategy with a one-bar peek — it should fail loudly. A validation layer you've never seen fail is a layer you don't understand.
More notes like this ship every week on this site.
Daily Picks
The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-11-16 08:46:12
Long
| Pair | Signal | Price | Take Profit | Stop Loss | R/R |
|---|---|---|---|---|---|
| SOXL | 高质量标的做多 | $57.98 | $59.8354 | $56.8204 | 1:1.6 |
Short
| Pair | Signal | Price | Take Profit | Stop Loss | R/R |
|---|---|---|---|---|---|
| EVAA | 热点跟涨 | $2.7639 | $2.6257 | $2.8606 | 1:1.4 |
2 picks selected. Scanner runs every 15 minutes.
Top comments (0)