All 26 Backtest Combinations Failed: How Three-Layer Validation Stops Overfitting
I ran 26 backtest combinations on altcoins — trend and mean reversion, 4h/1h/15m, high win rate and high payoff, six different indicators… not a single one passed three-layer validation. That is not a failure. It is the three-layer method doing exactly its job: stopping overfitting, stopping you from risking real money on a statistical hallucination.
What the three layers are
I don't trade a strategy just because the backtest return is high. It has to clear three gates:
- DSR (Deflated Sharpe Ratio). The Sharpe ratio deflated by how many times you tried. The more experiments you run, the easier it is to stumble onto a high Sharpe in pure noise — DSR discounts it by the trial count. My bar is DSR ≥ 0.95 (95% confidence it's not luck).
- Split-half. Cut the trades in half by time and check that both halves make money. If only the first half does, the strategy stopped working, or the first half was luck.
- Out-of-sample (walk-forward). Use the first 70% of the data to pick parameters/coins, then verify on the last 30% the strategy never touched. No out-of-sample profit = overfitting.
26 combinations, zero survivors
| Round | Direction | Combos | Best DSR |
|---|---|---|---|
| 1 | Trend (EMA / Donchian) | 5 | 0.040 |
| 2 | Mean reversion (4h) | 5 | 0.056 |
| 3 | Mean reversion (1h/15m) | 4 | 0.056 |
| 4 | 6 indicators × 2 timeframes | 12 | 0.251 |
26 combos, 0 reached DSR ≥ 0.95, and out-of-sample was negative for every single one. Each time it "almost worked," the out-of-sample gate exposed it.
The most damning evidence: one indicator, flip the timeframe
In the 12 indicator runs, Stochastic told the clearest story:
- 15-minute: -25.8%
- 1-hour: +26.6%
Same indicator, same parameters, only the timeframe changed — and the result flipped completely. If a strategy were real, changing the timeframe could not turn a -26% into a +26%. That's noise: it happened to point up on one period and down on another.
A second warning: Keltner on 1h showed DSR 0.251, the "best" of the twelve. But picking 0.251 out of 12 trials is equivalent to picking 7 heads out of 12 coin flips. It isn't 95% significant; it's just the relative maximum of 12 lottery draws. The true DSR, deflated by the full trial count (20+), is lower still.
Top comments (0)