DEV Community

mattleeee
mattleeee

Posted on

pubtest_2400

All 26 Backtest Combinations Failed: How Three-Layer Validation Stops Overfitting

I ran 26 backtest combinations on altcoins — trend and mean reversion, 4h/1h/15m, high win rate and high payoff, six different indicators… not a single one passed three-layer validation. That is not a failure. It is the three-layer method doing exactly its job: stopping overfitting, stopping you from risking real money on a statistical hallucination.

What the three layers are

I don't trade a strategy just because the backtest return is high. It has to clear three gates:

  1. DSR (Deflated Sharpe Ratio). The Sharpe ratio deflated by how many times you tried. The more experiments you run, the easier it is to stumble onto a high Sharpe in pure noise — DSR discounts it by the trial count. My bar is DSR ≥ 0.95 (95% confidence it's not luck).
  2. Split-half. Cut the trades in half by time and check that both halves make money. If only the first half does, the strategy stopped working, or the first half was luck.
  3. Out-of-sample (walk-forward). Use the first 70% of the data to pick parameters/coins, then verify on the last 30% the strategy never touched. No out-of-sample profit = overfitting.

26 combinations, zero survivors

Round Direction Combos Best DSR
1 Trend (EMA / Donchian) 5 0.040
2 Mean reversion (4h) 5 0.056
3 Mean reversion (1h/15m) 4 0.056
4 6 indicators × 2 timeframes 12 0.251

26 combos, 0 reached DSR ≥ 0.95, and out-of-sample was negative for every single one. Each time it "almost worked," the out-of-sample gate exposed it.

The most damning evidence: one indicator, flip the timeframe

In the 12 indicator runs, Stochastic told the clearest story:

  • 15-minute: -25.8%
  • 1-hour: +26.6%

Same indicator, same parameters, only the timeframe changed — and the result flipped completely. If a strategy were real, changing the timeframe could not turn a -26% into a +26%. That's noise: it happened to point up on one period and down on another.

A second warning: Keltner on 1h showed DSR 0.251, the "best" of the twelve. But picking 0.251 out of 12 trials is equivalent to picking 7 heads out of 12 coin flips. It isn't 95% significant; it's just the relative maximum of 12 lottery draws. The true DSR, deflated by the full trial count (20+), is lower still.

Top comments (0)