DEV Community

mattleeee
mattleeee

Posted on

Title70-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

All 26 Altcoin Backtests Failed: Validation Stops Overfitting

I ran 26 backtest combinations on altcoins — trend and mean reversion, 4h/1h/15m, high win rate and high payoff, six different indicators… not a single one passed three-layer validation. That is not a failure. It is the three-layer method doing exactly its job: stopping overfitting, stopping you from risking real money on a statistical hallucination.

What the three layers are

I don't trade a strategy just because the backtest return is high. It has to clear three gates:

  1. DSR (Deflated Sharpe Ratio). The Sharpe ratio deflated by how many times you tried. The more experiments you run, the easier it is to stumble onto a high Sharpe in pure noise — DSR discounts it by the trial count. My bar is DSR ≥ 0.95 (95% confidence it's not luck).
  2. Split-half. Cut the trades in half by time and check that both halves make money. If only the first half does, the strategy stopped working, or the first half was luck.
  3. Out-of-sample (walk-forward). Use the first 70% of the data to pick parameters/coins, then verify on the last 30% the strategy never touched. No out-of-sample profit = overfitting.

26 combinations, zero survivors

Round Direction Combos Best DSR
1 Trend (EMA / Donchian) 5 0.040
2 Mean reversion (4h) 5 0.056
3 Mean reversion (1h/15m) 4 0.056
4 6 indicators × 2 timeframes 12 0.251

26 combos, 0 reached DSR ≥ 0.95, and out-of-sample was negative for every single one. Each time it "almost worked," the out-of-sample gate exposed it.

The most damning evidence: one indicator, flip the timeframe

In the 12 indicator runs, Stochastic told the clearest story:

  • 15-minute: -25.8%
  • 1-hour: +26.6%

Same indicator, same parameters, only the timeframe changed — and the result flipped completely. If a strategy were real, changing the timeframe could not turn a -26% into a +26%. That's noise: it happened to point up on one period and down on another.

A second warning: Keltner on 1h showed DSR 0.251, the "best" of the twelve. But picking 0.251 out of 12 trials is equivalent to picking 7 heads out of 12 coin flips. It isn't 95% significant; it's just the relative maximum of 12 lottery draws. The true DSR, deflated by the full trial count (20+), is lower still.

Why you must stop when it's time to stop

Quant's biggest enemy isn't "the backtest fails" — it's "the backtest succeeds."

When you've tried 26 times and finally find a combo with "+18% in the training set, +0.5% out-of-sample," your instinct is "one more tweak and it passes." But:

  • Every additional trial raises the DSR penalty;
  • The "improvement" you see is you fitting into the noise;
  • The moment real money enters, it meets a market state you've never seen.

The three-layer method exists to stop overfitting with statistics before you have the rationality to stop yourself. It stopped me 26 times — and the money those 26 "failures" saved is more than any "almost-success" would ever have lost.

One-liner

Returns are measured, not tuned. If your strategy looks good only on one data slice and falls apart when the timeframe or period changes, it has no edge — you didn't miss the right parameter, the whole direction (altcoins × systematic small capital) has negative expectation. Learning to say "no" in front of the data is as important as learning to spot a signal.

Top comments (0)