All 26 Altcoin Backtests Failed: Validation Stops Overfitting
I ran 26 backtest combinations on altcoins — trend and mean reversion, 4h/1h/15m, high win rate and high payoff, six different indicators… not a single one passed three-layer validation. That is not a failure. It is the three-layer method doing exactly its job: stopping overfitting, stopping you from risking real money on a statistical hallucination.
What the three layers are
I don't trade a strategy just because the backtest return is high. It has to clear three gates:
- DSR (Deflated Sharpe Ratio). The Sharpe ratio deflated by how many times you tried. The more experiments you run, the easier it is to stumble onto a high Sharpe in pure noise — DSR discounts it by the trial count. My bar is DSR ≥ 0.95 (95% confidence it's not luck).
- Split-half. Cut the trades in half by time and check that both halves make money. If only the first half does, the strategy stopped working, or the first half was luck.
- Out-of-sample (walk-forward). Use the first 70% of the data to pick parameters/coins, then verify on the last 30% the strategy never touched. No out-of-sample profit = overfitting.
26 combinations, zero survivors
| Round | Direction | Combos | Best DSR |
|---|---|---|---|
| 1 | Trend (EMA / Donchian) | 5 | 0.040 |
| 2 | Mean reversion (4h) | 5 | 0.056 |
| 3 | Mean reversion (1h/15m) | 4 | 0.056 |
| 4 | 6 indicators × 2 timeframes | 12 | 0.251 |
26 combos, 0 reached DSR ≥ 0.95, and out-of-sample was negative for every single one. Each time it "almost worked," the out-of-sample gate exposed it.
The most damning evidence: one indicator, flip the timeframe
In the 12 indicator runs, Stochastic told the clearest story:
- 15-minute: -25.8%
- 1-hour: +26.6%
Same indicator, same parameters, only the timeframe changed — and the result flipped completely. If a strategy were real, changing the timeframe could not turn a -26% into a +26%. That's noise: it happened to point up on one period and down on another.
A second warning: Keltner on 1h showed DSR 0.251, the "best" of the twelve. But picking 0.251 out of 12 trials is equivalent to picking 7 heads out of 12 coin flips. It isn't 95% significant; it's just the relative maximum of 12 lottery draws. The true DSR, deflated by the full trial count (20+), is lower still.
Why you must stop when it's time to stop
Quant's biggest enemy isn't "the backtest fails" — it's "the backtest succeeds."
When you've tried 26 times and finally find a combo with "+18% in the training set, +0.5% out-of-sample," your instinct is "one more tweak and it passes." But:
- Every additional trial raises the DSR penalty;
- The "improvement" you see is you fitting into the noise;
- The moment real money enters, it meets a market state you've never seen.
The three-layer method exists to stop overfitting with statistics before you have the rationality to stop yourself. It stopped me 26 times — and the money those 26 "failures" saved is more than any "almost-success" would ever have lost.
One-liner
Returns are measured, not tuned. If your strategy looks good only on one data slice and falls apart when the timeframe or period changes, it has no edge — you didn't miss the right parameter, the whole direction (altcoins × systematic small capital) has negative expectation. Learning to say "no" in front of the data is as important as learning to spot a signal.
Top comments (0)