Cross-post. Original: stellarbytecapital.com/blog/backtesting-overfitting
A profitable backtest is the easiest thing to produce in all of quant trading, and the most worthless. Give a motivated person historical data and enough parameters, and they'll hand you a strategy that turned $10k into $10M — on paper, on data that already happened. The hard part was never getting a good backtest. It's getting one that predicts anything about tomorrow. The gap between those two is overfitting.
What overfitting actually is
Overfitting is when your strategy learns the noise in your historical data instead of a real, repeatable pattern. Markets are mostly noise with a little signal. A model with enough freedom will memorize the noise — every lucky spike, every specific dip — because that maximizes backtest performance. It fits the past perfectly and the future not at all.
A backtest tells you what would have happened. Overfitting is mistaking that for what will happen.
The mechanisms that make backtests lie
Multiple testing (the big one). If you try 1,000 variations and keep the best, you've almost certainly found one that looks great by chance. With enough attempts, random noise produces gorgeous Sharpe ratios. The strategy you selected is the survivor of a lottery — and lottery winners don't repeat. Every parameter you tune burns statistical power you rarely account for.
Lookahead bias. Using information that wouldn't have been available at decision time: the day's close to decide a trade at its open, a signal computed over the full dataset before splitting. Subtle — often a single misaligned index — and it vanishes the instant you go live. Backtest and live sharing one code path kills this class of bug.
Survivorship bias. Backtesting only on assets that exist today deletes every company that went bankrupt and every token that went to zero. Use point-in-time data that includes the dead.
Ignoring costs and fills. A high-turnover strategy can look brilliant with zero fees and perfect fills, then die on real spreads, slippage, and impact. If a small change in your cost assumption flips profit to loss, it never had an edge.
How to make a backtest honest
Out-of-sample and walk-forward. Never judge a strategy on data you used to build it. Develop in-sample, test once out-of-sample. Better: walk-forward — optimize on a rolling window, test on the next unseen window, roll, repeat. A strategy that survives many out-of-sample windows has something; one that only shines in-sample was memorizing.
Keep a locked holdout you touch once. Reserve recent history that neither you nor your optimizer look at during research. It only works if you look once — every re-run with a tweak contaminates it.
Prefer fewer parameters and robust plateaus. Every degree of freedom is room to overfit. Prefer a broad plateau of parameter values that all work over a single razor-sharp peak. If only one exact setting is profitable, you've found noise.
Account for the search you did. Be honest about how many things you tried and discount accordingly. A deflated Sharpe ratio adjusts for the number of trials; even "I tested 50 variants, so this p-value is meaningless" beats pretending the winner arrived in one shot.
What to avoid
- Optimizing on all your data — no out-of-sample means no evidence.
- Re-running until you like the holdout — then it's not a holdout.
- Chasing the highest backtest return — the best-looking backtest in a large search is usually the most overfit.
- Zero-cost, perfect-fill assumptions — model fees, slippage, impact, or your edge is imaginary.
- Too many parameters, one magic setting — brittleness is the signature of a curve fit.
Be adversarial toward your own results: assume every great backtest is overfit until it survives data it never saw, realistic costs, and a sober accounting of how hard you searched. A believable strategy usually looks modest in backtest. A result that seems too good to be true isn't a discovery — it's the warning.
We're Xingyao Byte — building quant trading systems, backtesting engines, secure AI-execution layers, and payment platforms. Remote, async-first → stellarbytecapital.com
Top comments (0)