Flip a fair coin 10 times. Getting 8 heads is unusual, about a 5% chance. Now get 100 people to do it. A few of them will hit 8 heads, and none of them has a special coin.
Backtesting works the same way. Each feature set, model, lookback window and threshold you try is another person flipping coins. The more you try, the better your best result gets, even if nothing works.
The question to ask
When someone shows you a great backtest, ask: out of how many?
If it's the best of 200 variations, the bar for "impressive" is much higher than if it was the only thing tested.
Correcting for it
The deflated Sharpe ratio (Bailey and López de Prado) asks: given how many trials were run, and how much their results varied, what Sharpe ratio would the best of them reach by luck alone? Then it measures how far the actual best result is above that bar, also accounting for skew and fat tails.
The idea in plain words:
- Count every configuration you tried, including the ones you threw away.
- Estimate how big the best result would be if none of them had skill.
- Only call it a finding if the real best clears that bar with room to spare.
What we do
Every model and setting we try for the stock-ranking models is logged as a trial. The leaderboard shows the trial count next to the results, and the deflated Sharpe of the best one. Across our twelve model families, none cleared the bar.
A habit worth keeping
Keep a trial log, even a plain CSV with one row per run: date, config, result. It costs nothing and it's the only honest way to answer "out of how many?", including to yourself.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/models
Educational, not investment advice.
Top comments (0)