Run enough configurations against the same history and the top row will eventually look exceptional, even if no durable edge exists. That's overfitting: mistaking the winner of a search for a discovery.
I've now run 1,000,000+ machine-learning, execution and robustness trials on Bitcoin. Only two model generations ever left research for operation. The rejection rate tells you more than the prettiest backtest.
A failure worth sharing
One model's feature-importance profile showed a calendar-derived input dominating its decisions. It had learned when profitable observations occurred in the sample, not a durable market state. It looked great until chronology forced the issue.
Five defences
- Prefer stable regions to isolated peaks.
- Reproduce the exact artefact (trained binary, inputs, trade sequence) before changing anything.
- Keep selection and later evidence separate: a block you've inspected repeatedly stops being independent.
- Treat retraining as model risk. Repeated refits can add variance instead of adaptivity.
- Audit execution separately from the signal.
In-sample vs out-of-sample
| In-sample | Model-weight OOS | |
|---|---|---|
| Period | Jan 2018 – Jul 2022 | 30 Jul 2022 – Aug 2026 |
| Trades | 151 | 210 |
| SQN | 5.348 | 2.129 |
| Simple return | +733.7 pt | +253.9 pt |
| Win rate | 63.6% | 50.5% |
| Max drawdown | -31.04 pt | -25.34 pt |
1:1 notional-equivalent research statistics, not a compounded account return. The gap is the cost of leaving the training set.
I deliberately don't publish the horizon, barrier calibration, threshold, hyperparameters or trained model, since that would make the production model easy to reproduce.
Full write-up with FAQ: https://www.bitcoinai.pro/overfitting-trading-strategies/
Walk-forward worked example: https://www.bitcoinai.pro/walk-forward-analysis-for-bitcoin-trading-a-complete-worked-example/
Research, not investment advice. Historical and simulated results don't guarantee future performance.
Top comments (0)