Overfitting in trading strategies: what a million trials taught me

Adrian Velai — Mon, 19 Oct 2026 17:30:28 +0000

Run enough configurations against the same history and the top row will eventually look exceptional, even if no durable edge exists. That's overfitting: mistaking the winner of a search for a discovery.

I've now run 1,000,000+ machine-learning, execution and robustness trials on Bitcoin. Only two model generations ever left research for operation. The rejection rate tells you more than the prettiest backtest.

A failure worth sharing

One model's feature-importance profile showed a calendar-derived input dominating its decisions. It had learned when profitable observations occurred in the sample, not a durable market state. It looked great until chronology forced the issue.

Five defences

Prefer stable regions to isolated peaks.
Reproduce the exact artefact (trained binary, inputs, trade sequence) before changing anything.
Keep selection and later evidence separate: a block you've inspected repeatedly stops being independent.
Treat retraining as model risk. Repeated refits can add variance instead of adaptivity.
Audit execution separately from the signal.

In-sample vs out-of-sample

	In-sample	Model-weight OOS
Period	Jan 2018 – Jul 2022	30 Jul 2022 – Aug 2026
Trades	151	210
SQN	5.348	2.129
Simple return	+733.7 pt	+253.9 pt
Win rate	63.6%	50.5%
Max drawdown	-31.04 pt	-25.34 pt

1:1 notional-equivalent research statistics, not a compounded account return. The gap is the cost of leaving the training set.

I deliberately don't publish the horizon, barrier calibration, threshold, hyperparameters or trained model, since that would make the production model easy to reproduce.

Full write-up with FAQ: https://www.bitcoinai.pro/overfitting-trading-strategies/
Walk-forward worked example: https://www.bitcoinai.pro/walk-forward-analysis-for-bitcoin-trading-a-complete-worked-example/

Research, not investment advice. Historical and simulated results don't guarantee future performance.

Why Gradient Boosting Beats Deep Learning

Adrian Velai — Tue, 29 Sep 2026 17:24:59 +0000

Practical lessons for XGBoost trading research

Use chronology before model sophistication. A sophisticated learner on contaminated splits is still contaminated.

Keep the feature set compact. More indicators mostly increase the search space in which overfitting can hide.

Ablate rather than trust importance. The only reliable test of a feature is what happens when it is removed under the same validation rule.

Do not select by TEST. Later blocks are evidence only if they were not used to choose the candidate.

Treat retraining as an experiment. A refit schedule should earn its place rather than being assumed to be adaptive.