A backtest does not need to crash to be broken.
The dangerous version runs cleanly, produces a smooth equity curve, and reports a Sharpe ratio that makes deployment feel obvious. Then live performance diverges because one quiet assumption allowed future information, ignored trading costs, or rewarded the luckiest configuration.
Before trusting a backtest, I now ask five questions.
1. Could later data change an earlier signal?
This is the simplest causal invariant I know:
- Run the strategy on the complete dataset.
- Delete the final 20% of the data.
- Run it again.
- Compare signals in the shared historical interval.
The earlier signals should be identical. If they change, investigate global normalization, centered windows, backward fills, full-sample feature selection, or any code that fits before splitting.
This test requires access to the strategy pipeline itself. A post-hoc CSV auditor cannot reconstruct source code it has never seen, so I treat this as a separate source-level invariant—not something a CSV report can magically prove.
2. Is the signal suspiciously aligned with returns it should not know?
For every row t, define the return that was actually knowable when the signal was created. Then inspect:
- same-row signal/return correlation;
- individual future offsets;
- cumulative forward returns over several horizons;
- the number of valid observations behind each result.
A correlation screen is a warning system, not proof of leakage. Legitimate strategies can correlate with later returns; contaminated strategies can also evade a simple threshold. The correct response to an extreme result is investigate, not “automatically guilty” or “automatically safe.”
3. Does performance survive chronological holdouts?
Random train/test splits are usually inappropriate for time-dependent strategies. Use chronological slices and preserve a gap between fitting and evaluation when labels or features overlap in time.
A useful sequence is:
earlier training evidence
→ purge gap
→ later evaluation slice
Repeat this over expanding windows. The purpose is not to demand identical performance everywhere. It is to expose whether one period carries the entire claim.
4. Are costs charged when the position changes?
Costs belong to turnover, not merely to rows where a position is non-zero.
For positions p[t], a minimal turnover proxy is:
turnover_t = abs(p_t - p_previous)
net_return_t = gross_return_t - turnover_t * cost_per_unit
The exact model depends on the venue and strategy. Fees, spread, slippage, borrowing, funding, and final liquidation may all matter. A generic audit can stress declared costs, but it cannot invent an execution model you did not supply.
5. Is a tiny fraction of the sample carrying the result?
Sort observation-level P&L contributions and ask how much of total profit comes from the largest few observations.
If five trades create nearly the entire backtest, the headline metric may be describing concentration rather than repeatable behaviour. That is not automatically invalid, but it should change how much confidence you place in the estimate.
A compact pre-deployment checklist
Before a strategy touches capital, record:
- the exact timestamp and return definitions;
- whether earlier outputs survive future-data deletion;
- chronological holdout results;
- declared trial count or configuration count;
- gross and cost-adjusted performance;
- profit concentration;
- every check that could not be evaluated.
The last item matters. Not evaluated is not the same as passed.
Why I packaged this
None of these ideas is secret. The tedious part is turning them into a repeatable local workflow with input validation, edge-case handling, and readable evidence.
I packaged the post-hoc checks into Backtest Guard, a pure-standard-library Python tool that audits exported CSV evidence locally and produces HTML, text, and JSON reports. It does not inspect strategy source code, catch every possible defect, prove profitability, or authorize deployment.
The contaminated demonstration and sample evidence are public here:
You can build these checks yourself. If you would rather keep the weekend, the packaged version is currently $29—about the cost of one delivered lunch.
What failure mode has produced the most convincing false-positive backtest in your own work?
Disclosure: I built Backtest Guard. This article explains the checks and their limitations; a clean report is not a profitability guarantee or deployment approval. AI tools assisted with editing and presentation; the technical claims and product behaviour were reviewed against the released implementation.
Top comments (0)