You've seen the shape before: equity curve going up and to the right, Sharpe ratio north of 2, max drawdown that looks survivable. It's convincing. It's also, on its own, close to worthless as evidence that the strategy will do anything useful going forward.
This isn't a claim that backtesting is pointless. It's that a single backtest, however clean, can hide four specific failure modes that don't show up in the summary stats you're staring at. You have to go looking for them in the trade list itself.
1. Return concentration
Open your closed-trade export and sort by PnL, descending. Sum the top 3 trades. Sum everything else. If the top 3 account for most of your total return, your backtest isn't describing a strategy — it's describing a small number of events that happened to occur in your sample window.
trade_id, pnl
1, 120
2, -40
3, 3800 <- one trade
4, -60
5, 90
...
47, -30
A strategy with 47 trades where one trade is 90% of total profit isn't a systematic edge, it's a lottery ticket that paid off once during backtesting. The Sharpe ratio doesn't tell you this. The equity curve doesn't tell you this, because one big trade still draws a smooth-looking line. You have to look at the distribution of individual trade PnL, not the cumulative sum.
The fix isn't to throw away the trade — maybe it was real. The fix is to know it's there before you size a live position based on "the strategy averages X% per trade."
2. Regime dependence
A strategy backtested from 2019 to 2023 has lived through exactly one dominant regime: a multi-year bull market interrupted by one sharp, fast-recovering crash. If your entries are long-biased and your backtest window doesn't include a prolonged sideways or bear regime, you haven't tested a strategy — you've tested a strategy's performance during one macro condition, and gotten a result that will not generalize.
The way this hides itself: date range selection feels neutral. "I used all the data I had" sounds responsible. But if all the data you had happens to be one regime, your Sharpe ratio is really a Sharpe ratio conditioned on that regime, and nothing in the report tells you that conditioning exists. You find it by segmenting trades by market condition (trend vs. chop, high vol vs. low vol) and checking whether performance holds up in each segment separately, not just in aggregate.
3. Execution assumptions doing the work
Every backtest makes assumptions about fills: what price you get, how much slippage, whether you can actually get filled at all at the size you're testing. These assumptions are usually buried in a config file or a default in your backtesting library, and they are frequently the entire source of the edge.
A mean-reversion strategy that assumes fills at the exact touch price on a thinly traded instrument is not describing a strategy — it's describing what happens if you had a magic wand for that one variable. Move the assumption from "fill at touch" to "fill at touch plus one tick" and watch a lot of "profitable" systematic strategies go flat or negative. This is worth doing as a deliberate stress test: take your existing trade list, degrade the fill assumption by a fixed amount, and recompute. If the strategy's profitability doesn't survive a small, realistic degradation, the edge was execution assumption, not signal.
4. Path luck
Even a strategy with real, non-concentrated, regime-robust edge produced one specific sequence of trades, in one specific order, during one specific slice of history. That sequence is a single sample from a distribution of possible sequences. Some of those alternate sequences look much worse — deeper drawdowns, longer flat periods, sequences where the losing trades cluster early and you'd have quit before the edge showed up.
The standard way to check this is to shuffle: take your trade returns, resample them (with replacement, or reorder them, depending on what you're testing for), and generate a distribution of possible equity curves instead of the one you happened to get. If your actual curve sits near the median of that distribution, the drawdown you experienced is typical. If your actual curve is near the best-case tail, you got lucky with path, and a live version of this strategy is more likely to look like the median case — which might include a drawdown you didn't plan for.
What this adds up to
None of these four checks are exotic. Concentration is a sort and a sum. Regime dependence is a segmentation. Execution sensitivity is a parameter sweep. Path luck is a resample. You can build all of this yourself in an afternoon with pandas and numpy, and if you're serious about running a systematic strategy with real money, you should — at minimum look at your trade list sorted by PnL and ask if 3 trades are secretly the whole story.
The uncomfortable conclusion is that a clean equity curve is weak evidence by default. It's evidence of what happened in one sample, once, under one set of fill assumptions. Turning it into evidence you can actually lean on means trying to break it, not admiring it.
Disclosure: I build Tradevo Verify, which runs a version of these checks (execution stress, path/Monte Carlo, concentration, regime tests) against a closed-trade CSV and returns a report on how much weight the evidence can carry — not a verdict that the strategy is good. It's a $99 one-time check, and it's built to return a fragile result whenever the trade list earns one.
Top comments (0)