Two configs, same strategy, same data. One becomes provable in under a year. The other cannot be proven in a working lifetime.
"Am I overfitting?" gets all the attention. The more common problem is quieter and harder to fix: you do not have enough data to tell, and no amount of cleverness changes that.
I had two configurations of the same breakout strategy. Same code, same decade, different instrument and parameter set.
| trades | per-trade vol | t | |
|---|---|---|---|
| Gold, k=0.75, 1:1 | 2,352 | 0.75% | 2.93 |
| BTC, k=0.50, 4R | 1,506 | 2.16% | 1.10 |
Both looked reasonable. Both had positive expectancy. One was nearly provable; the other was not going to be provable at all.
The formula
There is a closed form for the sample size you need to reach a given t-statistic, and it has no free parameters:
n = (3 × per-trade volatility ÷ per-trade edge)²
Read it as: the number of trades you need is the square of three divided by your signal-to-noise ratio. Not "how many trades did I run". Not "how many years of data". The ratio.
That is why "I have 1,500 trades, that's plenty" is not a statement about evidence. 1,500 trades with noisy returns is worth less than 300 with clean ones.
Applying it
From the table, the observed t and n give the ratio directly: edge/vol = t / sqrt(n).
| t | n | edge/vol | trades needed | still needed | at this rate | |
|---|---|---|---|---|---|---|
| Gold | 2.93 | 2,352 | 0.060 | 2,466 | 114 | ~0.9 yr |
| BTC | 1.10 | 1,506 | 0.028 | 11,202 | 9,696 | ~37 yr |
Gold needs about nine more months of trading. BTC needs thirty-seven years — and that assumes the edge and the volatility both stay exactly where they are, which is the optimistic case.
Both are the same strategy. The difference is that BTC's per-trade volatility is 2.9× gold's.
Why noise dominates silently
Look at the right panel of that figure: n grows with the square of the inverse ratio. Halve your signal-to-noise and you need four times the data. Drop it to a third and you need nine times.
BTC's ratio is less than half of gold's, so the requirement is roughly (0.060/0.028)² ≈ 4.5× more trades — and BTC also produces more trades per year, so the year count should improve. It does not, because the ratio entered as a square while the trade frequency is only linear. Frequency cannot outrun a squared penalty.
This is why a strategy with a great-looking annualised return can be the worst candidate for deployment. BTC had the highest annualised figure of anything I tested and the weakest evidence by a wide margin. The return was real in the sample. The evidence that it would repeat was not.
What I do differently now
1. Compute the required n before the strategy is finished. If the answer is thirty-seven years, I want to know that on day one, not after six months of work. The calculation takes one line and it has changed which projects I start.
2. Treat "needs a lifetime" as a rejection. Not "promising but unproven". If the required sample is longer than the instrument's plausible lifespan, the hypothesis is unfalsifiable in practice, and an unfalsifiable strategy is indistinguishable from a random one for as long as I would be trading it.
3. Prefer high signal-to-noise instruments for a first test. Same logic as the gold/BTC split: if you must prove something, prove it where the arithmetic permits it. Testing on the noisiest instrument first is the most expensive possible ordering — you spend the most time and learn the least.
4. Report n and the ratio, not just the t. A t-statistic alone hides whether the result is 300 clean trades or 3,000 noisy ones. The ratio makes the difference visible in one line.
The uncomfortable corollary
Some of the things I believed in earlier were not tested — they were untestable, and I mistook the absence of proof for the presence of evidence.
That is a worse error than a bug, because a bug can be fixed by editing a line. An unfalsifiable hypothesis can only be abandoned, and abandoning it costs whatever I already invested in it.
The formula is unforgiving in a useful way. It does not care how much I like the strategy, or how plausible the story is. It just says: with this much noise and this much edge, you need this much data. If you cannot get the data, you cannot get the answer.
The fold, sizing and selection helpers, plus this arithmetic, are in a small open-source kit (GitHub, MIT, no dependencies). If you have a config where the required n came out short and the strategy still failed, that is the case I want to hear about — it means the ratio was not the binding constraint.
If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*
Top comments (1)
Your size formula assumes independent trades plus a stable mean and variance. Several trades on one signal can be correlated positions, so ten simultaneous positions are not ten independent experiments.
I take the plug-in count as a planning estimate, not proof of the target statistical threshold, because the sample edge is an uncertain sample mean. Does the n in the table count trades or independent signal groups?