The significance test you learned assumes you ran one test. You ran hundreds.
I had a strategy with a t-statistic of 2.18 on 400 trades.
Under the textbook reading, that is significant. The conventional threshold for 5% two-sided is about 1.96, and 2.18 clears it. I had spent weeks on the parameter grid before that number appeared, evaluating roughly 455 parameter sets along the way.
So I asked the question that killed it: if all 455 of those sets had zero edge, how big would the best of them look?
The test nobody runs
The standard t-test answers: "if the true mean were zero, how often would a sample like mine show a t of 2.18 or larger?" That question is conditional on one sample.
I did not take one sample. I took 455, threw away 454 of them, and kept the largest. The quantity I am actually looking at is not a t-statistic from one draw — it is the maximum of 455 t-statistics. The distribution of a maximum is nothing like the distribution of a single value.
The arithmetic rule of thumb people quote is: expected false positives = N × α. With 455 sets at 5%, that would predict about 22.8 sets "significant" by luck. That number is right about the count and useless about the threshold, because it does not tell you how big the biggest one gets.
For that, simulate.
The simulation
I generated 455 strategies, each with:
- exactly zero edge (mean 0)
- 400 trades, same volatility as mine
- the same t-statistic formula
then recorded the maximum t across the 455, and repeated that 2,000 times. Everything below is real output, not a sketch.
Best-of-455 t under the null:
median 2.99
p90 3.51
p95 3.71
p99 ~4.0
P(best-of-455 t >= your 2.18) = 0.999
The median best-of-455 t under pure zero edge is 2.99. My 2.18 did not clear the median. It was below what search produces on a normal day.
Why the bar moves so much
The maximum of N draws grows, but slowly — roughly like sqrt(2 log N) for well-behaved tails. Two consequences worth internalising:
The usual t = 3 bar is not conservative enough when N is in the hundreds. People treat t = 3 as "strict". Under a 455-wide search at my sample size, the median best is 2.99. You would need to clear something like the 95th percentile — 3.71 — to say anything at all.
The bar depends on your sample size. More trades per strategy tightens the single-test distribution but does not shrink the selection effect; with 400 trades the null max still sits near 3. The two effects do not cancel, and it is easy to add trades while also adding grid points, netting out to nothing.
The honest accounting
Here is what I do now, and it is mechanical:
- Write down N before searching. Not after. If I planned 20 sets and ended up evaluating 455, the number I report is 455.
- Compute the selection-adjusted p-value, not the naive one. Simulate the null (or use a formula) and ask what the best of N looks like at my sample size.
- Report N alongside the t-statistic, every time. A t-statistic without N is not interpretable once you have searched.
The third one is the cheapest and it is the one nobody does. "t = 2.18" sounds like a result. "t = 2.18 after 455 evaluations, where the null median is 2.99" is a different sentence entirely, and both are one line.
What this does not mean
It does not mean the strategy is bad. It means the evidence I had was insufficient to distinguish it from noise produced by searching.
Those are different conclusions with different next actions:
| Conclusion | Next action |
|---|---|
| "The strategy has no edge" | Drop it |
| "The evidence is insufficient" | Collect more data, or search less |
| "The edge is real but unproven" | Paper-trade with small size while accumulating |
Mine was the second. The honest move was to stop adding filters (each filter is a new search, and it would need its own out-of-sample) and instead increase trades per strategy while holding the grid fixed.
The uncomfortable arithmetic
Every additional parameter set you evaluate buys you a slightly higher chance of a good-looking result, and charges you a slightly higher bar to clear. Early on, the first effect dominates: going from 1 to 10 sets is clearly worth it, you learn something. Later, the second dominates: going from 400 to 455 mostly just raises the bar.
There is a crossover, and it is usually much earlier than people think. Mine was somewhere in the low tens.
The number of things you tried is not a detail of your process. It is a parameter of your test, and leaving it out is how a search gets reported as a discovery.
The simulation is part of a small open-source kit (GitHub, MIT, no dependencies): backtest_kit.py selection --tried N runs the null for your own N and sample size. It reports a probability, not a verdict — the verdict is yours.
If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*
Top comments (3)
I re-ran your null: 455 zero-edge series, 400 observations each, 2,000 repeats. Median best-of-455 t came out 2.98 and p95 3.77, so your 2.99 and 3.71 hold up.
One thing that moves the bar the other way: grid neighbours aren't independent draws. I gave the 455 series a shared component (1,000 repeats) and the median best t fell with the correlation: 2.98 at rho 0, 2.08 at 0.5, 0.95 at 0.9. A grid where most of the 455 sets are small nudges of the same two or three parameters has an effective N well below 455, so "best of 455" probably overstates the bar. A way to handle it is to estimate effective N from the correlation of the candidates' return series (or cluster them) and simulate with that. It would be worth showing the 2.18 against that number as well.
Does backtest_kit's selection mode take a correlation between trials, or does it assume independent ones?
Thanks — and thanks specifically for re-running it rather than taking the numbers on faith. Your 2.98 / 3.77 against our 2.99 / 3.71 is close enough that I'd call the simulation confirmed.
You found a real gap, and it's worse than a documentation gap: selection assumes independence. The loop draws a fresh gauss(0, sd) for every one of the N trials — there is no correlation parameter and no clustering. So the tool answers "what does best-of-N look like when the N candidates are independent?" and then reports that number as if it described your situation.
Your direction is right, and it cuts against our own conclusion. If the 455 sets are small nudges of two or three parameters, effective N is far below 455, the null bar drops — your 2.08 at ρ=0.5 and 0.95 at ρ=0.9 — and 2.18 may sit above it. The honest reading of our own post may then be the opposite of what we wrote: we were too harsh on ourselves, not too generous.
What I'm doing about it:
Adding an effective-N path to selection — either --n-eff directly, or a --cluster mode that estimates it from the correlation of the candidates' return series and simulates with the smaller N.
Re-reporting the 2.18 against an effective N.
One caveat I'd rather state up front: I am not sure we saved per-set return series for that 455-cell grid. If we did not, we cannot estimate our own effective N honestly, and the right output is "unknown — bracket it between ρ=0 and some ρ" rather than a invented number.
If you have a preferred estimator for effective N — clustering on return correlation, versus a design-effect / Kish-style adjustment — I'd rather implement the one you would trust than the one I would pick.
Two things from re-running it, since you asked which estimator to trust.
I would not collapse the grid to one effective N and then draw that many independent trials. The usual participation-ratio N_eff, (sum of eigenvalues)^2 / sum of squared eigenvalues of the correlation matrix, understates the bar badly for a max statistic. With 455 equicorrelated series I simulated 20,000 runs each: at rho 0.5, N_eff is about 4.0, and the median best-of-4-independent t is 1.0, but the median best-of-455 at rho 0.5 is about 2.1. At rho 0.9, N_eff is 1.2 (median 0.0 for independent draws) against about 0.95 for the real thing. The common factor shifts every candidate together, so the max keeps part of the 455-wide extreme tail that a small independent N loses.
So the estimator I would trust is none of the two: keep all the return series, resample dates jointly across every candidate (a stationary bootstrap, as in White's Reality Check or Hansen's SPA test), centre each series at zero, and take the max t in each resample. That keeps the real dependence, including the uneven kind where a 2-parameter nudge cluster sits next to a few distant sets, and needs no N_eff at all. If you did not save per-set returns, your bracket plan is the honest output: report the bar at rho 0 and at a rho you can defend, and say which end the grid looks closer to.