DEV Community: 王伟

After 455 Parameter Sets, a t-Stat of 2.18 Means Nothing

王伟 — Wed, 21 Oct 2026 16:44:56 +0000

The significance test you learned assumes you ran one test. You ran hundreds.

I had a strategy with a t-statistic of 2.18 on 400 trades.

Under the textbook reading, that is significant. The conventional threshold for 5% two-sided is about 1.96, and 2.18 clears it. I had spent weeks on the parameter grid before that number appeared, evaluating roughly 455 parameter sets along the way.

So I asked the question that killed it: if all 455 of those sets had zero edge, how big would the best of them look?

The test nobody runs

The standard t-test answers: "if the true mean were zero, how often would a sample like mine show a t of 2.18 or larger?" That question is conditional on one sample.

I did not take one sample. I took 455, threw away 454 of them, and kept the largest. The quantity I am actually looking at is not a t-statistic from one draw — it is the maximum of 455 t-statistics. The distribution of a maximum is nothing like the distribution of a single value.

The arithmetic rule of thumb people quote is: expected false positives = N × α. With 455 sets at 5%, that would predict about 22.8 sets "significant" by luck. That number is right about the count and useless about the threshold, because it does not tell you how big the biggest one gets.

For that, simulate.

The simulation

I generated 455 strategies, each with:

exactly zero edge (mean 0)
400 trades, same volatility as mine
the same t-statistic formula

then recorded the maximum t across the 455, and repeated that 2,000 times. Everything below is real output, not a sketch.

Best-of-455 t under the null:
  median   2.99
  p90      3.51
  p95      3.71
  p99      ~4.0

P(best-of-455 t >= your 2.18) = 0.999

The median best-of-455 t under pure zero edge is 2.99. My 2.18 did not clear the median. It was below what search produces on a normal day.

Why the bar moves so much

The maximum of N draws grows, but slowly — roughly like sqrt(2 log N) for well-behaved tails. Two consequences worth internalising:

The usual t = 3 bar is not conservative enough when N is in the hundreds. People treat t = 3 as "strict". Under a 455-wide search at my sample size, the median best is 2.99. You would need to clear something like the 95th percentile — 3.71 — to say anything at all.

The bar depends on your sample size. More trades per strategy tightens the single-test distribution but does not shrink the selection effect; with 400 trades the null max still sits near 3. The two effects do not cancel, and it is easy to add trades while also adding grid points, netting out to nothing.

The honest accounting

Here is what I do now, and it is mechanical:

Write down N before searching. Not after. If I planned 20 sets and ended up evaluating 455, the number I report is 455.
Compute the selection-adjusted p-value, not the naive one. Simulate the null (or use a formula) and ask what the best of N looks like at my sample size.
Report N alongside the t-statistic, every time. A t-statistic without N is not interpretable once you have searched.

The third one is the cheapest and it is the one nobody does. "t = 2.18" sounds like a result. "t = 2.18 after 455 evaluations, where the null median is 2.99" is a different sentence entirely, and both are one line.

What this does not mean

It does not mean the strategy is bad. It means the evidence I had was insufficient to distinguish it from noise produced by searching.

Those are different conclusions with different next actions:

Conclusion	Next action
"The strategy has no edge"	Drop it
"The evidence is insufficient"	Collect more data, or search less
"The edge is real but unproven"	Paper-trade with small size while accumulating

Mine was the second. The honest move was to stop adding filters (each filter is a new search, and it would need its own out-of-sample) and instead increase trades per strategy while holding the grid fixed.

The uncomfortable arithmetic

Every additional parameter set you evaluate buys you a slightly higher chance of a good-looking result, and charges you a slightly higher bar to clear. Early on, the first effect dominates: going from 1 to 10 sets is clearly worth it, you learn something. Later, the second dominates: going from 400 to 455 mostly just raises the bar.

There is a crossover, and it is usually much earlier than people think. Mine was somewhere in the low tens.

The number of things you tried is not a detail of your process. It is a parameter of your test, and leaving it out is how a search gets reported as a discovery.

The simulation is part of a small open-source kit (GitHub, MIT, no dependencies): backtest_kit.py selection --tried N runs the null for your own N and sample size. It reports a probability, not a verdict — the verdict is yours.

If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

My Walk-Forward Said +29.2 Per Trade. Honest Out-of-Sample Said -4.6.

王伟 — Wed, 21 Oct 2026 16:44:29 +0000

A parameter set that won on folds 1-3 lost on folds 4-5. Same strategy, same data.

I ran the standard walk-forward check. Three parameter sets, five sequential folds, about 400 trades. For each fold I picked the set that was best on the previous folds only, then scored it on the fold it had never seen.

Here is the whole output:

fold   picked      oos net    oos t
2      k0.75        436.5     0.88
3      k0.75        827.9     1.56
4      k0.75       -941.2    -2.51
5      k1.00       -813.2    -1.96

Stitched out-of-sample: n=107  net=-489.99  mean=-4.58  t=-0.51
In-sample best mean   :  +29.17 per trade
Optimism gap          :   33.75 per trade

The parameter set k0.75 looked best for three consecutive folds. It was the worst set for the next two.

+29.17 per trade in sample. −4.58 out of sample. The gap is 33.75 per trade, which is larger than the entire in-sample edge — the sign flipped.

What "out-of-sample" actually means

Most people treat out-of-sample as a split. You cut the data at some date, fit on the left, evaluate on the right, and call the right side honest.

That is not what makes it honest. Out-of-sample is a procedure, not a file. What makes it honest is that no choice in the evaluation was informed by the data being evaluated.

Here is the distinction, concretely:

What you did	Is the score honest?
Optimised on 2015-2020, tested on 2021-2023, and you never looked at 2021-2023 while designing	yes
Optimised on 2015-2020, tested on 2021-2023, then went back and tweaked because 2021-2023 looked bad	no — that file is now in-sample
Picked the parameter that maximised the whole period, then reported the second half	no — the second half informed the pick
Picked on folds 1..i-1, scored on fold i, never revisited	yes

The third row is the one that catches people. If you choose a parameter using any statistic computed over the test period — total return, Sharpe, whether the equity curve "looked right" — the test period has leaked into the choice, and the score is in-sample no matter which file it lives in.

The gap is not a haircut

The natural instinct is to treat the optimism gap as a discount: knock 20% off the in-sample number and move on.

That does not work, because the gap is not proportional to the edge. It is roughly proportional to how much selection freedom you had. In this example:

in-sample best mean: +29.17
honest out-of-sample: −4.58

The gap (33.75) was bigger than the edge (29.17). There was no discount that turns this back into a positive number, because the honest number is negative.

When the gap exceeds the edge, what you measured was not a weak strategy. You measured the selection process.

Why this happens

Three mechanisms, all mundane:

1. Fold-to-fold variance is large. A fold with 20-25 trades has a standard error of roughly sd / sqrt(25). If per-trade sd is around 100, the standard error of the mean is about 20 per trade. So a "best mean" of 29 is roughly one to two standard errors above zero — entirely reachable by luck. With three candidate sets, picking the top one at each step reliably picks the noisiest winner.

2. Regime change is not announced. The demo I used has the edge decaying after ~60% of the sample on purpose, which is a clean version of what real markets do more subtly. A parameter that suits the first regime is not obliged to suit the second.

3. The selection is repeated, not once. You do not choose a parameter once. You choose at fold 2, again at fold 3, again at fold 4. Each choice is a fresh chance to pick the lucky one, and the errors compound across the stitched result.

How to run this yourself

You need one thing beyond a trade log: a column saying which parameter set produced each trade.

entry_time, exit_time, side, entry_price, exit_price, size, fee, param_set
2024-01-02T09:30:00,...,BUY,100.12,101.40,10,2.00,k0.75
2024-01-03T14:05:00,...,SELL,101.85,100.90,10,2.00,k1.00

With that column, the procedure is mechanical:

Split trades into K sequential folds by time.
For fold i (starting at i=2), compute each parameter set's mean on folds 1..i−1.
Pick the best one.
Score it on fold i only.
Stitch the fold-i results. That stitched series is your honest number.
Compare it to the best in-sample mean. The difference is the gap.

python backtest_kit.py folds trades.csv --k 5

Without the column, all you can do is split validation, which is weaker — it tells you about stability over time, not about selection. The tool says so rather than pretending otherwise.

The part that stings

I did not set out to fool myself. I had a parameter grid, I picked the winner on the development data, and I reported the number. That is the standard workflow, and it is exactly the workflow that produces this gap.

The gap is not a sign of carelessness. It is the expected output of selecting on noise. Which means the fix is not "be more careful" — it is to make the selection procedure mechanical and to report the stitched number, including when it is negative.

The uncomfortable question is not "was my gap large". It is: how many strategies do I still believe in because I never computed this number for them?

The fold walk and the gap calculation are in a small open-source kit (GitHub, MIT, no dependencies). If you have run this on your own grid and the gap came out smaller than the edge, I would like to see the setup — that is the interesting case.

There is also a *$0** download on Gumroad that bundles the tool with the demo
logs, if you want something to test against before pointing it at your own data —
name your price, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

The One-Line Fix That Took My t-Stat From 6.22 to 1.04

王伟 — Wed, 21 Oct 2026 16:42:36 +0000

A breakout strategy passed all six parameter cells. It was a fill assumption, not an edge.

I had a breakout strategy on 1-minute data come in at t = 6.22.

Six parameter cells, all six passing. That is the kind of number that makes you start thinking about position size and funding an account. It is also the kind of number that should make you suspicious, because in ten years of doing this I have almost never seen a real edge survive a clean audit with a t-stat that high.

It was not real. The strategy was fine. The fill assumption was wrong, and it was wrong in exactly the direction that flatters you.

Here is the whole story, because the same bug is probably sitting in a lot of backtesters, including yours.

The bug

My breakout entry looked like this:

if high[j] >= buy_level:
    entry = buy_level

buy_level came from the previous session's high. The logic reads fine: price trades up through my level, I get filled at my level. That is what a limit order does.

The problem is the gap. When the new session opens with a gap straight through buy_level, bar j satisfies high[j] >= buy_level on its very first tick — at the open. The open is already above my trigger, and I record a fill at buy_level, a price that never traded in that session.

I was gifting myself the gap. Systematically, on every gap-through day, in the direction that made the equity curve look better.

The X is the recorded fill. The dotted line is that bar's low. The fill sits 1.48 below the lowest price of the bar — a price that, by construction, no one could have transacted at. The backtester did not complain, because nothing in the code knows what a plausible fill looks like.

The fix

One line:

entry = max(buy_level, open[j])   # the worse of the two, for a long

If the bar opened above my trigger, I get the open — the price I could actually have paid. If it opened below, I get my trigger, as intended. Same for shorts with min(...).

That is the entire change. Not a filter, not a parameter, not a new feature.

Same trades, same period, same parameter grid. 6.22 → 1.04. Passing cells: 6 of 6 → 0 of 6.

The entire edge lived in the fill.

Why this is hard to catch

Three reasons, and they compound.

It is plausible. entry = buy_level reads like correct code. It is the natural way to express "I got filled at my level". Nothing about it looks like a mistake, so code review slides right over it.

It inflates results in the direction you want. There is no error, no crash, no unusual log line. The only symptom is a nicer equity curve, which is precisely the thing you were hoping for. Your brain is not going to flag good news as a bug.

It hides in the gap. If you trade a market that does not gap, you may never see it. It only appears on days where the open jumps past the trigger — and on those days it silently substitutes a favourable price for the one you would have got.

How often it actually bites

I measured the share of entries where the recorded fill fell outside the bar's [low, high]:

The pattern is exactly what you would expect: the more a market gaps overnight, the worse it gets. Index futures on a session-based breakout are the worst case — roughly one entry in six.

And note what this does to the interpretation of a backtest. On NQ, 17.4% of entries were priced at a level the market never traded. If your strategy's edge is on the order of a few basis points per trade, and one entry in six is being handed a free gap, then the measured edge is mostly the gap.

How I found it

Not by reading the code. I had read that code a dozen times.

I found it by looking at when the largest 1-minute moves happened. I ran a data-quality pass that printed the twelve biggest 1-minute jumps with their timestamps, and all twelve landed between 18:00 and 18:05 New York time.

That clustering was the tell. Twelve enormous 1-minute moves, all in a five-minute window, every time — those are not market moves. That is the session boundary. The "jumps" were cross-session gaps being recorded inside a session-labelled dataset.

Once I was looking at gaps specifically, the fill assumption fell apart in about ten minutes.

The general lesson here is worth more than the bug: when you audit a backtest, look at the time distribution of anomalies, not just their size. A 2% one-minute move at a random time is a market event. A 2% one-minute move that always happens at 18:00 is a data-structure artefact, and it is telling you that your data is not shaped the way your code assumes.

What this generalises to

Fill assumptions are where backtests lie most often, because they are the one part of the simulation that has no external check. Your signal logic produces a number you can inspect. Your fill price is just... whatever you wrote.

Three specific forms to look for:

Gap-through fills — the one above. Trigger levels set from prior session data, filled at the trigger regardless of the open.
Look-ahead on exits — if execute() is only called on bars where a signal fired, then exits are only checked on those bars. I had this one too: fixing it moved a different strategy from −5.7% to +17.9%, and inflated the win rate by 6–8 points. The original reported a 74.1% win rate; most of that was the bug.
Cost assumptions borrowed from another asset class — two futures strategies of mine were costed at 5bp per side. Actual round-trip cost was closer to 0.30bp (NQ) and 0.55bp (MGC). That is an 18–33× overstatement. Correcting it moved one of them from −94.0% to −45.5%. Rule of thumb: if cost ÷ per-trade risk exceeds about 5%, suspect the cost model before you reject the strategy.

Three checks you can run on your own log

You do not need my code for these; you need your trade log and your price bars.

1. Is every fill inside its bar?

For each trade, find the bar containing the entry timestamp and check
low <= entry_price <= high. Anything outside never traded. This single check catches the gap-fill bug directly, and it is the highest-yield check I know.

2. Is the fill better than the bar's open?

For breakout entries, if a long fill is below the open, either a limit order filled later in the bar (fine, but then model it that way) or you have gifted yourself the gap (not fine). This one has false positives — limit orders legitimately fill better — so treat it as a prompt to look, not a verdict.

3. Do your anomalies cluster in time?

Print your largest N moves with timestamps and histogram the hour. If the biggest moves all sit at a fixed time of day, you are looking at session boundaries, and your session handling is probably wrong.

The tool

I turned the checks I now run into a small script, because doing them by hand every time is how you stop doing them. It is one file, standard library only, no pandas:

python backtest_lint.py trades.csv --prices prices.csv

That image is real output from the tool on a demo log that has five bugs deliberately planted in it. It scores 1/6 and names each one with the row number and the actual numbers. The same repo also ships a clean control that scores 6/6 — a checker that always says "fail" would be worthless, so both directions are testable before you trust it.

Code: https://github.com/wei498766-ai/backtest-lie-detector — MIT, no dependencies.

The uncomfortable part

The uncomfortable part is not that I had a bug. Everyone has bugs.

The uncomfortable part is that the bug made the result better, and the result looked like a discovery. I did not find this because I was careless. I found it because I got suspicious of a number that was too good, and then went looking for how it could be too good.

That is not a scalable process. It depends on being suspicious at the right moment, about the right number — and the numbers you most want to believe are exactly the ones you are least likely to question.

So the process has to be external. Cheap, mechanical checks that run whether or not you are in the mood to doubt yourself.

That is the whole argument for this kind of tooling. Not that it makes you a better trader, but that it does not care what you want to be true.

If you have hit something in this family — particularly anything where the backtester's assumption is invisible in the output — I would like to hear about it. The ones that do not throw errors are the ones worth collecting.