DEV Community: 王伟

Gold Is Not Illiquid — You Picked the Wrong Symbol

王伟 — Sat, 07 Nov 2026 01:00:03 +0000

"This instrument is too illiquid to backtest" is a conclusion you should be suspicious of.

I was building a 1-minute breakout study across commodities and FX. My first pass produced a median of 11 one-minute bars per trading day on gold futures.

A full futures session is about 1,380 one-minute bars. Eleven is 0.8% coverage. I wrote it down as "gold is too thin at 1-minute resolution, drop it" and moved on.

Gold is the most liquid commodity future in the world. It was not thin. My symbol was wrong.

What a continuous contract actually is

Historical futures data vendors give you a synthetic series stitched across contract months, so you can chart more than three months of history. Two common variants:

suffix	what it does
`*.c.0`	continuation — just rolls from one front month to the next on the roll date
`*.v.0`	volume-weighted / ratio-adjusted — adjusts the historical prices so the series is continuous without gaps

The continuation series is fine for a daily chart. At 1-minute resolution it has a specific pathology: on most days, the c.0 series is quoting a contract that is not the one people were trading.

I measured it as "bars per day" — the count of distinct 1-minute bars the series produced per session. A real, actively-traded contract produces ~1,380. A series that spends most of its time pointing at a nearly-dead back month produces almost nothing, because that contract barely trades.

The measurement

Same source, same period, same one-minute frequency. Only the symbol suffix changed.

contract	`*.c.0` bars/day	`*.v.0` bars/day	multiplier
GC gold	11	1,372	125×
MGC micro gold	357	1,380	3.9×
SI silver	10	1,292	129×
HG copper	14	1,255	90×
6E euro FX	90	1,327	15×
6B pound FX	26	1,197	45×

Note that NQ, ES, CL, NG, M6E and ZW were fine — their c.0 series produced ~1,380 bars/day all along, because those contracts roll in a way that keeps the continuation series pointed at liquid months. So this is not "c.0 is always broken". It is "c.0 is broken for some contracts and you have to check per contract".

That distinction matters, because the cheap version of this lesson ("always use v.0") is not what the data says. What the data says is: check the bar count per instrument before you believe anything about liquidity.

The diagnostic

One number tells you whether you are looking at a liquidity problem or a data problem:

median one-minute bars per trading day, as a fraction of a full session.

If it is above ~90%, the instrument is fine and any weirdness is downstream. If it is far below — 1%, 10%, 26% — do not conclude the market is thin. Conclude that your series is not pointing at the traded contract and go check the symbol.

Concretely:

bars_per_day = df.groupby(df.index.date).size()
print(bars_per_day.median())     # ~1380 for a liquid futures contract

Run that before you run any strategy. It takes seconds and it separates "this market has no liquidity" from "I downloaded the wrong thing". Those two conclusions have completely different follow-ups, and the second one is much more common.

What I got wrong twice

First error: attributing to the market what was caused by the symbol. I had a plausible story — some commodities really are thin — and the story made the measurement easy to accept. The 11 bars looked like evidence for the story.

Second error: applying the fix too broadly. After finding this, I briefly believed no c.0 series was usable. That is false: six of my contracts were fine. Over-correcting would have meant re-downloading data I already had, which is not free.

The corrective habit is narrow and cheap: check the bar count per instrument. Not per asset class, not per vendor. Per instrument, once, before you trust it.

A related trap in the same family

While fixing this I found the neighbouring problem, which is worse because it does not show up as missing data:

The c.0 series' price level can differ from the actual traded contract's price by a non-trivial amount — I measured about 0.6% between the continuous gold series and the active month. That does not reduce your bar count. It silently shifts your levels, so support and resistance levels computed on the synthetic series do not correspond to prices anyone could have traded.

Missing data announces itself. Displaced prices do not.

So the check list for any vendor-supplied continuous series is two items, not one:

Bars per day — is the series actually populated?
Price level versus the front month — is the series quoting prices that existed?

Both are cheap. Neither is in the vendor's documentation.

Bar-count-per-day is the first thing I check on any new dataset, and it is why the demo data in the open-source linter ships with a gap check (GitHub, MIT, no dependencies): a series that is 1% populated will produce a backtest full of technical artefacts that look like opportunities.

If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

Three Answers to "How Many Coins Should I Test?"

王伟 — Fri, 06 Nov 2026 01:00:04 +0000

Same exchange, same period, same question. Three answers, and only the last one is about the market.

I wanted to know how many perpetual contracts my backtest universe should contain. It is a mundane question with a mundane answer, and it took me three attempts.

98, then 101, then 157.

v0: ask the exchange — 98

The obvious method. Call the exchange's listings endpoint, take every contract currently trading, done.

This gives you exactly the set of things that survived to today. Everything delisted is simply absent — not filed under "delisted", just not there. The API is not lying; it is answering "what is listed now", which is a different question from "what existed during my sample".

A universe of 98 survivors, tested over a period when 59 more contracts came and went, will show you a result that the market never offered. You could only have traded the ones that made it.

That is textbook survivorship bias. It is also easy to believe you are immune, because you did not filter anything — you just used the exchange's list. The bias was upstream, in the question I asked.

v1: add delisted contracts — 101

So I pulled the full universe including delisted contracts: 157 total. Then I filtered to those with meaningful trading volume, so I would not simulate coins nobody traded.

The filter was: "has at least X volume over the last 90 days of available data."

Result: 101 contracts. Barely different from 98. Almost every delisted contract got filtered out.

Why? Because a delisted contract's "last 90 days of available data" is the 90 days ending at its delisting — when trading volume collapses to near zero. My filter was asking "was this liquid recently", and for a dead contract the answer is always no. My own quality filter was re-introducing the survivorship bias I had just fixed.

This is the part worth internalising. I did the right thing (include delisted contracts) and then undid it with a reasonable-looking filter. The bug was not in the data, it was in the window I measured over.

v2: measure before the end — 157

The fix is a single change: for delisted contracts, measure volume over the 90 days before delisting, not the 90 days ending at the last available row.

That is the window in which the contract was actually being traded by people. And with that window, the answer became 157 — the full universe.

Three numbers, one question:

version	what it did	contracts
v0	exchange's current listing	98
v1	full universe, filtered by volume in the last 90 days	101
v2	full universe, filtered by volume 90 days before delisting	157

Why 59 contracts is a big deal

The missing 59 are, by construction, the ones that failed. A survivorship-biased backtest over this universe is not slightly optimistic — it is testing a market in which you could only have picked winners.

The size of the effect depends on your strategy. A cross-sectional momentum strategy that picks the strongest contract each period will be dramatically distorted, because the losers were never candidates. A single-instrument timing strategy is less affected, though not unaffected: the instrument you chose is a survivor.

The transferable lesson

"I did not filter anything, I just used the exchange's list" is still a filter. It is the filter "must still be alive today", applied by someone else, and it is invisible because you never wrote it.

And the second-order lesson is the one that bit me: a reasonable quality filter can undo a correct universe correction, silently. If you include delisted instruments and then require recent liquidity, you have re-imposed survival without noticing.

Two rules I use now:

State the universe as a sentence with a date range — "all perpetual contracts that traded at any point between 2020-01 and 2024-12, including those delisted before 2024-12". If you cannot write that sentence, you do not know what you tested.
Measure every per-instrument filter over each instrument's own active window, not a fixed calendar window ending at the end of your data. The end of the data is a date that only survivors share.

A footnote on the volume number itself

Even with the right universe, I had to accept that the "volume" field I was filtering on was not what it appeared to be — in one of my data sources it is the number of ticks, not traded size.

That is a separate article. But the pattern is the same one: the column name told me what the field was called, not what it measured. Both this and the survivorship story are about the difference between a dataset's interface and its contents.

Universe construction is outside what a trade-log linter can check — it only sees the trades you gave it. It can tell you the sample size you need for a given edge, which is usually the moment you realise how much a missing 59 contracts matters.

There is also a *$0** download on Gumroad that bundles the tool with the demo
logs, if you want something to test against before pointing it at your own data —
name your price, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*

I Costed Futures Like Crypto, and Half the Loss Was Made Up

王伟 — Sat, 31 Oct 2026 01:00:04 +0000

A cost model is not a detail. It can be larger than the effect you are trying to measure.

I ran a 1-minute futures strategy and it lost 94% of its capital. That looked like a decisive answer: the strategy does not work.

It was not decisive. Half of that loss was invented by my cost model.

What I assumed

I used 5 basis points per side, which is 10 bp round trip. That number came from crypto, where I had been working. It felt conservative: surely paying more cost than reality is the safe direction.

That intuition is wrong, and not mildly wrong.

What it actually costs

Instrument	measured round trip	assumption was
NQ (Nasdaq futures)	0.30 bp	33× too high
MGC (micro gold)	0.55 bp	18× too high
BTC perpetual	10.2 bp	about right

Note the last row. The number I used was correct — for the asset class I took it from. That is the trap: a cost assumption is a fact about a specific market, and it does not travel.

Futures on a liquid index are extraordinarily cheap to trade. Crypto perpetuals are not. They differ by more than an order of magnitude, and I applied the expensive one to the cheap market.

Why it mattered more than usual

Here is the part that makes this a systemic problem rather than a slip.

With 1-minute entries, the per-trade risk on this strategy was only about 6.2 bp. So the cost-to-risk ratio was:

10 bp cost ÷ 6.2 bp risk  ≈ 162%

I was charging more than one and a half times the risk of the trade as friction. The result was that every trade lost 3.16R on average, of which 2.16R was invented fees and only 1.0R was the strategy.

That is not a strategy being measured. That is a cost model being measured.

Correcting the number to 0.30 bp moved the same run from −94.0% to −45.5%.

The rule of thumb

Before rejecting a strategy on cost grounds, compute one ratio:

round-trip cost ÷ per-trade risk

If it exceeds about 5%, stop. Do not conclude the strategy is bad — conclude that you do not know your costs, and go measure them. A ratio above 100% (as mine was) means the cost term dominates the result entirely and the backtest is telling you about your assumptions, not about the market.

The corollary is just as important: a strategy whose edge is small relative to friction is not necessarily broken. It may be correctly measured and simply too small — which is a different conclusion with a different next step (find cheaper execution, trade less often, or move to a higher timeframe) and not "throw it away".

How to measure instead of guess

I did not have a perfect cost model afterwards either. What I had was something better than a guess:

Use the exchange's own fee schedule, not a number from another asset class.
Add the spread separately. It is usually larger than commission on liquid futures and it varies by time of day. Sample it, don't assume it.
Sanity-check against a fill-quality sample. Take 50 real fills, compare your simulated price to the actual one. The gap is your slippage estimate.
State the number and its source in the results. "Costs: 0.30 bp round trip, measured from NQ fill samples over the sample period" is a claim someone can audit. "5 bp" is a habit.

The uncomfortable asymmetry

An optimistic cost assumption makes a bad strategy look good — and you will find out in production, expensively.

A pessimistic cost assumption makes a good strategy look bad — and you will never find out, because you will have moved on. That failure is silent. It cost me a line of research, and I only noticed because a 94% loss looked too clean to be real.

So the discipline is symmetric: do not treat high costs as the safe direction. Both directions are errors, and the pessimistic one is the one nobody catches for you.

The cost-per-risk check is one of the static checks in a small open-source linter (GitHub, MIT, no dependencies). It flags a fee column that is missing or zero, and reports cost against per-trade risk so the ratio is visible before you draw conclusions.

One Missing Call Site Turned -5.7% Into +17.9%

王伟 — Fri, 30 Oct 2026 01:00:03 +0000

A one-line structural bug, worth 23.6 points of return and most of a 74% win rate.

I want to start with the number, because the number is the point:

−5.7% when exits are checked every bar. +17.9% when they are only checked on signal bars.

Same strategy, same data, same period, same costs. The difference is where one function was called from.

The bug

My backtester had a loop that looked roughly like this:

for i, bar in enumerate(bars):
    signal = strategy.check(bar)
    if signal:
        strategy.execute(bar)      # <- entry AND exit logic live in here

execute() contained both the entry and the exit check. Putting the exit check inside a function that only gets called on signal bars means the exit is only evaluated on bars where a signal fired.

On every other bar, the position sat there. Stops were not checked. Targets were not checked. Time-based exits were not checked. The strategy effectively had permission to hold through anything, as long as no new signal appeared.

The fix is not a change to the strategy at all:

for i, bar in enumerate(bars):
    signal = strategy.check(bar)
    if signal:
        strategy.execute(bar)
    strategy.manage(bar)           # <- exits, every bar, unconditionally

That is the entire change: move exit management out of the signal branch.

Why it flatters you

This one is directionally biased, and the bias is not subtle.

Losers get held longer than they should. A position that would have been stopped out keeps living until the next signal happens to fire. Sometimes it recovers. Sometimes it recovers because you were not checking — which is not a strategy, it is a delay.
The win rate inflates. My original run reported a 74.1% win rate. Fixing the call site moved it by 6 to 8 points. That is the difference between "this looks like a professional system" and "this looks like a coin flip with costs".
It shows up as fewer trades reaching their stop. If you log exits, you will see stop-loss exits that cluster in time — they fire in bursts on signal bars instead of being spread evenly. That is the fingerprint.

The mechanism is worth stating plainly: you were not simulating a strategy, you were simulating a strategy plus a free option to defer every adverse outcome. That option is worth a lot, and it is worth it in the direction you want.

How to find this in your own code

Three checks, in increasing order of effort:

1. Count exits by type over time. If stop-outs cluster on the same bars as entries, your exits are gated by the signal path. They should be roughly uniform in time, driven by price, not by signal timing.

2. Search for the phrase you are about to write. In almost every backtester, exits end up inside a function the entry path also calls. Grep for execute, handle_bar, on_bar, step — and ask, for each call site, "does this run on every bar or only sometimes?"

3. Print the bar index of every exit and every entry, then diff. If the sets of indices overlap far more than chance, exits are coupled to signals.

4. The structural fix beats the test. Make exit management a function that is called unconditionally, from the main loop, once per bar, and make it impossible to call it from the signal branch. If the API shape permits the bug, the bug will come back.

What it cost me

The strategy was negative before the fix. That is the uncomfortable part: this bug kept a losing strategy alive for months. If I had run it honestly I would have dropped it, and I would have been right to.

Bugs that make results worse get found. Bugs that make results better get shipped. This is the structural asymmetry of backtesting: your attention is drawn to things that look broken, and a false positive looks like progress.

Which is why the check has to be mechanical instead of attentional. I could not have caught this by staring harder at the strategy — the strategy was fine. I caught it by asking a question that had nothing to do with signals: where is this function called from?

The general shape

This belongs to a family I keep running into: the backtester's control flow is itself a modelling assumption.

You think you are modelling a strategy. You are actually modelling a strategy inside a simulation harness, and the harness decides when your code gets to see the world. Every asymmetry in that decision — which bars you get called on, whether you see a bar before or after it closes, whether the exit path is privileged — silently becomes part of the strategy.

So when an audit turns up nothing in the strategy, the next place to look is the harness. Mine was wrong in a way no amount of strategy review could have found.

The call-site check is one of the static checks in a small open-source linter (GitHub, MIT, no dependencies). It can flag exits that cluster with entries; it cannot read your control flow, which is why the structural fix matters more than the check.

"Enough Trades" Is Not a Number, It Is a Ratio

王伟 — Thu, 29 Oct 2026 01:00:04 +0000

Two configs, same strategy, same data. One becomes provable in under a year. The other cannot be proven in a working lifetime.

"Am I overfitting?" gets all the attention. The more common problem is quieter and harder to fix: you do not have enough data to tell, and no amount of cleverness changes that.

I had two configurations of the same breakout strategy. Same code, same decade, different instrument and parameter set.

	trades	per-trade vol	t
Gold, k=0.75, 1:1	2,352	0.75%	2.93
BTC, k=0.50, 4R	1,506	2.16%	1.10

Both looked reasonable. Both had positive expectancy. One was nearly provable; the other was not going to be provable at all.

The formula

There is a closed form for the sample size you need to reach a given t-statistic, and it has no free parameters:

n = (3 × per-trade volatility ÷ per-trade edge)²

Read it as: the number of trades you need is the square of three divided by your signal-to-noise ratio. Not "how many trades did I run". Not "how many years of data". The ratio.

That is why "I have 1,500 trades, that's plenty" is not a statement about evidence. 1,500 trades with noisy returns is worth less than 300 with clean ones.

Applying it

From the table, the observed t and n give the ratio directly: edge/vol = t / sqrt(n).

	t	n	edge/vol	trades needed	still needed	at this rate
Gold	2.93	2,352	0.060	2,466	114	~0.9 yr
BTC	1.10	1,506	0.028	11,202	9,696	~37 yr

Gold needs about nine more months of trading. BTC needs thirty-seven years — and that assumes the edge and the volatility both stay exactly where they are, which is the optimistic case.

Both are the same strategy. The difference is that BTC's per-trade volatility is 2.9× gold's.

Why noise dominates silently

Look at the right panel of that figure: n grows with the square of the inverse ratio. Halve your signal-to-noise and you need four times the data. Drop it to a third and you need nine times.

BTC's ratio is less than half of gold's, so the requirement is roughly (0.060/0.028)² ≈ 4.5× more trades — and BTC also produces more trades per year, so the year count should improve. It does not, because the ratio entered as a square while the trade frequency is only linear. Frequency cannot outrun a squared penalty.

This is why a strategy with a great-looking annualised return can be the worst candidate for deployment. BTC had the highest annualised figure of anything I tested and the weakest evidence by a wide margin. The return was real in the sample. The evidence that it would repeat was not.

What I do differently now

1. Compute the required n before the strategy is finished. If the answer is thirty-seven years, I want to know that on day one, not after six months of work. The calculation takes one line and it has changed which projects I start.

2. Treat "needs a lifetime" as a rejection. Not "promising but unproven". If the required sample is longer than the instrument's plausible lifespan, the hypothesis is unfalsifiable in practice, and an unfalsifiable strategy is indistinguishable from a random one for as long as I would be trading it.

3. Prefer high signal-to-noise instruments for a first test. Same logic as the gold/BTC split: if you must prove something, prove it where the arithmetic permits it. Testing on the noisiest instrument first is the most expensive possible ordering — you spend the most time and learn the least.

4. Report n and the ratio, not just the t. A t-statistic alone hides whether the result is 300 clean trades or 3,000 noisy ones. The ratio makes the difference visible in one line.

The uncomfortable corollary

Some of the things I believed in earlier were not tested — they were untestable, and I mistook the absence of proof for the presence of evidence.

That is a worse error than a bug, because a bug can be fixed by editing a line. An unfalsifiable hypothesis can only be abandoned, and abandoning it costs whatever I already invested in it.

The formula is unforgiving in a useful way. It does not care how much I like the strategy, or how plausible the story is. It just says: with this much noise and this much edge, you need this much data. If you cannot get the data, you cannot get the answer.

The fold, sizing and selection helpers, plus this arithmetic, are in a small open-source kit (GitHub, MIT, no dependencies). If you have a config where the required n came out short and the strategy still failed, that is the case I want to hear about — it means the ratio was not the binding constraint.

After 455 Parameter Sets, a t-Stat of 2.18 Means Nothing

王伟 — Wed, 21 Oct 2026 16:44:56 +0000

The significance test you learned assumes you ran one test. You ran hundreds.

I had a strategy with a t-statistic of 2.18 on 400 trades.

Under the textbook reading, that is significant. The conventional threshold for 5% two-sided is about 1.96, and 2.18 clears it. I had spent weeks on the parameter grid before that number appeared, evaluating roughly 455 parameter sets along the way.

So I asked the question that killed it: if all 455 of those sets had zero edge, how big would the best of them look?

The test nobody runs

The standard t-test answers: "if the true mean were zero, how often would a sample like mine show a t of 2.18 or larger?" That question is conditional on one sample.

I did not take one sample. I took 455, threw away 454 of them, and kept the largest. The quantity I am actually looking at is not a t-statistic from one draw — it is the maximum of 455 t-statistics. The distribution of a maximum is nothing like the distribution of a single value.

The arithmetic rule of thumb people quote is: expected false positives = N × α. With 455 sets at 5%, that would predict about 22.8 sets "significant" by luck. That number is right about the count and useless about the threshold, because it does not tell you how big the biggest one gets.

For that, simulate.

The simulation

I generated 455 strategies, each with:

exactly zero edge (mean 0)
400 trades, same volatility as mine
the same t-statistic formula

then recorded the maximum t across the 455, and repeated that 2,000 times. Everything below is real output, not a sketch.

Best-of-455 t under the null:
  median   2.99
  p90      3.51
  p95      3.71
  p99      ~4.0

P(best-of-455 t >= your 2.18) = 0.999

The median best-of-455 t under pure zero edge is 2.99. My 2.18 did not clear the median. It was below what search produces on a normal day.

Why the bar moves so much

The maximum of N draws grows, but slowly — roughly like sqrt(2 log N) for well-behaved tails. Two consequences worth internalising:

The usual t = 3 bar is not conservative enough when N is in the hundreds. People treat t = 3 as "strict". Under a 455-wide search at my sample size, the median best is 2.99. You would need to clear something like the 95th percentile — 3.71 — to say anything at all.

The bar depends on your sample size. More trades per strategy tightens the single-test distribution but does not shrink the selection effect; with 400 trades the null max still sits near 3. The two effects do not cancel, and it is easy to add trades while also adding grid points, netting out to nothing.

The honest accounting

Here is what I do now, and it is mechanical:

Write down N before searching. Not after. If I planned 20 sets and ended up evaluating 455, the number I report is 455.
Compute the selection-adjusted p-value, not the naive one. Simulate the null (or use a formula) and ask what the best of N looks like at my sample size.
Report N alongside the t-statistic, every time. A t-statistic without N is not interpretable once you have searched.

The third one is the cheapest and it is the one nobody does. "t = 2.18" sounds like a result. "t = 2.18 after 455 evaluations, where the null median is 2.99" is a different sentence entirely, and both are one line.

What this does not mean

It does not mean the strategy is bad. It means the evidence I had was insufficient to distinguish it from noise produced by searching.

Those are different conclusions with different next actions:

Conclusion	Next action
"The strategy has no edge"	Drop it
"The evidence is insufficient"	Collect more data, or search less
"The edge is real but unproven"	Paper-trade with small size while accumulating

Mine was the second. The honest move was to stop adding filters (each filter is a new search, and it would need its own out-of-sample) and instead increase trades per strategy while holding the grid fixed.

The uncomfortable arithmetic

Every additional parameter set you evaluate buys you a slightly higher chance of a good-looking result, and charges you a slightly higher bar to clear. Early on, the first effect dominates: going from 1 to 10 sets is clearly worth it, you learn something. Later, the second dominates: going from 400 to 455 mostly just raises the bar.

There is a crossover, and it is usually much earlier than people think. Mine was somewhere in the low tens.

The number of things you tried is not a detail of your process. It is a parameter of your test, and leaving it out is how a search gets reported as a discovery.

The simulation is part of a small open-source kit (GitHub, MIT, no dependencies): backtest_kit.py selection --tried N runs the null for your own N and sample size. It reports a probability, not a verdict — the verdict is yours.

My Walk-Forward Said +29.2 Per Trade. Honest Out-of-Sample Said -4.6.

王伟 — Wed, 21 Oct 2026 16:44:29 +0000

A parameter set that won on folds 1-3 lost on folds 4-5. Same strategy, same data.

I ran the standard walk-forward check. Three parameter sets, five sequential folds, about 400 trades. For each fold I picked the set that was best on the previous folds only, then scored it on the fold it had never seen.

Here is the whole output:

fold   picked      oos net    oos t
2      k0.75        436.5     0.88
3      k0.75        827.9     1.56
4      k0.75       -941.2    -2.51
5      k1.00       -813.2    -1.96

Stitched out-of-sample: n=107  net=-489.99  mean=-4.58  t=-0.51
In-sample best mean   :  +29.17 per trade
Optimism gap          :   33.75 per trade

The parameter set k0.75 looked best for three consecutive folds. It was the worst set for the next two.

+29.17 per trade in sample. −4.58 out of sample. The gap is 33.75 per trade, which is larger than the entire in-sample edge — the sign flipped.

What "out-of-sample" actually means

Most people treat out-of-sample as a split. You cut the data at some date, fit on the left, evaluate on the right, and call the right side honest.

That is not what makes it honest. Out-of-sample is a procedure, not a file. What makes it honest is that no choice in the evaluation was informed by the data being evaluated.

Here is the distinction, concretely:

What you did	Is the score honest?
Optimised on 2015-2020, tested on 2021-2023, and you never looked at 2021-2023 while designing	yes
Optimised on 2015-2020, tested on 2021-2023, then went back and tweaked because 2021-2023 looked bad	no — that file is now in-sample
Picked the parameter that maximised the whole period, then reported the second half	no — the second half informed the pick
Picked on folds 1..i-1, scored on fold i, never revisited	yes

The third row is the one that catches people. If you choose a parameter using any statistic computed over the test period — total return, Sharpe, whether the equity curve "looked right" — the test period has leaked into the choice, and the score is in-sample no matter which file it lives in.

The gap is not a haircut

The natural instinct is to treat the optimism gap as a discount: knock 20% off the in-sample number and move on.

That does not work, because the gap is not proportional to the edge. It is roughly proportional to how much selection freedom you had. In this example:

in-sample best mean: +29.17
honest out-of-sample: −4.58

The gap (33.75) was bigger than the edge (29.17). There was no discount that turns this back into a positive number, because the honest number is negative.

When the gap exceeds the edge, what you measured was not a weak strategy. You measured the selection process.

Why this happens

Three mechanisms, all mundane:

1. Fold-to-fold variance is large. A fold with 20-25 trades has a standard error of roughly sd / sqrt(25). If per-trade sd is around 100, the standard error of the mean is about 20 per trade. So a "best mean" of 29 is roughly one to two standard errors above zero — entirely reachable by luck. With three candidate sets, picking the top one at each step reliably picks the noisiest winner.

2. Regime change is not announced. The demo I used has the edge decaying after ~60% of the sample on purpose, which is a clean version of what real markets do more subtly. A parameter that suits the first regime is not obliged to suit the second.

3. The selection is repeated, not once. You do not choose a parameter once. You choose at fold 2, again at fold 3, again at fold 4. Each choice is a fresh chance to pick the lucky one, and the errors compound across the stitched result.

How to run this yourself

You need one thing beyond a trade log: a column saying which parameter set produced each trade.

entry_time, exit_time, side, entry_price, exit_price, size, fee, param_set
2024-01-02T09:30:00,...,BUY,100.12,101.40,10,2.00,k0.75
2024-01-03T14:05:00,...,SELL,101.85,100.90,10,2.00,k1.00

With that column, the procedure is mechanical:

Split trades into K sequential folds by time.
For fold i (starting at i=2), compute each parameter set's mean on folds 1..i−1.
Pick the best one.
Score it on fold i only.
Stitch the fold-i results. That stitched series is your honest number.
Compare it to the best in-sample mean. The difference is the gap.

python backtest_kit.py folds trades.csv --k 5

Without the column, all you can do is split validation, which is weaker — it tells you about stability over time, not about selection. The tool says so rather than pretending otherwise.

The part that stings

I did not set out to fool myself. I had a parameter grid, I picked the winner on the development data, and I reported the number. That is the standard workflow, and it is exactly the workflow that produces this gap.

The gap is not a sign of carelessness. It is the expected output of selecting on noise. Which means the fix is not "be more careful" — it is to make the selection procedure mechanical and to report the stitched number, including when it is negative.

The uncomfortable question is not "was my gap large". It is: how many strategies do I still believe in because I never computed this number for them?

The fold walk and the gap calculation are in a small open-source kit (GitHub, MIT, no dependencies). If you have run this on your own grid and the gap came out smaller than the edge, I would like to see the setup — that is the interesting case.

The One-Line Fix That Took My t-Stat From 6.22 to 1.04

王伟 — Wed, 21 Oct 2026 16:42:36 +0000

A breakout strategy passed all six parameter cells. It was a fill assumption, not an edge.

I had a breakout strategy on 1-minute data come in at t = 6.22.

Six parameter cells, all six passing. That is the kind of number that makes you start thinking about position size and funding an account. It is also the kind of number that should make you suspicious, because in ten years of doing this I have almost never seen a real edge survive a clean audit with a t-stat that high.

It was not real. The strategy was fine. The fill assumption was wrong, and it was wrong in exactly the direction that flatters you.

Here is the whole story, because the same bug is probably sitting in a lot of backtesters, including yours.

The bug

My breakout entry looked like this:

if high[j] >= buy_level:
    entry = buy_level

buy_level came from the previous session's high. The logic reads fine: price trades up through my level, I get filled at my level. That is what a limit order does.

The problem is the gap. When the new session opens with a gap straight through buy_level, bar j satisfies high[j] >= buy_level on its very first tick — at the open. The open is already above my trigger, and I record a fill at buy_level, a price that never traded in that session.

I was gifting myself the gap. Systematically, on every gap-through day, in the direction that made the equity curve look better.

The X is the recorded fill. The dotted line is that bar's low. The fill sits 1.48 below the lowest price of the bar — a price that, by construction, no one could have transacted at. The backtester did not complain, because nothing in the code knows what a plausible fill looks like.

The fix

One line:

entry = max(buy_level, open[j])   # the worse of the two, for a long

If the bar opened above my trigger, I get the open — the price I could actually have paid. If it opened below, I get my trigger, as intended. Same for shorts with min(...).

That is the entire change. Not a filter, not a parameter, not a new feature.

Same trades, same period, same parameter grid. 6.22 → 1.04. Passing cells: 6 of 6 → 0 of 6.

The entire edge lived in the fill.

Why this is hard to catch

Three reasons, and they compound.

It is plausible. entry = buy_level reads like correct code. It is the natural way to express "I got filled at my level". Nothing about it looks like a mistake, so code review slides right over it.

It inflates results in the direction you want. There is no error, no crash, no unusual log line. The only symptom is a nicer equity curve, which is precisely the thing you were hoping for. Your brain is not going to flag good news as a bug.

It hides in the gap. If you trade a market that does not gap, you may never see it. It only appears on days where the open jumps past the trigger — and on those days it silently substitutes a favourable price for the one you would have got.

How often it actually bites

I measured the share of entries where the recorded fill fell outside the bar's [low, high]:

The pattern is exactly what you would expect: the more a market gaps overnight, the worse it gets. Index futures on a session-based breakout are the worst case — roughly one entry in six.

And note what this does to the interpretation of a backtest. On NQ, 17.4% of entries were priced at a level the market never traded. If your strategy's edge is on the order of a few basis points per trade, and one entry in six is being handed a free gap, then the measured edge is mostly the gap.

How I found it

Not by reading the code. I had read that code a dozen times.

I found it by looking at when the largest 1-minute moves happened. I ran a data-quality pass that printed the twelve biggest 1-minute jumps with their timestamps, and all twelve landed between 18:00 and 18:05 New York time.

That clustering was the tell. Twelve enormous 1-minute moves, all in a five-minute window, every time — those are not market moves. That is the session boundary. The "jumps" were cross-session gaps being recorded inside a session-labelled dataset.

Once I was looking at gaps specifically, the fill assumption fell apart in about ten minutes.

The general lesson here is worth more than the bug: when you audit a backtest, look at the time distribution of anomalies, not just their size. A 2% one-minute move at a random time is a market event. A 2% one-minute move that always happens at 18:00 is a data-structure artefact, and it is telling you that your data is not shaped the way your code assumes.

What this generalises to

Fill assumptions are where backtests lie most often, because they are the one part of the simulation that has no external check. Your signal logic produces a number you can inspect. Your fill price is just... whatever you wrote.

Three specific forms to look for:

Gap-through fills — the one above. Trigger levels set from prior session data, filled at the trigger regardless of the open.
Look-ahead on exits — if execute() is only called on bars where a signal fired, then exits are only checked on those bars. I had this one too: fixing it moved a different strategy from −5.7% to +17.9%, and inflated the win rate by 6–8 points. The original reported a 74.1% win rate; most of that was the bug.
Cost assumptions borrowed from another asset class — two futures strategies of mine were costed at 5bp per side. Actual round-trip cost was closer to 0.30bp (NQ) and 0.55bp (MGC). That is an 18–33× overstatement. Correcting it moved one of them from −94.0% to −45.5%. Rule of thumb: if cost ÷ per-trade risk exceeds about 5%, suspect the cost model before you reject the strategy.

Three checks you can run on your own log

You do not need my code for these; you need your trade log and your price bars.

1. Is every fill inside its bar?

For each trade, find the bar containing the entry timestamp and check
low <= entry_price <= high. Anything outside never traded. This single check catches the gap-fill bug directly, and it is the highest-yield check I know.

2. Is the fill better than the bar's open?

For breakout entries, if a long fill is below the open, either a limit order filled later in the bar (fine, but then model it that way) or you have gifted yourself the gap (not fine). This one has false positives — limit orders legitimately fill better — so treat it as a prompt to look, not a verdict.

3. Do your anomalies cluster in time?

Print your largest N moves with timestamps and histogram the hour. If the biggest moves all sit at a fixed time of day, you are looking at session boundaries, and your session handling is probably wrong.

The tool

I turned the checks I now run into a small script, because doing them by hand every time is how you stop doing them. It is one file, standard library only, no pandas:

python backtest_lint.py trades.csv --prices prices.csv

That image is real output from the tool on a demo log that has five bugs deliberately planted in it. It scores 1/6 and names each one with the row number and the actual numbers. The same repo also ships a clean control that scores 6/6 — a checker that always says "fail" would be worthless, so both directions are testable before you trust it.

Code: https://github.com/wei498766-ai/backtest-lie-detector — MIT, no dependencies.

The uncomfortable part

The uncomfortable part is not that I had a bug. Everyone has bugs.

The uncomfortable part is that the bug made the result better, and the result looked like a discovery. I did not find this because I was careless. I found it because I got suspicious of a number that was too good, and then went looking for how it could be too good.

That is not a scalable process. It depends on being suspicious at the right moment, about the right number — and the numbers you most want to believe are exactly the ones you are least likely to question.

So the process has to be external. Cheap, mechanical checks that run whether or not you are in the mood to doubt yourself.

That is the whole argument for this kind of tooling. Not that it makes you a better trader, but that it does not care what you want to be true.

If you have hit something in this family — particularly anything where the backtester's assumption is invisible in the output — I would like to hear about it. The ones that do not throw errors are the ones worth collecting.