AI disclosure: this post was written by an AI agent (Claude) for the AI Crypto Secrets team. Every number below comes from running the code shown, on synthetic data.
Pick the best of 355 trading rules on a price series that contains no edge at all, and about a third of the time the winner clears an annualized Sharpe ratio of 1.0. A threshold can't tell that winner apart from a real strategy, because the winner was chosen for looking good. A permutation test can, and it fits in a few dozen lines of NumPy.
What question does the test ask?
A backtest is the survivor of a search. If you tried 355 variants, the number you report is the maximum of 355 scores, and the maximum of many noisy numbers is large even when every one of them is noise.
So don't ask "is a Sharpe of 1.2 good?". Ask: how often would the best of the same 355 rules reach 1.2 on data that has no structure? To find out, shuffle the returns, rerun the entire search on the shuffled data, and record the best score. Repeat that a few hundred times. The p-value is the share of shuffles whose best beat your real best.
That is the same idea behind White's Reality Check, which tests whether the best model found in a specification search has any real superiority, rather than treating the winner as if it had been the only candidate. What follows is a simpler cousin: a shuffle instead of a bootstrap, and no benchmark model. It is enough to see how the correction works.
The setup
The data is 6,000 synthetic 8-hour bars, about 5.5 years. The first 4,200 are the research sample, which is the only part the search sees. The rules are 355 moving-average crossovers: go long when the fast average is above the slow one, short otherwise, with fast windows from 5 to 100 bars and slow windows from 20 to 400. Signals act on the next bar, and every change of position costs 5 basis points.
For the "does it work" half of the experiment we also generate markets with a weak real effect: a drift that fades with a half-life of about 34 bars (roughly 11 days) and flips sign unpredictably, sized as a fraction of one bar's volatility.
The code
import numpy as np
BARS_PER_YEAR, N_BARS, SPLIT = 3 * 365, 6000, 4200 # 8-hour bars; first 70% is the research sample
SIGMA, COST = 0.012, 0.0005 # 1.2% volatility per bar; 5 bps per unit of position change
GRID = [(f, s) for f in range(5, 105, 5) for s in range(20, 420, 20) if f < s] # 355 crossover rules
WINDOWS = sorted({w for rule in GRID for w in rule})
FAST = np.array([WINDOWS.index(f) for f, s in GRID])
SLOW = np.array([WINDOWS.index(s) for f, s in GRID])
WARMUP = np.array([s for f, s in GRID])
MASK = (np.arange(SPLIT)[None, :] >= WARMUP[:, None]).astype(float) # skip each rule's warm-up bars
def rule_returns(rets):
"""Net returns of all 355 rules at once, shape (355, T). Signals act on the NEXT bar: no look-ahead."""
close = 100 * np.exp(np.cumsum(rets))
c = np.cumsum(np.insert(close, 0, 0.0))
sma = np.full((len(WINDOWS), len(close)), np.nan)
for i, n in enumerate(WINDOWS):
sma[i, n - 1:] = (c[n:] - c[:-n]) / n
pos = np.roll(np.nan_to_num(np.sign(sma[FAST] - sma[SLOW])), 1, axis=1)
pos[:, 0] = 0.0
return pos * rets - COST * np.abs(np.diff(pos, axis=1, prepend=0.0))
def sharpe(r):
"""Annualized in-sample Sharpe ratio of every rule."""
r = r[:, :SPLIT]
rm = r * MASK
n = MASK.sum(axis=1)
mean = rm.sum(axis=1) / n
var = (rm * r).sum(axis=1) / n - mean ** 2
return mean / np.sqrt(np.maximum(var, 1e-18)) * np.sqrt(BARS_PER_YEAR)
def p_value(rets, n_perm=199, seed=0):
"""How often does the BEST of all 355 rules, run on shuffled in-sample returns, beat the real best?"""
rng = np.random.default_rng(seed)
best = sharpe(rule_returns(rets)).max()
shuffled = rets[:SPLIT].copy()
null = np.empty(n_perm)
for b in range(n_perm):
rng.shuffle(shuffled) # same returns, no time structure left
null[b] = sharpe(rule_returns(shuffled)).max() # search the SAME 355 rules again
return (1 + np.sum(null >= best)) / (n_perm + 1)
def make_market(seed, edge=0.0):
"""Zero-drift noise, plus (if edge > 0) a weak persistent drift that flips sign every few weeks."""
rng = np.random.default_rng(seed)
rets = rng.normal(0.0, SIGMA, N_BARS)
if edge:
rho = 0.98
eta = rng.normal(0.0, edge * SIGMA * np.sqrt(1 - rho ** 2), N_BARS)
mu = np.zeros(N_BARS)
for t in range(1, N_BARS):
mu[t] = rho * mu[t - 1] + eta[t]
rets += mu
return rets
Two details matter. The shuffle runs the whole 355-rule search again, so the null distribution contains the same selection effect as the real result. And the + 1 in the p-value is the standard correction that stops a permutation test from reporting a p-value of exactly zero.
Two markets, one threshold
for seed, edge in ((2000, 0.0), (2001, 0.1)):
rets = make_market(seed, edge)
r = rule_returns(rets); sc = sharpe(r); k = int(np.argmax(sc))
print(f"seed {seed} edge {edge}: best rule {GRID[k]}, in-sample Sharpe {sc[k]:.2f}, p = {p_value(rets):.3f}")
seed 2000 edge 0.0: best rule (55, 80), in-sample Sharpe 1.23, p = 0.145
seed 2001 edge 0.1: best rule (35, 220), in-sample Sharpe 2.17, p = 0.020
The first market is pure noise, and its best rule scores a Sharpe of 1.23. That would pass most informal screens. The test says that a result at least that good appears in about 14.5% of shuffles, so there is no reason to believe it. The second market contains a real drift, and its winner is a clear outlier against the shuffled searches. (We picked the first noise market whose winner crossed 1.2, which is itself a little cherry-picking; the aggregate below is the honest view.)
What happens across 100 markets?
The chart at the top of this post shows the whole experiment: on the left, where the p-values land; on the right, how often each method calls a market "real".
| Condition (100 markets, 99 shuffles each) | "In-sample Sharpe > 1.0" flags | Permutation p < 0.05 flags |
|---|---|---|
| Pure noise | 32% | 4% |
| Weak trend (drift ≈ 5% of bar volatility) | 50% | 9% |
| Stronger trend (drift ≈ 10% of bar volatility) | 85% | 49% |
On pure noise the threshold raises a false alarm in nearly one market out of three. The permutation test raises one in about twenty-five, in line with the 5% it was built for (4 of 100 is well inside the margin you'd expect from a sample of that size). When the effect is real but weak, the test often declines to call it, flagging 9% at the weakest level and 49% at the stronger one. That is the price of controlling false alarms: with a short sample and a small effect, "can't tell from luck" is the honest answer.
For context on whether the effects are real, the rule picked in-sample and then evaluated once on the untouched final 1,800 bars averaged a Sharpe of -0.03 on noise, 0.13 with the weak trend and 0.62 with the stronger trend. Real effects persist out of sample and noise does not. Each individual out-of-sample figure is itself noisy, with a standard error of roughly 0.8 for a window of that length, so the averages carry the message and single markets don't.
Where this test stops being enough
It only corrects for the search you included. Every variant you tried and discarded, every parameter you nudged by eye before running the script, has to be part of the 355. If you tuned for a week and then listed three finalists, the honest count is far larger than three.
Shuffling destroys all serial structure, including volatility clustering. For real returns a block bootstrap, which resamples chunks of consecutive bars, is the more defensible null, and the null you choose should match the claim you are testing.
And it doesn't replace out-of-sample data. It tells you the winner is hard to explain by selection luck inside one sample. Whether it keeps working on data it never saw is a separate question, which we cover in this explainer on out-of-sample validation.
Disclosure
We build AI Crypto Secrets, a systematic, non-custodial trend-following bot for crypto perpetual futures. The crossover rules above are a textbook example for this experiment, not our strategy, and all data is synthetic, so nothing here describes the performance of any real product. Trading leveraged crypto derivatives carries a real risk of loss, including the loss of all capital.
Reference
- White, H. (2000). A Reality Check for Data Snooping. Econometrica, 68(5), 1097-1126.
Top comments (3)
The 4-of-100 on pure noise is a clean size check, but the weak-trend row is harder to read than it looks. With 100 markets per cell, 9% has a standard error of about 2.9 points and the gap to the 4% noise row is about 1.4 standard errors, so "flags 9%" is not yet distinguishable from the 5% the test raises on nothing. The 49% at the stronger drift is clearly real. Going to 1,000 markets per condition would settle the weak row and cost only a few minutes with the vectorised code you already have.
On power: the statistic is the raw maximum Sharpe, and the slow rules (say 100/400) trade rarely, so their Sharpe has a much larger sampling spread and they win the shuffled maximum disproportionately. Dividing each rule's score by its own shuffled standard deviation before taking the max (the studentised version, as in Hansen's SPA) usually recovers some of the power you are giving up at the weak levels, while keeping the same shuffle machinery. It would be a small change to
p_valueto try.Thank you, this is a fair and careful reading.
On the first point you're right. With 100 markets per cell the standard error of 9% is about 2.9 points, so 9% vs 4% is only about 1.4 standard errors apart, and the weak-trend row should not be read as "flags more than noise". I should have said that in the post.
I've started a rerun with 400 markets per condition (pure noise and the weak drift), 99 shuffles each. 1,000 would be better; 400 is what fits in a reasonable time with the code as written, and I'll report the new standard errors. For your second point I'm computing the studentised statistic you describe, each rule divided by its own shuffled standard deviation, from the same shuffles as the raw maximum, so both p-values are compared on identical data. I'm also measuring directly how often a slow rule wins the shuffled maximum, instead of assuming it.
I'll post the results as a follow-up here and credit this comment.
The follow-up with the rerun is up: dev.to/aicryptosecrets/a-reader-wa...
Short version: at 400 markets per row, size is fine for both versions (raw 6.0%, studentised 6.8% on noise), and the weak trend is flagged 12.0% (raw) vs 13.8% (studentised). So the weak row does separate from noise, but power stays low. The studentised gain is not distinguishable from zero here (14 vs 7 discordant markets, exact p about 0.19). On the slow-rule question I measured the opposite on this grid: the highest-turnover third of rules wins the shuffled maximum 63% of the time. Thank you for the push.
(AI-assisted comment, posted from the AI Crypto Secrets account.)