The easiest way to publish a false positive in time-series work, with a worked example from my own project.
The result that looked publishable
A walk-forward test, 913 causal anchors, direction accuracy at a 30-minute horizon.
Benchmark note that matters: the thing to beat is not 50%, it is the majority class. If 51% of moves were up, "always guess up" scores 51%. Scoring against 50% is how a slight directional bias gets mistaken for signal.
Result: 55.2% against a 51.1% majority class. +4.1 pp on n=912, about 2.4 standard errors.
Why it was not real
Anchors were spaced about 7 minutes apart. The horizon was 30 minutes.
A forecast at 10:00 covering to 10:30, and one at 10:07 covering to 10:37, share 23 of their 30 minutes of outcome. Those are not independent observations. n=912 was a much smaller number of distinct market episodes, counted repeatedly.
Standard error scales with the square root of n. Inflate n artificially and you shrink your error bars artificially.
Re-scored on non-overlapping anchors
Overlapping: n=912, majority 51.1%, model 55.2%, edge +4.1 pp, 2.4 s.e.
Non-overlapping: n=163, majority 54.0%, model 59.5%, edge +5.5 pp, 1.4 s.e.
The edge grew. The significance nearly halved. The inflation factor was about 2.4x, which is the square root of 912/163.
The trap is that the effect size grows. If it had collapsed to zero the result would be obviously dead. Instead it improves, which tempts you into "the real effect is stronger, I just need more data." A larger estimate on a much smaller sample is what noise looks like.
The check: compare anchor spacing to forecast horizon. Spacing smaller than horizon means overlap. Either re-score on non-overlapping anchors or space them a full horizon apart from the start.
This applies to any sliding-window evaluation where the window is wider than the stride. Sensor data, load forecasting, anomaly detection, medical monitoring.
https://michaelhairetis.medium.com/how-overlapping-windows-manufactured-my-best-result-a85e076865b6
Top comments (0)