The scariest bug in a backtest (the thing that estimates a strategy's returns on past price moves) isn't a crash.
It's accidentally using tomorrow's price.
It's called lookahead bias, and if you let even a sliver of tomorrow's close leak into today's math, your strategy turns into a prophet. The backtest numbers glow. Then you ship it live and it melts on contact. And the code never throws a single exception. If anything the numbers get better, so you're in a great mood.
(The worst kind of bug is the one that makes you feel good.)
This time I was writing the backtest engine for a pairs-trading strategy, and before I even started coding, the "no cheating" design got into a fight at the planning stage. More precisely, plans that past-me had written were fighting each other.
Shift once, or shift twice? The docs disagreed
The "no cheating" implementation basically comes down to "slide a series over by one" (shift(1) in pandas). Today's decision shows up in tomorrow's trade — you reproduce that real-world lag in code.
And my planning docs said this:
- Overall plan (past-me, a few days ago): "Shift the hedge ratio beta by
shift(1). Shift the position too, so it fills the next day." - Latest handoff note (also me): "Wait, isn't that a double delay?"
Right. Shift once for beta, shift again for the position, and you're now two days late in total.
Here's today's whole point:
Too few shifts (= cheating) and too many shifts (= trading a day late) both quietly move the numbers, and neither one raises an error.
Cheating inflates your returns, over-delay deflates them. What comes out either way is a plausible-looking Sharpe ratio (the score for how good the returns are). You glance at the output, go "yeah, seems reasonable," and it sails through. Be honest — are you confident you'd catch one extra shift(1) in code review? I'm not. So I gave up on eyeballing it.
I stopped deciding "where to shift" case by case, and pinned the invariant to one sentence
Here's the approach I took.
First, pin the rule you have to protect down to a single sentence.
The realized PnL of bar t depends only on information up to the close of day t−1.
And put the actual shift in exactly one place. The signal math (beta, spread, z-score, the buy/sell decision) can use everything up to today's close. But in the PnL calculation, exactly one shift(1) — "today's decision takes effect starting tomorrow."
Collect the shift into one spot and the thought "wait, do I need a shift here too?" disappears entirely. A double delay can't happen either — there's only one place that shifts.
But writing the policy in a docstring (the explanatory text inside the code) is just a promise. Promises get broken. By me, three months from now.
Fire a spike into the future, and prove the past doesn't budge
So I "proved" this invariant with a test. I'm fond of how it turned out, so let me show you.
Test 1: future spike injection.
Take the same price series twice, and into one of them only, inject a giant spike on some future day (day t+k) — like price +500%. Compute beta and z-score for each, and verify that every value before the spike is byte-for-byte identical (not approximately equal — numpy's array_equal, i.e. exact match).
Whatever happens in the future, the past computation doesn't move by a single bit. If the beta window were accidentally including the future, the spike would "bleed" back into the past and break the match.
I locked the other direction too: I also assert that beta on the spike day does change. Pin the rule "the compute window includes today's close" with a test, and if someone later adds a shift out of kindness, that test fails too.
(An extra shift tends to be born of kindness.)
Test 2: entries realize the day after the decision.
Build synthetic data where the z-score crosses the entry threshold on day 5, and strictly verify that PnL is zero on day 5 and only starts on day 6. If "decide today, profit from today's move" ever happens, it's an instant fail.
Now both "no cheating" and "not a day too late" are locked in as regression tests. From here on, if I refactor the engine and these tests stay green, I can at least say there's no time-direction cheating.
Aside: the correct engine failed a test
While I'm confessing — a sanity test that said "a mean-reverting series should turn a profit" failed at first. Bad luck on the RNG seed: with only 10 trades on the synthetic data, it just happened to land on the loss side. I ran the seed and parameters through a few combinations, swapped in synthetic data that reliably turns a profit, and that fixed it. I burned 30 minutes suspecting an engine bug — but what I should have suspected was the RNG in the test data.
Takeaways
Kill time-direction bugs with tests, not eyeball review.
- Pin the rule you protect to one sentence — "bar t's PnL depends only on info up to t−1" — and collect the shift into one place
- Fire a spike into the future and prove the past is byte-for-byte identical (catches a missing shift)
- Also pin that today's value does change (catches an extra shift)
- Strictly verify with synthetic data that entries realize the next day
A cheating backtest will praise you. When it does, try firing one spike into the future.
Top comments (0)