DEV Community

Quantooox
Quantooox

Posted on

Look-ahead bias: the backtest bug that uses tomorrow's newspaper

#python #datascience #statistics #beginners

Imagine testing a strategy for January 2024 using the unemployment rate for January 2024. Sounds fine, until you remember that number was published in early February. Your backtest just read tomorrow's newspaper.

That's look-ahead bias: using information that didn't exist yet at the moment of the decision.

Where it hides

  • Economic data. Most series are indexed by the period they describe, not the day they were released. The gap is often weeks.
  • Company reports. A quarter ends on 31 March; the filing may arrive in May. Joining on the quarter-end date leaks six weeks of the future.
  • Revisions. The GDP number you download today is not the one people saw at the time.
  • Index membership. Using today's list of index members for a past date (that one has its own name: survivorship bias).

The fix: store when you knew it

For every number we keep two dates: the period it describes, and the moment it became public. Then every join is an "as of" join on the publication time.

import pandas as pd

# facts: one row per value, with the time it was actually published
facts = facts.sort_values("published_at")
days = days.sort_values("decision_time")

# for each decision, take the latest value published *before* it
known = pd.merge_asof(days, facts, left_on="decision_time", right_on="published_at",
                      by="ticker", direction="backward")
Enter fullscreen mode Exit fullscreen mode

For US company filings we use the SEC's acceptance timestamp. (Gotcha we hit: in the SEC submissions feed, that timestamp is Eastern time but marked as UTC, so it has to be corrected.) For economic series we use release calendars. Where a release date is unknown, we apply a conservative lag rule rather than the period date.

The cost

Doing this makes backtests look worse. Usually noticeably worse. That's the point: the "better" version was never achievable in real time.

If a backtest's author can't tell you when each input was published, assume it leaks.

From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/economy

Educational, not investment advice.

Top comments (0)