train_test_split(X, y, shuffle=True) is fine for photos of cats. For anything that happens in time, it quietly lets the model learn from the future: Tuesday's training row sits next to Monday's test row, and markets on neighbouring days are related.
Walk-forward
Keep time in order:
- Train on everything up to a date.
- Test on the period right after it.
- Move forward and repeat.
Our schedule for the stock models:
- Training window: the trailing 756 trading days (about three years).
- Refit: every 63 trading days (about a quarter).
- Label: each stock's result versus SPY from the next open to the open 20 trading days later.
The gap (embargo)
There's a second, subtler leak. The label looks 20 days ahead. So the last training rows have labels that overlap with the first test days. The model has, in effect, seen part of the test period's outcome.
The fix is an embargo: leave a gap of 20 trading days between the end of the training data and the start of the test period, so no training label reaches into the test window.
def walk_forward(dates, train=756, step=63, embargo=20):
start = train + embargo
while start < len(dates):
tr = dates[start - embargo - train : start - embargo]
te = dates[start : start + step]
yield tr, te
start += step
What it costs
Fewer training rows, more refits, and results that look worse than a shuffled split. Every one of those is a more honest picture of what would have happened in real time.
If a time-series result was produced with a random split, treat it as unverified.
From Quantooo Lab, the makers of quantooo.com, a free, open and educational quant lab: every formula, every test and every failure is public. See it here: https://quantooo.com/method
Educational, not investment advice.
Top comments (0)