DEV Community

dustin liang
dustin liang

Posted on

sklearn TimeSeriesSplit with gap: Walk-Forward Cross-Validation That Does Not Leak the Future

If you cross-validate a forecasting model with KFold(shuffle=True), the score you get is usually too good to be true. Rows from next month end up in the training set, rows from last month end up in the test set, and the model gets to interpolate instead of forecast. scikit-learn's TimeSeriesSplit fixes the ordering problem, and its gap parameter fixes a quieter one: when your label is built from a future window (the next 5 days, the next 20 days), the last training rows already contain information from the test period.

This tutorial shows how sklearn TimeSeriesSplit and gap work on a daily series: what the folds look like, how test_size, gap and max_train_size change them, how to check that no label window crosses into the test fold, how to plug the splitter into cross_val_score and GridSearchCV, and what breaks when you have several series per date (a panel of stocks, products or stores).

Code runs on scikit-learn 1.9, pandas 3.1 and NumPy 2.5. The data is synthetic and seeded: an autocorrelated daily index, the kind of thing you get from news counts, web traffic or sensor readings. Every number below comes from running the code.

A daily series with a forward-looking label

The setup is a typical forecasting table: features computed from the past, and a label that is the average of the next H days.

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score
from sklearn.ensemble import RandomForestRegressor

rng = np.random.default_rng(42)
n = 600
dates = pd.bdate_range("2024-01-01", periods=n)
x = np.zeros(n)
for i in range(1, n):
    x[i] = 0.9 * x[i - 1] + rng.normal()
s = pd.Series(x, index=dates, name="x")

H = 5  # label horizon in days
df = pd.DataFrame({
    "x": s,
    "ma5": s.rolling(5).mean(),
    "ma20": s.rolling(20).mean(),
    "chg5": s - s.shift(5),
})
# mean of the NEXT H values (t+1 .. t+H)
df["y"] = s[::-1].rolling(H).mean()[::-1].shift(-1)
df = df.dropna()
X, y = df[["x", "ma5", "ma20", "chg5"]], df["y"]
print(df.shape, df.index[0].date(), "->", df.index[-1].date())
print(df.round(3).head(3))
Enter fullscreen mode Exit fullscreen mode
(576, 5) 2024-01-26 -> 2026-04-10
                x    ma5   ma20   chg5      y
2024-01-26  0.474  0.402 -0.518 -0.338  0.246
2024-01-29  0.377  0.237 -0.499 -0.822  0.176
2024-01-30  0.154  0.224 -0.506 -0.065  0.080
Enter fullscreen mode Exit fullscreen mode

The reversed rolling mean is a compact way to build a forward window: y on a given day is the mean of x over the next five rows. The first 19 rows are dropped because ma20 needs history, and the last 5 because their future is not known yet.

Shuffled KFold vs TimeSeriesSplit: the score you should not trust

model = RandomForestRegressor(n_estimators=200, min_samples_leaf=5,
                              random_state=0, n_jobs=-1)
for name, cv in [
    ("KFold(shuffle=True)", KFold(n_splits=5, shuffle=True, random_state=0)),
    ("TimeSeriesSplit", TimeSeriesSplit(n_splits=5, test_size=60)),
]:
    r2 = cross_val_score(model, X, y, cv=cv, scoring="r2")
    print(f"{name:20s} R2 per fold {r2.round(3)}  mean {r2.mean():.3f}")
Enter fullscreen mode Exit fullscreen mode
KFold(shuffle=True)  R2 per fold [0.525 0.482 0.524 0.474 0.487]  mean 0.499
TimeSeriesSplit      R2 per fold [ 0.25   0.487  0.098  0.521 -0.431]  mean 0.185
Enter fullscreen mode Exit fullscreen mode

Same model, same features, very different answer. With shuffling, each test row has training neighbours from the day before and the day after, and because the series is autocorrelated those neighbours almost give the answer away. The walk-forward score is lower and much more uneven across folds, which is what live forecasting actually looks like. If your CV score drops sharply when you switch to TimeSeriesSplit, the drop is information, not a bug.

What TimeSeriesSplit folds look like

By default TimeSeriesSplit(n_splits=5) cuts the data into n_splits + 1 blocks, trains on everything before a block and tests on the block. The training set grows each fold (an expanding window).

def show_folds(cv, index):
    for k, (tr, te) in enumerate(cv.split(np.arange(len(index)))):
        print(f"fold {k}: train {index[tr[0]].date()}..{index[tr[-1]].date()} "
              f"({len(tr):3d})  test {index[te[0]].date()}..{index[te[-1]].date()} "
              f"({len(te)})")

show_folds(TimeSeriesSplit(n_splits=5), df.index)
Enter fullscreen mode Exit fullscreen mode
fold 0: train 2024-01-26..2024-06-07 ( 96)  test 2024-06-10..2024-10-21 (96)
fold 1: train 2024-01-26..2024-10-21 (192)  test 2024-10-22..2025-03-04 (96)
fold 2: train 2024-01-26..2025-03-04 (288)  test 2025-03-05..2025-07-16 (96)
fold 3: train 2024-01-26..2025-07-16 (384)  test 2025-07-17..2025-11-27 (96)
fold 4: train 2024-01-26..2025-11-27 (480)  test 2025-11-28..2026-04-10 (96)
Enter fullscreen mode Exit fullscreen mode

Three parameters change this picture:

  • test_size fixes the length of each test block instead of letting it be n_samples // (n_splits + 1). Fixed blocks (for example 60 business days, about a quarter) make the folds comparable.
  • gap drops that many rows between the end of training and the start of the test block.
  • max_train_size caps the training window, turning the expanding window into a rolling one.
show_folds(TimeSeriesSplit(n_splits=4, test_size=60, gap=5, max_train_size=250), df.index)
Enter fullscreen mode Exit fullscreen mode
fold 0: train 2024-05-20..2025-05-02 (250)  test 2025-05-12..2025-08-01 (60)
fold 1: train 2024-08-12..2025-07-25 (250)  test 2025-08-04..2025-10-24 (60)
fold 2: train 2024-11-04..2025-10-17 (250)  test 2025-10-27..2026-01-16 (60)
fold 3: train 2025-01-27..2026-01-09 (250)  test 2026-01-19..2026-04-10 (60)
Enter fullscreen mode Exit fullscreen mode

Note what stays fixed: the test blocks are always the last n_splits * test_size rows, so changing gap or max_train_size changes only the training side. That makes it easy to compare settings on exactly the same test days.

Why gap matters when the label looks forward

With H = 5, the label of a training row on day t is built from days t+1 to t+5. If training ends the day before the test block starts, the last few training labels are averages of test-period values. The model is being fitted on numbers it will later be graded on. A small helper makes the overlap visible:

def label_overlap(cv, horizon):
    """Count training rows whose label window reaches into the test block."""
    out = []
    for tr, te in cv.split(np.arange(len(df))):
        last_needed = tr + horizon          # last row each training label uses
        out.append(int((last_needed >= te[0]).sum()))
    return out

for g in (0, 2, 5):
    cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=g)
    print(f"gap={g}: leaking training rows per fold = {label_overlap(cv, H)}")
Enter fullscreen mode Exit fullscreen mode
gap=0: leaking training rows per fold = [5, 5, 5, 5, 5]
gap=2: leaking training rows per fold = [3, 3, 3, 3, 3]
gap=5: leaking training rows per fold = [0, 0, 0, 0, 0]
Enter fullscreen mode Exit fullscreen mode

The rule is simple: for a label that uses the next H rows, set gap >= H. With gap=5 no training label touches the test block. (If your label is the value exactly H steps ahead rather than an average, the same rule applies: row t still needs row t+H.)

How much does this change the score? Often less than people expect:

for g in (0, H):
    cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=g)
    mae = -cross_val_score(model, X, y, cv=cv, scoring="neg_mean_absolute_error")
    print(f"gap={g}: MAE per fold {mae.round(3)}  mean {mae.mean():.4f}")
Enter fullscreen mode Exit fullscreen mode
gap=0: MAE per fold [1.258 1.224 1.367 1.093 1.005]  mean 1.1895
gap=5: MAE per fold [1.268 1.218 1.352 1.09  0.999]  mean 1.1854
Enter fullscreen mode Exit fullscreen mode

On this series the difference is in the third decimal. Only the first few test days of each block sit next to leaked labels, and a random forest on four smooth features cannot exploit them much. That is not a reason to skip the gap. The size of the leak grows with the horizon, with smaller test blocks, and with models that can memorise individual rows (nearest neighbours, deep trees, boosted models with many rounds). The gap costs you H training rows per fold and removes the question entirely. The big win in this example came from dropping the shuffle; the gap is what makes the remaining number honest.

Scaling and tuning inside the folds

Leakage also comes from preprocessing. If you fit a StandardScaler on the full table before cross-validation, the scaler has seen the test period's mean and variance. Put it in a Pipeline so it is refitted on each training fold, and pass the same splitter to GridSearchCV:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import GridSearchCV

cv = TimeSeriesSplit(n_splits=5, test_size=60, gap=H)
pipe = make_pipeline(StandardScaler(), Ridge())
grid = GridSearchCV(pipe, {"ridge__alpha": [0.1, 1, 10, 100, 1000]},
                    cv=cv, scoring="neg_mean_absolute_error")
grid.fit(X, y)
res = pd.DataFrame(grid.cv_results_)[["param_ridge__alpha", "mean_test_score", "std_test_score"]]
res["mean_test_score"] = -res["mean_test_score"]
print(res.rename(columns={"mean_test_score": "MAE", "std_test_score": "MAE_std"}).round(4).to_string(index=False))
print("best:", grid.best_params_)
Enter fullscreen mode Exit fullscreen mode
 param_ridge__alpha    MAE  MAE_std
                0.1 1.0772   0.0378
                1.0 1.0771   0.0378
               10.0 1.0797   0.0380
              100.0 1.1073   0.0388
             1000.0 1.3039   0.1297
best: {'ridge__alpha': 1}
Enter fullscreen mode Exit fullscreen mode

Notice that a plain ridge regression on the same four features beats the random forest from the earlier sections (MAE about 1.08 against 1.19): with honest folds, simple models often hold up well. Two details worth knowing. First, GridSearchCV refits the best model on all rows at the end (refit=True), which is what you want for the final model but means grid.best_score_ is the only out-of-sample number you have. Second, keep a final hold-out period that the grid search never sees; tuning on the same folds you report is a mild form of overfitting.

Panel data: several rows per date

TimeSeriesSplit splits rows, not dates. That is fine for one series, but with several series per day (one row per stock, product or store) a single date can land on both sides of a fold boundary.

tickers = ["AAA", "BBB", "CCC"]
panel = pd.DataFrame(
    [(d, t, rng.normal()) for d in dates[:20] for t in tickers],
    columns=["date", "ticker", "f1"],
)
cv = TimeSeriesSplit(n_splits=4, test_size=7)
for k, (tr, te) in enumerate(cv.split(panel)):
    shared = set(panel.loc[tr, "date"]) & set(panel.loc[te, "date"])
    print(f"fold {k}: test starts {panel.loc[te[0], 'date'].date()} {panel.loc[te[0], 'ticker']}"
          f"  dates in both train and test: {[d.date().isoformat() for d in sorted(shared)]}")
Enter fullscreen mode Exit fullscreen mode
fold 0: test starts 2024-01-15 CCC  dates in both train and test: ['2024-01-15']
fold 1: test starts 2024-01-18 AAA  dates in both train and test: []
fold 2: test starts 2024-01-22 BBB  dates in both train and test: ['2024-01-22']
fold 3: test starts 2024-01-24 CCC  dates in both train and test: ['2024-01-24']
Enter fullscreen mode Exit fullscreen mode

With 3 tickers and a test size of 7 rows, three of the four boundaries fall in the middle of a day; only fold 1 happens to start on a clean date. In practice this leaks same-day information: tickers move together on market-wide days, so training on AAA's Monday row and testing on BBB's Monday row is not a forecast. The fix is to split the unique dates and map the folds back to rows. A generator works anywhere cv= is accepted:

def date_time_series_split(dates_col, n_splits, test_days, gap_days=0):
    days = np.sort(dates_col.unique())
    pos = dates_col.map({d: i for i, d in enumerate(days)}).to_numpy()
    tss = TimeSeriesSplit(n_splits=n_splits, test_size=test_days, gap=gap_days)
    for tr_days, te_days in tss.split(days):
        yield np.flatnonzero(np.isin(pos, tr_days)), np.flatnonzero(np.isin(pos, te_days))

for k, (tr, te) in enumerate(date_time_series_split(panel["date"], 3, 4, gap_days=1)):
    d_tr, d_te = panel.loc[tr, "date"], panel.loc[te, "date"]
    print(f"fold {k}: train rows {len(tr):2d} up to {d_tr.max().date()}  "
          f"test rows {len(te):2d} {d_te.min().date()}..{d_te.max().date()}  "
          f"overlap {len(set(d_tr) & set(d_te))}")
Enter fullscreen mode Exit fullscreen mode
fold 0: train rows 21 up to 2024-01-09  test rows 12 2024-01-11..2024-01-16  overlap 0
fold 1: train rows 33 up to 2024-01-15  test rows 12 2024-01-17..2024-01-22  overlap 0
fold 2: train rows 45 up to 2024-01-19  test rows 12 2024-01-23..2024-01-26  overlap 0
Enter fullscreen mode Exit fullscreen mode

Now every fold holds whole days, gap_days is measured in days rather than rows, and the overlap is zero. Pass it as cv=list(date_time_series_split(...)) to cross_val_score or GridSearchCV; materialising the list lets the search reuse the same folds for every parameter setting.

A pre-flight check before you trust the score

Whatever splitter you use, a few assertions catch most mistakes:

def check_folds(folds, index, horizon):
    for k, (tr, te) in enumerate(folds):
        assert index[tr].max() < index[te].min(), f"fold {k}: train is not before test"
        assert tr.max() + horizon < te.min(), f"fold {k}: label window reaches test"
    return f"{len(folds)} folds OK"

print(check_folds(list(TimeSeriesSplit(5, test_size=60, gap=H).split(X)), df.index, H))
try:
    check_folds(list(TimeSeriesSplit(5, test_size=60).split(X)), df.index, H)
except AssertionError as e:
    print("AssertionError:", e)
Enter fullscreen mode Exit fullscreen mode
5 folds OK
AssertionError: fold 0: label window reaches test
Enter fullscreen mode Exit fullscreen mode

The check assumes rows are sorted by time, which TimeSeriesSplit also assumes and never verifies. Sort first (df.sort_index()), and sort panel data by date before building the folds.

FAQ

What does gap do in TimeSeriesSplit?
It removes gap rows between the end of each training set and the start of its test set. Use it when a label (or a feature) is computed from a window that would otherwise reach across the boundary.

How big should gap be?
At least the label horizon: for "mean of the next 5 days" or "value 5 days ahead", gap=5. If features use centred or forward windows, add those too, or better, fix the features.

Does gap shrink the test set?
No. Test blocks stay the same; the gap is taken from the end of the training data, so each fold trains on gap fewer rows.

Is TimeSeriesSplit the same as purged cross-validation?
Close, for forward-looking labels: a gap of H rows purges the overlapping training labels. Purged k-fold with an embargo goes further by also testing on blocks in the middle of the sample; TimeSeriesSplit only ever trains on the past.

Why is my TimeSeriesSplit score so much worse than KFold?
Usually because KFold was leaking. On autocorrelated data shuffled folds let the model interpolate between neighbouring days. The walk-forward number is the one to plan with.

Conclusion

For time-ordered data, KFold(shuffle=True) answers the wrong question. TimeSeriesSplit trains only on the past; test_size keeps the test blocks comparable; gap set to at least the label horizon stops forward-looking labels from leaking into the test period; and max_train_size gives you a rolling window when old data stops being relevant. Keep scaling inside a Pipeline, split on unique dates when there are several series per day, and run a one-line assertion on the folds before reading the score.

News-flow attention is a common example of a daily feature with exactly these properties: autocorrelated, published per stock and per day, and easy to misalign with a forward return window. Lava's stock heat monitoring page shows how that kind of per-stock and per-sector heat data is laid out for A-shares.

中文简介:本文用一条固定种子的合成自相关日度序列,讲解 scikit-learn 的 TimeSeriesSplit 与 gap 参数:先对比 KFold(shuffle=True) 与按时间切分的 R²,说明打乱切分在自相关数据上会严重高估;再展示 test_size、gap、max_train_size 对折的影响(测试块不变,只改训练端);用一个小函数统计“标签窗口跨进测试块”的训练行数,给出 gap ≥ 标签预测期 H 的规则,并如实说明本例中 gap 对得分影响很小但能消除泄漏;随后把 StandardScaler 放进 Pipeline 配合 GridSearchCV 调参;针对多只股票同一天多行的面板数据,说明按行切分会把同一天拆到训练和测试两边,给出按唯一日期切分的生成器;最后提供折检查断言。

Top comments (0)