Purged and Embargoed Cross-Validation for Options ML

#algorithms #python #datascience #machinelearning

Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it.

The Problem With k-Fold in Time Series

Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits
next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap. A label at
bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future.
The model leaks.

V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain
TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each
other.

Purged + Embargoed CV

For each test window [t0, t1]:

Purge any train row whose label window overlaps the test window.
Embargo max_training_horizon bars after the test window — drop those too.

Overlapping labels are not i.i.d. Purging + embargoing makes the split honest.

def purged_embargo_split(n, n_splits=5, embargo_frac=0.02):
    idx = np.arange(n)
    fold = np.array_split(idx, n_splits)
    splits = []
    for i in range(n_splits):
        test = fold[i]
        emb = int(len(test) * embargo_frac)
        lo, hi = max(0, test[0]-emb), min(n, test[-1]+emb+1)
        train_mask = np.ones(n, bool); train_mask[lo:hi] = False
        splits.append((idx[train_mask], test))
    return splits

Tune Only When You Have Enough

Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless.
Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the
date range or symbol basket first; don't trust the trial.

Three-Way Split, Always

train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/
isotonic) → test (untouched, final score only). V1 sometimes conflated validation and
calibration. Keep them separate.

The Promotion Gate

Log every trial's train/val/test gap, not just the winner's test score. Promote only if
replay AND shadow (≥1 live session) both beat baseline on buyer metrics: 1.5x/2.0x hit
rate, MAE-before-hit, time-to-hit, wrong-side rate.

Research only. Not investment advice.

DEV Community