DEV Community

shakti tiwari
shakti tiwari

Posted on

Purged Cross-Validation: overfitting kyun marta hai trading models ko

From the options-buyer-system-v2 blueprint — why plain k-fold silently destroys your trading backtest, and the 4-line fix that actually works.

The problem nobody talks about

Tumne XGBoost train kiya, train AUC 0.92, test AUC 0.91 — "model sahi hai!" Aur fir live mei lagaya, 31.6% win rate, −₹90k loss. Kya hua? Answer: tumhari validation galat thi. Financial data mei standard ML validation (k-fold, plain TimeSeriesSplit) silently leak karta hai, aur tumhe false confidence deta hai.

Why k-fold fails on time series

Financial data sequential hai. k-fold rows shuffle karta hai, toh training mei ek row Tuesday 2 PM ki, test mei ek row Monday 10 AM ki — spatially distant par temporally adjacent. Worse: triple-barrier labels overlap karte hain. Bar t ka label 6 bars aage dekhta hai; training row t+2 us future ka "pata" hai. Model leak kar gaya, tumhe pata bhi nahi.

Meri V1 history full thi "HIGH overfit" verdicts se — train AUC high, test flat. Plain TimeSeriesSplit bhi adjacent windows bleed karta hai.

Purged + Embargoed CV (the fix)

Har test window [t0, t1] ke liye:

  1. Purge — training rows jinki label window test se overlap karti hai, hatao.
  2. Embargo — test window ke baad max_training_horizon bars drop karo (ye bhi leak hain). 
def purged_embargo_split(n, n_splits=5, embargo_frac=0.02):
    idx = np.arange(n)
    fold = np.array_split(idx, n_splits)
    splits = []
    for i in range(n_splits):
        test = fold[i]
        emb = int(len(test) * embargo_frac)
        lo, hi = max(0, test[0]-emb), min(n, test[-1]+emb+1)
        m = np.ones(n, bool)
        m[lo:hi] = False
        splits.append((idx[m], test))
    return splits
Enter fullscreen mode Exit fullscreen mode

Ye 4-line function V1 ke overfit ko kaat deti hai. Overlapping labels i.i.d. nahi hote — purge+embargo honest split banata hai.

Tune only when you have enough

Optuna ek baar 4 decisive rows pe "jeet" gaya — statistically meaningless. Rule: decisive validation rows (non-abstained) < 30-50 ho toh date range widen karo ya symbol basket badhao, trial mat trust karo.

Meri V1 mei Optuna "won" on 4 rows — us "best" params ne live mei 10s gateway timeout diya kyunki n_estimators=1620 hardcoded tha. Validation choti thi, overfit thi, phir bhi production mei chala diya.

Three-way split, always 

train (fit) → validation (early stop + HP select)
            → disjoint calibration (sigmoid/isotonic)
            → test (untouched, final score only)
Enter fullscreen mode Exit fullscreen mode

V1 ne validation aur calibration conflate kiya tha. Calibration set alag rakho — wo model ke raw probability ko real probability mei convert karta hai. Test set kabhi bhi training mei mat lena.

The promotion gate (hard rule)

Har head ka overfit_gap = train_metric − test_metric. > 0.15 → BLOCK. Log har trial ka gap, sirf winner ka nahi. Promote sirf agar replay AND shadow (≥1 live session) dono baseline se beat karein buyer metrics pe: 1.5x/2.0x hit rate, MAE-before-hit, time-to-hit, wrong-side rate.

Meri V1 mei mean_overfit_gap >= 0.15 pe FAIL verdict aata tha, par manually invoke hota tha. V2 mei ye DEFAULT gate hai — har head, har retrain, automatic.

Underfit signal

Agar balanced accuracy 50% ke paas hai aur overfit gap bhi chota hai — fix "more features add karo" nahi hai (V1 ki repeat mistake). Solution: event sampling, better labels, ya check karo ki truth-quality filter silently bahut kam signal bacha toh nahi.

Real example from my build

Feature importance: top regression features prev_return_5 (21.3%), range_low (15.1%), atr (15.1%). Purged-CV ke baad test AUC 0.70-0.74 aaya (synthetic signal pe) — realistic, optimistic nahi. Overfit gap < 0.10 raha, so promote kiya gaya paper stage ke liye.

Why this matters for options buyers

CE/PE premium transformed signal hai. Agar tumne leak kiya validation mei, model "CE kharid" bolta hai kyunki usne future premium dekh liya — live mei wo future nahi hota. Purged CV ye rokta hai.

FAQ

Q: TimeSeriesSplit kyun nahi? Adjacent windows bleed karte hain, partial leak.
Q: Embargo kitna? max_training_horizon bars — label window length.
Q: 30-50 rows kaise laoon? Date range ya symbols badhao.
Q: Calibration zaruri hai? Haan, raw XGBoost probability calibrated nahi hoti.

Common mistakes

  1. Shuffled k-fold on time series — sabse badi.
  2. Optuna choti validation pe tune — false win.
  3. Validation=calibration conflate.
  4. Overfit gap ignore karke promote.

What I learned

Backtest "looks good" bolta hai jab validation leak ho. Purged CV ek band-aid nahi, architecture ka hissa hai. V1 ke saare "HIGH overfit" verdicts isi se the, aur V2 mei ye default hai.

Research only. Not investment advice.

Worked numerical example

Maan lo 100 bars hain, label horizon 6 bars. Plain k-fold 5 splits:

  • Split 1: train bars [0-79], test [80-99]
  • Leak: bar 80 ka label bars 80-86 dekhta hai. Bar 79 (train mei) bhi bars 79-85 dekhta hai. Overlap! Bar 79 ko bar 80-85 ka partial future pata hai.

Purged version:

  • Split 1: test [80-99], embargo = 2 bars (6×0.02≈0.12→1-2)
  • Purge bars [78-101] from train (test start 80 minus embargo 2 = 78, tak 101)
  • Train ab [0-77] only. Bar 79/80 leak removed.

Result: test pe model ne jo bhi seekha, wo future nahi dekh paya. Honest score.

Comparison table

Method Leak risk Use case
k-fold (shuffled) HIGH Never for time series
TimeSeriesSplit MEDIUM Fast baseline only
Purged+Embargo LOW Production research
Walk-forward LOW Final validation

How I integrated in training 

from sklearn.model_selection import KFold  # NOT used
splits = purged_embargo_split(len(X), n_splits=5, embargo_frac=0.02)
for train_idx, test_idx in splits:
    X_tr, X_te = X[train_idx], X[test_idx]
    y_tr, y_te = y[train_idx], y[test_idx]
    model.fit(X_tr, y_tr)
    gap = train_auc - test_auc
    if gap > 0.15:
        print("OVERFIT — block promotion")
Enter fullscreen mode Exit fullscreen mode

Epistemic note

Meri synthetic test mei AUC 0.70-0.74 aaya. Real NIFTY data pe shayad kam aaye (market noise). Par gap < 0.10 raha — matlab model generalize kar raha hai, memorize nahi.

Why retail traders blow up

Wo k-fold use karte hain, 90% accuracy dekhte hain, paid course bechte hain. Live mei 50% se kam. Purged CV unhe pehle hi bata deta: "tera model leak kar raha hai, real accuracy 55% hai."

My V1 autopsy

V1 ke 3 models mei mean_overfit_gap 0.40 tha kuch runs mei. Matlb train 0.85, test 0.45. Purged CV + shallow grid (depth 2-3) ne gap < 0.10 la diya V2 mei. Sirf architecture change, data same tha.

Action plan for you

  1. Kisi bhi time-series model se pehle purge+embargo lagao.
  2. Optuna tabhi tune karo jab 30+ decisive rows hon.
  3. 3-way split banaye rakho.
  4. Overfit gap > 0.15 pe promote band.

Research only. Not investment advice.

Extended FAQ

Q: Embargo fraction 0.02 kyun? Label horizon ÷ total bars ka rough fraction. 6 bars / 300 bars ≈ 0.02.
Q: Walk-forward vs purged CV? Purged CV ek split strategy hai, walk-forward sequential retraining. Dono use karo — purged CV har walk-forward window ke andar.
Q: Survival (AFT) labels pe bhi lagu? Haan, censor horizon same purge logic.
Q: Multi-asset? Har symbol alag purge karo, mat mix karo (cross-symbol leak).

The bigger lesson

ML mei sabse dangerous "looks too good" hai. Agar train/test dono 0.90+, doubt karo. Purged CV doubt ko proof mei convert karta hai. Meri V2 isliye survive karti hai kyunki har head pe gate hai.

Closing

Overfitting trading models ka silent killer hai. Plain validation use karke tum khud ko dhoka de rahe ho. Purged + embargoed CV 4-line fix hai jo V1 ke saare "HIGH overfit" verdicts ko rok deta hai. Lagao, warna live mei pachtayoge.

Research only. Not investment advice.

Code: full purged CV with overfit gate 

import numpy as np

def purged_embargo_split(n, n_splits=5, embargo_frac=0.02):
    idx = np.arange(n)
    fold = np.array_split(idx, n_splits)
    out = []
    for i in range(n_splits):
        test = fold[i]
        emb = int(len(test) * embargo_frac)
        lo, hi = max(0, test[0]-emb), min(n, test[-1]+emb+1)
        m = np.ones(n, bool); m[lo:hi] = False
        out.append((idx[m], test))
    return out

def train_with_gate(X, y, model_cls):
    gaps = []
    for tr, te in purged_embargo_split(len(X)):
        m = model_cls().fit(X[tr], y[tr])
        tr_auc = m.score(X[tr], y[tr])
        te_auc = m.score(X[te], y[te])
        gaps.append(tr_auc - te_auc)
    mean_gap = np.mean(gaps)
    return "BLOCK" if mean_gap > 0.15 else "PROMOTE", mean_gap
Enter fullscreen mode Exit fullscreen mode

Ye exact pattern maine V2 mei use kiya. mean_gap > 0.15 → block. Simple, automatic, non-negotiable.

Research only. Not investment advice.

Summary

Purged cross-validation trading models ko overfit hone se bachata hai by removing label-overlapping rows from training. Embargo + purge + 3-way split + overfit gate = honest backtest. Bina iske tumhara 90% accuracy dream live mei 50% ban jata hai. Lagao pehle, trading capital baad mei.

Agar tumhare paas koi model hai jo backtest mei acha hai par live mei fail ho raha, sabse pehle validation check karo. 90% chance hai ki leak hai. Purged CV lagao, sachai samne aa jayegi.

Research only. Not investment advice. SEBI compliance separate topic.

Next article mei hum dekhenge Options-Buyer V2 architecture — kaise ek model ki jagah multiple shallow heads use karke CE/PE confusion solve ki.

More From Shakti Tiwari

Top comments (0)