Three-Layer Validation Before You Trust a Backtest

Why One Good Backtest Means Nothing

A single equity curve is a sample of size one. Optimize hard enough over enough combinations and something will look great by luck alone. The question is not "does this rule make money in the backtest?" but "does this rule make money for reasons that survive contact with slightly different assumptions?"

The gate I use before any rule reaches the live scanner has three layers: a parameter sweep, a monthly consistency check, and a component ablation. Each layer answers a different question, and each has a hard pass/fail threshold. A rule has to clear all three. This post walks through the mechanics, then through a real rejection: a MACD divergence entry that passed layer 1 and died on layers 2 and 3.

Layer 1: Parameter Sweep

The question it answers

Is the edge a plateau or a spike? If performance only exists at fast=12, slow=26, signal=9, you have found noise, not a rule. Neighboring parameter values should produce comparable results, and the chosen point should sit near the center of a profitable region, not on its edge.

The metrics

For each parameter combination in the grid, record:

• Trade count — anything under ~50 is too thin to say much.
• Profit factor (PF) — gross wins / gross losses.
• Expectancy per trade — in R units (risk multiples), so results are comparable across parameter sets.
• Max drawdown — in R, not percent, so it doesn't depend on position sizing.
• Plateau score — the fraction of neighboring combinations (within one step in each dimension) that stay above the PF threshold.

The thresholds

• Median PF across the sweep ≥ 1.3
• At least 70% of combinations with ≥ 50 trades have PF > 1.0
• Plateau score at the chosen point ≥ 0.6
• Chosen point is not on the grid boundary

The code

import itertools
import numpy as np
import pandas as pd

def sweep(params_grid, run_backtest, min_trades=50):
    rows = []
    for combo in itertools.product(*params_grid.values()):
        params = dict(zip(params_grid.keys(), combo))
        res = run_backtest(**params)
        if res["trades"] < min_trades:
            continue
        rows.append({**params, **res})
    return pd.DataFrame(rows)

def plateau_score(df, param_names, chosen, pf_col="pf", threshold=1.0):
    """Fraction of 1-step neighbors of `chosen` with PF above threshold."""
    neighbors = []
    for name in param_names:
        values = sorted(df[name].unique())
        idx = values.index(chosen[name])
        for offset in (-1, 1):
            j = idx + offset
            if 0 <= j < len(values):
                mask = pd.Series(True, index=df.index)
                for n in param_names:
                    target = chosen[n] if n != name else values[j]
                    mask &= df[n] == target
                if mask.any():
                    neighbors.append(df.loc[mask, pf_col].iloc[0])
    if not neighbors:
        return 0.0
    return float(np.mean(np.array(neighbors) > threshold))

grid = {
    "fast":   [8, 10, 12, 14, 16],
    "slow":   [20, 24, 26, 30, 34],
    "signal": [7, 9, 11],
}
results = sweep(grid, run_macd_divergence_backtest)

chosen = {"fast": 12, "slow": 26, "signal": 9}
score = plateau_score(results, list(grid.keys()), chosen)

print(f"median PF: {results['pf'].median():.2f}")
print(f"share PF>1: {(results['pf'] > 1.0).mean():.0%}")
print(f"plateau score at chosen point: {score:.2f}")

A plateau score of 0.6 means most one-step neighbors still clear PF 1.0. That is a low bar, deliberately. If a rule can't survive a one-step nudge, it has no business being tested further.

Layer 2: Monthly Consistency

The question it answers

Is the annual result distributed across time, or is it one monster month doing all the work? A strategy with one +40% month and eleven flat months is not a strategy. It's a lottery ticket that already paid out, and the next ticket is unlikely to pay again.

The metrics

Bucket trades by calendar month and compute:

• Share of profitable months — count of months with positive PnL / total months.
• Best-month contribution — best month's PnL / total PnL.
• Top-3-month contribution — same, for the three best months.
• Median monthly return and worst month in R.

The thresholds

• ≥ 55% of months positive
• Best month ≤ 30% of total PnL
• Top 3 months ≤ 55% of total PnL
• Worst month no worse than −3R (scaled to your per-trade risk)

The best-month cap is the important one. It is the metric that catches curve-fit strategies that happen to overlap a single volatility event.

The code

def monthly_consistency(trades: pd.DataFrame, risk_per_trade=1.0):
    """trades: columns ['exit_time', 'pnl_r']"""
    monthly = (trades
               .assign(month=trades["exit_time"].dt.to_period("M"))
               .groupby("month")["pnl_r"]
               .sum()
               .sort_index())
    total = monthly.sum()
    if total <= 0:
        return {"pass": False, "reason": "non-positive total PnL"}

    best = monthly.max()
    top3 = monthly.nlargest(3).sum()
    stats = {
        "months": len(monthly),
        "positive_share": (monthly > 0).mean(),
        "best_share": best / total,
        "top3_share": top3 / total,
        "median_month": monthly.median(),
        "worst_month": monthly.min(),
    }
    stats["pass"] = (
        stats["months"] >= 24
        and stats["positive_share"] >= 0.55
        and stats["best_share"] <= 0.30
        and stats["top3_share"] <= 0.55
        and stats["worst_month"] >= -3 * risk_per_trade
    )
    return stats

The months >= 24 requirement matters. With twelve months, "positive share" is dominated by a handful of observations and the test is close to meaningless.

Layer 3: Robustness Ablation

The question it answers

Does each component of the rule contribute, and does the rule degrade gracefully when a component is removed? A robust rule loses some edge when you strip a filter. A fragile rule collapses to zero, or worse, only works with every component present — a sign the components were fitted to each other.

The metrics

For each component, run the backtest with that component removed (or replaced by a neutral version) and record:

• PF delta — component PF minus full-rule PF.
• Expectancy delta in R.
• Retention — component expectancy / full-rule expectancy.

The thresholds

• Removing any single component leaves expectancy ≥ 60% of the full rule (graceful degradation).
• Removing the core component (the entry trigger itself, not a filter) drops expectancy below 50% and PF below 1.1 (the core actually matters).
• No component removal improves expectancy by more than 15% (if it does, that component is hurting and should be cut).

The third criterion is the one people skip. It catches filters that were added to patch a specific losing streak in the sample and do nothing out of sample.

The code

def ablation(full_result, variants: dict, core="divergence"):
    """variants: name -> backtest result dict with 'pf' and 'expectancy_r'"""
    base_pf = full_result["pf"]
    base_exp = full_result["expectancy_r"]
    rows = []
    for name, res in variants.items():
        rows.append({
            "component": name,
            "pf": res["pf"],
            "expectancy_r": res["expectancy_r"],
            "retention": res["expectancy_r"] / base_exp if base_exp else 0.0,
            "pf_delta": res["pf"] - base_pf,
        })
    df = pd.DataFrame(rows)

    checks = {
        "graceful": (df.loc[df["component"] != core, "retention"] >= 0.60).all(),
        "core_matters": df.loc[df["component"] == core, "retention"].lt(0.50).all(),
        "no_harmful": (df["retention"] <= 1.15).all(),
    }
    return df, checks

The Case Study: A MACD Divergence Entry

The rule: enter long when price makes a lower low but MACD histogram makes a higher low (bullish divergence), confirmed by a close above the signal line. Exit on a fixed 2R target or a time stop of 20 bars. Filters included a trend filter (price above the 200-EMA), a volume filter (volume above its 20-bar average), and a volatility floor (ATR above its 20-bar median).

Layer 1: passed

The sweep over fast ∈ [8,16], slow ∈ [20,34], signal ∈ [7,11] produced 75 combinations, 68 with enough trades. Median PF was 1.41, 78% of combinations had PF above 1.0, and the plateau score at (12, 26, 9) was 0.71. The chosen point sat comfortably inside the grid. Nothing suspicious yet.

Layer 2: failed

Over 31 months, only 48% of months were positive. Worse, the best month contributed 44% of total PnL, and the top three months contributed 71%. The best month coincided with a broad volatility expansion — the rule caught one regime and did nothing in the other thirty. Median monthly PnL was slightly negative.

Verdict: fail on positive_share (0.48 < 0.55), best_share (0.44 > 0.30), and top3_share (0.71 > 0.55). This alone is enough to reject the rule.

Layer 3: failed

The ablation was run anyway, out of curiosity:

Removed	PF	Expectancy (R)	Retention
Trend filter	1.38	0.11	0.73
Volume filter	1.52	0.17	1.13
Volatility floor	1.09	0.06	0.40
Divergence (core)	0.94	−0.01	−0.07

Two problems. The volatility floor is doing most of the work — removing it drops retention to 0.40, well below the 0.60 floor. That means the "divergence" rule is largely a low-volatility regime filter wearing a costume. And removing the volume filter improves expectancy by 13%, which is under the 15% harm threshold but close enough to be a warning sign: the filter was probably added to patch a specific stretch of the sample.

The core divergence component does matter (removal sends expectancy negative), but that doesn't save the rule. A rule whose edge lives mostly in a regime filter, and whose only positive months come from one volatility event, is not a rule. It's a description of one period.

The rejection

Three-layer result: pass, fail, fail. The rule went into the rejected list with the metrics recorded, so future me doesn't re-test it in six months and get excited all over again. That bookkeeping is half the value of the process.

Practical Notes

• Run the layers in order. Layer 1 is cheap and filters out most junk. Layer 2 is cheaper than layer 3 and catches the most common failure mode. Don't run ablations on rules that can't pass a monthly check.
• Log every metric, including failures. A rejection is only useful if it's documented. Keep the sweep DataFrame and the ablation table next to the rule definition.
• Thresholds are conventions, not laws. The numbers above are tuned to my trade frequency and risk unit. If your rules trade twenty times a year, months >= 24 is too strict; if they trade daily, best_share <= 0.30 is probably too loose. Pick thresholds once, write them down, and don't move them after seeing a result you like.
• The ablation is the most informative layer. It tells you why a rule works, not just whether it did. When a rule fails layer 3, you often learn something reusable about the market — in this case, that low-volatility regimes are where this class of divergence entry lives, which is a hypothesis worth testing on its own.

More notes like this ship every week on this site.