Anti-Overfitting for Retail Backtests: Few Rules, Fewer Trades, More Skepticism

Why the Fifth Rule Is Usually Noise

Every discretionary backtest dies the same death. You start with a simple idea — buy the pullback, exit on a failed retest. It returns something mediocre. So you add a filter: skip trades when volatility is too high. The equity curve improves. You add another: only take longs when the trend filter agrees. Better. One more: require volume confirmation. Now the curve looks great, and you have five conditions, each of which was chosen because it made the past look better.

That is the entire failure mode in one sentence. Each added rule is a new degree of freedom, and every degree of freedom is a chance to fit noise rather than signal. A retail backtest with ten boolean conditions and a parameter grid is not a strategy; it is a compression algorithm for historical data. The compression is excellent. The generalization is not.

The defensive doctrine here is short enough to memorize:

• Few rules over many. Every added condition must survive its own audit.
• Fewer trades with clear invalidation over many marginal ones. A trade you cannot invalidate is not a trade, it is a hope.
• Distrust any edge that only appears after tuning. If it needs the search to exist, the search created it.
• Re-test on unseen data before believing anything. Including your own improvements.

Below is how to apply that in code and process.

The Rule That "Improved" the Backtest by Rejecting 52 Trades

A concrete case. The base strategy took ~400 trades over several years and had a modest expectancy. Adding a filter — "skip entries when the 20-day realized volatility is above its 80th percentile" — removed 52 trades and lifted the profit factor from 1.18 to 1.41. Tempting. Ship it.

But that filter was not proposed in advance. It was proposed after looking at the losing trades, noticing they clustered in high-volatility regimes, and encoding that observation as a rule. That is a hypothesis generated from the same data used to test it. The 52 rejected trades are not out-of-sample evidence; they are the training set for the filter.

The fix is an A/B audit of the rule itself, treated as a separate hypothesis with its own test:

import numpy as np
import pandas as pd

def ab_audit(trades: pd.DataFrame, filter_mask: pd.Series,
             n_perm: int = 5000, seed: int = 7) -> dict:
    """
    trades: one row per trade with columns ['pnl', 'entry_time'].
    filter_mask: True where the filter would REJECT the trade.
    Question: is the rejected subset's mean pnl meaningfully worse than
    the kept subset, beyond what random rejection of the same size gives?
    """
    rng = np.random.default_rng(seed)
    pnl = trades["pnl"].to_numpy()
    rejected = filter_mask.to_numpy()
    k = rejected.sum()

    observed_gap = pnl[~rejected].mean() - pnl[rejected].mean()

    gaps = np.empty(n_perm)
    for i in range(n_perm):
        idx = rng.choice(len(pnl), size=k, replace=False)
        mask = np.zeros(len(pnl), dtype=bool)
        mask[idx] = True
        gaps[i] = pnl[~mask].mean() - pnl[mask].mean()

    p_value = (gaps >= observed_gap).mean()
    return {
        "n_trades": len(pnl),
        "n_rejected": int(k),
        "observed_gap": round(observed_gap, 4),
        "p_value": round(p_value, 4),
    }

If the p-value is not small — and with 52 trades out of 400, it usually is not — the filter is not evidence of a regime effect. It is a random subset that happened to contain losers. The honest outcome is to drop the rule and keep the simpler system.

There is a second, cheaper audit: does the rule survive on a holdout period it never saw? Split by time, not randomly. Random splits leak regime information across the boundary; time splits mimic what you will actually face.

def time_split_ab(trades: pd.DataFrame, filter_mask: pd.Series,
                  split_frac: float = 0.6) -> pd.DataFrame:
    trades = trades.sort_values("entry_time").reset_index(drop=True)
    filter_mask = filter_mask.loc[trades.index]
    cut = int(len(trades) * split_frac)

    rows = []
    for name, sl in [("in_sample", slice(0, cut)), ("holdout", slice(cut, None))]:
        sub = trades.iloc[sl]
        rej = filter_mask.iloc[sl]
        rows.append({
            "period": name,
            "pf_all": profit_factor(sub["pnl"]),
            "pf_kept": profit_factor(sub.loc[~rej, "pnl"]),
            "n_rejected": int(rej.sum()),
        })
    return pd.DataFrame(rows)


def profit_factor(pnl: pd.Series) -> float:
    wins = pnl[pnl > 0].sum()
    losses = -pnl[pnl < 0].sum()
    return round(wins / losses, 3) if losses > 0 else float("inf")

A rule that lifts profit factor in-sample but does nothing in the holdout is a rule you invented, not one the market gave you.

The Multiple-Comparisons Trap in Parameter Sweeps

Parameter sweeps are where retail backtests quietly become data mining. Run 12 variants of a stop-loss, 8 of a take-profit, 5 of a trend filter — that is 480 configurations. The best one will look good even if every configuration is pure noise. The maximum of 480 random draws is not a discovery; it is the expected maximum of 480 random draws.

You can quantify this. Under the null hypothesis of no edge, the best in-sample Sharpe across N independent trials has an expected value roughly on the order of sqrt(2 * ln(N)) times the standard error of a single trial. That is why "I tested 500 combos and found one with Sharpe 2.1" is not impressive.

Two practical defenses:

1. Correct the significance threshold. If you test N configurations and want a family-wise error rate of α, require each individual result to beat a Bonferroni-style threshold of α/N. Crude, but it kills most false discoveries outright.

def sweep_report(results: pd.DataFrame, alpha: float = 0.05) -> pd.DataFrame:
    """
    results: columns ['config_id', 'sharpe', 'n_trades', 'p_value']
    p_value: from a permutation test of that config's expectancy vs zero.
    """
    n = len(results)
    threshold = alpha / n
    out = results.copy()
    out["bonferroni_threshold"] = threshold
    out["survives"] = out["p_value"] < threshold
    return out.sort_values("p_value")

2. Count your trials honestly. Every variant you ran and discarded is a trial. If you cannot remember how many you ran, your effective N is larger than you think, and your best result is weaker than it looks. Keep a log.

import json, datetime

def log_trial(path: str, config: dict, metrics: dict) -> None:
    with open(path, "a") as f:
        f.write(json.dumps({
            "ts": datetime.datetime.utcnow().isoformat(),
            "config": config,
            "metrics": metrics,
        }) + "\n")

The log is uncomfortable to keep precisely because it makes the search visible. That is the point.

Honest Metrics

Most retail backtests report the wrong numbers. Total return and win rate are the worst offenders — win rate in particular rewards strategies that take many small wins and one catastrophic loss. Report these instead:

Expectancy net of fees

Per-trade expectancy after realistic costs, including slippage. If your edge disappears at 2x your assumed fees, it was never an edge.

def expectancy(pnl: pd.Series, fees_per_trade: float) -> float:
    net = pnl - fees_per_trade
    return round(net.mean(), 4)

Profit factor

Gross wins over gross losses. Above ~1.3 net of costs is worth a second look; below 1.1 is noise territory. But profit factor alone hides path dependence — pair it with drawdown.

Monthly consistency

Bucket returns by calendar month. A strategy that makes all its money in three months out of forty-eight is a regime bet, not a system.

def monthly_consistency(returns: pd.Series) -> pd.DataFrame:
    monthly = (1 + returns).resample("ME").prod() - 1
    return pd.DataFrame({
        "month": monthly.index.strftime("%Y-%m"),
        "return": monthly.round(4),
        "positive": monthly > 0,
    })

Look at the fraction of positive months and the worst month. If 80% of the total return comes from the top 10% of months, you are not running a strategy; you are waiting for a handful of lucky regimes and paying fees the rest of the time.

The Person Who Built It Cannot Approve It

This is the cultural rule, and it matters more than any code above. The person who tuned the parameters has a vested interest in the curve. They have seen every variant. They know which filter "works." That knowledge is contamination, and it does not wash off.

The fix is procedural, not technical: the backtester must never be the approver. Someone else — a partner, a friend, a second account with a written checklist — reviews the out-of-sample report without seeing the parameter search. They get the final config, the holdout results, and the trial count. Nothing else. If the strategy only convinces the person who built it, it does not convince anyone.

A minimal approval checklist:

• Rule count is small and each rule has an A/B audit with a p-value.
• The trial log exists and the best result survives a multiple-comparisons correction.
• Expectancy is positive after 2x assumed fees.
• Profit factor and monthly consistency hold on the holdout period.
• A second person, who did not tune it, signed off.

If any line fails, the strategy goes back to the simplest version that passed. Usually that is the version with two rules and a lot of skipped trades — which is exactly the version most people discard on the way to a prettier curve.

More notes like this ship every week on this site.

---

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-11-06 08:46:43

Short

Pair	Signal	Price	Take Profit	Stop Loss	R/R
MUU	庄家做空	$13.57	$12.593	$14.045	1:2.1
KOMA	短平快做空	$0.0128	$0.0124	$0.0131	1:1.2

2 picks selected. Scanner runs every 15 minutes.