A/B Testing Trading Rules: The 52 Rejected Trades That Changed My Mind

The Question a Filter Should Answer

A pullback filter sits in front of an entry signal and decides which setups are allowed through. The usual debate goes like this: the strict variant looks cleaner on the equity curve, the relaxed variant trades more, and someone argues the extra trades are "noise." That argument is almost never tested. It is asserted from the shape of the curve.

The right question is narrower: what happens to the trades that exist only because of the relaxed rule? If that incremental set has positive expectancy, the strict filter is not cleaning up noise. It is deleting the strategy's core. In the case I worked through, the strict rule rejected 52 trades that the relaxed rule took. Those 52 trades had a 48.1% win rate, +0.131U expectancy per trade, and a profit factor of 1.50. The "worse" rule was rejecting the profitable part.

This post is about the method, not the specific rule: run both variants over the same window, diff the trade lists, and evaluate the rejected set on its own statistics.

Why the Headline Curve Can't Settle It

Two variants produce two equity curves. Comparing them tells you which variant made more money over one sample. It does not tell you whether the filter is doing what you think it is doing, because the curves aggregate everything: shared trades, rejected trades, and the timing of both.

A strict filter can improve a curve for three different reasons:

1. It removes genuinely bad trades (the intended effect).
2. It removes trades that were fine but happened to lose in this sample (overfitting to noise).
3. It changes position sizing or trade sequencing in a way that masks the removal of good trades.

Only the first is a reason to keep it. The diff isolates the trades that differ, and their statistics tell you which of the three you are looking at.

The diff is the unit of analysis

Think of the two variants as generating two sets of trades over an identical window. The shared trades are irrelevant to the filter decision — both rules took them, so they contribute equally to both curves. The decision lives entirely in the symmetric difference:

• Rejected-only: trades the strict rule skipped and the relaxed rule took.
• Accepted-only: trades the relaxed rule skipped and the strict rule took (this happens when the rules are not strictly nested).

If the rejected-only set has positive expectancy with a reasonable sample size, the filter is destroying value. If it has strongly negative expectancy, the filter is earning its keep. If the set is tiny, you have no evidence either way and should not pretend otherwise.

Building the Two Trade Lists

The prerequisite is that both variants log trades in the same schema, over the same data, with the same execution assumptions. Any difference in slippage model or fill logic between runs will contaminate the diff.

from dataclasses import dataclass
from datetime import datetime

@dataclass(frozen=True)
class Trade:
    symbol: str
    entry_time: datetime
    exit_time: datetime
    direction: str          # "long" / "short"
    entry_price: float
    exit_price: float
    size: float             # units
    pnl: float              # realized, in account currency
    rule_variant: str       # "strict" / "relaxed"

A trade key needs to identify the same opportunity across runs. Entry timestamp plus symbol plus direction is usually enough; if a strategy can re-enter the same symbol on the same bar, add an entry-sequence index.

def trade_key(t: Trade) -> tuple:
    return (t.symbol, t.entry_time, t.direction)

Run both variants over the same window and the same instrument universe:

strict_trades = run_backtest(strategy, data, params=STRICT_PARAMS)
relaxed_trades = run_backtest(strategy, data, params=RELAXED_PARAMS)

strict_by_key = {trade_key(t): t for t in strict_trades}
relaxed_by_key = {trade_key(t): t for t in relaxed_trades}

rejected_only = [t for k, t in relaxed_by_key.items() if k not in strict_by_key]
accepted_only = [t for k, t in strict_by_key.items() if k not in relaxed_by_key]
shared = [t for k, t in strict_by_key.items() if k in relaxed_by_key]

Sanity check before analyzing anything: len(shared) + len(rejected_only) should equal the relaxed run's trade count, and the PnL of the shared set should be identical in both runs. If it is not, the two runs are not comparable and the diff means nothing.

Statistics for the Incremental Set

The rejected-only set is just a list of trades. Evaluate it exactly as you would a standalone strategy, but report the full picture, not one number.

import statistics as st

def summarize(trades: list[Trade]) -> dict:
    n = len(trades)
    if n == 0:
        return {"n": 0}

    pnls = [t.pnl for t in trades]
    wins = [p for p in pnls if p > 0]
    losses = [p for p in pnls if p < 0]

    gross_win = sum(wins)
    gross_loss = -sum(losses)

    return {
        "n": n,
        "win_rate": len(wins) / n,
        "avg_win": st.mean(wins) if wins else 0.0,
        "avg_loss": st.mean(losses) if losses else 0.0,
        "expectancy": st.mean(pnls),
        "profit_factor": gross_win / gross_loss if gross_loss > 0 else float("inf"),
        "total_pnl": sum(pnls),
    }

For the 52 rejected trades this returns something like:

n=52  win_rate=0.481  avg_win=+0.71U  avg_loss=-0.41U
expectancy=+0.131U  profit_factor=1.50  total_pnl=+6.81U

Read it as a whole. A 48.1% win rate sounds unremarkable until you pair it with the payoff ratio: average win is roughly 1.7x average loss, so sub-50% accuracy still yields positive expectancy. Profit factor 1.50 over 52 trades is not spectacular, but it is clearly the wrong side of zero for a filter that is supposed to be removing bad trades. A filter that rejects a set with PF 1.50 is rejecting trades you would have wanted.

Interpret the shape, not just the sign

Three patterns are worth distinguishing when you look at the rejected-only set:

• Negative expectancy with a normal win rate: the filter is removing losers. Keep it.
• Near-zero expectancy with a large sample: the filter is removing noise. Keep it only if it materially reduces drawdown or costs; otherwise it is not earning its complexity.
• Positive expectancy: the filter is removing profit. The strict variant needs a different justification, or it should go.

The 52-trade case is the third pattern, and the win rate alone would have hidden it. This is why the diff has to be evaluated with expectancy and profit factor, not just hit rate.

Guarding Against One Lucky Trade

The most common way a diff analysis lies is concentration. Fifty-two trades with +0.131U expectancy can be twelve good trades and forty flat ones, or one trade that made the entire set. Before trusting the result, check:

Per-trade contribution

def concentration_report(trades: list[Trade], top_k: int = 5) -> dict:
    pnls = sorted((t.pnl for t in trades), reverse=True)
    total = sum(pnls)
    top = sum(pnls[:top_k])

    return {
        "total_pnl": total,
        "top_k_pnl": top,
        "top_k_share": top / total if total != 0 else float("inf"),
        "expectancy_ex_top_k": (total - top) / (len(pnls) - top_k) if len(pnls) > top_k else None,
    }

If the top five trades account for most of the set's PnL and expectancy excluding them collapses toward zero, the incremental set is not evidence of a better rule. It is evidence of a handful of outcomes.

Per-symbol breakdown

The diff can also be dominated by one instrument. Group the rejected-only trades by symbol and compare each group's count and expectancy against its share of the full sample.

from collections import defaultdict

def by_symbol(trades: list[Trade]) -> dict[str, dict]:
    groups = defaultdict(list)
    for t in trades:
        groups[t.symbol].append(t)
    return {sym: summarize(ts) for sym, ts in groups.items()}

If one symbol supplies most of the rejected trades and all of the positive expectancy, the honest conclusion is "the relaxed rule helps on this symbol in this window," not "the relaxed rule is better." A filter decision made on a single-symbol diff is a single-symbol decision.

Sample size and stability

Fifty-two trades is enough to see a signal but not enough to be confident about its magnitude. Split the rejected-only set in half by time and check whether expectancy holds in both halves. If the first half carries everything and the second half is flat or negative, the incremental set is not a stable property of the rule.

def split_half(trades: list[Trade]) -> tuple[list[Trade], list[Trade]]:
    ordered = sorted(trades, key=lambda t: t.entry_time)
    mid = len(ordered) // 2
    return ordered[:mid], ordered[mid:]

What the Method Actually Decides

The diff analysis does not tell you which rule is "correct." It tells you what each rule is doing. In this case, the strict pullback filter was not removing noise — it was removing a set of trades with positive expectancy, a 1.50 profit factor, and a payoff profile that made a sub-50% win rate profitable. The relaxed rule was not sloppier; the strict rule was over-filtered.

The general procedure generalizes beyond pullback filters. Any time you compare two entry conditions, two exit rules, or two position-sizing schemes, the shared trades cancel and the decision lives in the diff. Evaluate the incremental set with count, win rate, average win and loss, expectancy, and profit factor, then check whether that set is concentrated in a few trades or a single symbol. If it is, you have a hypothesis, not a result.

The headline curve is a summary of everything at once. The diff is the only place the filter's actual effect is visible.

More notes like this ship every week on this site.