When a Winning Backtest Goes Negative: Modeling the Execution-Time Gap

Here is a result that trips up almost everyone who builds a fast strategy: the backtest is clearly profitable, you go live, and the same logic loses money. No bug. No overfitting in the usual sense. The strategy is fine. What killed it is the gap between how a backtest fills orders and how the real world does.

I build execution software at BJF Trading Group, and this "execution-time gap" is the single most common reason a fast edge fails to survive contact with production. Let us model it so you can see the edge cross zero.

The setup

Take a simple latency-style edge: you act on a price move a few milliseconds before a slower quote catches up, capturing a tiny expected gain per trade. In a perfect-fill backtest you get that gain every time. In production, two things eat it:

1. Execution latency between your signal and your fill, during which the edge decays.
2. Asymmetric slippage on the fills you do get.

A minimal model

import numpy as np

rng = np.random.default_rng(0)

def simulate(edge_pips=0.8, decay_per_ms=0.15, latency_ms=0,
             slip_mean=0.0, slip_std=0.3, n=200_000):
    """
    edge_pips: raw edge captured at zero latency, perfect fill
    decay_per_ms: how fast the edge decays while you wait to fill
    latency_ms: execution delay
    slip_mean: mean slippage (negative = hostile), slip_std: noise
    """
    captured = edge_pips - decay_per_ms * latency_ms
    slippage = rng.normal(slip_mean, slip_std, size=n)
    pnl = captured + slippage
    return pnl.mean(), (pnl > 0).mean()

# perfect-fill backtest
mean, winrate = simulate(latency_ms=0, slip_mean=0.0)
print(f"backtest   : mean={mean:+.3f} pips  winrate={winrate:.1%}")

# realistic production
mean, winrate = simulate(latency_ms=4, slip_mean=-0.15)
print(f"production : mean={mean:+.3f} pips  winrate={winrate:.1%}")

Typical output:

backtest   : mean=+0.800 pips  winrate=99.6%
production : mean=+0.050 pips  winrate=56.3%

The backtest looks unbeatable. Add four milliseconds of latency and a small hostile slippage bias and the same strategy is barely above water, one broker policy change away from negative.

Find the latency where the edge dies

The useful thing is not a single number, it is the break-even latency. Sweep it.

for lat in range(0, 9):
    mean, winrate = simulate(latency_ms=lat, slip_mean=-0.15)
    flag = "  <-- negative" if mean < 0 else ""
    print(f"latency={lat}ms  mean={mean:+.3f} pips{flag}")

latency=0ms  mean=+0.650 pips
latency=1ms  mean=+0.500 pips
latency=2ms  mean=+0.350 pips
latency=3ms  mean=+0.200 pips
latency=4ms  mean=+0.050 pips
latency=5ms  mean=-0.100 pips  <-- negative
...

Now you know something concrete: this edge survives up to roughly a 4 ms execution-time budget and dies beyond it. That single figure tells you whether a given broker and VPS setup can host the strategy at all, before you risk a cent.

Why perfect-fill backtests lie

A backtester that assumes zero latency and zero slippage is not modeling your strategy. It is modeling a strategy that does not exist. The fix is not a better idea, it is a more honest simulation: real tick data, a configurable execution-latency budget, and variable spread applied per tick. An edge that stays positive under a realistic budget is worth deploying. One that only works at zero latency never existed.

I wrote up the empirical version of this, with the mathematical framework and validation, here: Quantifying the execution-time gap (full paper with DOI on Zenodo). The tool we build for backtesting on real ticks with modeled latency is the SharpTrader Optimizer.

Next: leaving execution for a moment to compute the net-APR of a swap-arbitrage setup, where the trap is different but the discipline is the same.

---

I develop arbitrage and execution software at BJF Trading Group.

Comments

[Arhan Canli]

I re-ran the arithmetic and it holds: with the mean edge at 0.8 − 0.15·latency − 0.15, break-even is 4.33 ms, so 4 ms printing +0.05 and 5 ms printing −0.10 is what the model should give. The win rate also checks (Φ(0.05/0.3) ≈ 57%, close to your 56.3%).

What I'd stress is the two assumptions that make the cliff look so clean: decay is linear in milliseconds, and slippage is independent of latency. In practice slippage std grows with the same delay (you are filled in a moved market), so the loss side is fatter than N(−0.15, 0.3) and the break-even sits earlier than 4.33 ms. A one-line test: make slip_std = 0.3 + 0.05·latency_ms and see how far the break-even and the win rate move. Per-trade costs shift it too: every 0.1 pip of commission takes about 0.67 ms off the budget.

Did you fit the 0.15 pips/ms decay from tick data, or is it a placeholder for the sweep?