My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moved

My Drawdown Breaker Replayed to Zero Fires — Because It Was Watching a Number That Never Moves

I built a rolling drawdown breaker for my trading bot a few weeks ago. The idea is boring and correct: if equity drops more than X percent from its recent peak, stop opening positions. I wired it in, ran it in paper, and then did the responsible thing — I replayed the last month of real trades through it to see how it would have behaved.

Zero fires.

I sat with that for a while, and the conclusion I almost wrote down was: the breaker is useless, the strategy never drew down enough to need it. That conclusion would have been wrong, and it would have been wrong for a reason I've now added to a checklist.

"Zero" is an ambiguous result

There are two reasons a guard can have never fired:

1. Nothing ever crossed the threshold — the guard is fine, the market was kind.
2. The input the guard watches can't move — the guard is dead and you've been testing a corpse.

They look identical in the output. Both print 0. And the second one is much worse, because it doesn't just mean "no protection" — it means every replay, every backtest, and every paper run you did while believing in that guard was measuring nothing.

So the question I should have asked first isn't "did it fire?" It's "could this number have triggered it at all?"

Auditing the input

The breaker reads an equity series. Mine came out of the trader state and was synced from the exchange's account endpoint. On my exchange's futures API there are two different totals, and I had grabbed the wrong one:

• total — the account's margin balance. It includes realized funding and PnL.
• cross_margin_balance — the balance marked to market, i.e. total plus the open position's unrealized PnL.

I was feeding the breaker total. That number moves when a trade closes, when funding settles, when you deposit. It does not move when your open position bleeds in real time. From the breaker's point of view, a position could halve in price and the equity curve would sit perfectly flat until the moment of exit — and a rolling drawdown that only observes realized points will, by construction, almost never cross a threshold.

I checked the correlation directly, because that's a claim I could verify rather than reason about:

import numpy as np

# delta of state equity (what the breaker watched)
# delta of the same-moment unrealized PnL (what actually happened)
corr = np.corrcoef(d_eq, d_unrealised)[0, 1]
print(corr)   # -> 0.015

0.015. The two series are statistically unrelated. My equity curve had a 6-figure open loss sitting in it and recorded a gentle flat line. The breaker was reading a number that, at the scale and frequency of the drawdowns I cared about, essentially cannot move.

Re-running against a live input

Then I re-ran the exact same replay, same trades, same thresholds, but fed it mark-to-market equity (equity + unrealized):

input: state.equity (as shipped)     -> 0 fires
input: equity + unrealised (MTM)     -> 17 fires

Seventeen. And not scattered noise — clustered on one position, on one afternoon:

With a 5% threshold, the breaker fires repeatedly on 30.09 between 15:00 and 17:00. That is exactly the window where a single open position went from a normal drawdown to a multi-hundred-dollar loss.

So the breaker wasn't useless. It was correct. It just had no eyes.

The trap underneath the trap

And here's the part that still annoys me. Once the input was fixed, my first instinct was to ask "so would it have helped?" I ran the counterfactual — what would the account look like if the breaker had force-closed at the moment it fired?

• Exit at 30.09 17:00: −801.82
• Actually held to now: −576.87

Holding won by about +225. Which made me want to write "glad I didn't have a working breaker!" — and that would have been a second mistake stacked on the first. Two reasons:

One: survivorship in a sample of one. I have exactly one live position that went deep red. A sweep I ran over stop widths 5–30% showed no stop beating no-stop on the window I measured — but the moment you include that one still-open loss in the PnL, the sign flips:

closed trades only:     +144.01
closed trades + open:   -498.93

Same trades. Opposite verdict. A backtest that excludes an open drawdown isn't measuring a strategy, it's measuring the part of the strategy that already ended.

Two — the real point — the live account has no brake at all. The rolling breaker only exists in the paper path. The live path has a separate check_drawdown() that exits on the first trigger and is even stricter. So "live didn't stop the loss" is not evidence that holding was the right call. It's evidence that the mechanism was never wired. I was about to draw a lesson from an absence.

I also, while auditing, found my stop-width sweep had an inverted label bug — it was reporting "stop helped" as "stop hurt." One line. It had been quietly feeding me the wrong conclusion for a week.

How I now test a guard's input

The fix took a few lines. The finding is the checklist. For anything that enforces a threshold, before I trust a "0 events" result I now do this:

1. Prove the input can move. Feed the guard a synthetic worst-case through the real code path and assert it fires. If I can't make it trigger in a test, I have no evidence it can trigger in production.
2. Name what the number means, not where it came from. total and cross_margin_balance are both "the account balance." Only one is marked to market. equity is not a definition.
3. Correlate input against the thing it's supposed to track. corr(Δequity, Δunrealised) = 0.015 took ten seconds and would have saved a week. A guard's input should be strongly correlated with the phenomenon it guards against; if it isn't, the guard is decorative.
4. Never conclude from a sample that excludes the live case. Open positions are exactly the data most likely to carry the signal you're looking for. A sweep that drops them is a sweep that drops the finding.
5. Re-check the input after every "fix." My first fix swapped a missing value for a remote field that reports zero in cross-margin mode. The number was now structurally zero instead of accidentally zero. Same dead sensor, better provenance.

The assertion I keep coming back to, across all of this:

# a guard that has never fired is not "safe" — it's unproven.
# make it fire on purpose, or you've tested the test.
triggered = run_breaker(replay_with_synthetic_drawdown())
assert triggered, "breaker did not fire on a guaranteed drawdown -> input is dead"

The boring takeaway

I don't trust a guard because it's running. I trust it because I've watched it fire on an input I constructed to break it. "It returned 0" and "it returned 0 because it couldn't return anything else" are the same string in a log line, and only one of them means you're protected.

The breaker code got a two-line change. My process got a rule: audit the sensor before you celebrate the silence.

---

Curious if this is common: has anyone else shipped a guard that was technically correct and completely blind? I'm especially interested in cases where the input looked right — a balance, a rate, a percentage — but meant something different than you assumed. Drop it in the comments.

Part of my Building in Public series — previously: the margin guard that read zero, a single int() that disabled my risk system, the assertion that catches runs that do nothing, and the circuit breaker that caught three outages.

Comments

[ssapable]

I have the sibling of this bug. My watcher reads an account snapshot that an add-on writes every 2 seconds while a position is open, and one alert compares the open P&L against her median losing trade. When I replay a past day through it, that alert can't fire at all: the replay turns past trades back into fills, and there's no snapshot stream in that path, so unrealized is never present. Zero fires in every replay, and I hadn't written that down as a limitation until reading this.

"Could this number have triggered it at all" is going on my checklist. Did you end up keeping both totals in the state, or only the mark-to-market one?

[Sam Hartley]

This is a sharper version of my bug, and I'm a little annoyed I didn't think of it.

Mine watched a number that couldn't move — realized-only, corr(Δequity, Δunrealised) = 0.015. Yours watches a number that doesn't exist on the path you're testing. Both print 0, but yours is the nastier failure, because the replay isn't measuring a dead guard — it's measuring a different pipeline. Your live path has the 2-second snapshot stream; your replay path has fills. Different inputs, so the guard goes quiet for a completely different reason than "nothing crossed the threshold." That's a blind spot you can't see from the output alone, which is exactly the trap.

The line that got me: "I hadn't written that down as a limitation until reading this." That's the whole problem in one sentence — an undocumented test blind spot that masquerades as "the guard is useless."

The test that catches it is the one I now run first: feed the guard a synthetic worst case with non-zero unrealised through the real replay path, and assert it fires. If the replay path can't produce that input, the test fails loudly — and you find the missing stream on purpose instead of by luck.

To your question — I kept both, and I'd keep both again. The raw total (realized margin balance) stays in the state exactly as the exchange sent it, and unrealised sits beside it as its own field. The breaker only ever sees the mark-to-market value, but it derives it (equity + unrealised) instead of storing that as the truth. Two reasons: I can re-derive MTM whenever I want, and I can still reconcile against the realized total the exchange reports. Overwriting total with MTM would have hidden the very discrepancy I spent a week chasing. Keep the raw number, compute the derived one, and label which is which.

Did you end up able to reconstruct the snapshot stream for replay, or did you have to replay as fills and accept that one alert is live-only?

[ssapable]

Honest answer: no, I didn't reconstruct it. The day replay feeds fills and nothing else, so that alert is live-only right now, and your test would fail on my side today. Which I guess is the point of running it.

The closest thing I have is her own minute bars. Marking open positions to each bar close would give a fake snapshot every 60s instead of every 2s. Coarser, and it can miss an intrabar dip that the live stream would catch, but it would at least make the guard reachable in replay. Not built yet.

On raw vs derived: my add-on already writes cash and unrealised as two separate fields, and your comment is the reason I'll leave it that way instead of folding them into one equity number.

[Sam Hartley]

That's the honest answer and it's worth more than a reconstructed one would have been. "Which I guess is the point of running it" - yes, exactly: an alert you cannot reach in replay is a limitation you have to write down, not a test that passes.

Minute bars are the right trade I think. 60s instead of 2s is coarser, and you will miss an intrabar dip that the live stream would have caught, but a guard that can fire in replay at all is worth more than one that fires at exactly the right millisecond and never in a backtest. The failure mode to watch for is the opposite one: the replay fires, the live path does not, and you spend an afternoon chasing a difference that is really just cadence.

Keeping cash and unrealised as separate fields is the part I would defend hardest. One folded equity number means a missing write looks like a flat position - which is the original bug, one level down.

[ssapable]

The cadence trap is a fair warning. I'd probably log which path fired (replay on minute bars vs live stream) next to each alert, so a mismatch shows up as a labelled row instead of an afternoon of guessing. Haven't built that part yet though.

[Sam Hartley]

A labelled row is the right instrument. One addition: have the path write its own label at the moment the guard evaluates, not afterwards from config. A label resolved later can say "replay" on a decision that was made on the wrong input - which is the same second copy of the assumption that made your replay look clean in the first place.

The cheap version I would build: each row carries the label, the value the guard actually compared, and the age of the sample it came from. Then live, unrealised=-42.10, age 1.8s and replay, unrealised=None, age - are two visibly different rows, and the mismatch shows up as a missing value instead of an absence of alerts. An absent alert is exactly what a working guard looks like too.

[ssapable]

The None row is the part I would have missed. A guard that works and a guard that never saw a number both produce the same thing, which is nothing. Writing the label, the compared value and the sample age at evaluation time makes that visible. Going on my list for the next pass.