The number said "-50%." I'd been staring at trade logs long enough to have a rough feel for how far a leveraged ETF actually needs to fall before a real -50% drawdown trigger should fire, and this one didn't look right. When I actually checked the math behind that specific row, the trigger had fired at roughly -2.9% below the average cost basis -- nowhere near the -50% the log claimed. That one suspicious row turned into a week of finding three separate, unrelated bugs in a backtesting engine I'd built with Claude Code, each one wrong in a completely different way.
The project, briefly
Outside of this blog, I've been using Claude Code to build and iterate on a backtesting simulator for a rules-based, dollar-cost-averaging-with-leverage trading strategy on leveraged ETFs -- buy more as price drops in defined stages, sell in defined stages as price recovers, all governed by explicit percentage thresholds and cooldown timers. None of the specifics matter for this post. What matters is that the simulator has to replay years of daily price data and decide, on every single day, whether a threshold condition is true. That's a lot of surface area for a rule to be implemented slightly differently than it was specified, and unlike an API call failing, a wrong threshold doesn't throw an exception -- it just quietly produces a plausible-looking wrong number.
Bug one: the trigger price formula didn't match the trigger rule
The rule, as written in my own spec, was simple: a buy-the-dip trigger at "-50% below average cost" should fire when price crosses average_cost * 0.5. What the code actually computed used a different reference point entirely, one that happened to produce trigger prices only a few percent below cost instead of half:
# spec: trigger fires 50% below average cost
trigger_price = average_cost * 0.5
# what the code actually computed -- a fixed step down from cost,
# which lands only a few percent below it, not half
trigger_price = average_cost - fixed_step_amountThe formula was internally consistent -- it always fired at the same (wrong) distance -- which is exactly what made it hard to catch by eyeballing outputs. It looked like a real trigger doing real work. It just wasn't the trigger the spec described. Every backtest run before this fix used trigger points that didn't match the strategy's own documentation, which meant weeks of "results" needed to be thrown out and rerun once the formula was corrected.
Bug two: a reactivation timer that was secretly a different mechanism
The second bug took a specific, hard-to-describe symptom to surface: a sell trigger reactivating suspiciously soon after its cooldown period should have still been active. The cooldown was supposed to be pure time-based -- N trading days pass, the trigger rearms, unconditionally. What was actually implemented was closer to hysteresis: the trigger would only rearm if price dropped back below the threshold line within that window; otherwise it stayed permanently disabled for the rest of that cycle. Those two behaviors look identical in the common case and diverge only in specific price paths, which is exactly why it survived several earlier rounds of eyeballing. Fixing it changed the simulated results substantially -- selling more often, it turns out, meaningfully changes how much of a multi-year uptrend a strategy captures, so the "wrong" and "right" versions of this rule didn't just differ by a rounding error, they told different stories about which strategy variant was better.
Bug three: a structural assumption that only broke on specific days
The third bug was the quietest of the three. The simulator originally computed everything on split-adjusted historical prices -- the standard convention, where past prices are rescaled so that today's share count lines up with history. Re-verifying against raw, unadjusted historical prices (to check whether the strategy's "only buy whole shares within a daily budget" rule would have actually been achievable in real life) surfaced a case where a stock split day wasn't handled at all: share counts weren't scaled up on the split date, so portfolio value briefly appeared to be cut in half on paper. It's the kind of bug that only exists on a handful of specific calendar days across sixteen years of data, invisible unless you specifically go looking at those days.
What ties these together
None of these three bugs would show up as a stack trace. They'd show up, if you were unlucky, as a strategy decision made on subtly wrong numbers -- the kind of bug that's genuinely dangerous specifically because the output still looks like a normal number in a normal range. The common thread across all three fixes was the same: stop trusting that a formula which "runs without error" is the formula that was actually specified, and go check a handful of individual data points by hand against the written rule instead of trusting the aggregate output. That's a much slower way to debug than reading an exception traceback, but for anything that computes silently-wrong numbers instead of throwing, it's the only way that actually works.
Top comments (1)
"Spot on! 🎯 The most dangerous bugs are the ones that don't crash your code; they just whisper plausible lies in the logs. When working with financial math or simulation engines, a syntax-valid formula or a green test suite means nothing if the underlying assumptions diverge from the spec. Manual spot-checking against raw baseline data is truly the only antidote to silent financial drift. Brilliant writeup!"