I shipped a safety feature that passed every test I had and could never have fired
in production. Not "rarely fired". Could not, structurally, under any market
condition, ever.
It took me two days to find out, and the only reason I found out at all is that I
stopped trusting green checkmarks.
Here's what happened, and the four habits I picked up from it. I build an automated
trading system, so my bugs settle in money rather than in tickets. That tends to
sharpen the lessons.
The feature
The system takes one position per day. I wanted a daily loss limit: if the account
drops by more than a set percentage today, stop trading until tomorrow.
Simple enough. Parse the threshold, compare it against the current drawdown, block
the entry if we're past it, log an alert.
I wrote it. Compilation clean. The test suite went green. The regression harness
confirmed the results matched the previous version byte for byte. I shipped it.
The part where nothing happened
Two days later I ran what I call a dose test. The idea is simple: set the threshold
so absurdly tight that the feature must fire. If it doesn't, something's wrong
with the feature, not with the market.
I set the daily loss limit to a value the account crosses within minutes of opening.
The results came back byte for byte identical to the run without the feature.
Not "mostly similar". Identical. The feature had done nothing at all, and it had
been doing nothing in production for two days.
Why every test was green
The check sat in the right place logically and in the wrong place temporally.
It ran before the entry block: "are we past the daily loss? then don't enter."
That reads correctly. But the system takes one position per day. By the time the
check ran, today's position had already been opened. The guard was standing in
front of a door nobody was going to walk through again until tomorrow.
Every test I'd written asked the same question in different clothes: given a
drawdown value, does the function return the correct decision? And it did. Every
time. Correctly.
Not one test asked the other question: does this function get called at a moment
when its answer can still change anything?
Those two things fail independently, and I'd only been testing one of them.
Habit 1: a test that can't go red isn't a test
This is the one that changed how I work.
A test proves nothing unless you've seen it fail. Until then you haven't tested
your code, you've tested the happy path of your test.
So now every meaningful check gets a dose: I deliberately break the thing under
test and require the suite to go red. Set the threshold absurdly tight and the
feature must trigger. Feed the parser a malformed input and it must refuse. Point
the monitor at a dead process and it must alert.
If the suite stays green while the system is visibly broken, the test was
decoration.
This has caught more of my mistakes than any coverage report ever did. Coverage
tells you a line was executed. It doesn't tell you that executing it proved
anything.
Habit 2: count, don't claim
While I was auditing tests, I found this in my own harness:
Top comments (2)
A guard that returns the right answer at a moment when no answer can change anything is the bug I'd never have found by reading the code. 'Does this function get called when its result still has consequences' really is a separate test from 'does it compute the right value', and byte-identical results with the feature enabled is a much faster way to prove it than any assertion.
The absurdly-tight threshold is a nice trick because it uses the market as an oracle instead of the test suite. Did you end up keeping that as a standing job, or only as a one-off investigation? My worry is that a forced-fire run needs a real order on the other side to be meaningful, and that's the part that rots.
Standing job, but only in the tester, and that distinction took me a while to
get right.
You're right that a forced-fire run against live execution rots. It depends on
market state I don't control, so it stops being repeatable within about a week.
I ran it live exactly once, on the day the feature shipped, and never again.
What I keep is the tester version. Set the threshold absurdly tight, run the
same period, and require the output to differ from the baseline. Byte-identical
output means the guard never executed. No order needed on the other side, fully
repeatable, and it runs as part of regression on every change that touches a
guard.
The rule I ended up with: regression proves I didn't break the baseline, the
dose proves the new thing is alive. Two different questions. I used to only ask
the first one.