The habit of letting each test intercept notification-sending leaked into a real send through one newly added alert path — a postmortem on why blocking side effects has to be the test harness's global default
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
There's a regression test suite I run every time I change the code.
In the evening I fold in that day's changes, run all the related tests, and check that nothing broke. Routine work.
What happened
I kicked off an evening regression run and went to do something else — and then push notifications started arriving on my phone. Not one, but six in a row within a few minutes.
They looked exactly like the operational alerts that show up during real production. The warning said "some reference value is missing."
The problem: there was no reason for a real production round to be running at that hour. The only thing running was the tests.
In other words, these weren't alerts sent by the production system — the tests had fired real messages into the real notification channel.
What I thought at first
My first thought was that maybe a real round had somehow fired without my knowing.
So I checked the production records first. If a real decision had run at that time, it should have left a trace in the production journal — but there was no evaluation record at all in that time window.
The notification send ledger (the log of which message went out when), on the other hand, had those six entries plainly recorded. Their send timestamps lined up exactly with the regression run window.
Production was silent; only the send ledger had rung. That left one conclusion — the sender was the tests, not production.
The real cause
This codebase funnels notification sending through a single function. How the tests had been handling that function is the heart of this incident.
The long-standing habit was that each test intercepts (mocks) the send function on its own. A test that verifies an alert would swap the send function for a fake inside itself, and just check how many times the fake was called.
The problem is that this is "opt-in." Whether sending gets blocked depended on each test's goodwill.
The regression I ran that day included one newly added code path: a branch that sends a warning when some input value is empty.
The test exercising that path had no idea it was supposed to intercept the send function. It was a new alert, so the existing interception habit had never been applied to it.
So when the test ran that branch, the send function — which nobody had blocked — fired a message into the real notification channel. There was no default barrier between the test environment and the live channel.
There was a secondary cause too. That alert path fetched its reference value from an external API before sending the warning. It wasn't just the send that leaked — the test very nearly called the real external API as well.
How I fixed it
The fix itself was short. I put one mechanism at the test-harness level: "block notification sending by default."
Concretely, I added a fixture that applies automatically to every test (autouse) and replaces the send function with a do-nothing fake by default.
The existing tests that verify sending override it with their own fake inside themselves, so they're unaffected. All that changed is the default — even if nobody explicitly blocks it, a real send can never go out.
In the same fixture I also pinned the external reference-value lookup to return "no value" by default. Tests that genuinely exercise that lookup likewise cover it with their own patch.
From spotting the incident to putting in the send block took a few minutes; including the re-run, under an hour. The code is a handful of lines. What mattered wasn't the line count but flipping the direction of the default.
The generalizable lesson
Blocking side effects should be opt-out, not opt-in. Any I/O that touches the world outside the test — sending notifications, calling external APIs, charging payments, sending mail — will inevitably leak if each test is left to block it on its own.
The structural weakness of opt-in mocking is that a hole opens up every time a new code path is added. Even if every existing test intercepts sending, one new path whose author doesn't know the habit is enough. When safety depends on "everybody knows the rule," it breaks the moment new code that doesn't know the rule arrives.
So this kind of I/O belongs at the test-harness level as a default-deny, with only the handful of tests that actually verify sending explicitly lifting the block. That way a lapse errs on the safe side, instead of a lapse firing for real.
A piece of observability that can tell apart who caused something makes diagnosis fast. This time, the contrast — no trace in the production journal, records only in the send ledger — let me separate "did production send this or did the tests?" within minutes. When you record a side effect, record "who / in what context" caused it too; in a moment like this, that one line names the culprit.
"Most tests block it" is a different claim from "the harness blocks it." If you leave a safeguard to the habits of individual members, it collapses the moment a new member who doesn't share that habit joins. Laying the default on the safe side — that's the only way that doesn't rely on human memory.
Top comments (0)