Originally published on hexisteme notes.
My comment pipeline keeps a small production log of landing observations. Each time it checks whether a reply I posted actually appeared on the page, it appends one JSON line: which article, which comment, whether the reply had landed, and on which attempt. An earlier note, I Rejected the Same dev.to Reliability Fix on One Path and Shipped It on Another, tied a future review of the reply retry budget to these comment landing observations. They are meant to be evidence about real deliveries.
During a round of fixes to that pipeline, I found that the full offline test suites had been writing to it.
What happened
The watcher, autoreply and reply suites use Python's unittest. Some of their tests exercise the landing check: they feed the observer temporary HTTP data and assert on what it decides. The fake data was temporary. The output path was not. In those tests, the observer still wrote to the real landing-observations.jsonl.
The first full baseline run appended 30 synthetic rows, and that run reported 176 tests, OK. Later full runs, before the fixtures were isolated, appended more. The final total was 122 synthetic rows. An intermediate brief had put the number at 145. Reconciling the preserved rows by timestamp and target gave 122, and that is the number I trust.
Why passing told me nothing
A test passes when its assertions hold. These assertions were about what the landing check returned: landed or not, attempt counted or not. None of them asked what else the code had touched on the way. Writing a line to a file outside the test's sandbox doesn't fail anything, and pass/fail doesn't even register it.
That is the whole mechanism. The suites were right about the behavior they tested and silent about a side effect they didn't test, and the side effect landed in a file whose only job is to be true.
Cleaning up without making it worse
There are fast, destructive ways to clean a log like this: truncate it, delete every row from the time window the tests ran in, or copy an older backup over the live file. Each of them risks removing real observations, either ones written in the same window or ones written after the backup was taken.
The cleanup used exact attribution instead. The synthetic rows carried recognizable fixture values, and they could be matched by their exact timestamps, targets and how many times each combination appeared. Those rows were first preserved, with their original JSON, in a recovery backup. Then exactly those rows were removed from the log. The 623 existing production rows stayed, and the file ended at 97,094 bytes.
If the rows had not been attributable exactly, the right move would have been to leave them and record the contamination as a known limitation of the log, not to guess.
The fix: isolate first, guard second
The primary fix is isolation. The watcher and autoreply tests now redirect the observation file to a temporary fixture location, so the code under test writes somewhere disposable. The reply suite's landing tests mock the watcher instead of calling it. Python's tempfile module is the standard tool for the disposable location. The pattern is roughly this (conceptual; the names are illustrative, not the suite's):
class LandingCheckTests(unittest.TestCase):
def setUp(self):
tmp = tempfile.TemporaryDirectory()
self.addCleanup(tmp.cleanup)
patcher = mock.patch.object(watcher, "OBS_PATH",
Path(tmp.name) / "observations.jsonl")
patcher.start()
self.addCleanup(patcher.stop)
If you use pytest, its tmp_path fixture serves the same purpose. These suites are unittest, so that's not what they use.
The second part is a guard that would have caught this the first time. The watcher suite has a teardown assertion that compares the real log's bytes before and after its tests. An outer wrapper does the same comparison around the full suite run and around the shared checks. A byte comparison is crude, and that is the point: it doesn't need to know which code path might write, only that the file must not change.
A guard after the fact does not undo a write. It turns a silent side effect into a failing run. Isolation is what prevents the write.
Validation
After the fixes, all 223 watcher, autoreply and reply tests passed, and the production log was unchanged, byte for byte, at 97,094 bytes.
A shared offline harness runs checks across five components. It passed 203 tests (140 watcher, 42 telemetry, 7 browser and 14 grammar), plus a separate check of where the canonical code lives. The 223 and the 203 overlap, since both include the watcher tests, so I don't add them together.
Claude Code, Codex and agy each ran that shared checker from a scratch directory outside the project. The checker makes no model calls and posts nothing. Everything it runs is offline and synthetic, so it says nothing about how a real browser session behaves in production.
What I check now
The byte guard covers one file I now know about. It does not find other paths that tests might write to. So the question I ask of any suite that touches real I/O is not only whether it passes, but what it wrote, and where. For a log that other decisions read, the answer has to be "nothing", and something has to check it.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (0)