I run an agent that wakes every four hours, does one thing, and writes its state back to
STATUS.md. That file is its entire memory across sessions — without it, every wake starts
from nothing.
Yesterday a write failed on a missing substring. I opened the file:
893,828 lines
49,414,452 bytes
49MB. It should be about 13KB.
Worse: it had been that way for three days, during which I read it, wrote to it, and
verified it daily. Every check passed.
The one character
The offending code replaces a section:
old = s[s.index('## Next') : s.index('### Index status')]
s = s.replace(old, new_section)
Looks fine. But what if ### Index status appears before ## Next in the file?
Then the second index returns a smaller number than the first, and a Python slice whose start
exceeds its stop doesn't raise. It returns an empty string.
So the line becomes:
s = s.replace('', new_section)
Which does something people rarely have cause to remember:
>>> 'abc'.replace('', '-')
'-a-b-c-'
replace('') inserts the replacement between every character.
A 13KB file with a 1KB section inserted this way becomes roughly 12.6MB. My agent then woke
every four hours and did the same thing again to the already-broken file. Three days later:
49MB.
That's not the interesting part
The bug is dumb and the story ends there. What's worth writing about is that I inspected that
file every day and the inspection kept saying fine.
Because replace('', x) deletes nothing. Every original character is still present. They're
just separated by thousands of copies of the same section.
So:
grep 'Last heartbeat' STATUS.md # found
sed -n '5,8p' STATUS.md # looks about right
grep -c 'published' STATUS.md # returns a number
Every check passed, because everything I was looking for really was there.
After each write I habitually sed the first few lines to confirm the update landed. Those
lines had updated. I never looked at the whole file, because I thought I knew what it looked
like.
This is the same failure as my last post
Last time I wrote that a local test passing proves nothing about production, and the conclusion
was: a difference between your test environment and the real one doesn't present as a
difference — it presents as a bug.
What lied to me this time wasn't the environment. It was the granularity I chose to check at.
I was checking whether content was present. This failure doesn't remove content — it dilutes
it. The metric I picked is structurally blind to this class of fault.
That's harder to defend against than an environment gap, because at least you know your test
environment differs from production. Check granularity is something you choose yourself, and
you'll naturally choose to check for failures you can imagine.
What I changed
1. Check the size after writing, not just the content.
wc -c STATUS.md
One command. File size tells you almost nothing about whether content is correct, and it is
extremely sensitive to whether structure is broken. 13KB becoming 7MB is instantly visible.
I missed it for three days because it never occurred to me to look.
Cheap, coarse, unlyable signals are badly undervalued. A precise check tells you "the thing
you asked about is fine." A coarse one tells you "something is off" — and only the second kind
catches the failure you didn't imagine.
2. Assert the ordering before slicing.
a, b = s.index(A), s.index(B)
assert 0 <= a < b, f'range inverted: {a} → {b}'
old = s[a:b]
assert old, 'empty slice — stop'
The second assert matters most. An empty first argument to replace is essentially always a
bug; no legitimate use needs that behaviour.
3. Assert the file's shape, not just its contents.
I now check that a section heading like ## Next appears exactly once. During the breakage
it appeared 12 times — an earlier and far clearer signal than size, and I simply wasn't looking
for it.
One last thing, and it's the part that stings.
This system's entire premise is autonomous operation. It checks its own state every four hours,
writes logs, verifies its output. And it destroyed its own core memory file for three days
without noticing.
It wasn't failing to check. It checked diligently — it just checked the places it expected to
break.
If you're building something like this, add one rule: periodically verify the things that
can't possibly be wrong. The parts that can break are already being watched. What bites you
is whatever you're confident doesn't need watching.
Top comments (1)
This is exactly why I like size and shape checks on state files, even when the content checks pass. A grep proves the marker survived. It says nothing about whether the file still has the structure you meant to preserve. I have been bitten by the same class of bug with generated ledgers where every row was valid and the file was still nonsense.