A line disappeared from my todo file.
Not corrupted. Not moved to the archive where completed items go. Just absent — present in one version, missing in the next, with nothing in between reporting a failure.
I run several AI coding agents in parallel, one per repository, and they share a single plain-text todo file as their cross-repo coordination point. I've written before about why plain text beat a SaaS board for this. This post is about the failure mode I hadn't accounted for.
The shape of the bug
Every writer to that file — a CLI, a terminal UI, and the agents themselves — does the same thing: read the whole file, change one line in memory, write the whole file back.
Three processes doing full-file read-modify-write on one path, with no locking anywhere.
That's a lost update, and it's textbook. If session A reads the file, session B reads it, A writes, then B writes, B's copy never contained A's change and now the file doesn't either. No error surfaces because nothing went wrong at the filesystem level. Both writes succeeded. One of them just described a world that no longer existed by the time it landed.
I had built a coordination mechanism whose entire job was to be shared, and left out the only part that makes sharing safe.
The part that was harder than the fix
Here's what I actually want to pass on, because the concurrency bug is the boring half.
When I noticed the file looked wrong, I did the obvious thing: diff it against the last backup. Which produced a wall of differences, because between those two points several agents had legitimately edited many lines — rewording items, compressing finished work, updating deadlines. Almost every line had changed somehow.
So "what's different" was useless. Nearly everything was different. The question I actually needed to answer was narrower: is there anything that exists in the old version and has no counterpart in the new one?
That's a set-difference question, not a line-diff question. And answering it meant picking a distinctive word from each candidate line — a repo name, an error string, a ticket number — and searching for that word across the current file. Most of the "missing" lines turned up immediately in reworded form. One didn't. That one was really gone.
A line diff tells you what changed. It doesn't tell you what's missing.
If your recovery instinct is diff old new, it will work beautifully on a file nobody edits and drown you on a file that several writers touch all day.
What I changed, and what I haven't
The honest status: the analysis is done, the fix isn't shipped. The candidates are file locking around every writer, or a modification-time check before each write that aborts if the file moved underneath you, or simply backing up more than once a day — because right now that daily snapshot is the only restore point I have, which is itself a finding.
I'm not going to pretend I've closed this. What I did do immediately was change my recovery procedure, since that cost nothing and would have saved the hour I spent chasing phantom deletions.
The same bug, one layer down
What makes this worth writing up rather than filing away is that I hit the identical structure the same week, in a product, in a database.
One of my apps had just gained a scheduler that sends payment reminders on a timer. Multiple workers, one queue of things to send — so exactly one worker must own a given run. The obvious implementation is: check whether the lock row is free, and if it is, claim it. Two lines of code, and a window between them where another worker can do the same check and reach the same conclusion. Same lost update, dressed as an invoice going out twice.
SQLite was one of the supported backends and it has no advisory locks, so the design couldn't lean on a database primitive. That constraint turned out to be a gift, because it forced the version that's actually testable: a plain table where acquiring is a single conditional UPDATE, and a first-time claim is a single INSERT on the primary key. No check-then-act window, because there's no gap between checking and acting — the database's own atomicity does the arbitration. Releasing verifies a holder token, so a worker can't release someone else's lock. A TTL reclaims locks from workers that died holding them. Six tests, verified against three database engines.
The todo file and the scheduler are the same bug at different altitudes. The database one got a rigorous fix because it was going to touch customers. The file one is still open because it only bites me.
I notice that's a real ordering, and also that it's how the file ended up unprotected in the first place.
Takeaways
If more than one process writes a whole file, you have a lost-update bug, whether or not you've seen it yet. Agent sessions count as processes. They're just processes that write convincing commit messages.
Make claiming a resource one statement, not two. Any "check, then act" pair has a gap in it, and the gap is where two workers agree they both won.
When you're recovering a multi-writer file, don't ask what changed — ask what has no counterpart. Pick a distinctive token from each old line and search for it in the new one. Reworded is not deleted, and a diff can't tell them apart.
The lock table, the single-statement claim, and its six tests are here: nene-clear#406. The todo file still has no lock.
What's the shared file in your setup that nobody has put a lock on?
── Hideyuki Mori (Ayane International) 🔗 hideyuki-mori.com
Top comments (0)