I removed every delete from my exactly-once guard. Then the bugs moved.

Jigon Yoo — Mon, 31 Aug 2026 01:16:36 +0000

Your monitoring will not flag a duplicate charge. Every one of them is a 200.

The refund succeeded. It succeeded again four hundred milliseconds later on a
second worker, and once more after a lost response made the client retry. Three
successes, three green spans, three log lines that say ok. The customer's
statement is the only place the incident exists.

Guardrails are usually built around may this happen? — permissions, scopes,
approval gates. This is the other question, and almost nothing asks it: did I
already?

I spent a week building a small answer to it, and then had it torn apart four
times. This is what each round moved, and why the last three rounds found
nothing where I was looking.

The easy half

The mechanism is one line and it is genuinely correct:

fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)

O_EXCL is a compare-and-swap on existence. Exactly one process creates the
file; everyone else gets FileExistsError. No daemon, no Redis, no dependency.

Forty OS processes — real processes, started before any of them are joined, not
threads — racing the same charge:

40 concurrent processes, identical call | executed 40 | executed 1 (always)

Threads would have passed on the GIL alone and proved nothing about production.

That number is real and it reproduces. It is also the easy half, and I spent
two review rounds learning how easy.

And then someone dies

A claim is a promise that someone is working. When the holder dies — OOM,
deploy, SIGKILL — the promise outlives the worker, and the file becomes a
tombstone that blocks that call forever.

So you need a TTL, and a way to take the claim back. And here is the thing
nobody tells you:

POSIX has no "delete this specific inode." unlink takes a path. Between
the moment you decide a claim is stale and the moment you remove it, that path
can point at something else — including a fresh claim that a faster process just
won.

Round one: unlink

My first reclaim was three lines and looked obviously right:

if time.time() - rec.get("at", time.time()) > ttl:
    os.unlink(path)      # holder died; race again

A reads the stale record. A unlinks. A wins O_EXCL and starts charging. B —
which read the same stale bytes a microsecond earlier — unlinks A's fresh
claim, wins, and charges too.

My test suite was green. It probed the expiry path with a single caller and the
concurrency path from an empty directory, so the bug lived exactly in the gap
between two passing tests.

The repo still carries that variant, and forces the interleaving instead of
hoping for it:

bug variants: claim-unlink 20/20

Twenty out of twenty. Not "flaky under load" — deterministic, in about a second.

Round two: I added a lock to guard a copy of the bug

The fix serialized takeover with a sentinel file and a POSIX flock. Tests
green again.

A second reviewer, who had not seen the first, replaced flock with a no-op —
which is what you get on NFS with local_lock, or after a lock file is rotated
out from under you. Detection went to zero. The lock had never been doing
the work; O_EXCL on the sentinel was.

And the branch the lock existed to protect was still unlink-then-recreate, on
a path, on a timestamp that can be stale:

bug variants: sentinel-unlink 20/20

I had added a lock to guard a copy of the bug I was fixing.

The same review found the lock was scoped to the claim directory, not to the
key — so four unrelated charges, racing nothing at all, blocked each other and
most of them were refused. The README calls a missed charge the worse half of the trade. I had
shipped it to calls that were never racing.

Round three: stop deleting

Two failures, one shape. Both were code that removed a path.

So I removed the removal. Claims are now generations — <sig>.gen0.claim,
<sig>.gen1.claim, and so on. An expired claim is never reclaimed. It stays
where it is, and a new generation opens beside it. There is no "take the claim
back" operation to get wrong, because there is no such operation.

The evidence for that is a command, not a paragraph:

$ grep -cE "unlink|os\.replace|rename|flock" once_guard.py
0

And the same forced interleaving that caught both earlier variants:

fixed generation: 0/20 double executions
16 workers x 40 trials: 0 double executions

The reclaim path was now provably clean. I thought I was done.

Round four: the bugs moved

The next review found eleven more defects. Not one of them was in the writing
side. Every single one was in the reading side — deciding whether a claim is
old, whether a half-written result is finished, whose clock to believe.

The one worth your time is claim age. A claim records when it was made. That
record is written by the claim, so it can be wrong — a worker whose clock
jumped backward writes a timestamp from the past and its live claim looks
ancient. Meanwhile the filesystem records an mtime, which that worker cannot
forge.

return min(record_age, mtime_age)

The younger of the two. Not the more accurate one — the safer one. If the
record lies and I believe it, I steal a live worker's claim and charge twice. If
mtime bounds the age and I am wrong, a dead worker's claim stays locked a while
longer and a refund is late. I picked the direction I would rather be wrong in,
and that is the whole design decision.

Writing the claim took one syscall. Deciding whether a claim is currently
valid took eleven fixes.

Round five: my tests believed the same lie

With that rule in the library, two tests started failing intermittently. They
seeded an expired claim like this:

claim.write_text('{"holder":"dead","at":0}')

at=0 is 1970. The file's mtime is now. And the library believes the younger
of the two — because I made it.

freshly written file, at=0:  0.0000s   → not expired (ttl 0.01)
50ms later:                  0.0503s   → expired
mtime backdated to 1970:     1788115004s

So a worker that reached the age check within the TTL saw a fresh claim and
correctly refused. The library was doing exactly what I designed. The test was
believing the claim's own story about itself.

The fix was one os.utime in a shared helper. Remove that single line again and
nine tests across four files fail — which is how I know the helper is load
bearing and not decoration.

Three rounds, three layers, one mistake: trusting what a claim says about
itself without checking it against something the claim cannot forge.

Then I stopped looking for defects

Five rounds all asked the same question — what breaks? At some point that
becomes the failure mode. So the last pass asked a different one: does the
documentation say what the code does?

I pulled every claim out of the README — every behavioral assertion, every
number, every command, every support-range statement — numbered them, and
checked each one against the code or against a command I actually ran. One
hundred and fifty-eight claims. One hundred and thirty matched. Seven were
wrong. Three I could not check in this environment. Eighteen had no supporting
evidence in the repository at all.

That third category is the interesting one. Those sentences were not false. They
just had nothing behind them — the kind of line a stranger reads and asks "how
do you know that?" and you have no answer.

Two of mine, both embarrassing:

I published latency figures and called them reproducible. There was no script in the repo that produced them. They are gone now.
I described five sibling repos as "each one file." One click disproves it.

No amount of adversarial defect review would ever have caught either. The code
was right. The document was writing checks the repo could not cash.

What it still does not do

The demo has a sixth row, and it exists to be embarrassing:

the side effect outlives its TTL | off: executed 2 | on: executed 2

Two, and two. The guard does not help. Once a real side effect has outlived its
TTL, the old owner and its legitimate replacement have both genuinely executed,
and no bookkeeping turns two real charges into one.

There is a control row too:

the same four, with note wrongly inside the key | off: executed 4 | on: executed 4

Put a value the model rewrites into the idempotency key and every retry gets a
new identity. $126.00 extra in both columns. The guard does not fix a
misconfigured key, and I report that row separately from the headline instead of
averaging it in.

And what is untested is not a limitation I discovered, it is one I am telling
you about: power loss is not covered — the result file is fsynced, the parent
directory is not. NFS is not covered; the warning in the README about O_EXCL
on older servers is judgement, not measurement. I have marked those as
judgement in the README rather than letting them sit next to the measured
numbers looking equally solid.

What to check in your own code

If you have an exactly-once guard anywhere — a Redis SETNX, a unique index, a
lock table — the interesting question is not how you take the lock.

What happens when the holder dies? If the answer is a TTL, you have a
reclaim path, and that is where the bugs are.

Does your guard ask the state how old it is, or check it against something the
state cannot forge? Redis TTLs, an updated_at column, a heartbeat_at in a
lock table — every one of those is a value the record wrote about itself. For
each place you read one, write down in a single sentence which way you fail if
that value is a lie. If the answer is "double execution," that is your next
incident.

Is your race test's sensitivity a function of your CPU count? Mine was. The
same suite on a 4-core machine and a 2-core one gave different answers, and the
statistical version passed on both while the bug was live. Hooking the read and
forcing the loser to lose finds it 20 times out of 20 in about a second.

Have you ever counted the claims in your own README? How many of them can
you prove with one command?

Take it

git clone https://github.com/jigonyoo/once-guard
cd once-guard && python3 demo.py

Fifteen seconds, six scenarios, the failures priced in dollars — including the
two rows where the guard does nothing. 71 tests, no dependencies, MIT, CI on
Python 3.8 and 3.11.

The most useful thing you can do with it is point the demo at your own guard and
see which rows you lose.

DEV Community: Jigon Yoo