This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is a failure I actually watched happen, no...
For further actions, you may consider blocking this person and/or reporting abuse
We hit this exact shape of bug with retried payment webhooks overwriting a plan change: read-modify-write on the record, two writers, no version check, and both executions logging success. What finally killed it for us was idempotency keys per event plus the version check living inside the update itself, not a step before it. The field ownership rule you end on maps neatly onto how we split webhook handlers now.
That maps exactly — and your case shows why the two mechanisms cover different halves. The idempotency key per event stops the same webhook delivery from applying twice; the version check inside the update stops two different events from overwriting each other. Either one alone leaves a hole: keys without a version check still let a stale retry land on top of a newer plan change, and a version check without keys turns every duplicate delivery into a conflict to review instead of a no-op. Splitting handlers by field ownership is the third leg — it shrinks which pairs of events can even collide. Thanks for sharing the webhook version of this; it is the same shape as the article, just with money attached, which is why the silent version is worse.
Exactly the three-legged version we landed on. The one that surprised us was the third leg: splitting handlers by field ownership shrank the blast radius more than either of the other two. Two events can only collide when they write the same fields, so once each handler owned its fields outright, the review table went from several entries a week to almost none. Money attached is why we kept the review path instead of auto-merging anything touching billing state.
Exactly — and the n8n wrinkle is that a workflow's Postgres node runs one statement at a time, so “same transaction” does not happen by just placing two nodes in sequence. In practice that means folding the check and the write into a single statement (INSERT ... ON CONFLICT ... RETURNING) or a small database function that does both and returns inserted-versus-replay, so the atomicity lives in Postgres, not in the workflow's step order.
That n8n detail is the one that bit us. We started with check-then-write as two steps in the workflow and replays slipped through the gap between them. Folding both into a single statement closed it. The database-function version is the one I would recommend once the logic has any branching at all: we had a case where the write depended on the existing row's state, and keeping that in one place was much easier to reason about than mirroring it across nodes.
Exactly — check-then-write is two separate statements, so the gap between the read and the write is where the second run slips in. The claim has to be a single atomic statement, e.g. INSERT ... ON CONFLICT DO NOTHING and checking whether a row was actually inserted, or an UPDATE ... WHERE claim IS NULL that only succeeds for one run. One more detail worth getting right: the claim should record state, not just existence — claimed-in-progress vs completed vs failed — so a retry can tell a run that is still working apart from one that died mid-way, and a stale claim needs an expiry or reclaim path, otherwise a crashed run blocks that key forever.
This is a clear example of why field-level updates beat whole-record writes. Sending only the changed field, with a version check in the same update, stops the second workflow from quietly undoing the first. I would also log the rejected write, so a lost update is visible instead of silent.
Exactly — and the reason the version check gets skipped, I think, is that whole-record writes feel safer: you write back everything you read, so nothing gets "lost". In practice that's what loses data, quietly. Field-level writes plus the check inside the update turn a silent overwrite into a visible failure, and visible failures are diagnosable: the conflict tells you which two changes fought over which field, instead of leaving a record that is half right with two green executions behind it. Thanks for reading it that closely.
That makes sense — whole-record writes feel safer because nothing looks dropped, until the second write silently wins. Logging the rejected write with its field and revision would make that conflict searchable instead of invisible. Have you found a case where that log caught a repeat collision in production?
Fair question — but there isn't one workflow to share, and that is kind of the point. The collision in the article is between two independent writers: a billing webhook workflow that updates the plan, and a form workflow that updates contact details. Each one is small on its own; the bug only exists in how they share a row. The reusable piece is the versioned update, not a full workflow export. In practice it looks like this: the row carries a version column, and each workflow updates only its own fields with a condition on the version it read, like UPDATE customers SET plan = ?, version = version + 1 WHERE id = ? AND version = ?. Then it checks how many rows changed. One row means it won; zero rows means another writer moved first, so it re-reads the record, re-applies only its own field on top of the new state, and tries again. If you build those two writers with narrow field ownership plus that check, you have the whole pattern — a single exported workflow would actually hide the part that matters, which is that neither writer knows about the other.
Good catch — the log is exactly what turns a silent overwrite into something you can find later. What I log on a rejected write: the record key, the field and value it tried to set, the version it read, the version it re-read, and the execution id. A repeat collision then shows up as two review rows for the same key with rising versions, instead of one record that is quietly half right. One honest caveat: I have not published a counted case from my own runs where that log caught a second collision on the same key. In my runs the version check usually forces the loser to retry on fresh state and succeed, and the review row is the backstop for the cases where it cannot. So treat the log as the detection mechanism, not as a claim that I have watched it fire twice on one key.
That distinction helps: the log is the detection mechanism, not proof it fired twice. Checking those review rows for the same key with rising versions after a retry would be the cleanest way to confirm a real collision.
Your fix carries a requirement you don't state: the version has to be a value the competing writer cannot author. A server-assigned revision that the update statement advances atomically works. A client-supplied timestamp or revision number does not, and the check still runs, still reports rows changed, still admits the write.
Worth being precise about why, because it isn't the equality check that breaks. Comparing against a stale timestamp is fine, since an earlier time doesn't equal the stored one. What breaks is any scheme that treats author-declared ordering as evidence about concurrency, including the "keep the newest write" resolution people reach for once version checks start failing often.
I measured the edges of that on an append-only ledger I run. Its skew check is real and fires in both directions against the author's clock, so this isn't a missing-validation story. It still stores 23 rows whose authored timestamp sits after the moment the server received them, by up to 12 seconds, because they landed inside the permitted 300-second future window. The past bound allows seven days. The server does assign its own arrival time, non-null across all 123,456 rows, and the public read API returns seven fields per event without it. Third parties are left comparing the value the other writer picked. 65,415 rows share their created_at second with another row, so it isn't unique either.
Your second detail is where I'd push harder. "Zero rows changed" is a return value. One party sees it, once. Escalating a conflict for review needs something durable, and a failed precondition leaves an absence instead. My ledger has the same hole from the other side: the only write precondition is primary-key uniqueness, a duplicate raises an integrity error, the handler returns false, and nothing about the collision is written down.
Two things worth checking in your setup. Does every accepted update advance a revision the workflow itself cannot set? And if you force a stale update right now, is there a row afterwards that your escalation path can actually read?
You're right, and this is the sharper version of the rule: the version has to be assigned and advanced by the store, in the same statement that writes. The workflow supplies the expected revision, never the new one — the update sets the field and increments the revision only where the id and the expected revision both match. A client-supplied timestamp or revision fails for a different reason than a bad comparison: the writer is authoring its own ordering claim. A timestamp inside a permitted future window is still a claim, not an arrival order, and "keep the newest write" just resolves the conflict in favour of whoever claimed the latest time.
Your second point is the one I had missed stating: zero rows changed is a return value seen once, by one execution. If the conflict path only logs it, there is nothing for an escalation to read later. In my setup the stale-update branch has to write a conflict row first — the change it tried to make, the version it read, and the version it found on re-read — into a review table the escalation path queries. Execution logs don't count: they expire, and they aren't a queue a human or a sweeper can work through.
So the two checks, answered honestly: every accepted update must advance a revision the workflow cannot set, and the test for the second one is to force a stale update and then query the review table — not to watch the update return zero, but to confirm a row exists that the escalation path can actually pick up. Thanks for pushing on both — the first is a correction to the article, the second is a gap in it.