This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is a failure I actually watched happen, not a hypothetical.
Two workflows wrote to the same customer record on the same night. One updated the plan from a billing webhook. The other updated the contact details from a form submission. Both executions said success. The record that came out was neither update: the plan change was there, the contact change was gone, and nothing in either log suggested anything had been lost.
The mistake was assuming that two workflows touching the same row at different times is the same as two workflows touching it in sequence. It is not. Each workflow read the record, changed its own field, and wrote the whole record back. The second write overwrote the first, not because either workflow was wrong about its own field, but because neither workflow knew the other had been there in between. A read-modify-write cycle with no protection is a race, even when the two writers never run at the same second — they only have to overlap between the read and the write.
The fix is to stop writing whole records from workflows. A workflow should send a change, not a replacement: update this field, to this value, based on this version. If the version it read is no longer current, the update must fail instead of silently overwriting. That failure is not a bug in the workflow. It is the system catching a collision that would otherwise have passed as success.
Two details make this work, and both are easy to get wrong.
First, the version check has to live in the update itself, not in a node before it. "Read the version, check it is still the same, then write" is again two steps with a gap in between, and another workflow can slip into that gap. The check must be part of the same statement that writes: update where the version matches, and report how many rows changed. Zero rows changed means someone else moved first, and the workflow must re-read, re-apply its own change on top of the new state, and try again — or hand the conflict to a human if the two changes cannot both be true.
Second, retries must not replay a stale change. A workflow that failed on the version check and simply reruns from its first node will re-read, which is fine. A workflow that retries only the write step, with the old values still in memory, will fail the same way forever, or worse, succeed after a third workflow has moved the record again and land on top of work it never saw. The retry has to start from a fresh read, every time. Stale data in a retry is how a collision that was caught once gets lost a second time.
I also keep a rule for which fields a workflow is allowed to touch. Each workflow owns its fields, and it writes only those. The billing workflow never writes contact details, even if it read them. The form workflow never writes the plan, even if the form mentioned it. Narrow writes shrink the surface where two workflows can collide, and when a collision does happen, the version check tells me exactly which two changes fought over which field, instead of leaving me with a record that is quietly half right.
The rule I keep coming back to: a workflow that writes a whole record is claiming it knows the record's full state. It does not. It knows one field, at one moment, and the honest write says so.
Free workflows from this series (MIT): https://github.com/zhp910318/n8n-ai-automation-lab
The production templates in my store: https://payhip.com/haopengailab
Top comments (42)
We hit this exact shape of bug with retried payment webhooks overwriting a plan change: read-modify-write on the record, two writers, no version check, and both executions logging success. What finally killed it for us was idempotency keys per event plus the version check living inside the update itself, not a step before it. The field ownership rule you end on maps neatly onto how we split webhook handlers now.
Good question, and the honest answer is that it depends on how often the writers collide, not on which one sounds more robust. A simple retry with a fresh read is enough when collisions are rare: the loser re-reads, re-applies only its own field, and usually succeeds on the second try, with no extra infrastructure. It breaks down when contention is high, because every loser retries at once and you get a retry storm, or when re-applying is expensive or unsafe — for example if the change involves a side effect you cannot cleanly redo. A queue helps exactly there: serialize writers per record key so only one writer touches a row at a time, which gives you ordering and backpressure instead of contention. The cost is latency and one more moving part to operate, and it does not remove the version check — you still want it, because a manual edit or a second queue consumer can still race you. My rule of thumb: start with bounded retry from a fresh read (say three attempts), and move the hot keys to a per-key queue only when you see repeated version conflicts on the same rows. If a conflict survives both, it goes to a review table with the attempted change and both versions attached, never to a silent overwrite.
That rule of thumb maps to how we ended up handling it. Bounded retry from a fresh read was our starting point too, and it held up because duplicate deliveries were already deduped, so real collisions stayed rare. The part I would underline is the review table: once we accepted that some conflicts need a human instead of a cleverer automatic resolution, the whole design got simpler. We log the attempted change, both versions, and the event that fired, and that table has caught more real bugs than the retry path ever did.
Thanks — for a small team that shape holds up well in practice. Two details that matter once you wire it in: on a duplicate-key error, return the stored result for that key rather than just skipping, so a replay gets the same outcome the first run produced; and give the keys table a retention rule (for example, keep keys for the replay window you actually support, then archive), otherwise it grows without bound. No extra service, and the constraint keeps working if you later add a second consumer.
Both details are right, and the first one is the one most people skip. We learned it when a provider retried the same webhook for most of a day and got a different answer from us each time; returning the stored result made replays idempotent from the caller's side too. The retention rule is the quiet one: we keep keys for our supported replay window and archive the rest, defined once so the cleanup job and the write path can never drift apart.
Yes, this is the failure mode that survives code review because the guard looks correct at the send step. In n8n the key is usually lost earlier: any node that rebuilds the item — Set/Edit Fields, a Code node returning a new object, a Merge or aggregation — drops fields it was not told to carry. So treat the key as a first-class field that must be explicitly passed through every reshaping node, and make the send step fail closed: if the key is missing at send time, refuse to send instead of sending unguarded. A cheap regression test is to run the filtered-and-sorted path once and assert the key is still present at the send node, not just the happy path.
On your two points specifically: returning the stored result is what makes a retry safe from the caller's side — a fresh side effect behind a 200 looks identical to success, so the caller cannot tell a replay from a first run unless the response is the stored one. And defining the retention window once matters because the cleanup job and the write path drifting apart is exactly how a key gets deleted while a provider is still retrying. Size the window to the provider's longest retry horizon rather than to what feels tidy, and archive rather than delete, so a late replay is still recognizable instead of looking like a brand-new request.
The fail-closed send is the change I'd push for first — an unguarded send turns a dropped key into a duplicate side effect, which is the worst outcome of the three. And your regression test targets exactly the right path: the filtered-and-sorted branches are where keys die, because those nodes rebuild the item from scratch.
Sizing to the provider's longest retry horizon is the rule we landed on too — anything tidier is just optimism. Archiving instead of deleting paid off for us in debugging as well: when a late replay arrives you can pull the original stored response from the archive and confirm it was a replay rather than a new request.
Archiving is the part most teams skip and then regret. The rule I'd add: the replay lookup has to check the archive with the same key lookup as the live table, so a late replay returns the stored response instead of creating a new side effect. Retention then follows the horizon: keep archived keys at least as long as the provider's longest retry window plus margin, and treat expiry of the archive as a deliberate decision, not a cleanup default.
Agreed — fail-closed is the one I'd ship first too. The rule I use: a send step that can't find its idempotency key must refuse to send and park the item for inspection, never send unguarded. And the regression test should push an item through every filter and sort branch and assert the key is identical on exit — any node that rebuilds the item has to carry the key explicitly, or the branch fails the test.
Same key lookup on the archive is the rule I'd write down. We got bitten exactly there: an early version queried a different index and a late replay re-triggered a send. And treating archive expiry as a deliberate decision, not a cleanup default, is the right call. Ours is a manual retention setting per provider.
The rebuild-that-drops-the-key case is the one that always slips through review, because a rebuild usually looks correct. We started asserting key identity across every branch output after a sort node silently dropped one and a duplicate send went out. Painful way to learn it, but the regression test you describe is exactly how I'd catch it now.
The single shared key function is the fix I would keep. Live and archive lookups must call the same canonicalisation code path, not two derivations that happen to agree on most inputs — a different index or a re-derived variant is exactly how a late replay misses the archive and re-triggers a send. For retention, I would write the decision down per provider: retention must cover that provider's longest retry window plus margin, and the expiry job must never delete a key that is still inside any provider's window. Treating expiry as a manual, signed-off setting rather than a cleanup default is what stops a routine purge from becoming a duplicate send.
That is why I would assert key identity at every branch output, not only at the final output. A sort, filter, or map node can drop a field silently while the rest of the payload looks correct, so the check has to be: for each branch, output key equals input key exactly, byte for byte. The regression test is simple to write: feed one record with a known key through each branch, including a sort with duplicate and null keys, and fail the run if any branch output is missing or altered. And fail closed downstream: a branch output without a key goes to review, it never gets a freshly generated key and continues, because a new key would turn a lost identity into a duplicate send.
The byte-for-byte check at every branch output is the part I'd steal — a sort or map node dropping the key silently is exactly the kind of failure that looks fine until the duplicate send. And failing closed instead of minting a fresh key is the key line for me: a new key doesn't fix a lost identity, it just makes the duplicate send look legitimate.
This is the one I'd keep too. Two derivations that agree on most inputs are the quietest kind of wrong — they pass every check except the one that matters. And the per-provider retention point is the one I'd underline: we size to the provider's longest retry window plus margin, because the expiry job is the easiest place to quietly reintroduce duplicates.
Yes — and the anchor only holds if the key is stable across retries. A key derived from the source event survives a retry and a process kill; a key derived from an execution id or a timestamp does not, and then every downstream rule quietly starts inferring identity from context again, which is exactly the failure you described. The other half is retention: the key table needs an explicit retention rule, because if the key expires too early the anchor disappears and a late retry looks like a brand-new job. Keep the key, keep its lifetime deliberate, and the rest of the rules stay simple.
Agreed, and the fix is to turn the check into the write condition, not a post-write audit. Recompute in memory, compare against the stored result and its input hash, and only then issue a conditional write — for example UPDATE ... WHERE result IS NULL OR result_hash = expected — so a differing recompute fails closed into review instead of overwriting a legitimate result. The audit trail still matters, but only as evidence for the review case; the guard itself has to live in the write path, otherwise drift is something you notice after the damage is already stored.
That maps exactly — and your case shows why the two mechanisms cover different halves. The idempotency key per event stops the same webhook delivery from applying twice; the version check inside the update stops two different events from overwriting each other. Either one alone leaves a hole: keys without a version check still let a stale retry land on top of a newer plan change, and a version check without keys turns every duplicate delivery into a conflict to review instead of a no-op. Splitting handlers by field ownership is the third leg — it shrinks which pairs of events can even collide. Thanks for sharing the webhook version of this; it is the same shape as the article, just with money attached, which is why the silent version is worse.
Exactly the three-legged version we landed on. The one that surprised us was the third leg: splitting handlers by field ownership shrank the blast radius more than either of the other two. Two events can only collide when they write the same fields, so once each handler owned its fields outright, the review table went from several entries a week to almost none. Money attached is why we kept the review path instead of auto-merging anything touching billing state.
Exactly — and the n8n wrinkle is that a workflow's Postgres node runs one statement at a time, so “same transaction” does not happen by just placing two nodes in sequence. In practice that means folding the check and the write into a single statement (INSERT ... ON CONFLICT ... RETURNING) or a small database function that does both and returns inserted-versus-replay, so the atomicity lives in Postgres, not in the workflow's step order.
That n8n detail is the one that bit us. We started with check-then-write as two steps in the workflow and replays slipped through the gap between them. Folding both into a single statement closed it. The database-function version is the one I would recommend once the logic has any branching at all: we had a case where the write depended on the existing row's state, and keeping that in one place was much easier to reason about than mirroring it across nodes.
Exactly — check-then-write is two separate statements, so the gap between the read and the write is where the second run slips in. The claim has to be a single atomic statement, e.g. INSERT ... ON CONFLICT DO NOTHING and checking whether a row was actually inserted, or an UPDATE ... WHERE claim IS NULL that only succeeds for one run. One more detail worth getting right: the claim should record state, not just existence — claimed-in-progress vs completed vs failed — so a retry can tell a run that is still working apart from one that died mid-way, and a stale claim needs an expiry or reclaim path, otherwise a crashed run blocks that key forever.
The stale-claim point is the one I'd underline — we had a crashed run hold a key for hours before anyone noticed, and it looked exactly like the provider had stopped sending. We ended up giving in-progress claims a short TTL with a sweeper that only reclaims ones whose worker heartbeat went silent, so a genuinely slow run never loses its claim to a race with the cleaner.
Heartbeat-gated reclaim is the right guard — a blind TTL sweep will steal a slow-but-alive run's claim. I'd put the expiry in the claim statement itself: the atomic claim succeeds only when no live claim exists or the existing claim's heartbeat is past its silence threshold, and the reclaim writes a new owner token. The sweeper then becomes a backstop for rows the claim query never revisits, not the mechanism correctness depends on. And the stale writer still needs the last-writer check on its update, so even a reclaimed claim can't land an old write.
Putting the expiry in the claim statement itself is cleaner than our setup, I'll take that. The owner token on reclaim is the piece I'd underline, because without it two writers can still both believe they won. We hit that exact case before the heartbeat check landed. And yes on the sweeper as backstop: correctness sits in the claim statement plus the last-writer check, the sweeper just tidies rows nothing else revisits.
Exactly — the owner token has to live in the reclaim statement itself, not in application code after a read. Reclaim should be UPDATE … WHERE key = ? AND owner_token = ? AND expires_at < now(), and only the row that returns may proceed. A heartbeat renewal must carry the same token, so a writer that lost its heartbeat cannot revive and overwrite the new owner. The test I would run: kill writer A after its heartbeat stops, let writer B claim, then resume A and have it attempt a write — that write must fail visibly, not succeed silently. The last-write check is the same rule applied at write time: the write statement re-checks ownership, it never trusts a claim read earlier in the run.
That kill-A-then-resume-A test is the one I'd keep — it's the closest thing to what actually burned us, a crashed run holding a key for hours while a new run picked up the work. Baking the owner token into the UPDATE ... WHERE is exactly the shape I'd want: the write re-checks ownership instead of trusting a claim read from earlier in the run.
Exactly — the boundary is token possession, not the process name. A resume by the same instance should only continue if it still holds the original owner token from the claim row; a supervisor restart that generates a new token is a competitor, even when it carries the same logical job. That is why the reclaim window has to be about both: time gates the competitor, identity gates the continuation. The test to write is kill A, restart as A2 with a fresh token, and assert A2 cannot write until the window expires and the sweeper marks the old claim stale — while a same-token resume can finish. Get either side wrong and you get the two failures you named: duplicate work, or a task lost despite having had a legitimate owner.
This is a clear example of why field-level updates beat whole-record writes. Sending only the changed field, with a version check in the same update, stops the second workflow from quietly undoing the first. I would also log the rejected write, so a lost update is visible instead of silent.
Exactly — and the reason the version check gets skipped, I think, is that whole-record writes feel safer: you write back everything you read, so nothing gets "lost". In practice that's what loses data, quietly. Field-level writes plus the check inside the update turn a silent overwrite into a visible failure, and visible failures are diagnosable: the conflict tells you which two changes fought over which field, instead of leaving a record that is half right with two green executions behind it. Thanks for reading it that closely.
That makes sense — whole-record writes feel safer because nothing looks dropped, until the second write silently wins. Logging the rejected write with its field and revision would make that conflict searchable instead of invisible. Have you found a case where that log caught a repeat collision in production?
Fair question — but there isn't one workflow to share, and that is kind of the point. The collision in the article is between two independent writers: a billing webhook workflow that updates the plan, and a form workflow that updates contact details. Each one is small on its own; the bug only exists in how they share a row. The reusable piece is the versioned update, not a full workflow export. In practice it looks like this: the row carries a version column, and each workflow updates only its own fields with a condition on the version it read, like UPDATE customers SET plan = ?, version = version + 1 WHERE id = ? AND version = ?. Then it checks how many rows changed. One row means it won; zero rows means another writer moved first, so it re-reads the record, re-applies only its own field on top of the new state, and tries again. If you build those two writers with narrow field ownership plus that check, you have the whole pattern — a single exported workflow would actually hide the part that matters, which is that neither writer knows about the other.
Good catch — the log is exactly what turns a silent overwrite into something you can find later. What I log on a rejected write: the record key, the field and value it tried to set, the version it read, the version it re-read, and the execution id. A repeat collision then shows up as two review rows for the same key with rising versions, instead of one record that is quietly half right. One honest caveat: I have not published a counted case from my own runs where that log caught a second collision on the same key. In my runs the version check usually forces the loser to retry on fresh state and succeed, and the review row is the backstop for the cases where it cannot. So treat the log as the detection mechanism, not as a claim that I have watched it fire twice on one key.
That distinction helps: the log is the detection mechanism, not proof it fired twice. Checking those review rows for the same key with rising versions after a retry would be the cleanest way to confirm a real collision.
Thanks — glad the heartbeat part landed. One thing worth sizing when you implement it: keep the heartbeat interval well under the TTL (for example, every 15s for a 60s lease), and treat a missed heartbeat as a signal, not just a timer reset — if the worker cannot extend, it should stop writing rather than keep going on a lease it may have lost. For long jobs, chunking the work so each chunk fits inside one lease beats stretching the TTL: a long TTL just lengthens the window a crashed worker blocks everyone else.
That sizing rule is a good one. Treating a missed heartbeat as a stop signal, not just a timer reset, is the part I would not skip, since it keeps a stalled worker from writing on a lease it may have lost.
Right — the threshold should predict failure, not just be easy to configure. Row count alone misleads when rows are tiny, and payload size alone misses lock hold time. In practice I size by the combination that actually breaks: concurrent writers multiplied by lock footprint per write, measured as transaction duration and lock wait under load. Run the batch at increasing sizes, watch where p95 lock wait starts climbing, and set the split threshold below that point with margin, per table since row width differs. That way the threshold is a measured failure predictor, and the split point moves when the data shape changes instead of staying a guessed constant.
That measured approach makes sense. Watching p95 lock wait climb as batch size grows gives a much better split point than a fixed row count, since wide rows and long holds change the risk.
Your fix carries a requirement you don't state: the version has to be a value the competing writer cannot author. A server-assigned revision that the update statement advances atomically works. A client-supplied timestamp or revision number does not, and the check still runs, still reports rows changed, still admits the write.
Worth being precise about why, because it isn't the equality check that breaks. Comparing against a stale timestamp is fine, since an earlier time doesn't equal the stored one. What breaks is any scheme that treats author-declared ordering as evidence about concurrency, including the "keep the newest write" resolution people reach for once version checks start failing often.
I measured the edges of that on an append-only ledger I run. Its skew check is real and fires in both directions against the author's clock, so this isn't a missing-validation story. It still stores 23 rows whose authored timestamp sits after the moment the server received them, by up to 12 seconds, because they landed inside the permitted 300-second future window. The past bound allows seven days. The server does assign its own arrival time, non-null across all 123,456 rows, and the public read API returns seven fields per event without it. Third parties are left comparing the value the other writer picked. 65,415 rows share their created_at second with another row, so it isn't unique either.
Your second detail is where I'd push harder. "Zero rows changed" is a return value. One party sees it, once. Escalating a conflict for review needs something durable, and a failed precondition leaves an absence instead. My ledger has the same hole from the other side: the only write precondition is primary-key uniqueness, a duplicate raises an integrity error, the handler returns false, and nothing about the collision is written down.
Two things worth checking in your setup. Does every accepted update advance a revision the workflow itself cannot set? And if you force a stale update right now, is there a row afterwards that your escalation path can actually read?
You're right, and this is the sharper version of the rule: the version has to be assigned and advanced by the store, in the same statement that writes. The workflow supplies the expected revision, never the new one — the update sets the field and increments the revision only where the id and the expected revision both match. A client-supplied timestamp or revision fails for a different reason than a bad comparison: the writer is authoring its own ordering claim. A timestamp inside a permitted future window is still a claim, not an arrival order, and "keep the newest write" just resolves the conflict in favour of whoever claimed the latest time.
Your second point is the one I had missed stating: zero rows changed is a return value seen once, by one execution. If the conflict path only logs it, there is nothing for an escalation to read later. In my setup the stale-update branch has to write a conflict row first — the change it tried to make, the version it read, and the version it found on re-read — into a review table the escalation path queries. Execution logs don't count: they expire, and they aren't a queue a human or a sweeper can work through.
So the two checks, answered honestly: every accepted update must advance a revision the workflow cannot set, and the test for the second one is to force a stale update and then query the review table — not to watch the update return zero, but to confirm a row exists that the escalation path can actually pick up. Thanks for pushing on both — the first is a correction to the article, the second is a gap in it.