This is part of my build-in-public series: I am shipping a small store of n8n workflow templates (AI Automation Lab) and a free open-source lab on GitHub. Previous parts: The Crash Test Nobody Runs and No Idempotency Key? How I Make Gmail Sends Safe to Retry Anyway.
Part 1 of this series was about where crashes bite. Part 2 was about giving yourself a reconciliation handle when the provider offers no idempotency key. This part is about the component that does the actual reconciling: the sweeper.
A sweeper is a scheduled job that scans for work left in an ambiguous state and decides its fate. It exists because crashes don't clean up after themselves. Somewhere in your queue there is a row that says claimed, and nobody knows whether the email went out, the charge happened, or the webhook fired. The sweeper's entire job is to turn "unknown" into a decision — without guessing.
The lease is the whole trick
The sweeper only works if "claimed" has an expiry. Every job row carries a lease:
-- jobs table: the lease is what makes "claimed" a temporary state
-- id comes from the SOURCE EVENT (Stripe payment id, message hash)
-- never from $execution.id or a fresh uuid
CREATE TABLE jobs (
id TEXT PRIMARY KEY, -- stable key from the SOURCE EVENT
state TEXT NOT NULL, -- queued | claimed | done | review
payload JSONB NOT NULL,
provider TEXT NOT NULL,
provider_ref TEXT, -- reference the provider gave back, if any
attempts INT DEFAULT 0,
claimed_at TIMESTAMPTZ, -- when the lease started
lease_seconds INT DEFAULT 600 -- 10 min default; see discussion below
);
Workers claim atomically — only the worker that gets the row proceeds:
-- atomic claim: exactly one worker wins, everyone else gets zero rows
UPDATE jobs
SET state = 'claimed', claimed_at = now(), attempts = attempts + 1
WHERE id = $1 AND state = 'queued'
RETURNING *;
And the sweeper touches only expired leases:
-- the sweeper's scan: only rows whose lease has expired
-- this is the line between "the worker is slow" and "the worker is dead"
SELECT * FROM jobs
WHERE state = 'claimed'
AND claimed_at < now() - (lease_seconds || ' seconds')::interval;
The lease is the line between "the worker is slow" and "the worker is dead." Sweep too early and you reconcile a job that was about to finish — now you have two actors working the same row. Sweep too late and ambiguous jobs sit there for hours. Ten minutes is my default; a charge that takes longer than your lease doesn't belong in this queue, it belongs in an async-confirmation flow.
The sweeper's decision table
For each expired-lease row, the sweeper asks the provider — never n8n — and follows this table:
| Provider class | Handle available | Action |
|---|---|---|
| Forced-unique (Stripe Idempotency-Key, natural-key upsert, reference-backed order API) | idempotency key or provider reference | Re-drive with the same key, or query the provider by reference. Safe by construction. Ends done. |
Suggestion-key (SMTP, plain webhooks, Gmail raw messages.send) |
reconciliation handle (deterministic Message-ID, searchable log) | Look it up. Found → done. Still absent after the settle window → review. Never auto-resend.
|
| Suggestion-key | no handle at all |
review. Always. A human decides, with the evidence the row carries. |
Note the asymmetry, which is the entire point: for forced-unique providers the sweeper recovers; for suggestion-key providers the sweeper classifies. Recovery and classification are different verbs. Conflating them is how you get double-charges and double-emails.
Five ways to screw up a sweeper
1. Re-running the workflow as the recovery action. The sweeper must reconcile state, not re-run logic. If your recovery path is "put the job back to queued and let the workflow run again," you have built a retry loop with extra steps. Re-running a workflow re-executes side effects. The sweeper should perform the smallest provider-side check that closes the question, then write done or review directly.
2. Letting the sweeper crash unsafely. The sweeper is also a process that can die mid-sweep. So every recovery action must be idempotent, and the sweeper's own claim on a row must be a lease too — if the sweeper dies halfway, the next sweep re-runs against the same rows and gets the same answers. A sweeper that cannot survive its own crash is just another worker with a fancier name.
3. Sweeping without single-flight. Two sweeper instances racing over the same expired leases will double-recover. Solve it with the same primitive: an atomic claim on the sweep itself. One row in a sweep_locks table, one INSERT ... ON CONFLICT DO NOTHING, one holder. The runner-up exits quietly.
4. Treating "not found" as "not sent." Provider search has indexing delay — I measured this on Gmail in part 2. A negative result five seconds after a crash is inconclusive, not evidence. The sweeper needs a settle window: mark the row sweeping, wait, re-scan, and only then decide. If the channel can't tolerate the latency, that row was never a sweeper candidate — it was review from the start.
5. Counting in n8n. The sweeper's verdict must come from provider-side evidence — the provider's dashboard, its API, your relay's acceptance log that records the actual effect. n8n execution history tells you which workflows ran, not which side effects landed. A green execution next to a dead provider connection is exactly the crash-boundary from part 1.
The kill matrix
I test the sweeper the same way I test everything in this series: kill things and count on the provider's side.
-
Kill after the effect, before mark-done. Effect happened, row still says
claimed. Run the sweeper. Provider must show exactly one effect; row endsdone. No duplicate. - Kill mid-sweep. Sweeper dies halfway through recovery. Re-run it. Still exactly one effect, no double-recovery. (This is test 2 for rule 2 above.)
-
Kill before the effect. Row says
claimed, provider shows nothing. After the settle window, the sweeper re-drives (forced-unique) or reconciles (suggestion-key). Exactly one effect. -
Relay forward-then-drop. The ambiguous class: provider accepted, response never observed. Sweeper must land the row in
reviewand never auto-resend.
Assert all four against provider-side counts, and report mock results as mock results. A green kill matrix on a mock relay is a tested design, not a tested deployment — the real ambiguous send still has to be watched in production before you claim it works.
What the sweeper doesn't give you
It does not give you exactly-once. Exactly-once is a property of the receiving system; no caller-side component can impose it. What the sweeper gives you is smaller and more honest: a bounded recovery path, a review state that means "a human looked at this" instead of "this silently duplicated," and a kill matrix you can re-run after every change to prove the behavior survived.
Most n8n reliability advice stops at "add a retry." Retries answer the wrong question. The question after a crash is never "should I try again" — it's "did it already happen." The sweeper is the component that answers it, on a schedule, without guessing.
I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.
Top comments (2)
@challan116ux Exactly — and the EDI-acknowledgement case is a sharper version of the same point than my invoice example, because "wait for the ack" is already what the business does; the sweeper just makes the waiting explicit and bounded.
One nuance I'd add: classifying without provider-side evidence doesn't just fail to recover — it actively creates the next incident. The moment a row is marked "recovered" with no evidence, every later pass (human or automated) treats it as settled and stops looking. A row sent to human review stays suspicious, which is the correct state when nothing is proven. My working rule: never let a reconciliation step reduce the uncertainty level below what the evidence supports.
Your "lease expiry plus a settle window beats a confident guess" generalizes nicely too. In the crash-boundary post I frame it as the reconciliation horizon: by T+Δ the row must be settled or escalated, not merely "checked". Sounds like you found the same invariant in EDI land — curious whether your settle window is fixed or scales with the provider's ack SLA.
The distinction between recovering and merely classifying is the sharpest point here, and it is one people blur constantly. Sending a row to human review when there is no provider-side evidence is not a failure of automation in my book — it is the automation being honest about what it cannot know. We hit the same wall with EDI acknowledgements that arrive late or never, and a lease expiry plus a settle window beats a confident guess every time.