Quarantine, Don't Drop: What I Do With Webhook Payloads That Fail Validation

Failing validation doesn't make a payload worthless — it makes it evidence. Here's the quarantine pattern that recovered 214 events for me last month.

Every n8n webhook flow I've ever built starts with the same lie: "the payload will always look like the docs."

It won't. Providers ship undocumented fields, change types overnight ("price": 19.99 becomes "price": "19.99"), and send null where the schema says string. My validation node catches these — and for the first year, I dropped them. Log the error, move on. Clean pipeline, zero noise.

Then a provider migrated timezones and my date parser rejected three days of orders before I noticed. The payloads were gone. No raw body, no replay, no recovery. I re-fetched what I could from their API, but the webhooks carrying one-time events (cancellations, refunds) were unrecoverable. Dropping bad payloads is the pipeline equivalent of deleting an error message.

Now every webhook flow has the same shape:

Validate → if valid, process → if invalid, quarantine.

The quarantine is not a log line. It's a table row with four things:

1. The raw body, byte-for-byte, before any parsing. Once you parse-and-drop, you've destroyed evidence.
2. The validation errors, field by field, so a human can see at a glance whether it's schema drift or garbage.
3. Received timestamp + source (which workflow, which provider, which event type).
4. A status column: new, reviewed, replayed.

The n8n implementation is boring on purpose: an IF node on the validation result, a Postgres (or even SQLite) Insert on the false branch. Total added latency: one INSERT. The interesting part is what happens after.

Alert on quarantine depth, not on every item. One bad payload is a Tuesday. Twenty in an hour is a provider migration. A scheduled workflow counts status='new' rows per hour per source, and pages me only when a source crosses its baseline. Everything else goes into a morning digest. This is the same principle as the retry-budget pattern I wrote about earlier — don't page humans for things that can wait, page them for regime changes.

Replay is the whole point. When I fix the parser, I don't just deploy and hope. I flip the quarantined rows back through the fixed flow:

• The replay workflow reads status='new' rows and feeds each raw body into the same processing sub-workflow the live path uses. One code path, not two.
• Every downstream action is idempotent — this is where the delivery-ID dedup pattern from my last post pays off again. Replayed payloads get the same idempotency key as if they'd arrived live.
• On success, mark replayed. On failure, leave it new with the new error appended.

Last month this recovered 214 events from a provider's undocumented field rename. Total human time: one digest email, one parser fix, one replay click.

The one decision you have to make up front: fail-open or fail-closed. Quarantine is fail-open — the pipeline keeps running on the good payloads while bad ones wait. For payment and fulfillment events, fail-open is usually right: losing the good 95% to protect against the bad 5% is backwards. But there are flows where fail-closed is correct — anything that spends money or sends customer-facing messages. My rule: if the downstream action is reversible, quarantine (fail-open). If it's irreversible, hold the whole batch and page (fail-closed). The quarantine table works for both; the difference is whether the valid branch keeps running.

The uncomfortable corollary: if you don't have a quarantine, you don't have a data pipeline — you have a data sieve. And sieves don't tell you what fell through.

---

I'm building an n8n automation lab in public — production patterns from workflows that actually run. Previous posts covered the retry budget, the sweeper pattern, and deduping webhook replays when the provider sends no delivery ID.

Comments

[xcl]

Alerting on quarantine depth per source instead of every bad payload makes the pattern much more usable, since a single type change from 19.99 to "19.99" should not page anyone, but twenty in an hour clearly signals drift. One trade-off worth spelling out is retention for the raw bodies: quarantined payloads can contain payment or personal data, so how long should those rows be kept, and should the raw body be encrypted or redacted after a successful replay?