The Webhook Burst That Buried My Queue: Backpressure in n8n

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

One quiet minute, then a thousand webhooks

A partner flipped a switch on their side and backfilled a day of events in one go. My webhook workflow had been happily handling a few events a minute. In sixty seconds it received over a thousand.

Nothing crashed, exactly. That was the problem. Every webhook execution started, every one called the downstream API, and the downstream API answered with 429s. Retries piled on top of the burst. The queue grew faster than it drained, latencies stretched from seconds to tens of minutes, and by the time the backlog cleared, half the events were stale — the customer had already seen the newer state. The workflow had "processed" everything and delivered almost nothing on time.

The mistake was treating arrival rate as something I controlled. I did not. A webhook endpoint accepts whatever the sender sends, whenever the sender sends it. If my workflow cannot keep up, the honest options are to queue, to slow the sender down, or to refuse — not to pretend the burst did not happen.

Accept fast, work slowly

The first rule I now follow: the webhook handler does almost nothing. It validates the signature, checks the delivery id for duplicates, writes the payload to a durable queue, and returns 200. All the real work — calling the model, writing to the CRM, sending the email — happens in a separate worker workflow that pulls from that queue.

This separation matters for two reasons. First, the webhook response time stops depending on the slowest downstream call. Second, a crash in the worker no longer loses the event: the row is still in the queue, still marked unprocessed, and the sweeper will pick it up. The webhook handler is the one place in the system where speed is the feature.

Bound the concurrency, and say so

The worker pulls a fixed number of jobs at a time. Mine is small — two concurrent executions for a provider with a documented rate limit, more only where I have measured headroom. The bound is explicit, in one place, not spread across five workflows each guessing.

An unbounded worker is just a burst with extra steps. Ten parallel executions hitting the same provider at once will trip the same 429 that a burst would, except now the retries are mine and I own the bill. Concurrency is a setting to choose, not a default to accept.

Shed load with a signal, not silence

When the queue depth passes a threshold, the webhook handler stops returning 200 and starts returning 429 with a Retry-After header. This is the part people resist: deliberately refusing work feels like failure. It is the opposite. A 429 tells a well-behaved sender "try again in sixty seconds," which is a plan. Silently accepting work I cannot do in time is a lie that surfaces later as stale deliveries and a queue nobody can drain.

The threshold itself is derived, not guessed: queue depth at which the oldest unprocessed job would miss its freshness deadline, given the worker's measured drain rate. If the worker drains ten jobs a minute and a job goes stale after thirty minutes, a depth of three hundred means new arrivals are already too late. Better to refuse them loudly than to bury them quietly.

Order the queue by value, not arrival

Not every event deserves the same urgency. A cancellation beats a status update; a new order beats a nightly sync. My queue rows carry a priority, and the worker claims the highest-priority oldest job first. Without this, a burst of low-value events can starve the one event the customer is waiting on, even while the system looks "busy."

Priority is also how I keep the digest honest: when I shed load, I shed the lowest-priority work first, and I log exactly what was refused, so the sender's retry — or a human — can recover it.

Measure the drain rate, not just the depth

Queue depth alone lies. A depth of five hundred draining at a hundred a minute is a five-minute delay. The same depth draining at five a minute is a hundred-minute delay. I track both: depth, and jobs completed per minute over the last ten minutes. The alert fires on projected staleness — depth divided by drain rate exceeding the freshness deadline — not on depth alone. That is the alert that pages me before the queue is buried, instead of after.

The test is a burst, not a trickle

I test this the way it actually fails: fire two hundred webhooks at the endpoint in one minute against a mock provider with a strict rate limit, then assert four things. First, the webhook handler returns 200 or 429 quickly — never a timeout. Second, the worker never exceeds its concurrency bound, even at peak. Third, no event is processed twice and none is silently lost: every accepted event reaches done, every refused event is logged with its delivery id. Fourth, the highest-priority event finishes before the backlog of low-priority events, even when it arrives last.

A test that sends one webhook at a time proves the happy path. The burst test proves the system.

Checklist before your next webhook goes live

• Webhook handler: validate, dedup, enqueue, respond. Nothing else.
• Worker concurrency: one explicit bound, measured against the provider's limit.
• Shed threshold: derived from drain rate and freshness deadline, returning 429 with Retry-After.
• Queue priority: value-ordered claiming, so a burst cannot starve the important event.
• Alerting: projected staleness, not raw depth.
• Recovery: refused events logged by delivery id, so a retry or a human can replay them.

Backpressure is not a performance optimisation. It is the difference between a system that degrades gracefully — slower, louder, recoverable — and one that accepts everything, delivers late, and calls it success.

---

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at https://github.com/zhp910318/n8n-ai-automation-lab. The store is at https://payhip.com/haopengailab, and the free workflow is at https://payhip.com/b/ulER8.