DEV Community

Shahdin Salman
Shahdin Salman

Posted on

Designing Resilient Event-Driven Webhook Queues in n8n and Node.js for High-Volume Systems

#architecture #javascript #node #systemdesign

The failure mode nobody notices until it's expensive

Most webhook handlers look correct in a code review and fail in
production for the same reason: they conflate receiving an event
with processing it.

// The naive (and common) pattern
app.post('/webhook/order', async (req, res) => {
  const order = req.body;
  await classifyOrder(order);       // LLM call
  await syncToCRM(order);           // external API
  await sendConfirmation(order);    // messaging API
  res.status(200).send('ok');
});
Enter fullscreen mode Exit fullscreen mode

This works in every test you'll ever run manually. It fails the first
time:

  • your LLM call takes 4 seconds instead of 400ms under load,
  • the sender (Shopify, WhatsApp Business API, Stripe) times out waiting for your 200, and
  • retries the same webhook, or worse, gives up and drops it.

Under a traffic spike, this pattern doesn't degrade gracefully it
silently loses data. No stack trace. No error log. Just a missing row.

The fix: Ingest-and-Acknowledge, then process asynchronously

The core architectural change is simple to state and easy to get wrong
in implementation: separate the act of receiving an event from the
act of processing it, and make the receiving step as close to
instantaneous and failure-proof as possible.

// Ingest-and-Acknowledge pattern
app.post('/webhook/order', async (req, res) => {
  const eventId = generateIdempotencyKey(req.body);

  // Step 1: persist raw event immediately before any processing
  await rawEventStore.insert({
    id: eventId,
    payload: req.body,
    receivedAt: Date.now(),
    status: 'pending'
  });

  // Step 2: acknowledge the sender instantly
  res.status(200).send('ok');

  // Step 3: hand off to the queue this does NOT block the response
  await queue.enqueue('process-order', { eventId });
});
Enter fullscreen mode Exit fullscreen mode

Three properties this buys you:

  1. Idempotency at the door. Every event gets a deterministic ID (hash of payload + source, or a provider-supplied event ID) checked against the raw store before insert. Duplicate webhook deliveries which every major provider will eventually send you become a no-op instead of a duplicate order.
  2. Zero data loss on downstream failure. If the LLM call, CRM sync, or messaging API fails, the raw event is already safely stored. You replay from the queue you never lose the original payload.
  3. Decoupled scaling. The webhook endpoint's only job is "write one row, respond 200." That can handle enormous throughput. The heavier processing work scales independently, on its own schedule.

Why n8n's default pattern needs this layer added manually

n8n's out-of-the-box Webhook node executes the entire workflow
synchronously by default great for prototyping, risky at volume. For
production systems, we typically:

  • Put a lightweight ingestion endpoint (a thin Node.js/Express layer, or n8n's Webhook node set to Respond Immediately) in front of the workflow.
  • Write the raw payload straight to Postgres or Redis before anything else runs.
  • Trigger the actual n8n workflow from a queue consumer, not directly from the incoming webhook. 
// Queue consumer this is what actually calls the heavy logic
queue.process('process-order', async (job) => {
  const event = await rawEventStore.findById(job.data.eventId);

  try {
    const classified = await classifyOrder(event.payload);
    await syncToCRM(classified);
    await sendConfirmation(classified);

    await rawEventStore.updateStatus(event.id, 'completed');
  } catch (err) {
    await rawEventStore.updateStatus(event.id, 'failed');
    throw err; // let the queue's retry policy handle it
  }
});
Enter fullscreen mode Exit fullscreen mode

Retry policy and backoff don't hand-roll this

Configure exponential backoff at the queue level rather than inside
your business logic:

queue.add('process-order', { eventId }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnComplete: true,
  removeOnFail: false // keep failed jobs for manual inspection
});
Enter fullscreen mode Exit fullscreen mode

removeOnFail: false matters more than it looks it's what turns a
silent failure into a queryable, debuggable dead-letter queue instead
of data that simply vanishes.

Rate-limiting the downstream calls, not the ingestion

A common mistake: rate-limiting the webhook endpoint itself. Don't.
The endpoint should always accept and store the event instantly rate
limits belong on the queue consumer, where you control concurrency
against your LLM provider's or CRM's actual rate limits.

queue.process('process-order', { concurrency: 5 }, async (job) => {
  // only 5 concurrent downstream calls at a time,
  // regardless of how many events arrived
});
Enter fullscreen mode Exit fullscreen mode

This means a 500-order spike doesn't crash your OpenAI/Claude rate
limit or your CRM's API quota it just takes slightly longer to fully
process, with zero events lost in the meantime.

The result

A system built this way doesn't "handle load better." It removes the
category of failure entirely a dropped webhook under a traffic spike
stops being possible, because the moment of receiving and the moment
of processing are no longer the same moment.

If you're running order intake, lead capture, or any webhook-triggered
AI workflow at real volume, this pattern is the first thing worth
auditing it's usually a half-day fix that prevents a class of bugs
that otherwise only surfaces on your worst (best) day: the traffic spike.

I write about production-grade AI automation architecture at
SpaceAI360. If you're dealing with a webhook reliability problem at
scale, I'm happy to compare notes.

Top comments (0)