Return 200 fast: why your webhook handler should do almost nothing

The payment provider sent an order.paid webhook. The handler created the order, generated a PDF invoice, emailed it and synced the CRM. That took 14 seconds. The provider's timeout was 10. It marked the delivery as failed and retried. Then again. Each customer got three invoices.

The handler never errored. It was just too slow to say "got it".

TL;DR

• Webhook senders have short timeouts (often 5 to 30 seconds) and retry on anything other than a fast 2xx.
• Do four things inline: verify, store, dedupe, acknowledge. Do everything else asynchronously.
• Returning 200 means "I have it safely", not "I've finished processing it".
• My opinion: any webhook handler that calls a third-party API before responding is a retry storm waiting for a slow day.

What senders actually do

Most providers follow the same pattern: POST the event, wait a few seconds, treat a timeout or non-2xx as failure, retry with backoff for hours or days. Some disable your endpoint after enough failures.

So a slow handler doesn't just risk one timeout. It creates duplicates during the retries and can get your endpoint switched off during an incident, which is exactly when you need the events most.

The four-step handler

1. verify signature       (cheap, rejects junk)
2. store raw event        (durable, before anything else)
3. dedupe on event id     (unique constraint)
4. return 2xx             (target: < 500 ms)
--- async ---
5. process from storage   (your worker, your timeouts, your retries)

Node (Express)

import express from 'express';
import crypto from 'node:crypto';

const app = express();

app.post('/webhooks/payments', express.raw({ type: 'application/json' }), async (req, res) => {
  if (!verify(req.headers['x-signature'], req.body)) return res.sendStatus(401);

  const event = JSON.parse(req.body);
  try {
    await db.query(
      `INSERT INTO inbound_events (id, type, payload, received_at, status)
       VALUES ($1, $2, $3, now(), 'pending')
       ON CONFLICT (id) DO NOTHING`,
      [event.id, event.type, req.body.toString()]
    );
  } catch (e) {
    return res.sendStatus(500); // storage failed: let the sender retry
  }
  res.sendStatus(200);          // stored (or already had it): acknowledge
});

function verify(sig, raw) {
  const mac = crypto.createHmac('sha256', process.env.WEBHOOK_SECRET).update(raw).digest('hex');
  return sig && mac.length === sig.length &&
    crypto.timingSafeEqual(Buffer.from(mac), Buffer.from(sig));
}

Two details matter. Verification runs on the raw body, before JSON parsing, or the signature won't match. And a duplicate insert still returns 200, because telling the sender to retry something you already have just creates more duplicates. The signature details are covered in How to verify HMAC-SHA256 signed webhooks in Node and Python.

The worker

import time

def work_loop():
    while True:
        rows = db.execute("""
            UPDATE inbound_events SET status='processing', locked_at=now()
            WHERE id IN (
              SELECT id FROM inbound_events
              WHERE status='pending'
                 OR (status='processing' AND locked_at < now() - interval '10 minutes')
              ORDER BY received_at
              LIMIT 20
              FOR UPDATE SKIP LOCKED)
            RETURNING id, type, payload""").fetchall()
        if not rows:
            time.sleep(1)
            continue
        for r in rows:
            try:
                handle(r.type, r.payload)       # must be idempotent
                db.execute("UPDATE inbound_events SET status='done' WHERE id=%s", (r.id,))
            except Exception as e:
                db.execute("""UPDATE inbound_events
                              SET status='pending', attempts=attempts+1, last_error=%s
                              WHERE id=%s""", (str(e)[:500], r.id))

This assumes the table also has attempts, locked_at and last_error columns. FOR UPDATE SKIP LOCKED lets you run several workers on the same table without them grabbing the same event. Stuck "processing" rows get picked up again after ten minutes, which means handle can run twice for the same event. That's why it needs to be idempotent: see the previous post on at-least-once delivery.

Add an attempts limit and move poison events aside instead of retrying forever. A small dead-letter table is enough.

"But I don't want another table"

Fair. Alternatives, from simplest to heaviest:

• Background task in the same process (e.g. a framework's background tasks): fast to build, but events are lost if the process dies before finishing. Fine for non-critical events.
• A queue you already run (Redis, SQS, a job library): push the raw event, acknowledge, let workers consume.
• An inbox table, as above: durable, queryable, replayable. My default for anything involving money.

The key property is the same: the event is durable before you return 200.

Ordering

Retries and parallel workers mean events can be processed out of order: subscription.updated before subscription.created. Don't trust arrival order. Either re-fetch the current state from the provider's API when processing, or compare a version or timestamp field and ignore stale events.

Testing it

# time your handler's response, not your processing
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' \
  -X POST https://example.com/webhooks/payments \
  -H "x-signature: $SIG" --data-binary @event.json

Anything over a second deserves a look. Send the same event twice and confirm only one row and one side effect.

If you're on the sending side, for example firing scheduled HTTP callbacks from cron or a service like delay.run, the same rule applies to your receivers. Keep the endpoint you call fast and do the heavy lifting afterwards. How to schedule an HTTP request covers the sending side.

My take

A webhook endpoint is a mailbox, not an office. Its only job is to accept the letter safely and close the door. Everything you do before responding is a chance to time out and get the same letter again.

Over to you

Do you process webhooks inline or through an inbox/queue, and what made you choose? If you've been burned by a provider disabling your endpoint after timeouts, I'd like to hear how you recovered the missed events.

Comments

[Livinus Okoz]

Structuring webhook receivers to immediately acknowledge payloads and offload execution to a durable inbox queue with row-level locks prevents timeout storms and duplicate invoices entirely. Let me know what payment gateway or CRM webhooks you are currently integrating and I can share a robust asynchronous event-handling pattern right here.

[delay.run]

Agreed, ack first and process later is the core of it. The inbox table with row-level locks (SELECT ... FOR UPDATE SKIP LOCKED in Postgres) is a nice way to get there without adding a separate queue service, and a unique constraint on the provider's event id handles duplicates at the same time.

The part I see teams miss is the provider's retry schedule. Stripe, GitHub and Shopify all retry differently, so "how long can my worker be down before events are gone for good" has a different answer per provider. Have you run into one where the retry window was shorter than you expected?