DEV Community

Young Gao
Young Gao

Posted on

Building Reliable Webhook Delivery: Retries, Signatures, and Failure Handling (2026)

#api #webhooks #architecture #backend

Your webhook fires. The receiver is down. The event is lost forever.

The Problem With Fire-and-Forget

Most webhook implementations: serialize payload, POST to URL, move on. If the receiver returns 500 or times out, the event vanishes. No retry. No record. No way to recover.

Webhook Architecture 

Event -> Queue -> Delivery Worker -> HTTP POST -> Receiver
                    | (on failure)
               Retry Queue -> Exponential Backoff -> DLQ
Enter fullscreen mode Exit fullscreen mode

Store every event in a database. The delivery worker reads from the queue and attempts delivery. On failure, schedule a retry with exponential backoff.

Signing Webhooks

Never trust the sender without verification. Sign every payload with HMAC-SHA256:

import crypto from "crypto";
function signPayload(payload: string, secret: string): string {
  return crypto.createHmac("sha256", secret).update(payload).digest("hex");
}
function verifyWebhook(payload: string, signature: string, secret: string): boolean {
  const expected = signPayload(payload, secret);
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}
Enter fullscreen mode Exit fullscreen mode

Include timestamp in the signature to prevent replay attacks. Stripe adds t=timestamp to the signature header and rejects events older than 5 minutes.

The Delivery Worker 

interface WebhookEvent {
  id: string; type: string; payload: unknown;
  url: string; secret: string; attempts: number;
  nextRetry: Date | null;
  status: "pending" | "delivered" | "failed";
}
async function deliver(event: WebhookEvent): Promise<void> {
  const body = JSON.stringify(event.payload);
  const signature = signPayload(body, event.secret);
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 10000);
  try {
    const res = await fetch(event.url, {
      method: "POST",
      headers: { "Content-Type": "application/json", "X-Webhook-Signature": signature, "X-Webhook-ID": event.id },
      body, signal: controller.signal,
    });
    clearTimeout(timeout);
    if (res.ok) event.status = "delivered";
    else scheduleRetry(event);
  } catch { scheduleRetry(event); }
}
function scheduleRetry(event: WebhookEvent): void {
  event.attempts++;
  if (event.attempts >= 5) { event.status = "failed"; return; }
  const delay = Math.min(1000 * Math.pow(2, event.attempts), 3600000);
  event.nextRetry = new Date(Date.now() + delay);
}
Enter fullscreen mode Exit fullscreen mode

Idempotency on the Receiver Side

Receivers must handle duplicate deliveries. Use the webhook ID as an idempotency key:

app.post("/webhooks", async (req, res) => {
  const webhookId = req.headers["x-webhook-id"];
  const exists = await redis.get(`webhook:${webhookId}`);
  if (exists) return res.status(200).json({ status: "already_processed" });
  await processEvent(req.body);
  await redis.setex(`webhook:${webhookId}`, 604800, "processed");
  res.status(200).json({ status: "ok" });
});
Enter fullscreen mode Exit fullscreen mode

What Stripe Gets Right

  1. Event log: Every event stored with full payload, queryable via API
  2. Automatic retries: Up to 3 days with exponential backoff
  3. Webhook signatures: HMAC-SHA256 with timestamp for replay protection
  4. Granular event types: invoice.paid, not just invoice
  5. Test mode: Separate webhook endpoints for test vs live

Common Mistakes

  1. No timeout: Receiver hangs forever, blocking your delivery worker
  2. Synchronous delivery: Sending webhooks in the request path blocks your API
  3. No signature verification: Accepting any POST to your webhook URL
  4. No idempotency: Processing the same event twice charges the customer twice
  5. Logging payloads with PII: Webhook payloads often contain sensitive data ***

Part of my Production Backend Patterns series. Follow for more practical backend engineering.

You Might Also Like

Follow me for more production-ready backend content!

If this helped you, buy me a coffee on Ko-fi!

Top comments (0)