How Should You Design Reliable Webhooks?

TL;DR

Design reliable webhooks by implementing exponential backoff retry (5-10 attempts), idempotency keys, HMAC signature verification, and strict 5-second timeouts. Always return 2xx immediately and process asynchronously. Modern PetstoreAPI provides examples of production-grade webhooks for order updates, pet adoptions, and payment notifications, including robust retry and security mechanisms.

Try Apidog today

---

Introduction

Suppose you send a webhook to notify a client that their pet was adopted, but their server is down. Your webhook fails—should you retry? How many times? What if the client processes the webhook twice and double-charges the customer?

Webhooks are HTTP callbacks that push events to client URLs. While simple in concept, real-world deployments require attention to retries, idempotency, security, and monitoring to handle failures and duplicate events.

Modern PetstoreAPI implements production-ready webhooks for order updates, pet adoptions, and payment notifications. Each webhook includes retry logic, signature verification, and idempotency features.

💡 Tip: If you’re building or testing webhooks, Apidog helps you test webhook delivery, validate signatures, and simulate failure scenarios. You can test retry logic and verify idempotency handling.

This guide demonstrates actionable patterns for designing reliable webhooks using Modern PetstoreAPI as a reference.

---

Webhook Basics

Webhooks are HTTP POST requests sent to client-defined URLs when specific events occur.

How Webhooks Work

1. Client registers webhook URL:

POST /webhooks
{
  "url": "https://client.com/webhooks/petstore",
  "events": ["pet.adopted", "order.completed"]
}

2. An event occurs (e.g., pet adopted)

3. Server sends webhook:

POST https://client.com/webhooks/petstore
Content-Type: application/json
X-Webhook-Signature: sha256=abc123...

{
  "event": "pet.adopted",
  "data": {
    "petId": "019b4132",
    "userId": "user-456",
    "timestamp": "2026-03-13T10:30:00Z"
  }
}

4. Client responds:

200 OK

---

The Reliability Problem

Webhooks can fail for reasons such as:

• Client server downtime
• Network timeouts
• 5xx server errors
• Slow client processing
• Client crashes before processing

Without retry logic, events may be lost. Without idempotency, duplicate webhooks can cause duplicate actions.

---

Retry Logic with Exponential Backoff

Always retry failed webhooks with exponential backoff to maximize delivery reliability.

Exponential Backoff Strategy

Attempt 1: Immediate
Attempt 2: 1 second later
Attempt 3: 2 seconds later
Attempt 4: 4 seconds later
Attempt 5: 8 seconds later
Attempt 6: 16 seconds later

Rationale: Exponential delays prevent overwhelming a downed client and provide time for recovery.

Implementation Example

async function sendWebhook(url, payload, attempt = 1, maxAttempts = 6) {
  try {
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': generateSignature(payload)
      },
      body: JSON.stringify(payload),
      timeout: 5000 // 5 second timeout
    });

    if (response.ok) {
      return { success: true, attempt };
    }

    // Retry on 5xx errors
    if (response.status >= 500 && attempt < maxAttempts) {
      const delay = Math.pow(2, attempt - 1) * 1000;
      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }

    // Don't retry 4xx errors (client error)
    return { success: false, status: response.status };

  } catch (error) {
    // Network error or timeout - retry
    if (attempt < maxAttempts) {
      const delay = Math.pow(2, attempt - 1) * 1000;
      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }
    return { success: false, error: error.message };
  }
}

When to Retry

Retry on:

• 5xx server errors (500, 502, 503, 504)
• Network timeouts
• Connection refused
• DNS failures

Do NOT retry on:

• 4xx client errors (400, 401, 404)
• 2xx success responses

Dead Letter Queue

After exhausting max retries, move failed webhooks to a dead letter queue for manual review:

if (!result.success) {
  await deadLetterQueue.add({
    url,
    payload,
    attempts: maxAttempts,
    lastError: result.error,
    timestamp: new Date()
  });
}

---

Idempotency for Duplicate Prevention

Clients may receive the same webhook multiple times due to retries. Implement idempotency to avoid duplicate processing.

Idempotency Keys

Include a unique ID in each webhook:

{
  "id": "webhook_019b4132",
  "event": "pet.adopted",
  "data": {...}
}

Sample handler to ensure idempotency:

app.post('/webhooks/petstore', async (req, res) => {
  const webhookId = req.body.id;

  // Check if already processed
  const processed = await db.webhooks.findOne({ id: webhookId });
  if (processed) {
    return res.status(200).json({ message: 'Already processed' });
  }

  // Process webhook
  await processPetAdoption(req.body.data);

  // Mark as processed
  await db.webhooks.insert({ id: webhookId, processedAt: new Date() });

  res.status(200).json({ message: 'Processed' });
});

Idempotent Operations

Design business logic to be idempotent:

Not idempotent (bad):

// Charging twice causes double charge
await chargeCustomer(userId, amount);

Idempotent (good):

// Charging with idempotency key prevents double charge
await chargeCustomer(userId, amount, { idempotencyKey: webhookId });

---

Signature Verification for Security

Verify that webhooks are sent by your API, not an attacker.

HMAC Signature

Server generates signature:

const crypto = require('crypto');

function generateSignature(payload, secret) {
  const hmac = crypto.createHmac('sha256', secret);
  hmac.update(JSON.stringify(payload));
  return hmac.digest('hex');
}

// Include in header
headers['X-Webhook-Signature'] = `sha256=${generateSignature(payload, webhookSecret)}`;

Client verifies signature:

function verifySignature(payload, signature, secret) {
  const expected = generateSignature(payload, secret);
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(`sha256=${expected}`)
  );
}

app.post('/webhooks/petstore', (req, res) => {
  const signature = req.headers['x-webhook-signature'];

  if (!verifySignature(req.body, signature, WEBHOOK_SECRET)) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  // Process webhook
  ...
});

Timestamp Validation (Prevent Replay Attacks)

Include timestamp in payload:

{
  "id": "webhook_019b4132",
  "timestamp": "2026-03-13T10:30:00Z",
  "event": "pet.adopted",
  "data": {...}
}

Reject old webhooks:

const webhookAge = Date.now() - new Date(req.body.timestamp);
if (webhookAge > 5 * 60 * 1000) { // 5 minutes
  return res.status(400).json({ error: 'Webhook too old' });
}

---

Timeout Handling

Configure aggressive timeouts to prevent slow clients from blocking your system.

5-Second Timeout

const response = await fetch(url, {
  method: 'POST',
  body: JSON.stringify(payload),
  timeout: 5000 // 5 seconds
});

Why 5 seconds?

Webhooks should return 2xx quickly. Long client processing is a sign of incorrect synchronous logic.

Async Processing Pattern

Incorrect (synchronous processing):

app.post('/webhooks/petstore', async (req, res) => {
  // This takes 30 seconds - webhook will timeout
  await processOrder(req.body.data);
  await sendEmail(req.body.data);
  await updateInventory(req.body.data);

  res.status(200).json({ message: 'Processed' });
});

Correct (asynchronous processing):

app.post('/webhooks/petstore', async (req, res) => {
  // Return immediately
  res.status(200).json({ message: 'Received' });

  // Process asynchronously
  queue.add('process-webhook', req.body);
});

---

How Modern PetstoreAPI Implements Webhooks

Modern PetstoreAPI provides a reference implementation of robust webhooks.

Webhook Events

pet.adopted         - Pet was adopted
pet.status_changed  - Pet status changed
order.created       - Order created
order.completed     - Order completed
payment.succeeded   - Payment succeeded
payment.failed      - Payment failed

Webhook Payload Example

{
  "id": "webhook_019b4132-70aa-764f-b315-e2803d882a24",
  "event": "pet.adopted",
  "timestamp": "2026-03-13T10:30:00Z",
  "data": {
    "petId": "019b4132-70aa-764f-b315-e2803d882a24",
    "userId": "user-456",
    "orderId": "order-789",
    "adoptionDate": "2026-03-13"
  },
  "apiVersion": "v1"
}

Retry Configuration

• Max attempts: 10
• Backoff: Exponential (1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s)
• Total retry window: ~17 minutes
• Dead letter queue after max retries

Security

• HMAC-SHA256 signature in X-Webhook-Signature header
• Timestamp validation (reject > 5 minutes old)
• HTTPS required for webhook URLs

---

Testing Webhooks with Apidog

Apidog streamlines webhook testing.

Test Webhook Delivery

1. Create a mock webhook endpoint in Apidog
2. Register the endpoint with PetstoreAPI
3. Trigger an event (e.g., adopt a pet)
4. Verify webhook receipt in Apidog
5. Check payload structure

Test Signature Verification

// Apidog test script
const signature = pm.request.headers.get('X-Webhook-Signature');
const payload = pm.request.body.raw;
const secret = pm.environment.get('WEBHOOK_SECRET');

const expected = generateSignature(payload, secret);
pm.test('Signature valid', () => {
  pm.expect(signature).to.equal(`sha256=${expected}`);
});

Test Retry Logic

1. Return a 500 error from the mock endpoint
2. Confirm multiple retry attempts with exponential intervals
3. Check that failed deliveries appear in the dead letter queue after max retries

Test Idempotency

1. Receive webhook
2. Return 200
3. Simulate receiving the same webhook again (retry)
4. Confirm no duplicate processing occurs

---

Conclusion

To build reliable, production-grade webhooks:

• Use exponential backoff retry (5-10 attempts)
• Include idempotency keys to prevent duplicates
• Implement HMAC signature verification for security
• Enforce 5-second timeouts
• Always process webhooks asynchronously on the client
• Move failed webhooks to a dead letter queue for review

Modern PetstoreAPI implements all of these patterns. See the webhook documentation for detailed examples.

Test your webhooks with Apidog to verify retry handling, signature validation, and idempotency before going live.

---

FAQ

How many retry attempts should webhooks have?

Use 5–10 attempts with exponential backoff. This covers temporary outages (5–17 minutes) without overwhelming the client.

Should webhooks retry on 4xx errors?

No. 4xx errors indicate client problems (bad URL, authentication failure). Only retry 5xx errors and network failures.

How long should webhook timeouts be?

5 seconds maximum. Clients should return 200 immediately and process the webhook asynchronously.

What if a client never responds to webhooks?

After max retries, move the event to a dead letter queue and alert the client. Consider disabling webhooks for clients with repeated failures.

Should webhook URLs be HTTPS?

Yes, always require HTTPS. HTTP can be intercepted and modified. Modern PetstoreAPI rejects non-HTTPS webhook URLs.

How do you prevent replay attacks?

Include a timestamp in the payload and reject webhooks older than 5 minutes. Always use signature verification.

Can clients request webhook redelivery?

Yes. Modern PetstoreAPI offers a POST /webhooks/{id}/redeliver endpoint for redelivery.

How do you test webhooks locally?

Expose localhost using tools like ngrok, or use Apidog’s mock server to simulate webhook endpoints during development.