Preecha

Posted on Aug 30

How Should You Design Reliable Webhooks?

Designing Reliable Webhooks with Modern PetstoreAPI

TL;DR

Design reliable webhooks with exponential backoff retries, idempotency keys, HMAC signature verification, and 5-second timeouts. Return a 2xx response as soon as the event is durably accepted, then process it asynchronously.

Modern PetstoreAPI implements webhooks for order updates, pet adoptions, and payment notifications with retry and security controls.

Try Apidog today

Introduction

You send a webhook to notify a client that their pet was adopted. The client’s server is down, so delivery fails. Should you retry? How many times? What happens if the client receives the webhook twice and charges the customer twice?

Webhooks are HTTP callbacks that push events to client-provided URLs. They are simple in theory but complex in production. Networks fail, servers crash, and client applications may process a request successfully before losing the response.

A production-ready webhook system needs:

Retry logic
Idempotency
Signature verification
Short timeouts
Asynchronous processing
Monitoring and failure recovery

Modern PetstoreAPI implements these patterns for order updates, pet adoptions, and payment notifications. Every webhook includes retry logic, signature verification, and an idempotency identifier.

In this guide, you’ll learn how to design reliable webhooks using Modern PetstoreAPI patterns.

Webhook Basics

A webhook is an HTTP POST request sent to a client URL when an event occurs.

How Webhooks Work

1. The client registers a webhook URL

POST /webhooks
Content-Type: application/json

{
  "url": "https://client.com/webhooks/petstore",
  "events": [
    "pet.adopted",
    "order.completed"
  ]
}

2. An event occurs

For example, a pet is adopted.

3. The server sends the webhook

POST https://client.com/webhooks/petstore
Content-Type: application/json
X-Webhook-Signature: sha256=abc123...

{
  "id": "webhook_019b4132",
  "event": "pet.adopted",
  "timestamp": "2026-03-13T10:30:00Z",
  "data": {
    "petId": "019b4132",
    "userId": "user-456"
  }
}

4. The client acknowledges receipt

HTTP/1.1 200 OK

The Reliability Problem

Webhook delivery can fail when:

The client server is unavailable
The network connection times out
The client returns a 500-series error
The client takes too long to respond
The client processes the request but crashes before sending a response

Without retry logic, events can be lost. Without idempotency, duplicate deliveries can trigger duplicate actions.

Retry Logic with Exponential Backoff

Retry failed webhook deliveries with progressively longer delays.

Exponential Backoff Strategy

A typical schedule looks like this:

Attempt	Delay
1	Immediate
2	1 second
3	2 seconds
4	4 seconds
5	8 seconds
6	16 seconds

If the client is temporarily unavailable, immediately sending repeated requests can increase the load on an already unhealthy system. Exponential backoff gives the client time to recover.

Implementing Retries

The following example retries network failures and 5xx responses. It does not retry 4xx responses.

const sleep = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function sendWebhook(
  url,
  payload,
  attempt = 1,
  maxAttempts = 6
) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), 5000);

  try {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "X-Webhook-Signature": generateSignature(payload)
      },
      body: JSON.stringify(payload),
      signal: controller.signal
    });

    if (response.ok) {
      return {
        success: true,
        attempt
      };
    }

    // Retry only server errors.
    if (response.status >= 500 && attempt < maxAttempts) {
      const delay = 2 ** (attempt - 1) * 1000;

      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }

    // A 4xx response usually requires client configuration changes.
    return {
      success: false,
      status: response.status,
      attempt
    };
  } catch (error) {
    // Network errors and timeouts are retryable.
    if (attempt < maxAttempts) {
      const delay = 2 ** (attempt - 1) * 1000;

      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }

    return {
      success: false,
      error: error.message,
      attempt
    };
  } finally {
    clearTimeout(timeout);
  }
}

The exact timeout API depends on the HTTP client you use. In the native fetch API, use AbortController as shown above rather than relying on a non-standard timeout option.

When to Retry

Retry on:

5xx server errors, including 500, 502, 503, and 504
Network timeouts
Connection refused errors
DNS failures

Do not retry on:

2xx responses, because delivery succeeded
4xx errors such as 400, 401, or 404, because the client configuration must usually be fixed first

Dead-Letter Queues

After the maximum number of attempts, move the failed webhook to a dead-letter queue for investigation or later redelivery.

const result = await sendWebhook(webhook.url, webhook.payload);

if (!result.success) {
  await deadLetterQueue.add({
    url: webhook.url,
    payload: webhook.payload,
    attempts: result.attempt,
    lastError: result.error || `HTTP ${result.status}`,
    timestamp: new Date()
  });
}

A dead-letter queue prevents permanent failures from disappearing silently. It also gives your operations team a place to inspect failed requests and retry them after correcting the client configuration.

Idempotency for Duplicate Prevention

A client can receive the same webhook more than once. This is expected when a request succeeds but the response is lost, causing the sender to retry.

Idempotency ensures that processing the same event repeatedly has the same effect as processing it once.

Add an Idempotency Key

Include a unique identifier in every webhook payload:

{
  "id": "webhook_019b4132",
  "event": "pet.adopted",
  "data": {
    "petId": "019b4132",
    "userId": "user-456"
  }
}

The client can store processed webhook IDs:

app.post("/webhooks/petstore", async (req, res) => {
  const webhookId = req.body.id;

  const processed = await db.webhooks.findOne({
    id: webhookId
  });

  if (processed) {
    return res.status(200).json({
      message: "Already processed"
    });
  }

  await processPetAdoption(req.body.data);

  await db.webhooks.insert({
    id: webhookId,
    processedAt: new Date()
  });

  return res.status(200).json({
    message: "Processed"
  });
});

Create a unique database constraint on the webhook ID. A simple “check, then insert” sequence can still process an event twice if two duplicate requests arrive concurrently.

Make Operations Idempotent

The operation itself should also tolerate retries.

Not idempotent:

// A retry can charge the customer twice.
await chargeCustomer(userId, amount);

Idempotent:

// The payment provider uses the webhook ID to deduplicate the charge.
await chargeCustomer(userId, amount, {
  idempotencyKey: webhookId
});

Use the webhook ID, event ID, or another stable identifier as the idempotency key. Do not generate a new key for every retry.

Signature Verification for Security

A webhook endpoint is publicly reachable, so it must verify that requests came from your API rather than an attacker.

Generate an HMAC Signature

The sender can generate an HMAC-SHA256 signature using a shared [REDACTED CREDENTIAL]
const crypto = require("crypto");

function generateSignature(payload, secret) {
const body = JSON.stringify(payload);

return crypto
.createHmac("sha256", secret)
.update(body)
.digest("hex");
}

const signature = generateSignature(payload, webhookSecret);

const headers = {
"Content-Type": "application/json",
"X-Webhook-Signature": sha256=${signature}
};


The receiver calculates the expected signature and compares it using a constant-time comparison:

js
function verifySignature(payload, signature, secret) {
const expected = sha256=${generateSignature(payload, secret)};

const receivedBuffer = Buffer.from(signature || "");
const expectedBuffer = Buffer.from(expected);

if (receivedBuffer.length !== expectedBuffer.length) {
return false;
}

return crypto.timingSafeEqual(receivedBuffer, expectedBuffer);
}


The sender and receiver must sign exactly the same bytes. For JSON requests, verify the raw request body before parsing it because whitespace or property-order changes can produce a different signature.

js
app.post(
"/webhooks/petstore",
express.raw({ type: "application/json" }),
async (req, res) => {
const signature = req.headers["x-webhook-signature"];
const rawBody = req.body.toString("utf8");

const valid = verifyRawSignature(
  rawBody,
  signature,
  process.env.WEBHOOK_SECRET
);

if (!valid) {
  return res.status(401).json({
    error: "Invalid signature"
  });
}

const payload = JSON.parse(rawBody);

await webhookQueue.add("process-webhook", payload);

return res.status(200).json({
  message: "Received"
});

}
);


### Validate the Timestamp

Include a timestamp in the payload:

json
{
"id": "webhook_019b4132",
"timestamp": "2026-03-13T10:30:00Z",
"event": "pet.adopted",
"data": {
"petId": "019b4132"
}
}


Reject old requests to reduce the risk of replay attacks:

js
const webhookAge =
Date.now() - new Date(req.body.timestamp).getTime();

if (webhookAge > 5 * 60 * 1000) {
return res.status(400).json({
error: "Webhook too old"
});
}


Use timestamp validation together with HMAC verification. A timestamp alone is not a security mechanism because an attacker could modify it without a valid signature.

## Timeout Handling

Set a short timeout so a slow client does not block your delivery workers.

### Use a Five-Second Timeout

js
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 5000);

try {
await fetch(url, {
method: "POST",
body: JSON.stringify(payload),
signal: controller.signal
});
} finally {
clearTimeout(timeout);
}


Five seconds is a practical maximum for webhook acknowledgement. The receiving application should acknowledge the request quickly and perform expensive work asynchronously.

### Process Webhooks Asynchronously

A synchronous handler keeps the sender waiting while it performs every downstream operation:

js
// Slow synchronous processing
app.post("/webhooks/petstore", async (req, res) => {
await processOrder(req.body.data);
await sendEmail(req.body.data);
await updateInventory(req.body.data);

return res.status(200).json({
message: "Processed"
});
});


Instead, enqueue the event and acknowledge it after the queue accepts it:

js
app.post("/webhooks/petstore", async (req, res) => {
await queue.add("process-webhook", req.body);

return res.status(200).json({
message: "Received"
});
});


The receiver should return a 2xx response once the event is durably accepted. If queue insertion fails, return an error so the sender can retry.

## How Modern PetstoreAPI Implements Webhooks

Modern PetstoreAPI provides production-ready webhooks for several event types.

### Webhook Events

- `pet.adopted` — A pet was adopted
- `pet.status_changed` — A pet’s status changed
- `order.created` — An order was created
- `order.completed` — An order was completed
- `payment.succeeded` — A payment succeeded
- `payment.failed` — A payment failed

### Webhook Payload

json
{
"id": "webhook_019b4132-70aa-764f-b315-e2803d882a24",
"event": "pet.adopted",
"timestamp": "2026-03-13T10:30:00Z",
"data": {
"petId": "019b4132-70aa-764f-b315-e2803d882a24",
"userId": "user-456",
"orderId": "order-789",
"adoptionDate": "2026-03-13"
},
"apiVersion": "v1"
}


### Retry Configuration

Modern PetstoreAPI uses:

- Maximum attempts: 10
- Backoff: Exponential
- Delays: `1s`, `2s`, `4s`, `8s`, `16s`, `32s`, `64s`, `128s`, `256s`, and `512s`
- Total retry window: approximately 17 minutes
- Dead-letter queue after the maximum number of retries

### Security Controls

- HMAC-SHA256 signatures in the `X-Webhook-Signature` header
- Timestamp validation that rejects requests more than 5 minutes old
- HTTPS required for webhook URLs

## Testing Webhooks with Apidog

Apidog can help you test delivery, signatures, retries, and duplicate handling.

### Test Webhook Delivery

1. Create a mock webhook endpoint in Apidog.
2. Register the endpoint with PetstoreAPI.
3. Trigger an event, such as adopting a pet.
4. Verify that the mock endpoint receives the request.
5. Check the event ID, payload format, timestamp, and headers.

### Test Signature Verification

Use the exact raw request body when calculating the expected signature:

js
// Apidog test script
const signature = pm.request.headers.get("X-Webhook-Signature");
const payload = pm.request.body.raw;
const secret = pm.environment.get("WEBHOOK_SECRET");

const digest = CryptoJS.HmacSHA256(payload, secret)
.toString(CryptoJS.enc.Hex);

pm.test("Signature is valid", () => {
pm.expect(signature).to.equal(sha256=${digest});
});


### Test Retry Logic

1. Configure the mock endpoint to return a `500` response.
2. Trigger a webhook event.
3. Verify that the sender retries the request.
4. Compare the timestamps between attempts to confirm exponential backoff.
5. Check the dead-letter queue after the maximum number of attempts.

### Test Idempotency

1. Receive a webhook.
2. Return `200 OK`.
3. Send the same webhook again to simulate a retry.
4. Verify that the client does not process the event twice.
5. Confirm that the second request returns a successful duplicate response.

## Conclusion

Reliable webhooks require more than an HTTP `POST`. Use:

- Exponential backoff with 5–10 attempts
- Idempotency keys to prevent duplicate processing
- HMAC signature verification
- Timestamp validation
- Five-second delivery timeouts
- Asynchronous processing on the client side
- A dead-letter queue for failed deliveries

Modern PetstoreAPI implements these patterns for pet, order, and payment events. Check the webhook documentation for complete examples.

Test webhook delivery, retry behavior, signatures, and idempotency with Apidog before sending events to production.

## FAQ

### How many retry attempts should webhooks have?

Use 5–10 attempts with exponential backoff. This covers temporary outages over a retry window of roughly 5–17 minutes without continuously overwhelming the client.

### Should webhooks retry on 4xx errors?

Usually, no. A `4xx` response generally indicates a client configuration problem, such as a bad URL or authentication failure. Retrying will not fix it.

Retry `5xx` responses and network failures instead.

### How long should webhook timeouts be?

Use a maximum timeout of approximately 5 seconds. The receiving application should return a 2xx response quickly and process the event asynchronously.

### What if a client never responds to webhooks?

Move the event to a dead-letter queue after the maximum number of retries. Alert the client and consider disabling the webhook after repeated failures.

### Should webhook URLs use HTTPS?

Yes. HTTPS protects webhook requests from being intercepted or modified in transit. Modern PetstoreAPI rejects HTTP webhook URLs.

### How do you prevent replay attacks?

Include a timestamp in the payload and reject webhooks older than 5 minutes. Combine timestamp validation with HMAC signature verification.

### Can clients request webhook redelivery?

Yes. Modern PetstoreAPI provides an endpoint to redeliver a specific webhook:

http
POST /webhooks/{id}/redeliver




### How do you test webhooks locally?

Use a tool such as ngrok to expose a local endpoint to the internet, or use an Apidog mock server to simulate webhook delivery during development.

DEV Community