Building a high-throughput webhook delivery engine means handling remote failures gracefully. When you deliver millions of webhooks a day to thousands of customer endpoints, naive retry loops, such as one exponential backoff schedule applied to every HTTP error, lead to worker starvation, retry storms and duplicated side effects.
There is no single retry standard to copy. Stripe retries a live-mode event for up to three days with exponential backoff, and test-mode events are retried three times over a few hours. GitHub does not automatically redeliver failed deliveries at all. Shopify documents up to eight retries over roughly four hours and may remove the subscription if failures persist. Provider timeouts also differ: GitHub expects a 2xx within 10 seconds and Shopify within 5. These figures come from provider documentation and may change, so check the current docs before relying on them.
The community Standard Webhooks specification offers non-binding recommendations for senders: treat 2xx as success, retry with exponential backoff and jitter, treat redirects as failures, and stop on 410 Gone. It is not an IETF standard.
This guide covers parsing rate-limit signals, isolating bad events in batches, containing multi-tenant outages, and classifying transient versus permanent failures.
1. Stop Retrying 429s Blindly: Honor Retry-After and Rate-Limit Headers
The Flaw of Blind Exponential Backoff
Standard exponential backoff with a cap calculates delay like this:
Tdelay=min(Tmax,Tbase×2attempt)T_{\text{delay}} = \min\left(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}\right)Tdelay=min(Tmax,Tbase×2attempt)
The widely used “full jitter” variant randomizes the delay so that many failing deliveries do not retry in lockstep:
Tdelay=random(0,min(Tmax,Tbase×2attempt))T_{\text{delay}} = \text{random}\left(0, \min\left(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}\right)\right)Tdelay=random(0,min(Tmax,Tbase×2attempt))
This works well for transient network glitches. Applied blindly to 429 Too Many Requests, it ignores explicit feedback from the receiver. If a receiver says “come back in 120 seconds” and your sender retries at 2s, 4s, 8s and 16s, every one of those attempts is rejected. Depending on the receiver’s limiter, rejected requests may also count against its quota, and they consume your own retry budget either way.
+-------------------+ +-----------------------+
| Webhook Dispatch | | Downstream Receiver |
+-------------------+ +-----------------------+
| |
|-------- Delivery Attempt -------->|
| | (Rate Limit Exceeded)
|<-- 429 [Retry-After: 120] --------|
| |
|-- Naive Retry in 2s (BREACH) ---->| (Rejected: 429)
|-- Naive Retry in 4s (BREACH) ---->| (Rejected: 429)
| |
|-- Header-Aware Retry (Wait 120s)->| (SUCCESS: 200 OK)
Deconstructing Rate-Limit Headers
Receivers signal backoff through standard and vendor-specific headers. Check them in this order:
-
Retry-After(RFC 9110, Section 10.2.3). The429status code itself comes from RFC 6585. The value is either a non-negative integer of seconds or an HTTP-date.- Seconds:
Retry-After: 120 - HTTP-date:
Retry-After: Fri, 27 Nov 2026 07:28:00 GMT - RFC 9110 also allows
Retry-Afteron503 Service Unavailableresponses, so check it there too.
- Seconds:
-
IETF
RateLimitandRateLimit-Policyfields. These are still an Internet-Draft, not an RFC. As ofdraft-ietf-httpapi-ratelimit-headers-11(May 2026) the working group’s draft defines two structured-field headers:-
RateLimit: "default";r=0;t=30, whereris the remaining quota andtis the number of seconds in the effective window. -
RateLimit-Policy: "default";q=100;w=60, which describes the quota policy:qis the quota andwis the window in seconds. - Older drafts (and many blog posts) describe separate
RateLimit-Limit,RateLimit-RemainingandRateLimit-Resetheaders, and aRateLimit-Policyvalue such as100, 100;w=60. The current draft has replaced that syntax, so parse both forms only if you must support legacy senders. - The draft says that if both
RateLimitandRetry-Afterare present,Retry-Aftertakes precedence.
-
-
Legacy vendor headers.
X-RateLimit-Resetis common but its meaning is not consistent. The draft’s own survey notes implementations that send seconds until reset, milliseconds, a Unix timestamp in seconds, or a datetime. Prefer per-provider configuration over guessing. Where you must guess, a magnitude heuristic is a reasonable fallback.
Priority Resolution and Header Parsing
A webhook dispatcher that gets a 429 (or a 503) should run a deterministic header pipeline and clamp the result. Clamping matters because the draft’s security considerations warn that a misconfigured or malicious server can send an enormous value, for example Retry-After: 1000000. Providers cap it in practice. Kit clamps Retry-After between 1 second and 24 hours, and Webhook Relay caps it at one hour.
The code below assumes lowercase header names, as Node.js’s http module provides.
interface RateLimitDelay {
delayMs: number;
source:
| 'Retry-After-Seconds'
| 'Retry-After-Date'
| 'RateLimit'
| 'X-RateLimit-Reset'
| 'Default-Backoff';
}
type Headers = Record<string, string | string[] | undefined>;
/**
* Parses an IETF-draft `RateLimit` field (simplified Structured Fields parsing),
* e.g. `"default";r=0;t=30`. Returns the longest wait among exhausted policies (r=0).
*/
function parseRateLimitWaitSeconds(value: string): number | null {
let wait: number | null = null;
const items = value.matchAll(
/"[^"]*"((?:\s*;\s*[a-z*][a-z0-9_.*-]*(?:=[^;,\s]+)?)*)/g
);
for (const item of items) {
const params = item[1];
const r = /;\s*r=(\d+)/.exec(params);
const t = /;\s*t=(\d+)/.exec(params);
if (r && t && Number(r[1]) === 0) {
wait = Math.max(wait ?? 0, Number(t[1]));
}
}
return wait;
}
export function calculateRateLimitDelay(
headers: Headers,
defaultBackoffMs: number,
nowMs: number = Date.now(),
maxDelayMs: number = 60 * 60 * 1000 // cap untrusted values (1 hour)
): RateLimitDelay {
const getHeader = (name: string): string | undefined => {
const val = headers[name];
return Array.isArray(val) ? val[0] : val;
};
const clamp = (ms: number) => Math.min(Math.max(ms, 0), maxDelayMs);
// 1. Retry-After (RFC 9110): delay-seconds or HTTP-date. Takes precedence.
const retryAfter = getHeader('retry-after')?.trim();
if (retryAfter) {
if (/^\d+$/.test(retryAfter)) {
return { delayMs: clamp(Number(retryAfter) * 1000), source: 'Retry-After-Seconds' };
}
const dateMs = Date.parse(retryAfter);
if (!Number.isNaN(dateMs)) {
return { delayMs: clamp(dateMs - nowMs), source: 'Retry-After-Date' };
}
}
// 2. IETF draft RateLimit field (r = remaining quota, t = seconds in window)
const rateLimit = getHeader('ratelimit');
if (rateLimit) {
const seconds = parseRateLimitWaitSeconds(rateLimit);
if (seconds !== null) {
return { delayMs: clamp(seconds * 1000), source: 'RateLimit' };
}
}
// 3. Legacy X-RateLimit-Reset: epoch ms, epoch seconds, or delta seconds
const xReset = getHeader('x-ratelimit-reset')?.trim();
if (xReset && /^\d+$/.test(xReset)) {
const n = Number(xReset);
if (n > 1e12) return { delayMs: clamp(n - nowMs), source: 'X-RateLimit-Reset' };
if (n > 1e9) return { delayMs: clamp(n * 1000 - nowMs), source: 'X-RateLimit-Reset' };
return { delayMs: clamp(n * 1000), source: 'X-RateLimit-Reset' };
}
return { delayMs: defaultBackoffMs, source: 'Default-Backoff' };
}
When the server tells every client to return at the same instant, you get a thundering herd. The IETF draft warns that this applies to Retry-After too and suggests adding jitter to the window. A good rule is to add random positive jitter on top of the server’s value, never subtract from it.
Integrating Dynamic Cooldowns into Scheduling Queues
Once you have the delay, the task should not occupy a worker. Requeue it in a delayed-scheduling structure:
-
Redis sorted sets. Store each task with a score of
now_ms + delayMs. Workers poll for due tasks withZRANGE retries -inf <now_ms> BYSCORE LIMIT 0 100. The olderZRANGEBYSCOREcommand has been deprecated since Redis 6.2 in favor ofZRANGE ... BYSCORE. Claim each member withZREMand process it only ifZREMreturns1, so two workers never take the same task. A worker that crashes after claiming a task loses it unless you add a lease or visibility-timeout pattern. -
Amazon SQS. The delay is limited to 15 minutes (900 seconds), set with
DelaySecondson the queue or per message. FIFO queues do not support per-message timers, andMessageGroupIdis a FIFO concept unrelated to delay. AWS recommends EventBridge Scheduler for delays beyond 15 minutes. -
RabbitMQ. The
rabbitmq_delayed_message_exchangeplugin (exchange typex-delayed-message, delay in thex-delayheader, in milliseconds) is a community plugin that does not ship with RabbitMQ. Its repository says it is no longer actively maintained and describes limitations, including delayed messages being held in a Mnesia table with a single disk replica. Evaluate it carefully for new deployments. A common alternative is a set of TTL queues, one per delay tier, that dead-letter into the work queue.
2. Partial Batch Failures: Isolating Bad Events Inside Multi-Event Payloads
To lower connection overhead and raise throughput, dispatchers often combine multiple events for the same endpoint into one request:
{
"batch_id": "btc_9081237",
"events": [
{ "id": "evt_001", "type": "order.created", "payload": { "order_id": 101 } },
{ "id": "evt_002", "type": "order.updated", "payload": { "order_id": 102 } },
{ "id": "evt_003", "type": "user.deleted", "payload": { "user_id": null } }
]
}
The All-or-Nothing Problem
Suppose evt_003 carries data that triggers an unhandled exception on the customer’s server, and the server answers the whole batch with HTTP 500. If the dispatcher treats that as a single binary outcome and retries the entire payload:
-
evt_001andevt_002are delivered again, producing duplicate side effects unless the receiver deduplicates. -
evt_003keeps failing, so its healthy siblings are stuck in the retry loop until the batch exhausts its retry limit.
Real systems already behave this way. Kit’s documentation states that a retry re-sends the delivery’s entire event array under the same delivery_id, and it tells receivers to deduplicate on each event’s UUID id.
UNHANDLED BATCH RETRY (BAD)
+-------------------------------------------------------+
| Batch [evt_001 (OK), evt_002 (OK), evt_003 (FAIL)] |
+-------------------------------------------------------+
|
Delivery Try #1 -> HTTP 500
|
Delivery Try #2 -> HTTP 500
(evt_001 and evt_002 re-executed, causing duplication!)
Mitigation Strategies
Pattern A: Receiver-Side Inbox and Deduplication Keys
Every event carries a stable unique id, and the receiver records it in an inbox table with a unique constraint. Processing each event in its own transaction (or savepoint) lets one failure leave the others committed.
-- PostgreSQL / SQLite syntax
BEGIN;
INSERT INTO inbox_events (event_id, received_at)
VALUES ('evt_001', NOW())
ON CONFLICT (event_id) DO NOTHING;
-- If 0 rows were inserted, the event was already processed: skip it.
-- Otherwise apply the business logic here, in the SAME transaction.
COMMIT;
Recording the ID and applying the side effect in one transaction is what makes the pattern safe. If you commit the inbox row first and the side effect fails afterward, the event is silently lost on retry.
The sender’s job is to keep the same event ID across every attempt and to treat delivery as at-least-once.
Pattern B: Itemized Partial-Success Responses
Some APIs return a per-event result map in the response body. HTTP 207 Multi-Status is sometimes borrowed for this. It originates in WebDAV (RFC 4918), and no webhook standard defines it for batches, so this is a contract you agree with your receivers rather than something to assume. A 207 is a 2xx, which most senders treat as plain success, so your dispatcher must explicitly parse the body or failed items will be silently counted as delivered.
HTTP/1.1 207 Multi-Status
Content-Type: application/json
{
"batch_id": "btc_9081237",
"summary": { "succeeded": 2, "failed": 1 },
"results": [
{ "id": "evt_001", "status": 200 },
{ "id": "evt_002", "status": 200 },
{ "id": "evt_003", "status": 422, "error": "Invalid user_id null payload" }
]
}
When the dispatcher receives an itemized response:
- It marks
evt_001andevt_002asDELIVERED. - It retries
evt_003alone if its status is retryable, or sends it straight to the dead-letter queue (DLQ) if it is not (such as422).
Pattern C: Dispatcher-Side Atomization
For endpoints that return no itemized results, fall back to splitting the batch:
interface WebhookEvent {
id: string;
type: string;
payload: Record<string, unknown>;
}
interface BatchDeliveryPayload {
batchId: string;
events: WebhookEvent[];
}
interface ErrorResponse {
statusCode: number;
body: string;
}
async function handleBatchDeliveryFailure(
batch: BatchDeliveryPayload,
error: ErrorResponse,
enqueueSingleEvent: (event: WebhookEvent, delayMs: number) => Promise<void>,
processSingleEventFailure: (event: WebhookEvent, error: ErrorResponse) => Promise<void>
): Promise<void> {
if (batch.events.length > 1) {
console.warn(
`Batch ${batch.batchId} failed with HTTP ${error.statusCode}. ` +
`Splitting ${batch.events.length} events into single-event deliveries.`
);
for (const event of batch.events) {
const jitterMs = Math.floor(Math.random() * 500);
await enqueueSingleEvent(event, jitterMs);
}
} else {
// A single event failed: run the normal classification path (see Section 4).
await processSingleEventFailure(batch.events[0], error);
}
}
Atomization multiplies request volume. If the original failure was an outage rather than a poisoned event, splitting a 100-event batch sends 100 requests to an endpoint that is already struggling. Only atomize on failures that plausibly point to payload content. Let 429, 502, 503 and 504 follow the normal retry path, and route the split single-event deliveries through the per-endpoint rate limiter described in Section 3. A bisection strategy (splitting into halves repeatedly) is a lower-amplification variant.
3. Multi-Tenant Retry Storm Containment: Token Buckets vs. Circuit Breakers
In multi-tenant systems, thousands of customer webhooks share queue infrastructure and worker pools. If Tenant A’s receiver goes down, its backlog generates a flood of failed attempts. Without isolation, those retries saturate the shared consumers and delay deliveries for healthy tenants. This is a form of head-of-line blocking, often called worker starvation.
SHARED WORKER POOL (WITHOUT CONTAINMENT)
[ Tenant A Retry ] [ Tenant A Retry ] [ Tenant A Retry ] [ Tenant A Retry ]
^ (pool saturated by one tenant)
[ Tenant B Normal ] -- STARVED IN QUEUE
Containment Mechanics
| Mechanism | Operating Layer | Primary Objective | Pros | Cons |
|---|---|---|---|---|
| Circuit breaker | Per endpoint or domain | Stop sending to a destination that is clearly down. | Saves worker time and network connections. | Thresholds can misfire; breaking per domain can block healthy endpoints that share a host. |
| Per-endpoint outbound token bucket | Dispatcher rate limiter | Cap delivery velocity per destination. | Prevents overwhelming a receiver, including during recovery. | Needs shared state (e.g., Redis) across workers. |
| Separate retry and real-time queues | Queue infrastructure | Keep retry traffic from delaying first-attempt deliveries. | Protects fresh-traffic latency regardless of retry volume. | More queues to operate and monitor. |
Per-Endpoint Token Buckets
Before a worker calls https://api.customer-a.com/webhook, it must take a token from the bucket for that destination. Tokens refill continuously over the elapsed interval Δt\Delta t
Δt:
Tavailable=min(Bcapacity,Tcurrent+Δt×Rfill)T_{\text{available}} = \min\left(B_{\text{capacity}}, T_{\text{current}} + \Delta t \times R_{\text{fill}}\right)Tavailable=min(Bcapacity,Tcurrent+Δt×Rfill)
- BcapacityB_{\text{capacity}} Bcapacity is the maximum burst size.
- RfillR_{\text{fill}} Rfill is the refill rate in tokens per second.
An atomic Lua script keeps the read-modify-write safe across many concurrent workers. This version reads the clock from Redis itself (TIME, safe in scripts on Redis 5.0+) so worker clock skew cannot corrupt the bucket. It also uses HSET, because HMSET has been deprecated since Redis 4.0. On denial it returns how long to wait, so the task can be rescheduled precisely instead of spinning.
-- KEYS[1]: bucket key, e.g. "ratelimit:endpoint:api.customer-a.com"
-- ARGV[1]: capacity (max burst), e.g. 20
-- ARGV[2]: refill rate in tokens per second, e.g. 5
-- ARGV[3]: tokens requested, e.g. 1
-- Returns: { allowed (1 or 0), retry_after_ms }
local capacity = tonumber(ARGV[1])
local fill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])
local t = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000
local bucket = redis.call("HMGET", KEYS[1], "tokens", "last_updated")
local tokens = tonumber(bucket[1])
local last_updated = tonumber(bucket[2])
if tokens == nil then
tokens = capacity
else
tokens = math.min(capacity, tokens + math.max(0, now - last_updated) * fill_rate)
end
local allowed = 0
local retry_after_ms = 0
if tokens >= requested then
tokens = tokens - requested
allowed = 1
else
retry_after_ms = math.ceil((requested - tokens) / fill_rate * 1000)
end
redis.call("HSET", KEYS[1], "tokens", tokens, "last_updated", now)
redis.call("EXPIRE", KEYS[1], math.max(1, math.ceil(capacity / fill_rate) * 2))
return { allowed, retry_after_ms }
If a receiver publishes its own limits through Retry-After or the IETF RateLimit fields, feed that information back into the bucket so your outbound rate tracks what the receiver says it can handle.
Circuit Breakers for Endpoint Cooldowns
When an endpoint fails many consecutive requests, or returns repeated 5xx responses or timeouts within a window, a circuit breaker trips to OPEN.
failure threshold reached
+----------+ ---------------------------> +----------+
| CLOSED | | OPEN |
+----------+ <--+ +----------+
^ | |
| | probe fails | cooldown elapsed
| | +-------------+ |
| +----- | HALF-OPEN | <--------+
| +-------------+
| probe succeeds |
+----------------------+
- CLOSED. Deliveries flow normally while failures are counted over a rolling window.
- OPEN. Live requests to the endpoint are suppressed. Events are not dropped; they are parked in a delayed queue so no worker time or connections are spent on a dead destination.
-
HALF-OPEN. After a cooldown (for example, several minutes), a limited number of probe requests are allowed. If they succeed, the breaker returns to
CLOSED. If they fail, it returns toOPENand the cooldown restarts.
Pair the breaker with a longer-term health policy. Stripe, for example, emails account owners when an endpoint has not returned a 2xx for multiple days and discloses when it will be automatically disabled. Always pair auto-disabling with notification and a way to re-enable and replay.
4. Deterministic Error Classification: Transient vs. Permanent
A common weakness in failure handling is putting every non-2xx response and network error on the same retry schedule.
Retrying a 401 Unauthorized response ten times over a day consumes worker capacity and fills your logs, and an invalid credential is unlikely to fix itself. But “permanent” is a policy decision, not a fact, and providers differ. Stripe retries any non-2xx response, while Standard Webhooks-style senders such as Kit stop immediately only on 410 Gone. A receiver that rolls out a fix or rotates a secret can recover within hours.
A practical middle path is to stop automatic retries early for errors that need human action, but never discard the payload. Quarantine it, notify the customer and allow replay after the fix.
OUTBOUND WEBHOOK RESPONSE
|
+--------------------+--------------------+
| |
HTTP Status Code / Network Response Body
| |
+----------+----------+ +----------+----------+
| | | |
Success (2xx) Failure (non-2xx) Structured error Unstructured
| | (RFC 9457 JSON) text
[ Delivered ] +---------+---------+ | |
| | [ Match known [ Don't guess:
Transient Permanent error codes ] use status ]
(408/429/5xx/network) (most other 4xx)
| |
[ Retry queue ] [ Quarantine / DLQ ]
Classification Matrix
| Category | Trigger | Action |
|---|---|---|
| Transient | 429 Too Many Requests |
Retry, honoring Retry-After / RateLimit
|
| Transient | 503 Service Unavailable |
Retry, honoring Retry-After if present |
| Transient |
500, 502, 504, 408
|
Exponential backoff with jitter |
| Transient |
ETIMEDOUT, ECONNRESET, EAI_AGAIN (temporary DNS failure) |
Backoff with jitter |
| Ambiguous |
ENOTFOUND (DNS name did not resolve) |
Limited retries with long delays, then escalate |
| Ambiguous | 404 Not Found |
Limited retries, then disable the endpoint after sustained failure |
| Permanent | 410 Gone |
Disable the endpoint; no retries (the receiver is explicitly asking you to stop) |
| Permanent |
400, 422, 415
|
Quarantine |
| Permanent |
401, 403
|
Quarantine and notify the customer |
| Permanent | 413 Content Too Large |
Quarantine, or split the batch and resend |
| Permanent | Certificate errors (expired, self-signed, hostname mismatch) | Quarantine and notify |
Response-Body Heuristics (RFC 9457 Problem Details)
Status codes don’t tell the whole story. Many APIs return 4xx or 5xx with a structured JSON body. The standard format is RFC 9457, Problem Details for HTTP APIs, which obsoleted RFC 7807 in 2023. Problem documents can carry extension members such as an application error code:
{
"type": "https://api.example.com/errors/invalid-signature",
"title": "Invalid Webhook HMAC Signature",
"status": 400,
"detail": "The calculated HMAC-SHA256 signature did not match the signature header.",
"code": "ERR_HMAC_MISMATCH"
}
If a customer’s endpoint reports a signature mismatch, retrying will not help until the secret is corrected. Before blaming the customer, rule out sender-side causes such as an in-progress secret rotation. Sign each attempt freshly, with a new timestamp and the currently valid secrets, as Kit’s retry behavior does.
The IETF rate-limit draft also defines problem types registered under https://iana.org/assignments/http-problem-types: quota-exceeded and abnormal-usage-detected (both recommended with 429), and temporary-reduced-capacity (recommended with 503). Receivers that adopt them give you machine-readable transient signals.
Match on parsed, structured fields (type, code) against an explicit allowlist. Avoid substring searches over free-form text. A phrase like unauthorized appearing in a 5xx page is not a reliable permanent-failure signal.
Deterministic Failure Classifier
import { calculateRateLimitDelay } from './rate-limit'; // from Section 1
export enum FailureAction {
RETRY_HEADER_AWARE = 'RETRY_HEADER_AWARE',
RETRY_EXPONENTIAL = 'RETRY_EXPONENTIAL',
QUARANTINE_PERMANENT = 'QUARANTINE_PERMANENT',
DISABLE_ENDPOINT = 'DISABLE_ENDPOINT',
}
export interface ClassificationResult {
action: FailureAction;
reason: string;
suggestedDelayMs?: number;
}
type Headers = Record<string, string | string[] | undefined>;
export class WebhookErrorClassifier {
// Codes that your integration contract defines as unrecoverable.
private static PERMANENT_ERROR_CODES = new Set([
'ERR_HMAC_MISMATCH',
'ERR_UNKNOWN_EVENT_TYPE',
'ERR_SCHEMA_VALIDATION',
]);
private static PERMANENT_STATUS_CODES = new Set([400, 401, 403, 413, 415, 422]);
private static TRANSIENT_STATUS_CODES = new Set([408, 500, 502, 503, 504]);
private static PERMANENT_NETWORK_CODES = new Set([
'DEPTH_ZERO_SELF_SIGNED_CERT',
'CERT_HAS_EXPIRED',
'UNABLE_TO_VERIFY_LEAF_SIGNATURE',
'ERR_TLS_CERT_ALTNAME_INVALID',
]);
private static extractErrorCode(body: string | null): string | null {
if (!body) return null;
try {
const parsed = JSON.parse(body);
return typeof parsed?.code === 'string' ? parsed.code : null;
} catch {
return null; // not JSON: do not guess from free text
}
}
public static classify(
statusCode: number | null,
body: string | null,
networkErrorCode: string | null,
headers: Headers,
defaultBackoffMs = 30_000
): ClassificationResult {
// 1. Low-level network faults
if (networkErrorCode) {
if (this.PERMANENT_NETWORK_CODES.has(networkErrorCode)) {
return {
action: FailureAction.QUARANTINE_PERMANENT,
reason: `TLS certificate validation failed: ${networkErrorCode}`,
};
}
return {
action: FailureAction.RETRY_EXPONENTIAL,
reason: `Network fault: ${networkErrorCode}`, // ECONNRESET, ETIMEDOUT, EAI_AGAIN, ENOTFOUND...
};
}
// 2. Explicit "stop sending" signal
if (statusCode === 410) {
return {
action: FailureAction.DISABLE_ENDPOINT,
reason: 'Receiver returned 410 Gone: endpoint no longer wants webhooks.',
};
}
// 3. Throttling and unavailability: honor the receiver's own timing
if (statusCode === 429 || (statusCode === 503 && headers['retry-after'])) {
const { delayMs, source } = calculateRateLimitDelay(headers, defaultBackoffMs);
return {
action: FailureAction.RETRY_HEADER_AWARE,
reason: `HTTP ${statusCode}; delay from ${source}.`,
suggestedDelayMs: delayMs,
};
}
// 4. Structured body markers (any status): explicit unrecoverable codes
const code = this.extractErrorCode(body);
if (code && this.PERMANENT_ERROR_CODES.has(code)) {
return {
action: FailureAction.QUARANTINE_PERMANENT,
reason: `Unrecoverable error code in response body: ${code}`,
};
}
// 5. Permanent status codes
if (statusCode !== null && this.PERMANENT_STATUS_CODES.has(statusCode)) {
return {
action: FailureAction.QUARANTINE_PERMANENT,
reason: `Terminal HTTP status ${statusCode}.`,
};
}
// 6. Transient server errors, and 404 (ambiguous: a route may be mid-deploy).
// Let endpoint-health tracking disable endpoints with sustained 404s.
if (statusCode !== null && (this.TRANSIENT_STATUS_CODES.has(statusCode) || statusCode === 404)) {
return {
action: FailureAction.RETRY_EXPONENTIAL,
reason: `Retryable status ${statusCode}.`,
};
}
// Default for unmapped responses (including unexpected 3xx): bounded retry
return {
action: FailureAction.RETRY_EXPONENTIAL,
reason: `Unclassified response (status: ${statusCode}). Defaulting to bounded retry.`,
};
}
}
Note that Standard Webhooks recommends treating redirects as failures rather than following them, since a redirect can break signatures and open SSRF paths. The default branch above handles that case.
Dead-Letter Quarantine (DLQ) and Remediation
When an event is marked QUARANTINE_PERMANENT:
- Payload vaulting. Store the full delivery context (request headers, raw body, response status, response body and failure trace) in a searchable store such as PostgreSQL with JSONB or Elasticsearch.
- Customer alerting. Send an in-app alert or email digest showing the endpoint errors, with a link to the failed deliveries.
- Manual and bulk replay. Offer an API and UI to re-dispatch single events or batches once the customer has fixed the problem. Replays must reuse the original event IDs so receivers can deduplicate:
POST /api/v1/webhooks/quarantine/replay
Content-Type: application/json
{
"quarantine_ids": ["q_8801", "q_8802", "q_8803"],
"override_endpoint_url": "https://api.customer.com/v2/webhook"
}
Stripe and GitHub both offer bounded manual resend or redelivery windows, which is the same idea. Because a replay can create duplicates, receiver-side idempotency (Section 2, Pattern A) remains essential.
Complete Architectural Checklist for Production Webhook Systems
-
Header parsing: Honor
Retry-After(on429and503) first, then the IETFRateLimitfield (r/tparameters), then legacy vendor headers, before falling back to computed backoff. Clamp every server-supplied delay to a sane maximum. - Jitter: Apply full jitter to exponential backoff, and add positive jitter on top of server-specified waits, to avoid synchronized retry spikes.
- Stable event IDs: Keep the same event ID across every attempt and replay, and re-sign each attempt with a fresh timestamp.
- Batch isolation: Support itemized partial-success responses where receivers opt in, and fall back to rate-limited atomization or bisection when a batched request fails.
- Tenant isolation: Enforce per-endpoint token buckets in shared state, and keep retry traffic separate from first-attempt traffic.
- Circuit breakers: Use the full CLOSED, OPEN and HALF-OPEN cycle, park events instead of dropping them, and notify customers before auto-disabling an endpoint.
-
Deterministic classification: Combine network codes, status codes and structured error bodies. Treat
410 Goneas the explicit stop signal, and be deliberate about which4xxcodes you stop retrying. - Operational tooling: Provide DLQ inspection, failure alerts, and replay APIs, and document your retry policy for customers (timeouts, retry window, backoff, and what happens on exhaustion).
Originally published at https://instawebhook.com/blog/webhook-engineering-retries-dynamic-backoff-and-multi-tenant-failure-recovery
Top comments (0)