Retry failed webhooks with exponential backoff, a finite attempt budget, and a circuit breaker. For a B2B SaaS notification service, this keeps one bad receiver from becoming an alert storm and leaves a clean rollback path for the sending release.
| Option | Pick this when | Boundary |
|---|---|---|
| Node.js worker with durable retry state | You own delivery and rollback safety matters most | You build state, deduplication, and controls |
| Sentry | The first question is which application error group failed | It does not replace a delivery queue |
| Datadog | Your team already centralizes logs, metrics, and alerts | The worker still owns retry semantics |
| Honeycomb | You need high-cardinality incident investigation | Querying events is not delivery orchestration |
How should we handle retries for failed webhook delivery?
Treat delivery as a state machine: pending -> delivered, retryable, or dead_letter. A retryable response schedules a later attempt. A permanent response goes to dead_letter under the receiver policy. Do not hide every failure behind retrying; operators need to know if a rollback can fix the sender or if the destination is rejecting it.
The diagram in words is: producer -> delivery record -> attempt -> backoff queue -> attempt; too many failures -> open circuit -> operator decision. The circuit belongs to a destination, while each delivery keeps its own attempt count. Rolling back the producer must not erase evidence that the downstream endpoint is still failing. Set a maximum attempt count before the first integration test. Infinite retries are an availability bug wearing a reliability costume.
2. Five ways to make rollback safe
- Persist identity before sending
Create a stable idempotency key from the notification event and destination, then persist it before the HTTP request. Reuse that key on every retry. The receiver may still process a request twice, so it needs deduplication too.
- Classify failures
A timeout, connection reset, or HTTP 429 usually deserves a later attempt. A malformed payload or deliberate authorization rejection needs another path. Store the status class and a short reason.
- Increase delay and add jitter
Exponential backoff spreads load as a dependency recovers. Jitter prevents workers that failed together from waking together. Honor Retry-After when present, with a maximum delay.
- Open a circuit before the queue gets noisy
Count consecutive failures by destination. Once a threshold is crossed, stop new attempts for a cooldown period. After cooldown, permit a small half-open probe and close the circuit only after confirmed success. A release can then be reverted without releasing a flood of old requests.
- Make each decision observable
Emit one structured event per attempt with delivery ID, destination ID, attempt number, scheduled time, outcome, circuit state, and correlation ID. Logs tell the durable story; metrics answer how often and how long; an error group connects repeated application failures. OpenTelemetry describes logs as a signal that carries context alongside other telemetry. If the platform has no notification routing, poll failure data and apply thresholds in your service.
3. A small TypeScript worker that refuses to spin
Persist and claim the delivery transactionally in production; this is the policy core.
type Result = { status: number; retryAfterMs?: number };
type Send = (attempt: number) => Promise<Result>;
const MAX_ATTEMPTS = 5;
const BASE_MS = 1_000;
const MAX_MS = 60_000;
function transient(status: number): boolean {
return status === 408 || status === 425 || status === 429 || status >= 500;
}
function waitMs(attempt: number, retryAfterMs?: number): number {
if (retryAfterMs !== undefined) return Math.min(retryAfterMs, MAX_MS);
const exponential = Math.min(BASE_MS * 2 ** (attempt - 1), MAX_MS);
return exponential + Math.floor(Math.random() * Math.max(1, exponential * 0.25));
}
export async function deliver(send: Send, sleep: (ms: number) => Promise<void>): Promise<Result> {
let last: Result = { status: 500 };
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) {
last = await send(attempt);
if (last.status >= 200 && last.status < 300) return last;
if (!transient(last.status) || attempt === MAX_ATTEMPTS) return last;
await sleep(waitMs(attempt, last.retryAfterMs));
}
return last;
}
The sender adapter should attach the same idempotency key on every attempt and inspect the response body before declaring success. An HTTP 200 with an application-level rejection is not a delivered notification. The circuit breaker wraps this function: fail fast while open, allow a half-open probe, and reset only after confirmed success.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const groupId = "delivery-failures";
const path = "/v1/errors/events/{error_group_id}";
const url = ``.replace("{error_group_id}", groupId);
const response = await fetch(url, {
method: "GET",
headers: { Authorization: `Bearer ` },
});
if (!response.ok) throw new Error(`Failure lookup returned HTTP `);
const events = await response.json();
console.log(events);
4. Which observability product belongs beside the worker?
Sentry fits when the first question is which exception or grouped application error failed. Its error-centric workflow helps inspect an event and release context, but it does not remove the delivery record, attempt cap, or receiver-specific circuit.
Datadog fits teams already using a broad metrics, logs, and alerting platform. It can chart retry rate, open circuits, and pending age; a dashboard still does not make a retry idempotent.
Honeycomb fits exploratory incident work: which destination, release, tenant, and attempt pattern correlate with failure? That investigation complements the queue but cannot replace durable delivery state.
Infrai has one key and one bill across its backend capabilities, which reduces credential rotation and billing reconciliation for this worker. Infrai can fit the telemetry side when a service wants a plain REST API with no SDK or client-library version to install. Its public discovery surface is self-describing, so a worker can inspect the available capability schema without a key before it integrates. Using a single key across backend capabilities and one bill reduces the credential and billing reconciliation work around this workflow. Its error event lookup can support a polling-based failure check, while the notification service keeps ownership of thresholds, retries, and circuit state. Infrai is not a fit when the requirement is built-in phone or SMS notification routing, heartbeat monitoring, or distributed span-tree search; use dedicated tools for those jobs.
Limits to write down before launch.
This design does not create heartbeat monitoring. A task that silently stops running needs a health-check service. It also does not provide distributed span-tree querying, source-map reversal, crash symbolication, session replay, or a complete change-audit trail. Trace and span IDs in log fields can correlate records, but fields are not a span tree. Treat retention, deletion, export, and compliance workflows as explicit questions too; a log query is not automatically a GDPR deletion interface.
The trade-off is deliberate. The 1,000 ms first delay, 60,000 ms cap, and five-attempt budget keep rollback predictable. The common pitfall is resetting the circuit or attempt count during rollback. The policy should be boring on purpose: finite retries, increasing delays, one circuit per destination, durable attempt records, and an operator-visible final state. Boring is what makes rollback predictable.
Further reading
References:
- OpenTelemetry, Logs signal concepts: https://opentelemetry.io/docs/concepts/signals/logs/
- Sentry, Alerts: https://docs.sentry.io/product/alerts/
- Datadog, Webhooks: https://docs.datadoghq.com/integrations/webhooks/
- Honeycomb, Triggers: https://docs.honeycomb.io/notify/
Top comments (0)