TL;DR: retry a failed notification only when the attempt has a durable identity, a bounded schedule, and an observable outcome. For scheduled imports, the useful alert is not the fifth POST. It is the chain showing which import stopped producing results, which attempts happened, and why the next action is safe.
A small retry loop around fetch ships quickly, but it erases the difference between a timeout before a receiver saw a request and a timeout after the receiver accepted it. That difference controls duplicate handling and incident reconstruction. The practical design is one notification event per failed import, with append-only attempt records beneath it.
That is the boundary.
For a scheduled import, the evidence chain is more important than the retry loop itself. Suppose the 03:00 run produces no records and emits one notification. The sender starts attempt 1, receives no response before its deadline, and the worker restarts. If the system only stores a generic error, the next worker cannot tell whether it should send again, wait for a receiver-side lookup, or ask an operator to inspect the destination. If it stores the event, the attempt, the deadline, the idempotency key, and the policy version, the uncertainty is visible and bounded. That extra state consumes storage and adds a conditional write, but it prevents a plausible success from being silently converted into either a duplicate page or a missing alert.
How should handling retries change after failed webhook delivery?
A retry should leave enough evidence to answer three questions later: what event was delivered, what was attempted, and what ended the attempt? Give each event an import_run_id, notification_id, attempt number, state, timestamps, response status when available, and transport error when no response exists. Add an idempotency key and policy version. A digest or schema version can connect the record to stored payload data without copying sensitive content into logs.
The difficult state is unknown. A client deadline can fire while the receiver is processing the request. Calling that a guaranteed rejection invites a duplicate; calling it success hides a missed notification. Retries therefore need a receiver-side idempotency contract or a review queue for outcomes that cannot be established.
The trade-off is deliberate: stronger evidence requires durable writes and retention, while a memory-only retry loop is cheaper to operate but cannot reconstruct a process crash. A concrete pitfall is marking an attempt failed at the deadline and then sending the same event while the first request is still running.
OpenTelemetry treats logs as a signal with timestamped records and attributes, and describes correlating signals with shared context. Use trace_id and span_id when available, but keep notification_id as the business key. A trace is context, not durable delivery state.
A small state machine for scheduled imports
Use explicit transitions instead of inferring state from the last log line. A useful set is pending, in_flight, accepted, retryable_failure, permanent_failure, and unknown. The example keeps the policy independent from a queue or notification provider. Its limits are examples; configuration and tests should own the final values.
type Result =
| { kind: "response"; status: number }
| { kind: "transport_error"; code: string }
| { kind: "deadline" };
type Decision = {
state: "accepted" | "retryable_failure" | "permanent_failure" | "unknown";
retryAt?: Date;
reason: string;
};
export function decide(
result: Result,
attempt: number,
now: Date,
maxAttempts = 6,
): Decision {
if (result.kind === "response" && result.status >= 200 && result.status < 300) {
return { state: "accepted", reason: "receiver acknowledged the event" };
}
if (result.kind === "deadline") {
return {
state: attempt < maxAttempts ? "unknown" : "permanent_failure",
retryAt: attempt < maxAttempts
? new Date(now.getTime() + 60_000)
: undefined,
reason: "request outcome was not established",
};
}
if (attempt >= maxAttempts) {
return { state: "permanent_failure", reason: "retry budget exhausted" };
}
const delayMs = Math.min(15 * 60_000, 2 ** (attempt - 1) * 30_000);
return {
state: "retryable_failure",
retryAt: new Date(now.getTime() + delayMs),
reason: result.kind === "response"
? "receiver returned status " + result.status
: result.code,
};
}
Six attempts and a fifteen-minute cap may fit a low-volume import monitor, or may be too slow for an urgent pipeline. Record the selected policy with the event and test the schedule with a fake clock. A retry budget without a deadline is a delayed incident.
How do you reconstruct the failed notification?
Start from the import run, then follow its notification event, then its attempts. Searching for a 500 in a busy worker log can find an unrelated request. Searching by notification_id can show the whole sequence even if a trace was sampled or a process restarted.
| Field | Incident value |
|---|---|
| import_run_id | Connects the alert to the scheduled job and input window |
| notification_id | Keeps every attempt attached to one event |
| attempt | Exposes duplicate scheduling and missing attempts |
| state | Separates accepted, retryable, permanent, and unknown outcomes |
| http_status / error_type | Distinguishes receiver response from local failure |
| policy_version | Explains the chosen limit and delay |
The alert should summarize current state and point to this evidence. It should not fire once per retry. A retryable event is one incident with changing evidence; when it becomes permanent_failure, page according to the import's operational importance. When it becomes accepted, close the delivery incident while retaining the timeline.
The worker must not mark an attempt complete before capturing the response or transport error. A process exit after the POST but before persistence is the gap that makes retries contentious. Let a sweeper find stale in_flight rows, move them to unknown under a documented rule, and emit that transition as an event. Conditional state changes must prevent two workers from claiming the same attempt.
Choosing retry behavior without lock-in
The portable boundary is a sender interface plus a durable store. The sender needs a deadline, cancellation, response classification, and an idempotency key. The store needs conditional transitions. A queue can change later without changing the delivery semantics.
HTTP status is evidence, not a complete policy. A 2xx response is commonly treated as acceptance; 429 and many 5xx responses are candidates for delay and retry. Honor a receiver's Retry-After value when present, cap the delay, and record its source. Do not assume every 4xx is permanent: authentication, authorization, validation, and rate limiting need different remedies.
Emit stable dimensions such as delivery.state, delivery.attempt, notification.channel, and failure.kind. Keep transport-specific details at the edge. That makes dashboards portable and exposes cost through attempt counts, bytes, and retention rather than through a vendor's proprietary status names.
Test the failure modes that alter the evidence: a receiver returns 503 twice and accepts; a deadline fires after acceptance; two workers claim one row; the process exits after sending; and a stale schema is rejected. Each test should assert the next state and the operator's query path.
Before choosing a schedule, measure response classes, time to acceptance, unknown outcomes, duplicate deliveries, and alerts open past the import's useful recovery window. Break results down by channel and receiver; a global average can hide one slow integration.
The decision rule is direct: retry transient failures when duplicates are safe, isolate unknown outcomes when they are not, and page on the durable event rather than individual attempts. That preserves the evidence needed to explain a stalled import without creating an alert storm.
Top comments (1)
The unknown state is the part most retry code pretends does not exist. A deadline firing tells you about your client, not about the receiver, and treating it as a failure is exactly how you get the duplicate page at 3am.
One thing I'd add: the sweeper that moves stale in_flight rows to unknown is itself a writer, so it needs the same conditional transition as the workers. Otherwise a slow worker finishes, writes accepted, and the sweeper overwrites it a second later. Making every state change a compare-and-set on the previous state closes that race and makes the audit trail trustworthy.
Also fully agree on paging per event, not per attempt. An alert storm from retries teaches people to mute the channel, which is worse than no alert.