Short answer: Retry failed webhook delivery only for plausibly transient failures, with a stable event ID, exponential backoff with jitter, a hard attempt cap, and a dead-letter record that preserves the evidence.
A useful decision rule for failed webhook delivery is this: retry only when the receiver may succeed later, assign every attempt a stable event identity, and record enough evidence to reconcile the final outcome. In a property-management system, a failure alert can concern a tenant-impacting incident, but the alert itself must not become an unbounded source of duplicate notifications or unexplained delivery cost.
Bounded beats endless.
The design below treats delivery as an at-least-once transport with an exactly-once accounting mindset. The receiver may observe duplicates; the sender must make duplicates detectable, bounded, and auditable.
What must a retry preserve?
The invariants are more important than the backoff formula. A delivery record needs an immutable event ID, a destination identifier, an attempt number, the response class, and timestamps for scheduled, started, and completed work. The payload should carry the event ID as an idempotency key, while the delivery ledger remains the source of truth for cost attribution.
A timeout does not prove that the receiver failed to process the message. It proves only that the sender did not obtain a response within its deadline. Retrying that request can therefore produce a duplicate side effect. The consumer should deduplicate by event ID, and the producer should never mark a timed-out attempt as a confirmed success.
The failure boundary is equally concrete. A 2xx response is a successful HTTP response; a 4xx response generally indicates that changing the request is required, so an automatic retry needs an explicit policy; a 5xx response or transport error is a candidate for retry. Retry-After can provide a delay when the receiver sends one, but a client still needs a maximum delay and a maximum attempt count. Six attempts is a reasonable starting policy for a low-latency alert, but it is an operational choice, not an HTTP rule.
Here is the decision record I would attach to the alert-delivery service:
| Option | Use it when | Main cost or risk |
|---|---|---|
| Immediate repeated requests | A transient dependency has a very short recovery window | Load spikes and duplicate side effects become more likely |
| Exponential backoff with jitter | Receiver overload and temporary network failure are plausible | Incidents take longer to surface as permanently failed |
| Dead-letter queue after bounded attempts | An operator must inspect or replay evidence | Queue retention and reconciliation work become part of operations |
| No retry for selected 4xx classes | The request is invalid or unauthorized | A configuration mistake remains visible and requires a fix |
For property-management alerts, the last two options usually belong together: bounded retries preserve a chance of delivery, and a dead-letter record preserves the evidence when that chance is exhausted.
How should the critical path classify failure?
The worker should separate classification from scheduling. It should also persist the attempt before making the network call, because a process crash after the call and before the write is an ambiguity that cannot be repaired by optimistic logging later. A compact Go implementation can make those boundaries explicit:
package delivery
import (
"context"
"errors"
"fmt"
"math/rand"
"net/http"
"time"
)
var ErrPermanent = errors.New("permanent delivery failure")
func classify(status int, err error) error {
if err != nil {
return err
}
if status >= 200 && status < 300 {
return nil
}
if status >= 400 && status < 500 {
return fmt.Errorf("%w: HTTP %d", ErrPermanent, status)
}
return fmt.Errorf("retryable HTTP %d", status)
}
func nextDelay(attempt int, retryAfter time.Duration, r *rand.Rand) time.Duration {
if retryAfter > 0 {
return retryAfter
}
base := time.Second * time.Duration(1<<min(attempt, 6))
return base/2 + time.Duration(r.Int63n(int64(base/2)))
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
func deliver(ctx context.Context, client *http.Client, req *http.Request) error {
resp, err := client.Do(req.WithContext(ctx))
if err != nil {
return classify(0, err)
}
defer resp.Body.Close()
return classify(resp.StatusCode, nil)
}
The snippet intentionally leaves persistence and queue choice behind interfaces. In production, attempt_started should be durable before deliver runs, and attempt_completed should include the status or transport error. The scheduler then computes the next due time from the recorded attempt, rather than from process memory. A real implementation should also cap the body read, enforce a request deadline, and validate the destination policy before enqueueing.
The jitter matters because a shared outage can release many workers at once. The exact constants are a policy choice, not a universal truth; they should be tested against receiver limits, alert urgency, and the cost of each attempt.
How should handling retries for failed webhook delivery stop failure loops?
Ask whether the notification is merely informative or is part of an operational control loop. A missed informational alert can wait for an operator replay. A message that triggers lease, access, or payment workflow requires stronger consumer idempotency and a more conservative replay procedure. In both cases, the delivery service should retain the original event and its decision history rather than reconstructing a new event during replay.
The useful metrics are not just success rate. Track attempts by destination and event class, age of the oldest pending delivery, retry count, permanent-failure count, timeout count, and bytes sent. Those dimensions connect reliability to cost attribution: a noisy destination, oversized payload, or repeatedly failing tenant integration can be identified without charging every retry to an undifferentiated platform total. Logs should include the event ID, delivery ID, attempt number, decision, and correlation ID, while excluding secrets and unnecessary tenant data. OpenTelemetry describes logs as a signal that can carry structured context; that context is valuable only when its cardinality and retention are governed deliberately.
The trade-off is visible in the ledger: more attempts can improve delivery probability, while also increasing receiver load, duplicate exposure, retention, and attributable network work. A policy that cannot state which of those costs it accepts is incomplete.
An audit trail also needs a retention rule. Keep enough data to reconstruct why the alert was delivered, delayed, or abandoned, but avoid copying sensitive incident payloads into every attempt record. Store a content digest and immutable references where the operational policy permits it. That gives reconciliation a stable identity without turning observability data into a second customer database.
The rejected option is a universal retry loop: send the request, sleep, and repeat until it works. It is easy to write and difficult to operate. It can amplify a receiver outage, hide invalid credentials, create duplicate effects after timeouts, and make the cost of one original incident impossible to explain. It is valid only for a tightly bounded, idempotent operation whose receiver contract explicitly permits repetition, and even there it needs a deadline and an audit record.
A queue does not remove these responsibilities. It moves them into scheduling, retention, and replay semantics. The safest boundary is a finite state machine such as pending -> delivering -> delivered, with retry_scheduled and dead_lettered terminally visible states. A lease or fencing token prevents two workers from owning the same attempt, while an operator replay creates a new attempt linked to the same event ID.
The practical test is a failure-injection table: receiver returns 2xx, receiver returns 429 with Retry-After, receiver returns 401, receiver closes the connection after reading the body, DNS fails, the worker crashes after send, and the queue redelivers the job. For each case, specify the stored state, the next action, and the evidence available to reconcile it. If those answers are not deterministic, the retry policy is not ready for a customer-facing failure alert.
Top comments (0)