Webhooks are the circulatory system of modern integrations. When a payment clears, a subscription renews, or a digital code is issued, a webhook usually carries that news from one service to another. And when a webhook fails — a timeout, a 429, a deploy landing at the wrong second — the customer never hears about the event at all.
A few years of operating webhook-driven fulfillment in the digital goods space taught me that delivery semantics deserve the same engineering rigor as the payment flow itself. Here's a practical playbook.
1. Assume failures are normal
Receivers go down. Networks flap. A deploy can restart your consumer mid-processing. Design the sender as if every delivery will fail at least once, and make "at least once" delivery the contract rather than a hope.
2. Exponential backoff with jitter
Fixed-interval retries create thundering herds: every failed delivery retries at the same moment, hammering a receiver that may already be struggling. Exponential backoff spaces attempts out; full jitter randomizes them:
import random
import time
def next_delay(attempt: int, base: float = 1.0, cap: float = 3600.0) -> float:
# Full jitter (AWS architecture blog style)
return min(cap, random.uniform(0, base * 2 ** attempt))
def deliver(url, payload, max_attempts=8):
for attempt in range(max_attempts):
try:
resp = requests.post(url, json=payload, timeout=10)
if resp.status_code < 500 and resp.status_code != 429:
return True # other 4xx = receiver rejected it; don't retry
except requests.RequestException:
pass
time.sleep(next_delay(attempt))
return False
A common schedule: attempt 1 immediately, then roughly 2s, 8s, 30s, 2min, 10min, 1h, 6h — about 24 hours of coverage in 8 attempts.
3. Only retry what is retryable
Retrying a 400 Bad Request eight times is just noise. Classify failures explicitly:
- Retry: timeouts, connection errors, 429, 5xx.
- Don't retry: 400, 401, 403, 404, 422 — the receiver understood the event and refused it. Log it and move on.
This one distinction will cut your retry volume dramatically.
4. Dead-letter queues, not silent drops
After the final attempt, failed deliveries go to a dead-letter queue with full context: payload, headers, attempt history, and the receiver's last response. A DLQ turns "the customer never got their code" into a searchable, replayable incident. Pair it with an alert — for example, more than 5 DLQ entries in 10 minutes pages someone.
5. Receivers must be idempotent
"At least once" means the receiver will occasionally see the same event twice — a retry racing the original, or a redelivery after a timeout. Receivers should key on the event ID: store it in a unique column and skip duplicates. A shared secret plus an HMAC signature on the payload (Stripe-style) lets the receiver verify the sender before doing anything.
6. Observability: log the whole lifecycle
Log each attempt with a correlation ID shared between sender and receiver: event_id, attempt, status_code, latency_ms, next_retry_at. When support asks "why didn't the fulfillment trigger?", you answer with one query instead of grep archaeology.
Putting it together
A robust webhook pipeline has four parts: a durable outbox (never lose an event), a worker applying backoff with jitter, explicit classification of retryable failures, and a dead-letter queue with alerting. At kadokey.com — where digital gift cards and game codes are delivered within minutes of payment — this exact architecture keeps fulfillment reliable through provider incidents and traffic spikes.
Checklist before you ship:
- [ ] Backoff with jitter, not fixed intervals
- [ ] Retryable vs non-retryable status codes classified
- [ ] Dead-letter queue plus alerting on final failures
- [ ] Receiver verifies signatures and dedupes event IDs
- [ ] Correlation IDs in every attempt log
Build for the failure case and the happy path takes care of itself.
Top comments (0)