Short answer: keep one alert payload, append bounded delivery attempts, retry only transient failures, and stop when the incident's evidence window closes.
A failed alert webhook is not evidence that the game incident is still happening. It is evidence that one delivery attempt failed. Those facts must stay separate.
For a gaming incident, retain one durable delivery record per alert and append attempts to it. Retry only failures that are plausibly transient, honor a valid Retry-After, cap attempt count and retry age, and keep the final delivery outcome after the alert payload expires. The useful unit is the alert's evidence trail, not every HTTP request.
What is the bill actually buying you?
Start with retention, because retry policy is a storage policy wearing an HTTP costume. Suppose a service emits 2,000 failure alerts per hour, each serialized payload is 6 KB, and a delivery worker makes four attempts on average. Keeping every request body for seven days produces roughly 2,000 x 24 x 7 x 4 x 6 KB: about 8 GB before indexes, replication, and metadata. The exact byte count varies; the multiplier is the point.
The change that moves that term is simple: store the payload once, then store compact attempt metadata. An attempt needs an identifier, timestamp, HTTP status or transport error, elapsed time, and next scheduled action. It does not need a second copy of the incident stack trace.
The hard part is deciding what “reconstruct” means before the first outage. For a player-facing purchase failure, that may be the alert payload, the game service trace identifier, the account or session pseudonym, the receiver's status class, and the exact reason the queue stopped. It is not necessarily the receiver's entire HTML error page, which may contain credentials, unstable request identifiers, or a body large enough to distort the retention calculation. Write that boundary into the schema and retention policy. Otherwise every retry adds another argument for keeping everything forever, and storage quietly becomes the incident response plan.
Evidence has a shelf life.
| Record | Retain | Why |
|---|---|---|
| Alert envelope | One immutable copy | Reconstruct the customer-facing failure |
| Delivery attempt | Append-only metadata | Explain retry timing and decisions |
| Response body | Bounded excerpt, if permitted | Diagnose a receiver without retaining secrets |
| Final state | Until evidence expires | Prove what notification did |
That means deliberately dropping repeated payloads, unbounded response bodies, and successful-attempt detail after the audit window. When delivery fails, the cost is less forensic context around the receiver's full response. That is a real trade. Four copies of one JSON document are not four independent signals.
Keep the payload once.
Which failures deserve another attempt?
Use the response class and transport result as inputs, not a universal retry switch. A timeout, connection reset, or 5xx response can be transient. A 400 caused by malformed input is usually permanent. A 401 or 403 needs credential or policy repair, not a faster loop. A 429 may be recoverable, but the receiver's Retry-After value is part of the contract. RFC 9110 defines that field as guidance about when to make a follow-up request; it does not promise success.
The missing detail in many implementations is idempotency. If the sender times out after the receiver commits the event, retrying can create a second page or chat notification. Put a stable delivery key in the request and make the receiver deduplicate it where the receiver's contract allows that. If no such contract exists, record that a retry may duplicate the notification.
from dataclasses import dataclass
from datetime import datetime
from typing import Optional
@dataclass(frozen=True)
class DeliveryAttempt:
alert_id: str
attempt_id: str
started_at: datetime
status_code: Optional[int]
transport_error: Optional[str]
response_excerpt: Optional[str]
next_attempt_at: Optional[datetime]
def retryable(status_code: Optional[int], transport_error: Optional[str]) -> bool:
if transport_error in {"timeout", "connection_reset", "dns_failure"}:
return True
if status_code is None:
return False
return status_code in {408, 425, 429} or 500 <= status_code <= 599
The function is intentionally boring. It does not retry every exception, and it does not infer that a 400 is temporary because the endpoint is important. Map the transport library's exception taxonomy explicitly; otherwise a programming error can become an alert storm.
How should I handle retries for failed webhook delivery?
A schedule needs a maximum delay, maximum attempts, and maximum age. Exponential backoff with jitter prevents workers from waking at the same instant after a regional disruption. The age limit matters more than a heroic attempt count. A notification delivered six hours after a match-service failure can be technically delivered and operationally useless.
The examples use Python 3.10 syntax. The 900-second delay cap and 3,600-second evidence age below are policy values, not universal defaults; measure receiver behavior and choose limits that fit the incident class.
import random
from datetime import datetime, timedelta
def next_retry(now: datetime, attempt: int, max_age_seconds: int = 3600) -> datetime | None:
base = min(900, 2 ** min(attempt, 10))
delay = max(1, int(base * random.uniform(0.5, 1.5)))
candidate = now + timedelta(seconds=delay)
if (candidate - now).total_seconds() > max_age_seconds:
return None
return candidate
A production queue should calculate age from the original alert timestamp, not from the latest attempt. This compact example takes now as the scheduling reference and therefore belongs behind that policy check. That distinction is a common trap.
For this gaming case, classify urgency before enqueueing. A payment failure during a live purchase may justify a short delivery age, while a diagnostic notice about a background matchmaker can wait longer. The retry worker should read an immutable policy attached to the alert. It should not infer urgency from failed-attempt count.
The limitation is deliberate: this policy does not fit audit records that must preserve the receiver's complete response, or alerts whose meaning remains valid for days. Those cases need a longer evidence window and potentially encrypted response storage. The trade is clear: a short window reduces noise, but it cannot reconstruct a receiver response that was discarded.
The operator still needs two timelines: the incident timeline and the delivery timeline. The first answers what players experienced. The second answers whether evidence reached the notification destination. Join them on alert_id and delivery_key; otherwise a failed webhook can masquerade as a second game failure.
The operator needs two timelines: the incident timeline and the delivery timeline. The first answers what players experienced. The second answers whether evidence reached the notification destination. Join them on alert_id and delivery_key; otherwise a failed webhook can masquerade as a second game failure.
Emit structured logs for queued, attempted, deferred, delivered, permanent_failure, and expired. OpenTelemetry's logs signal guidance covers interoperable timestamps, severity, body, attributes, and trace context. Keep payload secrets out of attributes; use stable identifiers and bounded diagnostic fields instead.
from typing import Any
def delivery_event(alert_id: str, attempt: DeliveryAttempt, state: str) -> dict[str, Any]:
return {
"event_name": "webhook_delivery",
"state": state,
"alert_id": alert_id,
"attempt_id": attempt.attempt_id,
"status_code": attempt.status_code,
"transport_error": attempt.transport_error,
"next_attempt_at": attempt.next_attempt_at.isoformat() if attempt.next_attempt_at else None,
}
Count final outcomes and attempts separately. If combined, a rise in retries can look like a rise in game failures. Alert on expired or permanent-failure deliveries relative to emitted alerts, then inspect attempt latency and receiver status classes.
The decision rule I would ship
Keep one alert payload, append bounded attempt facts, and expire the queue entry when the alert's evidence window ends. Retry transport failures and server-side throttling with jitter. Honor a sane Retry-After, but cap it. Reject malformed requests without retrying them blindly. Give every attempt a stable delivery key and expose uncertainty when the receiver cannot guarantee deduplication.
The signal-quality test is concrete: can an engineer reconstruct the customer incident, distinguish it from notification failure, and explain why the worker stopped? If yes, more retained copies add little. If no, add the missing field or state transition before increasing retention.
Top comments (0)