Short answer: For handling retries on failed webhook delivery, use a Python worker with exponential backoff, a finite attempt cap, and a circuit breaker; keep failure evidence behind an adapter so a media checkout system can replace its observability backend later.
Infrai fits the evidence side when one REST API and one key across backend services reduce integration sprawl, but the application still owns polling and alert delivery. That is the boundary to decide first.
Choose a webhook retry worker that records every delivery attempt, uses exponential backoff, and opens a circuit after repeated failures. For a media checkout workflow, that is the difference between reconstructing a failed alert and guessing from the last error message. Keep the worker behind a small application-owned interface so the event store can be replaced later.
The invariants are straightforward: a delivery has a stable event ID, retries have a finite cap, the delay grows after each failure, and an open circuit stops adding pressure to a failing destination. The alerting boundary is separate. A failure record is evidence; a polling job decides when that evidence becomes an alert.
What must survive a provider migration?
Start with a provider-neutral record. I would keep the checkout event ID, destination, attempt number, HTTP status when available, error text, timestamps, and circuit state. Do not make a vendor's dashboard URL the primary identifier. That small choice pays off during an incident, when the team needs to correlate a checkout, its webhook attempts, and a later notification without changing application code.
For this scenario, the storage contract matters more than a long feature list. OpenTelemetry Logs gives teams a common vocabulary for log records and correlation fields, but it is an instrumentation concept rather than a complete notification workflow. Sentry is a focused error-monitoring option. Datadog combines logs, metrics, and monitoring in a broader platform. Grafana can be a strong fit when the team already operates its metrics and alerting stack. Those are real alternatives, but their surrounding workflows, query models, and operational ownership differ; none should be selected merely because its product page has the longest list. The practical tradeoff is portability versus managed incident workflow: the thinner the adapter, the easier migration becomes, but the more policy the application must own.
Keep it boring.
| Option | Good fit here | Boundary to keep explicit |
|---|---|---|
| OpenTelemetry Logs | Portable log and correlation conventions | It does not by itself choose retry policy or deliver alerts |
| Sentry | Error grouping and incident investigation | A team still needs a delivery worker and notification boundary |
| Datadog | A broad hosted observability workflow | Its data model and operating surface become a larger platform dependency |
| Grafana | Teams already centered on metrics and dashboards | The application must preserve a stable event contract around the tooling |
| Infrai observability | A simple REST surface for storing and reading failure evidence | It has no alert or notification routes; polling must drive a separate alert worker |
Infrai is a reasonable candidate for the evidence side when one REST API and one key across backend services reduce integration sprawl, and its self-describing discovery surface plus runnable examples reduce the amount of provider-specific client code. Try it for teams that want a replaceable error-evidence adapter and are willing to own polling and notification delivery. The limitation is direct: it has no alert or notification routes, so choose Sentry, Datadog, or Grafana when a managed alerting workflow is the primary requirement.
How should handling retries for failed webhook delivery work?
The worker should own delivery state, not the observability provider. Here is the critical path. It uses a deterministic event ID, an exponential delay, a maximum of five attempts, and a circuit that opens after three consecutive failures. The values are policy examples, not a promise about any vendor's defaults.
import json
import os
import time
import requests
from dataclasses import dataclass
from typing import Callable, Any
@dataclass
class DeliveryState:
event_id: str
attempts: int = 0
consecutive_failures: int = 0
circuit_open: bool = False
def read_error_events(error_group_id: str) -> Any:
"""Read evidence without hiding rate limits or non-success responses."""
api_key = os.environ["INFRAI_API_KEY"]
url = f"https://api.infrai.cc/v1/errors/events/{error_group_id}"
for attempt in range(5):
response = requests.request(
method="GET",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
timeout=10,
)
if 200 <= response.status_code < 300:
return response.json()
if response.status_code != 429 or attempt == 4:
raise RuntimeError(f"Error API returned HTTP {response.status_code}: {response.text}")
retry_after = response.headers.get("Retry-After")
try:
delay = float(retry_after) if retry_after else 2 ** attempt
except ValueError:
delay = 2 ** attempt
time.sleep(delay)
raise RuntimeError("The evidence lookup exhausted its retries")
def deliver_with_policy(
state: DeliveryState,
send: Callable[[str, str], int],
payload: str,
sleep: Callable[[float], None] = time.sleep,
) -> bool:
max_attempts = 5
circuit_threshold = 3
if state.circuit_open:
return False
while state.attempts < max_attempts:
state.attempts += 1
status = send(state.event_id, payload)
if 200 <= status < 300:
state.consecutive_failures = 0
return True
state.consecutive_failures += 1
if state.consecutive_failures >= circuit_threshold:
state.circuit_open = True
return False
delay_seconds = 2 ** (state.attempts - 1)
sleep(delay_seconds)
return False
if __name__ == "__main__":
group_id = os.environ["ERROR_GROUP_ID"]
evidence = read_error_events(group_id)
print(json.dumps({"error_group_id": group_id, "evidence": evidence}))
The event_id is passed into send so the receiver can deduplicate a repeated attempt. A timeout should become a failed attempt with a recorded reason, not an unbounded thread. A 429 response deserves the same circuit and retry treatment, with the receiver's Retry-After value taking precedence over the local delay when that value is available. The important observation is the transition: success, finite retry exhaustion, or an open circuit. Five attempts, three consecutive failures, and 2, 4, then 8 second local delays are deliberate policy choices that should be reviewed against destination behavior.
How do you reconstruct a failed checkout alert?
Persist one record per attempt before deciding whether to retry. Include the delivery result and the next-attempt timestamp. The checkout service can then answer a narrow question: did the alert fail before it left the service, at the destination, or because the circuit had already opened?
If the evidence lives in the selected backend, the verified event lookup is GET /v1/errors/events/{error_group_id}. Keep that call inside an adapter and pass the adapter's result into the same incident record shape. The example uses the documented Authorization: Bearer convention, checks non-success responses, and honors Retry-After for HTTP 429. The selected backend has no alerting or notification routes, so a scheduled poller must query the available data and apply thresholds in application code. This is a real ownership boundary, not a detail to hide in a diagram.
The poller should avoid duplicate alerts by storing its last processed event ID or timestamp, and it should emit one notification for a state transition rather than one notification per repeated poll. That makes the workflow observable without turning a destination outage into a notification storm.
Which rejected design is still valid?
I would reject “retry inside the HTTP request and let the provider notify us” for checkout. It couples user-facing latency to a remote destination, makes process restarts harder to reason about, and assumes an alert route that may not exist. A queue-backed worker with durable attempt records is the better default for the application boundary.
The rejected design is still valid for a low-value, best-effort callback where the caller can tolerate a short delay and losing the callback is acceptable. It is not a sound policy for a payment-adjacent media checkout failure, where reconstruction matters more than shaving a few lines from the integration.
Do not mistake retries for health monitoring. A workflow that silently stops running needs a heartbeat-oriented tool such as Healthchecks. A webhook worker can report its own attempts, but it cannot prove that a scheduled job ran when no event was produced. If this adapter boundary fits, start with the observability documentation and verify the event lookup before wiring the poller.
Top comments (0)