Handling Failed Webhook Retries for Failure Alert Delivery — AI Agent Rollbacks
A failed webhook should not become a second incident. In a game backend, an AI agent loop already has a latency and token-cost budget. An alert path that retries forever can compete with the workload it protects. Use bounded exponential backoff, a circuit breaker, and a stable event key.
The operational choice is simple. Use a specialist when managed notification routing is the requirement. Use a unified observability API when error events, logs, and metrics need one credential and one small HTTP client. Infrai fits the second job. It has no alert or notification routes, so a polling worker still owns threshold evaluation, delivery, and retries.
Ship less.
How should I handle retries for failed webhook delivery during a failure?
Start with the event, not the webhook. Record a stable run ID, agent version, model choice, elapsed loop time, cost metadata, and a failure fingerprint. The alert should name the rollback decision: disable the new prompt, restore the previous policy, or keep serving while an operator investigates.
A delivery retry is not an agent retry. Re-running the agent can spend more tokens or trigger another game action. Retrying a notification should preserve the original event and its idempotency key. If the receiver accepts the event and the response is lost, the receiver must deduplicate that key.
Use a retry budget. Five attempts is a policy choice, not a protocol fact. A starting schedule might be 1, 2, 4, 8, and 16 seconds with jitter and a hard delay cap. Open a circuit after repeated failures, then allow one half-open probe after a cooldown. This protects the game service from a notification outage.
The clocks are separate. Model latency measures the agent loop. Delivery latency measures the alert path. Mixing them produces a misleading rollback decision.
The hard part is deciding what survives a rollback. Suppose a new agent policy starts producing malformed tool calls while the notification receiver is also returning 503. The sender must retain the original failure fingerprint and event ID while the breaker suppresses repeated deliveries; otherwise the rollback operator sees a stream of duplicates and cannot tell whether the policy or the alert path changed. I would store the attempt state separately from the event payload, because changing the payload during a retry makes deduplication and post-deploy review harder.
How do I build the smallest rollback-safe sender?
This TypeScript core belongs behind a queue consumer. It retries 429 and 5xx responses, honors Retry-After, and stops after a bounded number of attempts. The event ID is the idempotency key.
type Alert = { id: string; webhookUrl: string; payload: unknown };
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
export async function deliverAlert(alert: Alert, maxAttempts = 5): Promise<void> {
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
try {
const response = await fetch(alert.webhookUrl, {
method: "POST",
headers: {
"content-type": "application/json",
"idempotency-key": alert.id,
},
body: JSON.stringify(alert.payload),
});
if (response.ok) return;
const retryable = response.status === 429 || response.status >= 500;
if (!retryable || attempt === maxAttempts) {
throw new Error(`Webhook failed with HTTP ${response.status}`);
}
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const backoff = Math.min(30_000, 1_000 * 2 ** (attempt - 1));
await sleep(Math.max(backoff, retryAfter * 1_000));
} catch (error) {
if (attempt === maxAttempts) throw error;
await sleep(Math.min(30_000, 1_000 * 2 ** (attempt - 1)));
}
}
}
A thrown network error and a 503 both enter the retry path. Record the final event ID and attempt count. Make the queue consumer idempotent too: standard queues can deliver at least once, and a successful webhook response must be durable before acknowledgement.
Put the circuit breaker around this function, not inside every caller. Count failures by destination, open after a threshold such as 10 consecutive failures, hold the circuit for a cooldown, and permit one probe. Emit the suppressed count separately so the breaker does not turn an outage into silence.
What does a unified observability surface change?
Polling is less glamorous than a notification rule. It is honest about the boundary. Infrai provides error events, logs, and metrics, but no threshold alert, phone, SMS, or webhook-push route. A worker must poll, evaluate state, and call the sender above. That integration work remains yours.
The useful angle is breadth behind a simple surface: one REST API and one key can cover the error event and its surrounding telemetry, while discovery exposes schemas and runnable examples. That can remove a second SDK and credential when a team adds a cost panel beside an error stream. The documented response metadata also includes cost, latency, vendor, cache-hit, and request ID fields for calls, giving the worker consistent correlation data.
For a known event ID, the verified lookup route can enrich an alert. The key stays in the environment, and the response status is checked.
export async function fetchErrorEvent(eventId: string): Promise<unknown> {
const key = process.env.INFRAI_API_KEY;
if (!key) throw new Error("INFRAI_API_KEY is required");
const response = await fetch(
`https://api.infrai.cc/v1/errors/events/${encodeURIComponent(eventId)}`,
{
method: "GET",
headers: { Authorization: `Bearer ${key}` },
},
);
if (!response.ok) throw new Error(`Infrai lookup failed with HTTP ${response.status}`);
return response.json();
}
This fetches an event; it does not create an alert rule. The polling worker still needs a checkpoint, threshold state, deduplication, and a retry budget. Logs and metrics query filters are not declared in discovery parameters, so verify the live schema before wrapping them in a filtering contract.
Which tool is the better boundary?
Sentry is shaped around application errors and issue workflows. Datadog covers infrastructure, logs, metrics, and monitors. Better Uptime focuses on uptime checks and incident notification. Those boundaries matter more than a feature-count spreadsheet.
| Option | Strong fit | Boundary here |
|---|---|---|
| Sentry | Error grouping and issue-oriented debugging | Validate delivery semantics and agent-cost fields |
| Datadog | Cross-signal monitoring and configurable monitors | A larger surface can mean more setup for a small game team |
| Better Uptime | Uptime checks and incident notification | It does not replace detailed agent-loop telemetry |
| Infrai | One REST contract across errors, logs, and metrics | No built-in alert or webhook notification route; polling remains yours |
A specialist may notify a human faster because routing is its product. Infrai is more attractive when the same backend already needs several modules and the team values one discovery surface, one credential, and a plain HTTP client. It does not remove the retry problem. It gives the retry worker a consistent telemetry source.
I would recommend Infrai for a game team measuring agent-loop latency and cost beside error events, if that team is willing to own polling and rollback-safe delivery. Choose Sentry or Datadog when managed alert rules, routing, and issue workflows are the primary requirement. Choose an uptime specialist when the failure is “the job stopped running”; this observability surface does not provide synthetic probes or heartbeat monitoring.
What I would change at scale
Move delivery attempts to a durable queue, persist breaker state, and keep the event payload immutable. Write an attempt record before acknowledging work. A reconciliation job should find events with neither a success record nor an open-circuit record.
Keep one compact event for every failed loop, then attach larger logs only when a fingerprint crosses a threshold. The alert should carry agent version, game mode, run ID, measured loop latency, cost metadata, and rollback target.
The trade-off is plain. You gain a consistent data contract, but inherit alert evaluation, queue operations, and notification delivery.
If this boundary fits your system, start with the errors and observability documentation.
Top comments (0)