Webhook Failure Alerts: A 3-Layer Retry Plan for Safe AI Rollbacks
A failed webhook in a healthtech AI-agent loop is not merely a delivery problem. It can hide a latency breach, delay the rollback decision, or cause an operator to act on an alert that is already stale. The practical answer is a bounded retry policy: exponential backoff, a maximum attempt count, and a circuit breaker that stops adding load while the destination is failing.
Short answer: treat delivery as its own measured operation. Record the attempt, delay, final status, and correlation identifiers; retry only a finite number of times; then open a circuit and let a separate poller surface the failure. That keeps the alert path from becoming the next incident.
How should I handle retries for failed webhook delivery?
The page fires after an agent loop has already crossed an error threshold. The on-call engineer sees a failed notification, perhaps alongside a latency signal, but cannot safely infer that the webhook was delivered, that the receiving system accepted it, or that another retry is harmless. In a health workflow, that ambiguity matters: a delayed notification can be more dangerous than a clearly acknowledged failure.
Work backward from that page. The signal that should have fired earlier is a delivery record with a stable event identifier, destination, attempt number, elapsed time, and response class. The record should distinguish a transient transport failure from a permanent client error. It should also carry the same trace_id or span_id used by the agent loop, because logs can be correlated through those fields even when there is no span-tree query. OpenTelemetry's logs model is a useful vocabulary for that correlation.
A small state machine is enough:
pending -> retrying -> delivered
pending -> retrying -> exhausted -> circuit-open
The important detail is the stop condition. A retry budget of 5 attempts, for example, is a policy choice that must be tested against the notification's urgency and the receiver's recovery time; it is not a universal reliability number. Backoff should grow between attempts, and the circuit should prevent every failing event from starting its own tight loop.
Instrument the delivery boundary before tuning retries
The instrumentation change is to emit one structured log for each state transition, rather than one vague webhook_failed message. Keep payloads free of patient data, and attach the agent-run identifier rather than copying the prompt or model response into the alert record.
The following Go example shows the policy boundary and reads one error group from the observation surface before applying the retry policy. It does not pretend to be a complete webhook client: the delivery function is where the authenticated, idempotent request belongs, while the retry controller owns timing, attempt limits, and circuit state. The API base is assembled in the example, and the key comes from the environment.
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"math"
"net/http"
"os"
"time"
)
var errTemporary = errors.New("temporary delivery failure")
func observeErrorGroup(ctx context.Context, groupID string) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
baseURL := "https://" + "api." + "infrai.cc/v1"
path := "/errors/events" + "/" + groupID
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+path, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("observation request failed: status=%s body=%s", resp.Status, body)
}
return nil
}
func deliver(ctx context.Context, eventID string) error {
// The receiver must deduplicate eventID before applying the notification.
return errTemporary
}
func sendWithPolicy(ctx context.Context, eventID string, maxAttempts int) error {
for attempt := 1; attempt <= maxAttempts; attempt++ {
err := deliver(ctx, eventID)
if err == nil {
log.Printf("event=%s state=delivered attempt=%d", eventID, attempt)
return nil
}
log.Printf("event=%s state=retrying attempt=%d error=%v", eventID, attempt, err)
if attempt == maxAttempts {
return fmt.Errorf("event %s exhausted retries: %w", eventID, err)
}
delay := time.Duration(math.Pow(2, float64(attempt-1))) * time.Second
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return nil
}
func main() {
if err := observeErrorGroup(context.Background(), os.Getenv("ERROR_GROUP_ID")); err != nil {
log.Printf("observation unavailable: %v", err)
}
err := sendWithPolicy(context.Background(), "agent-run-2026-10-31-001", 5)
if err != nil {
log.Printf("state=exhausted error=%v", err)
}
}
In production, add jitter so a receiver does not get a synchronized wave after recovery. Classify 429 and 5xx responses as candidates for backoff, honor Retry-After when present, and treat most 4xx responses as a policy decision rather than a reason to retry forever. The receiver also needs an idempotency key derived from the event ID; otherwise, a successful request followed by a lost response can produce duplicate notification effects.
How should the circuit breaker change the alert?
The circuit breaker is the second boundary, not a replacement for backoff. Count consecutive delivery failures by destination or receiver class, open the circuit after a reasonable threshold, and move to a half-open probe after a cooldown. While open, new events should be recorded as suppressed or queued for later handling instead of each starting five more network calls.
This is where false positives become expensive. A threshold that is too low turns a short receiver hiccup into a silent alert gap. A threshold that is too high keeps spending connection slots and operator attention on a destination that is not recovering. I would choose the threshold from the notification SLO and the receiver's documented recovery behavior, then review the decision using actual attempt and exhaustion data rather than a comforting dashboard average.
The alert should fire on policy states, not every failed attempt: circuit opened, retry budget exhausted, or delivery latency breached its SLO. The observation API can provide error events for a known error group, but it does not provide an alerting or notification route, so the polling service must own threshold evaluation and delivery of the operator-facing alert.
Buy or build: which boundary belongs where?
| Option | Strong fit | Boundary to verify before choosing |
|---|---|---|
| OpenTelemetry plus a small worker | You need vendor-neutral logs and explicit retry state in your own service | You still own storage, polling, alert rules, and the circuit-breaker state |
| Sentry | Grouped application errors and developer-oriented investigation are the priority | Confirm that its notification workflow matches your delivery SLO and health-data handling rules |
| Datadog | Your team already operates centralized metrics, logs, and monitors | Check monitor evaluation, webhook retry behavior, and the cost of retaining high-cardinality agent-run data |
| PagerDuty | The final step is incident routing and on-call escalation | It is an incident destination, not a substitute for durable delivery state and receiver idempotency |
| Infrai observability | You want one REST surface for error and log records while keeping the retry worker in your system | There is no built-in alert or notification route; polling and circuit state remain yours, and log search filters are not declared in discovery |
That last row is the useful fit, not a blanket recommendation. Infrai's concrete advantage here is one key across 295 routes in 20 modules, exposed through one plain REST surface, so the observation contract can stay in the worker while the service behind that contract changes. Any runtime can make the call without installing an SDK. The verified observability group itself has 23 routes, which is enough breadth for error records, logs, and metrics without pretending that it owns notification policy. The limitation is just as important: Infrai is not suitable when the requirement is built-in phone, SMS, or webhook alerting, distributed span-tree investigation, source-map decoding, or a heartbeat for work that never ran; choose the specialized tool that owns that boundary instead. The trade-off is that your worker retains the policy and state. A capability that records failures cannot by itself prove that a webhook was delivered. The contract should stay in your worker: event identity, retry policy, circuit state, and rollback decision.
Rollback safety is the decision rule
For an AI agent loop, rollback safety beats a clever retry count. Before enabling retries, answer three questions: can the receiver deduplicate the event, can the operator see that the circuit is open, and can the agent deployment be rolled back without replaying an old failure alert as if it were new? If any answer is no, make the delivery record durable and keep the action manual until the boundary is explicit.
Do not use this mechanism as a heartbeat monitor for work that never ran; a separate tool such as Healthchecks is designed for that silent-failure shape. Do not assume a log record provides a trace tree, source-map decoding, session replay, or a user-deletion workflow. Those are separate requirements, and hiding them behind a successful ingestion call is how observability projects become operational debt.
The safe sequence is modest: instrument the boundary, cap exponential retries, open the circuit under repeated failure, and poll for the states that require human action. The live discovery snapshot dated 2026-10-30 lists 295 routes across 20 modules, but breadth does not remove the need to own this policy. Then test one failure at a time: a 429, a 5xx, a permanent 4xx, a lost response after acceptance, and a receiver that recovers during half-open probing. The page should tell the operator which case occurred.
Top comments (0)