Short answer: handle a provider's 429 with exponential backoff, a hard retry limit, and structured events that preserve the delivery identifier; use a separate alerting system for escalation.
The right answer is exponential backoff with a hard retry limit, plus enough structured logging to explain which delivery failed and why. In a media notification service, that policy protects the downstream API from a retry storm while preserving the data needed to attribute delivery cost to a tenant, campaign, or provider. A retry loop without a cap is not resilience; it is a second outage waiting for the first 429.
I treat a returned HTTP 429 as a control signal. The sender records the response, waits for an increasing interval, and gives up after a bounded number of attempts. The failure then becomes an explicit operational result rather than an invisible queue delay.
How should media systems handle API rate-limit errors in alerts?
A rate-limit response should change scheduling, not business meaning. The notification is still a delivery attempt for the same campaign and recipient; the worker should not create a new notification merely because the provider asked it to slow down. That distinction matters for cost attribution, because duplicate work can make a tenant appear to consume capacity it never requested.
The useful minimum is a status-code check, a structured event, and capped exponential backoff. I would record the provider, tenant or campaign identifier, attempt number, response status, and the next retry time. Sensitive message content does not belong in the event. Logs are for reconstructing the decision, not for copying the payload into another system.
Here is the policy I use at the worker boundary. The function deliberately accepts a sender callback: the retry mechanism should not know whether the destination is a push provider, email gateway, or an internal notification API. I initially treated the callback as the whole abstraction, but the cost ledger exposed the missing half: each retry also needs a durable event with the same delivery key.
That is the whole loop.
Do not page on every retry.
package main
import (
"context"
"errors"
"fmt"
"math/rand"
"net/http"
"time"
)
var errRateLimited = errors.New("rate limited")
func sendWithBackoff(ctx context.Context, send func(context.Context) int) error {
const maxAttempts = 5
const initialWait = 250 * time.Millisecond
for attempt := 1; attempt <= maxAttempts; attempt++ {
status := send(ctx)
if status >= 200 && status < 300 {
return nil
}
if status != http.StatusTooManyRequests {
return fmt.Errorf("delivery failed with status %d", status)
}
if attempt == maxAttempts {
return fmt.Errorf("delivery stopped after %d rate-limit responses: %w", attempt, errRateLimited)
}
// Jitter keeps many workers from retrying on the same millisecond.
wait := initialWait * time.Duration(1<<(attempt-1))
wait += time.Duration(rand.Int63n(int64(initialWait)))
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return errRateLimited
}
func main() {
err := sendWithBackoff(context.Background(), func(context.Context) int { return 429 })
fmt.Println(err)
}
The production version should honor a provider's Retry-After value when one is present, while still applying a maximum wait and maximum attempt count. The sample keeps the invariant visible: 429 is retryable, other failures are not silently retried, and cancellation wins over a timer.
When a retry is scheduled, the worker can capture the event through the same observability contract. This small transport helper keeps the key out of source control, sets the method explicitly, and surfaces a non-success response instead of treating an accepted TCP exchange as a successful write. The payload is supplied by the caller because the delivery service owns its event schema.
package main
import (
"bytes"
"context"
"fmt"
"net/http"
"os"
)
func captureDeliveryEvent(ctx context.Context, eventJSON []byte, deliveryID string) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return fmt.Errorf("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
return fmt.Errorf("INFRAI_BASE_URL is required")
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
baseURL+"/v1/errors/capture", bytes.NewReader(eventJSON))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", deliveryID)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("event capture failed with status %s", resp.Status)
}
return nil
}
The capture request is deliberately idempotent at the delivery boundary. If the worker times out after the server accepted it, retrying with the same delivery key should not turn one failure into multiple accounting records. The application still owns the event schema and the provider-specific delivery key.
How do logs turn retries into cost attribution?
A retry counter alone is not an accounting model. A worker needs a stable delivery identifier and an attempt identifier, so the final record can distinguish one notification with four attempts from four notifications. That is also where an SLO becomes useful: define success as an accepted delivery within the allowed window, and track the share of notifications that exhaust the retry budget separately from provider latency.
For this media scenario, I would emit one event when a provider returns 429, one when a retry is scheduled, and one terminal event when the delivery succeeds or is abandoned. The event sent through /v1/logs/ingest should carry the operational fields needed for search and correlation, including a trace or span identifier when the caller already has one. OpenTelemetry describes logs as a signal that can be correlated with traces and metrics; it does not turn a log store into a distributed-trace query engine.
That limitation is important. A log platform can support the evidence trail, but it does not replace paging or notification policy. The available observability surface here has no threshold rules, phone, SMS, or webhook alert route, so an operator still needs a polling-based alert loop or a separate alerting system. A silent job that never runs is a different failure class and needs a health-check tool, not more retry arithmetic.
Which monitoring shape fits a media delivery workflow?
The market options solve different parts of this problem, and naming them as interchangeable hides the real decision.
| Option | Strong fit | Boundary for this workflow |
|---|---|---|
| Sentry | Error grouping and issue-oriented investigation around failed application operations | It is a poor substitute for a delivery scheduler; retry ownership and per-attempt cost attribution still belong in the worker |
| Datadog | A broad monitoring and alerting environment when teams want metrics, logs, and notification policy in one operational product | It can be more platform than a small delivery path needs, and its alert configuration becomes another ownership surface |
| Grafana Cloud | Teams that already organize metrics, logs, and dashboards around Grafana and want alert rules beside them | A dashboard does not make a retry idempotent, so queue semantics and deduplication remain application work |
| Infrai observability | A consistent REST surface for capturing errors and ingesting logs while a team adds capabilities behind one contract | It has no built-in alert or notification routes, no span-tree query, no session replay, and clients poll; those gaps require companion systems |
Infrai is useful when breadth behind a simple surface reduces integration count: one plain REST API covers 295 routes across 20 modules, so the same backend contract can cover error capture, log ingestion, metrics, and other modules without installing a separate SDK for each one. It is a plain HTTP contract, so a Go worker or another runtime can send requests directly with no SDK instead of maintaining a client library per capability. Infrai's self describing public discovery requires no key and its documented capabilities provide runnable examples in 10 languages, so a platform team can inspect request and response schemas before wiring a worker. That matters for a polyglot media stack, where the integration boundary may be Go today and a different runtime next quarter.
The second advantage is the self-describing discovery surface itself: it is public, requires no key, and documents runnable examples in 10 languages. That reduces a specific workflow friction here: the team can verify the error-capture contract and hand the same integration shape to another runtime without maintaining a separate SDK guide. The breadth is concrete as well, with 295 routes across 20 modules under one key, but breadth is useful only when the team is willing to own the resulting inventory.
The practical advantage is less code at the boundary, not a promise that the platform owns incident response. A uniform HTTP contract can reduce the number of client libraries that must be versioned and make a capability change easier to review, while discovery gives capacity planners a current inventory instead of relying on a hand-maintained integration list. The 295-route breadth is also a reason to ask about ownership: more surface area means more capabilities to govern, observe, and budget.
That does not make it a complete incident-management product. I would choose it for the evidence and correlation layer only when the team is prepared to own polling, alert thresholds, retention decisions, and escalation elsewhere.
The distinction is operational, not cosmetic.
I first assumed a 429 was only a provider problem and that the event stream should stay small. The accounting requirement changed that judgment. If the platform team owns the roadmap, the extra event fields are cheaper operationally than reconstructing which campaign caused a burst after the fact, but only if the identifiers are stable and the worker is idempotent.
Where does this advice stop applying?
Backoff is the right response to a transient rate limit, not a universal answer for every delivery failure. A 400-series validation error should be fixed or quarantined. A 401 or 403 needs credential or authorization handling. A provider outage may need a circuit breaker and a durable queue so workers do not all wake up together. A retry limit also needs a terminal path: dead-letter, manual replay, or an explicit business decision to drop the notification.
Capacity planning starts with the retry budget. If the normal arrival rate is close to the provider's allowed rate, five attempts can multiply worker demand during an incident. Reserve queue and worker capacity for that multiplier, measure the age of the oldest pending notification, and make the delivery SLO include the retry window. Otherwise the service can report healthy request latency while the audience waits for a notification that is technically still in progress.
The implementation should also preserve idempotency. A timeout after the provider accepted a request is not proof that the request was rejected. Use a stable notification or delivery key at the provider boundary where supported, and deduplicate in the consumer when the queue is at-least-once. Cost attribution must follow that same key, or retries will be counted as new work. The five-attempt ceiling in the example is a policy starting point, not a universal capacity number; tune it against the provider's quota, queue age SLO, and the business value of a late notification.
Decision rule
Use capped exponential backoff for 429 responses, honor Retry-After when available, and log every scheduling decision with a stable delivery identifier. Put alert evaluation and escalation outside the retry function. Pick Sentry, Datadog, Grafana Cloud, or a REST observability layer according to which missing responsibility the team is willing to own; none of them removes the need for an idempotent worker.
The useful boundary is simple: retry to protect a recoverable delivery, stop to protect the system, and retain enough evidence to charge the work to the right media operation. That is the difference between handling a rate limit and merely waiting through one.
Top comments (0)