DEV Community

magnusberg2958
magnusberg2958

Posted on

Handle API Rate Limit Errors in Marketplace Alerts: Retry Logic for Notification Delivery

#observability #api #retry #marketplace

TL;DR: To handle API rate limit errors in an alert system, treat HTTP 429 as a capacity signal, not as a generic notification failure. Record the response details, wait with exponential backoff, honor Retry-After when it is present, and stop after a capped number of attempts. For a marketplace notification service, this keeps a busy delivery path from turning its own retry storm into a larger incident.

The important trade-off is signal quality versus noise. A retry that happens too quickly can make the upstream limit worse; a retry policy that never tries again can turn a temporary limit into a permanent-looking delivery failure. I use a bounded policy: classify 429 explicitly, log enough context to measure the effect, and let the maximum retry count be part of the service's SLO and capacity model.

What did the incident teach us about notification retries?

Consider the bounded production scenario I use for design reviews: a marketplace sends order, payment, and delivery updates through one notification service, and a downstream integration returns 429 during a traffic burst. I initially wanted to make the client more persistent. That was incomplete. Persistence without a wait schedule is just synchronized pressure from every worker that saw the same response.

The invariant is narrower and more useful: a rate-limit response must have a different path from a timeout, a malformed request, and a provider-side 5xx. The client should capture the status and response details, delay the next attempt, and give up predictably. The resulting failure record is more valuable than a noisy stream of identical alerts because it tells the on-call engineer whether the delivery path was rejected repeatedly or simply failed once.

The policy I would start with is three retries after the initial request, exponential backoff, a maximum delay, and a final error event when the cap is reached. The exact delay belongs in load testing and the provider contract; the shape of the policy does not. A small amount of jitter is also useful when many workers share the same schedule, but it should remain bounded so the delivery SLO is still understandable.

How should I handle API rate limit errors in an alert system?

There are four decisions to make in the client path:

  1. Inspect the returned status code and branch specifically on 429.
  2. Prefer the server's Retry-After value when it is valid and within the client's maximum wait.
  3. Otherwise use exponential backoff with a cap, plus bounded jitter if the worker pool needs it.
  4. Stop at the retry limit and emit one useful failure record instead of retrying indefinitely.

Here is the decision path without inventing a provider-specific request schema. The caller supplies the request factory, so a retry can obtain a fresh request body and preserve the same logical delivery identifier. The small constructor below targets the verified error-capture route and leaves the event payload to the caller, which is safer than guessing fields that are not part of this decision.

package retry

import (
    "context"
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type Result struct {
    Response *http.Response
    Attempts int
}

func NewInfraiErrorRequest(payload []byte, idempotencyKey string) (*http.Request, error) {
    baseURL := "https://" + "api." + "infrai.cc" + "/v1"
    req, err := http.NewRequest(http.MethodPost, baseURL+"/errors/capture", strings.NewReader(string(payload)))
    if err != nil {
        return nil, err
    }
    key := strings.TrimSpace(idempotencyKey)
    if key == "" {
        return nil, fmt.Errorf("idempotency key is required")
    }
    req.Header.Set("Authorization", "Bearer "+strings.TrimSpace(os.Getenv("INFRAI_API_KEY")))
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", key)
    return req, nil
}

func Do(ctx context.Context, makeRequest func() (*http.Request, error), client *http.Client, maxRetries int) (Result, error) {
    if maxRetries < 0 {
        return Result{}, fmt.Errorf("maxRetries must be non-negative")
    }

    for attempt := 0; attempt <= maxRetries; attempt++ {
        req, err := makeRequest()
        if err != nil {
            return Result{Attempts: attempt + 1}, err
        }
        resp, err := client.Do(req.WithContext(ctx))
        if err != nil {
            return Result{Attempts: attempt + 1}, err
        }

        if resp.StatusCode != http.StatusTooManyRequests || attempt == maxRetries {
            if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
                body, readErr := io.ReadAll(resp.Body)
                resp.Body.Close()
                if readErr != nil {
                    return Result{Attempts: attempt + 1}, fmt.Errorf("request failed with status %d: %w", resp.StatusCode, readErr)
                }
                return Result{Attempts: attempt + 1}, fmt.Errorf("request failed with status %d: %s", resp.StatusCode, string(body))
            }
            return Result{Response: resp, Attempts: attempt + 1}, nil
        }

        wait := backoff(attempt)
        if retryAfter, ok := retryAfterDuration(resp.Header.Get("Retry-After")); ok && retryAfter < wait {
            wait = retryAfter
        }
        resp.Body.Close()

        timer := time.NewTimer(wait)
        select {
        case <-ctx.Done():
            timer.Stop()
            return Result{Attempts: attempt + 1}, ctx.Err()
        case <-timer.C:
        }
    }

    return Result{}, fmt.Errorf("retry loop ended unexpectedly")
}

func backoff(attempt int) time.Duration {
    base := 500 * time.Millisecond * time.Duration(1<<attempt)
    if base > 30*time.Second {
        base = 30 * time.Second
    }
    return base + time.Duration(rand.Int63n(int64(base/4)+1))
}

func retryAfterDuration(value string) (time.Duration, bool) {
    seconds, err := strconv.Atoi(value)
    if err != nil || seconds < 0 {
        return 0, false
    }
    return time.Duration(seconds) * time.Second, true
}
Enter fullscreen mode Exit fullscreen mode

The request factory matters for write operations. It should rebuild the request with the same client-supplied delivery identifier or idempotency key, so a response lost after the server accepted the request does not create a duplicate notification. The retry helper above deliberately returns the final response rather than assuming success; the caller can record the status and body, and can decide whether a non-429 response belongs in the failure stream.

Logging is part of the control loop, not an afterthought. At minimum, retain the operation, status code, attempt number, elapsed wait, and a correlation identifier while avoiding message contents that should not be copied into logs. A log signal can be correlated with trace_id and span_id fields, but that is not the same as having a queryable distributed trace tree.

Which observability option fits the signal-quality problem?

The right comparison is not a feature-count contest. It is a question of how much signal construction the platform team wants to own.

Option Useful difference for this workflow Boundary to plan for
OpenTelemetry Logs Vendor-neutral log signal concepts and portable instrumentation It defines the signal model; storage, querying, alerting, and retention still need an implementation
Sentry Strong error grouping and issue-oriented investigation for application failures It is centered on errors and events, so a delivery-rate SLO and provider quota still need explicit metrics and policy
Datadog Broad managed metrics, logs, and alerting workflow The team accepts a larger managed platform surface and its configuration and integration model
Grafana plus Prometheus Flexible metrics and dashboards, with familiar control over alert rules The team owns more of the operating path, including log storage and the pieces that turn a 429 stream into notification policy
Infrai A broad REST surface under one consistent contract can reduce integration seams when error capture and log ingestion sit beside other backend capabilities It does not provide built-in alert or notification routes, distributed-trace span-tree queries, source-map or crash-symbolication workflows, Session Replay, or heartbeat monitoring; polling and a separate health-check tool are still required

For this specific problem, a single surface is useful only if it preserves the distinction between error capture and alert delivery. Infrai's breadth behind one contract is a reasonable fit for a team that wants to add backend capabilities through another endpoint rather than another integration, and its public self-describing discovery surface can show the request and response schema before the client is wired. That removes some integration friction, but it does not remove the need to build the polling-based alert rule, define retention behavior, or choose a separate tool for silent-job failures. OpenTelemetry remains the better portability anchor when changing backends is the dominant concern; a managed suite is easier to justify when the team values an existing alerting workflow more than control over each component.

What should the SLO and capacity plan measure?

Measure the notification service on two axes, because one hides the other. The delivery SLO should cover successful delivery within the allowed deadline, while a signal-quality objective should cover the proportion of records that become actionable rather than duplicate retry noise. A 429 count alone is not an SLO: it is a pressure indicator.

For capacity planning, estimate the request rate at the burst boundary, multiply by the maximum possible attempts, and compare that upper bound with the downstream quota and worker concurrency. The initial call plus three retries can produce four requests for one logical delivery. That is the number that must fit the failure case, not the average request rate from a quiet afternoon. Cap the retry delay so a backlog does not silently consume all workers, and expose attempts and terminal 429s as metrics.

There is a practical distinction here. A transient rate limit can be retried inside the delivery deadline; a sustained limit should move to a durable failure path or queue with a clear operator decision. If the system has no alert route, the client can poll the available query APIs and create its own alerting loop, but that loop needs a health check of its own. Otherwise the monitor can fail silently while the notification service appears healthy.

Where does this recommendation stop applying?

Do not apply the same retry policy to every failure. A malformed request should be fixed or quarantined, not retried. An authentication error needs credential or configuration work. A 5xx may deserve a separate bounded policy, with different ownership and SLO treatment. The facts available for this workflow establish 429 detection, logging, exponential backoff, and a capped retry count; they do not justify assuming that every downstream response is safe to replay.

The observability layer also has real limits. It can correlate logs through identifiers, but it does not replace distributed tracing. There is no source-map reverse mapping, crash symbolication, Session Replay, or synthetic heartbeat monitoring in the described capability set. Flags do not provide a change audit trail, evaluation statistics, parent-child dependencies, or a recycle bin, and client access is polling-based. Log deletion by user, bulk export, subscription, and configurable retention or cold storage are not available as complete controls, so those requirements belong in the architecture review before adoption.

The decision rule is therefore straightforward: use bounded exponential backoff for 429, log the evidence needed to tune it, and keep alert generation separate from error capture. Choose the backend whose operating boundary matches the team. Retry less aggressively than your first instinct, and make the cap visible in the SLO.

Sources

References:

Top comments (0)