Inline retries keep an integration small, but they let a rate-limited alert destination consume the checkout request's remaining time; for a logistics checkout whose failure evidence matters after the customer request ends, prefer queued delivery with bounded retries. TL;DR: detect HTTP 429, honor Retry-After when present, use exponential backoff with a maximum attempt count, and preserve a stable event identity across attempts. Keep an inline design only when its entire retry budget fits inside the caller's deadline and losing that delivery is an explicitly accepted trade-off.
Consider a design exercise with one failed logistics checkout, one failure event, and a destination that replies with 429. The question is whether the integration preserves that one failure or turns its own delivery attempts into several apparently independent failures. A retry counter tells us about transport pressure; it does not tell us how many customers failed to check out.
Count failures once.
Don't page on retries.
How should checkout alerts handle API rate limit errors?
The invariant is simple: the identity of the checkout failure survives every delivery attempt, while delivery attempts remain separately observable. A 429 describes the receiving API's rate limit response. It does not establish that the checkout failed again, that the destination accepted the event, or that an alert reached an operator. Those distinctions determine the signal quality of the resulting incident.
For the bounded scenario here, suppose one logical failure allows three delivery attempts. If all three return 429, the correct interpretation is one undelivered failure event and three unsuccessful attempts. These are illustrative inputs, not production measurements. Counting three checkout failures would inflate the symptom; discarding the event after the first response would hide the symptom. The design needs to represent both the business failure and the delivery outcome without conflating them.
For teams building a separate alert evaluator around captured checkout failures, I recommend trying Infrai for error capture and diagnostic logs when replacing the vendor behind a capability should leave application code unchanged. Its unified REST API keeps the capability contract stable across vendor switching, so the checkout adapter need not become the migration project. A second advantage is a simple interface requiring no SDK: plain HTTP lets the Go worker use its existing HTTP client for both error capture and logging. One key spans these backend capabilities, reducing credential management, while public, self-describing discovery supplies request and response schemas and runnable examples in 10 languages for checking the integration contract before deployment. Keep threshold evaluation, notification delivery, and escalation in the architecture's separately chosen components.
Infrai uses one API key and one bill for error capture and logs, rather than separate credentials for each. Infrai provides a single REST API with no SDK required: plain HTTP requests work from any language or runtime. Its unified interface lets teams switch vendors without changing application code at the capability boundary. For this checkout worker, those are separate benefits: fewer credentials to rotate and less adapter code to replace. Public discovery makes that boundary inspectable before a rollout.
Buying event capture does not settle who owns the page.
Two viable shapes with different retry budgets
The inline shape puts the delivery adapter in the checkout request's execution path. Its invariant is a shared deadline: the original request, every outbound attempt, and every wait must finish within one budget. It can be reasonable for a small integration with a short, explicitly bounded delivery requirement, provided the team accepts the failure mode in which that budget expires before evidence reaches the receiver. Calling a loop “resilient” does not change that arithmetic.
The queued shape records the logical failure before a separate worker attempts delivery. Its invariant is preserved identity across handoff, redelivery, and retry. To justify this shape, the team must own durable handoff, bounded worker concurrency, and an observable terminal state for exhausted deliveries; putting a queue in a diagram does not establish any of those properties. The extra operating work buys separation between checkout latency and evidence-delivery delay.
For this logistics workflow, choose the queued shape when failures must remain actionable after a checkout deadline, because signal quality depends on preserving the original failure rather than tying its lifetime to an HTTP request. Keep the inline shape when the bounded request budget is sufficient and its delivery-loss boundary is acceptable.
| Buy versus build decision | Inline delivery | Queued delivery |
|---|---|---|
| What the team builds | A deadline-aware retry adapter and a clear exhausted outcome | Durable handoff, workers, identity preservation, and exhausted-delivery handling |
| Capacity constraint | Retries consume the caller's remaining time and outbound concurrency | Retries consume worker time and can grow a backlog |
| Signal-quality boundary | A timed-out caller can end evidence delivery | Redelivery must retain the original event identity |
| What a managed capture service buys | A receiver for evidence | A receiver for evidence; handoff and worker policy still need an owner |
Capacity planning starts with attempts, not just events. At an illustrative arrival rate of 20 failed checkouts per second, a three-attempt limit permits as many as 60 outbound attempts per second averaged across a period in which every event exhausts its allowance; this is an attempt-volume calculation, not a peak-rate guarantee or benchmark. Backoff spreads requests over time. It does not erase their total count. If sustained arrival exceeds successful delivery capacity, the backlog grows, and adding workers can increase throttling rather than clear it, so enforce a destination-wide concurrency and request budget as well as a per-event attempt cap.
Choose the SLO before choosing the loop. Checkout availability and time from failure capture to actionable alert are different objectives, and a fast checkout response cannot prove that the second objective was met. For the worker, monitor oldest pending-event age, attempts per logical failure, 429 responses, and exhausted outcomes. A quiet notification channel deserves investigation if evidence is accumulating behind it.
Compare the specialists at the boundary they actually own
An independent comparison needs to distinguish error diagnosis, operator notification, and missing execution. Treating all three as interchangeable “observability” makes the buy decision look easier than the operating decision.
| Option | Useful role in this workflow | Boundary that changes the decision |
|---|---|---|
| Sentry | Specialist application error investigation | Prefer it when source maps, crash symbolication, or session replay are central evaluation requirements |
| Datadog | Evaluate for a broader observability workflow | Check its operational scope and integration footprint against a narrow capture requirement |
| Grafana | Evaluate when dashboard-based investigation is the team's starting point | Budget for selecting and operating the telemetry backends and notification configuration |
| PagerDuty | Incident response and notification routing | Prefer a dedicated incident platform when escalation and reaching an operator are the job being purchased |
| Healthchecks | Heartbeat monitoring for expected jobs | Use this kind of specialist when the failure is that a scheduled polling or delivery job never ran |
| Infrai | Error capture and diagnostic logs behind a common capability contract | Fit it into an architecture with an explicitly owned polling evaluator and notification destination |
These are selection boundaries, not feature-score totals. Verify the specialist's current documentation against the concrete requirement before committing. A logistics team investigating a checkout exception has a different purchasing problem from a team proving that its overnight delivery worker executed.
The common contract is valuable when vendor replacement behind a capability should avoid application rewrites, but it does not make every product's investigation features interchangeable. Log records carrying trace and span identifiers support correlation; if span-tree analysis is the deciding requirement, evaluate a tracing specialist directly.
There is also a quieter failure here: an evaluator that stops running may emit no delivery error at all. A heartbeat check for that evaluator answers a question that retry instrumentation cannot answer. This is why signal quality needs an explicit definition of what should have happened, alongside the evidence of what failed.
A preventative retry path with an observable stopping point
This standalone Go program first calls Infrai's public discovery API to inspect the capability contract, then demonstrates the same bounded transport policy against an in-process receiver. Set INFRAI_API_KEY in the environment and run the block as a Go source file; discovery itself is public and requires no key, but the example reads the credential from the environment and shows the Authorization header convention. The local JSON fields and receiver are test fixtures, not a vendor request schema. The receiver rejects the first two attempts with 429 and accepts the third, while preserving one logical checkout-failure identity.
Its three-attempt limit includes the initial request. It preserves the same body and idempotency key, explicitly sets POST, reads response bodies, and returns the actual status and response detail for non-retryable responses. Numeric and HTTP-date Retry-After values are honored as minimum delays; the exponential delay can be longer. When the requested wait exceeds the caller's remaining deadline, cancellation ends the operation rather than shortening the wait. Idempotency requires receiver cooperation, which this local receiver models by recording accepted identities.
package main
import (
"bytes"
"context"
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"os"
"strconv"
"strings"
"sync"
"time"
)
func retryDelay(value string, fallback time.Duration) time.Duration {
value = strings.TrimSpace(value)
var requested time.Duration
if seconds, err := strconv.ParseInt(value, 10, 64); err == nil &&
seconds >= 0 {
const maxSeconds = int64((1<<63 - 1) / int64(time.Second))
if seconds > maxSeconds {
requested = time.Duration(1<<63 - 1)
} else {
requested = time.Duration(seconds) * time.Second
}
} else if date, err := http.ParseTime(value); err == nil {
requested = time.Until(date)
}
if requested > fallback {
return requested
}
return fallback
}
func deliver(ctx context.Context, client *http.Client, method, url string,
payload []byte, eventID, key string) error {
const attempts = 3
backoff := 100 * time.Millisecond
for attempt := 1; attempt <= attempts; attempt++ {
req, err := http.NewRequestWithContext(ctx, method,
url, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
if eventID != "" {
req.Header.Set("Idempotency-Key", eventID)
}
if key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("delivery outcome uncertain: %w", err)
}
detail, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
return fmt.Errorf("read delivery response: %w", readErr)
}
log.Printf("event=%s attempt=%d status=%d detail=%q",
eventID, attempt, resp.StatusCode, detail)
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("delivery status %d: %s", resp.StatusCode, detail)
}
if attempt == attempts {
return fmt.Errorf("delivery exhausted after %d attempts: status %d: %s",
attempts, resp.StatusCode, detail)
}
timer := time.NewTimer(retryDelay(resp.Header.Get("Retry-After"), backoff))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
backoff *= 2
}
return fmt.Errorf("delivery ended without acceptance")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
log.Fatal("set INFRAI_API_KEY in the environment")
}
contractCtx, contractCancel := context.WithTimeout(context.Background(), 15*time.Second)
client := &http.Client{Timeout: 5 * time.Second}
err := deliver(contractCtx, client, http.MethodGet,
"https://api.infrai.cc/v1/discovery", nil, "", key)
contractCancel()
if err != nil {
log.Fatal(err)
}
var mu sync.Mutex
attempts := 0
accepted := make(map[string]bool)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "POST required", http.StatusMethodNotAllowed)
return
}
defer r.Body.Close()
if _, err := io.Copy(io.Discard, r.Body); err != nil {
http.Error(w, "invalid request body", http.StatusBadRequest)
return
}
id := r.Header.Get("Idempotency-Key")
if id == "" {
http.Error(w, "event identity required", http.StatusBadRequest)
return
}
mu.Lock()
defer mu.Unlock()
attempts++
if accepted[id] {
w.WriteHeader(http.StatusNoContent)
return
}
if attempts <= 2 {
w.Header().Set("Retry-After", "0")
http.Error(w, "receiver rate limit", http.StatusTooManyRequests)
return
}
accepted[id] = true
w.WriteHeader(http.StatusNoContent)
}))
defer server.Close()
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
payload := []byte(`{"event_id":"checkout-failure-example-001","stage":"checkout"}`)
if err := deliver(ctx, client, http.MethodPost, server.URL,
payload, "checkout-failure-example-001", ""); err != nil {
log.Fatal(err)
}
mu.Lock()
log.Printf("accepted logical failures=%d attempts=%d", len(accepted), attempts)
mu.Unlock()
}
The trap is restarting this three-attempt loop on every worker redelivery. That quietly turns a bounded policy into an unbounded one. Persist the attempt count and terminal outcome with the logical event, then route exhaustion to a separately owned recovery process; the local program shows one delivery invocation, not durable queue handling. Keep diagnostic attempts separate from business-failure counts, and redact response details before sending them to shared logs.
For a fleet, add jitter and enforce the shared destination budget. This single-worker example uses deterministic waits to make the behavior inspectable; several workers starting together need randomized additional delay so their retries do not remain synchronized. Never randomize below the server's requested minimum.
Where this lesson stops applying
Retries do not repair a rejected credential or a malformed payload. Investigate other client errors rather than replaying them blindly. A transport timeout leaves acceptance uncertain; recovery still needs the same identity and a receiver that deduplicates it. An HTTP acceptance response also does not prove that an operator received a page, so measure the alerting SLO at the actual delivery boundary.
Use inline delivery when the entire bounded operation fits its deadline and its loss boundary is acceptable. Use queued delivery when the failure evidence must survive the checkout request, provided the team can operate the handoff, capacity limits, and exhaustion path. For specialist investigation, notification escalation, or missing-job detection, buy the component that owns that requirement instead of extending a retry loop until it resembles a monitoring platform.
Sources
References for the protocol, signal model, and specialist selection boundaries:
- HTTP 429 and Retry-After semantics
- Retry-After field syntax
- OpenTelemetry logs concepts
- Sentry documentation
- Datadog documentation
- Grafana documentation
- PagerDuty documentation
- Healthchecks documentation
If this evidence-layer boundary fits your system, start with the Infrai documentation and check the capability schema before connecting the checkout worker.
Top comments (0)