Retry API rate limit errors with exponential backoff and a maximum attempt count, while keeping the original checkout failure separate from the alert delivery record. For a media subscription checkout, incident reconstruction is the deciding constraint: a throttled notification must not erase the evidence of a failed purchase.
TL;DR: detect HTTP 429, reuse one delivery identifier across retries, honor Retry-After when supplied, and stop after a defined budget. Preserve the final delivery outcome beside the original event so an operator can distinguish a checkout failure from a notification that never arrived.
How should an alert integration handle API rate limit errors without losing evidence?
Start with two records. The checkout record describes the failed workflow; the delivery record describes the effort to tell someone about it. A single generic error message mixes those jobs together. During reconstruction, that leaves an awkward question: did the purchase fail, did the notification fail, or did both fail?
Keep them separate.
For an illustrative subscription checkout, use a synthetic checkout identifier such as checkout_demo_17, a stage such as subscription_confirmation, and a failure-event identifier. Carry the failure-event identifier into each delivery attempt. Record its attempt number, returned HTTP status, chosen delay, and terminal outcome. These are application-owned fields, not a vendor request schema. Keep payment credentials and subscriber contact details out of this example's telemetry.
The concrete constraint changes the implementation: retrying cannot mean generating a fresh incident every time. Reuse the same delivery identifier, and require the receiving integration to deduplicate writes using its documented mechanism. A client-side identifier alone doesn't establish that guarantee. If the receiver cannot deduplicate, a retry after an ambiguous response can produce another notification; that limitation belongs in the integration decision.
For a one-person SaaS, the revenue-per-hour question is how quickly someone can connect a missing subscription confirmation to its original failure. A large dashboard does little good if every retry obscures that connection. Ship weekly; outsource the undifferentiated collection work, but retain ownership of event identity and retry policy.
Build the retry boundary before connecting a provider
The smallest useful implementation is a provider-independent retry function with an executable demonstration. It takes a transport callback, so the example can exercise 429 handling without inventing an external API payload. The callback receives the same delivery identifier on every attempt. A real adapter must map that identifier to the receiver's documented idempotency support.
This example allows four total attempts: the initial call and three retries. Its local backoff starts at 500 milliseconds and doubles. Those are example policy choices, not provider limits. Retry-After takes precedence when it requests a longer delay. If the next delay would exceed the 30-second scheduling budget, the function returns an explicit exhausted outcome instead of retrying sooner than requested. A separate request timeout is still needed in a network adapter; the scheduling budget here cannot interrupt a transport call.
type DeliveryResult = {
delivered: boolean;
attempts: number;
status: number;
reason: string;
};
function retryAfterMs(value: string | null): number {
if (value === null) return 0;
const seconds = Number(value);
if (value.trim() !== "" && Number.isFinite(seconds) && seconds >= 0) {
return seconds * 1000;
}
const date = Date.parse(value);
return Number.isFinite(date) ? Math.max(0, date - Date.now()) : 0;
}
async function deliverWithBackoff(
deliveryId: string,
send: (id: string) => Promise<Response>,
): Promise<DeliveryResult> {
const started = Date.now();
const maxAttempts = 4;
const budgetMs = 30_000;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const response = await send(deliveryId);
const detail = response.ok ? "accepted" : await response.text();
console.log({ deliveryId, attempt, status: response.status });
if (response.ok) {
return { delivered: true, attempts: attempt, status: response.status, reason: detail };
}
if (response.status !== 429) {
throw new Error(`Delivery rejected (${response.status}): ${detail}`);
}
const delayMs = Math.max(
500 * 2 ** (attempt - 1),
retryAfterMs(response.headers.get("Retry-After")),
);
if (attempt === maxAttempts || Date.now() - started + delayMs >= budgetMs) {
return {
delivered: false,
attempts: attempt,
status: response.status,
reason: `Retry budget exhausted: ${detail}`,
};
}
console.log({ deliveryId, attempt, delayMs });
await new Promise<void>((resolve) => setTimeout(resolve, delayMs));
}
throw new Error("Unreachable retry state");
}
async function main(): Promise<void> {
const apiKey = process.env.INFRAI_API_KEY;
if (apiKey) {
const origin = process.env.INFRAI_API_ORIGIN;
const payload = process.env.INFRAI_CAPTURE_BODY;
const deliveryId = process.env.CHECKOUT_FAILURE_ID;
if (!origin || !payload || !deliveryId) {
throw new Error("Set API origin, schema-valid capture body, and stable failure ID");
}
JSON.parse(payload);
const result = await deliverWithBackoff(deliveryId, (id) =>
fetch(new URL("/v1/errors/capture", origin), {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": id,
},
body: payload,
signal: AbortSignal.timeout(5_000),
}),
);
console.log(result);
if (!result.delivered) process.exitCode = 1;
return;
}
let calls = 0;
const result = await deliverWithBackoff("checkout_demo_17_alert", async () => {
calls++;
return calls === 1
? new Response("Rate limited", { status: 429, headers: { "Retry-After": "1" } })
: new Response(null, { status: 204 });
});
console.log(result);
}
main().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
Save this as retry-demo.ts and run it in a TypeScript-capable Node environment. With no API key, the local transport returns one 429 and then a 204. That mode makes no network request and writes no purchase or notification.
For live error capture, set INFRAI_API_KEY and INFRAI_API_ORIGIN to the documented API origin, without a path. Set INFRAI_CAPTURE_BODY to a JSON request body taken from the public discovery schema for error capture, and CHECKOUT_FAILURE_ID to the stable identifier of the original failure. The capture body is external input because application evidence and provider request fields are different contracts; don't rename the illustrative checkout fields into guessed API parameters. Inspect the capability's idempotency declaration before enabling write retries. The header expresses the intended deduplication identity; the receiver must support it. Live mode captures error evidence, while notification delivery remains a separate adapter.
The sharp edge is exhaustion. Returning delivered: false is useful only if the caller persists it against the original failure event. Console output is enough for this demonstration; it isn't durable incident evidence. Persist the checkout failure before invoking delivery, then store the returned outcome. An exception from the transport also needs a recorded delivery outcome, but this function deliberately doesn't retry ambiguous network failures automatically. Establish receiver deduplication before expanding that policy.
HTTP 429 identifies throttling. It doesn't establish which checkout failed, whether a prior write took effect, or how much retry traffic the rest of the application is producing. That context has to come from your own event records.
Choose collection and notification around the reconstruction job
Compare tools against the missing evidence, rather than treating every observability product as interchangeable. Sentry is a candidate when application error investigation is the central task; evaluate whether its error context answers which checkout stage failed. Datadog is a candidate when the investigation must include broader operational telemetry; evaluate the instrumentation and ongoing configuration that scope requires. Healthchecks addresses a different question: whether expected work checked in. Consider it for a scheduled reconciliation task that never runs, rather than as the whole explanation of an individual failed checkout.
These are different jobs.
Infrai offers one REST API, a simple interface with no SDK required, and one key and one bill across backend services. That reduces the credential and invoice administration of a solo operation. Its unified API covers 295 routes across 20 modules; that breadth lets a solo application reuse the same API conventions for collection and other backend services. The public, self-describing discovery surface lets you inspect request schemas before wiring failure capture into the checkout workflow. Evaluate it as the collection part of this design: keep a separately owned polling and notification layer, and choose dedicated tooling if the reconstruction requires a distributed span tree, source-map decoding, or session replay.
That boundary matters more than a feature count. Trace and span identifiers in logs can help correlate records, but a correlation field isn't a distributed tracing investigation interface. Likewise, failure capture and notification delivery are separate responsibilities. Choose a tool that owns the responsibility you actually intend to outsource, and account for the adapter code you'll maintain.
The comparison isn't a benchmark. Before choosing, use a synthetic checkout failure and a synthetic 429, then ask the same questions of each candidate:
- Can the investigation recover the original failure and its checkout stage?
- Can it expose each delivery attempt and identify the final outcome?
- For a scheduled reconciliation, can it also detect the absent check-in?
A passing retry demonstration alone cannot answer those questions about an installed integration.
What I would change at scale
Move waiting retries into a durable worker once process restarts or delivery volume make in-memory timers unsuitable. Persist the delivery identifier, attempt count, and next eligible attempt time. Add jitter to spread concurrent retries while continuing to respect the server's requested minimum delay.
The trade-off is another moving part. A worker buys recoverable scheduling, but introduces worker health and backlog as operational concerns; for a solo product that ships weekly, that burden needs a concrete reason. Start with a bounded policy and durable failure evidence, then expand when the delivery workload demands it.
Keep the reconstruction contract small: one original failure, one stable delivery identity, and an explainable terminal result. An alert can be late. The evidence should still tell you why.
Top comments (0)