A rate-limited alert destination is not a reason to retry every failed request. In a healthtech incident, the useful record is the smallest set of events that lets an engineer reconstruct what happened: the source failure, the delivery attempt, the quota response, and the final disposition. Retrying blindly can turn one patient-impacting symptom into a burst of duplicate notifications and a noisier audit trail.
Short answer: treat HTTP 429 as a scheduling signal, honor Retry-After when it is valid, apply exponential backoff with jitter when it is absent, cap the number and age of attempts, and write one structured event for every attempt. Preserve the original alert identity across retries. The alert can be delayed; its evidence must not be rewritten.
Start with identity.
Then preserve time.
How should an alert handle API rate limit errors?
Start with the incident reconstruction question, not the retry library: what would someone need six hours later to distinguish a slow delivery from a lost alert? For a synthetic healthtech event, the durable record might describe a medication-service timeout for tenant clinic-042, correlation ID inc-7f31, and source timestamp 2026-10-29T09:14:03Z. It should not contain a patient name or an unnecessary payload copy.
The four signals worth retaining are:
| Signal | Why it matters |
|---|---|
| Original alert identity and source time | Joins every attempt to one failure |
| Attempt number and send time | Shows delay and retry amplification |
Destination response class and Retry-After value |
Separates quota pressure from authentication, validation, and server failure |
| Terminal disposition and retention pointer | Establishes whether the alert was delivered, deferred, or exhausted |
A 429 Too Many Requests response indicates that the client has sent too many requests in a given time window; RFC 6585 defines that status and permits a Retry-After response header. The header may contain a delay in seconds or an HTTP date. Parse it, validate it, and record the raw value for diagnosis. Do not treat a missing header as permission to retry immediately.
The boundary is important. A rate-limit event is operational evidence, not the alert itself. Store a compact envelope for every attempt and keep the original event immutable. This prevents a later retry from appearing to have happened at the time of the clinical or service failure.
The trade-off is deliberate: a compact envelope gives a cleaner reconstruction trail, while a full payload copy can make forensic review easier at the cost of more sensitive data and more retention work. The right boundary depends on the access policy and the incident's evidentiary requirements.
Which failures should retry logic treat differently?
The first design decision is classification. “The request failed” is too coarse to drive a safe queue. A quota response, a temporary network timeout, a malformed request, and an expired credential have different recovery paths.
| Failure | Default action | Evidence to retain |
|---|---|---|
429 |
Delay using Retry-After, then back off |
Status, header, attempt, next eligible time |
408 or a transport timeout |
Retry within a bounded budget | Timeout class and elapsed time |
500, 502, 503, or 504
|
Retry only when the operation is safe to repeat | Status and response request ID if supplied |
400, 401, 403, or 422
|
Stop and route for correction or credential handling | Sanitized response class and terminal reason |
This table is a policy starting point, not a promise that every endpoint behaves identically. A POST that creates a notification may duplicate work when repeated unless the receiver supports an idempotency key or the sender has a deduplication record. A retry decision therefore needs two inputs: the response class and the delivery operation's repeatability.
A common mistake is to use one exponential schedule for every failure. That hides configuration errors behind delayed noise and can keep sending protected data after the destination has rejected the request permanently. Another mistake is to retry a 429 at a fixed one-second interval. A fleet of workers then wakes together, recreates the burst, and produces another quota response.
This method has limits. It cannot make an unsafe, non-idempotent operation safe, and it cannot recover an alert after the source has already discarded it. For a destination with no deduplication contract, the safer choice may be to stop after a short budget and hand the event to a human-reviewed queue rather than pursue delivery indefinitely.
That boundary matters most during a busy clinical workflow. Suppose the source emits one medication-service failure, the first delivery receives 429, and four workers independently notice the same pending item. A retry policy that only checks status codes may schedule four more sends, each with a plausible delay, while the receiver sees five notifications for one failure. Identity, a single queue claim, and a terminal age rule resolve that ambiguity; adding more retries does not.
How should a bounded retry schedule work?
The schedule needs four limits: a maximum attempts count, a maximum elapsed age, a maximum delay, and a queue-level concurrency limit. The exact values belong to the incident policy. For example, a team might allow 5 delivery attempts over 15 minutes, cap an individual delay at 60 seconds, and move the alert to a durable review queue afterward. Those numbers are policy examples, not universal healthtech defaults.
Use full jitter so workers do not share the same wake-up time. If the server supplies a valid delay, it takes precedence, subject to the local maximum delay and maximum age. If it supplies no usable delay, calculate a backoff such as min(cap, base * 2 ** (attempt - 1)) and choose a random delay from zero through that value. The random choice changes timing; it must never change the alert identity or attempt number.
Here is a small policy function. It deliberately returns a decision instead of performing I/O, which makes the boundary testable without contacting an alert destination.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
import email.utils
import random
from typing import Optional
@dataclass(frozen=True)
class RetryDecision:
retry: bool
delay_seconds: float
reason: str
def parse_retry_after(value: Optional[str], now: datetime) -> Optional[float]:
if not value:
return None
try:
seconds = float(value.strip())
return max(0.0, seconds)
except ValueError:
pass
try:
retry_at = email.utils.parsedate_to_datetime(value)
if retry_at.tzinfo is None:
retry_at = retry_at.replace(tzinfo=timezone.utc)
return max(0.0, (retry_at - now).total_seconds())
except (TypeError, ValueError, OverflowError):
return None
def decide_retry(
status: int,
attempt: int,
first_attempt_at: datetime,
now: datetime,
retry_after: Optional[str],
rng: random.Random,
max_attempts: int = 5,
max_age: timedelta = timedelta(minutes=15),
base_seconds: float = 1.0,
max_delay_seconds: float = 60.0,
) -> RetryDecision:
retryable_status = status == 429 or status in {408, 500, 502, 503, 504}
age = now - first_attempt_at
if not retryable_status:
return RetryDecision(False, 0.0, "non_retryable_status")
if attempt >= max_attempts:
return RetryDecision(False, 0.0, "attempt_limit")
if age >= max_age:
return RetryDecision(False, 0.0, "age_limit")
server_delay = parse_retry_after(retry_after, now) if status == 429 else None
ceiling = min(
max_delay_seconds,
server_delay if server_delay is not None else base_seconds * (2 ** (attempt - 1)),
)
delay = rng.uniform(0.0, max(0.0, ceiling))
if age + timedelta(seconds=delay) > max_age:
return RetryDecision(False, 0.0, "age_limit")
return RetryDecision(True, delay, "rate_limited" if status == 429 else "transient_failure")
The function does not claim that a retry succeeded. It only schedules an eligible attempt. The worker must record the decision, sleep or defer through a durable queue, and then create a new attempt event. If the process dies during the wait, the queue should retain enough state to resume without inventing a new alert.
There is a subtle boundary around Retry-After: honoring a value of several hours may violate the alert's useful age, while ignoring it may prolong quota pressure. The local age limit is the tie-breaker. When it wins, mark the event deferred or exhausted and preserve the server's instruction as evidence.
How do you prevent retries from becoming a second incident?
Retries need an identity model. Use a stable alert ID for the original condition and a separate attempt ID for each delivery. Include a deduplication key derived from the alert ID at the destination when the destination supports one. Do not derive it from the current timestamp; that makes every retry look new.
The event envelope can stay small:
from datetime import datetime
def attempt_event(
alert_id: str,
attempt_id: str,
attempt: int,
status: int | None,
disposition: str,
occurred_at: datetime,
next_attempt_at: datetime | None,
) -> dict:
return {
"alert_id": alert_id,
"attempt_id": attempt_id,
"attempt": attempt,
"status": status,
"disposition": disposition,
"occurred_at": occurred_at.isoformat(),
"next_attempt_at": next_attempt_at.isoformat() if next_attempt_at else None,
}
The payload is intentionally absent. An evidence record should point to a controlled, access-logged payload store when reconstruction requires more context. In healthtech, retention and access policy are part of the design; copying the full alert into every retry log increases exposure without improving the decision.
Noise has a second source: alerting on each retry. Page on the original high-severity condition and expose retry count, age, and terminal disposition as correlated fields. A separate operational alert can fire when a queue's rate-limit ratio crosses a tested threshold. The two alerts answer different questions.
What should be tested before a rollout?
Test the policy as a state machine, including the awkward edges that a happy-path integration test skips: a numeric Retry-After, an HTTP-date value in the past, a malformed header, a response with no header, a worker restart after scheduling, and two workers claiming the same alert. Assert that the original alert ID stays constant, attempt IDs do not repeat, and the age limit stops retries even when the response remains 429.
Use a deterministic random generator in unit tests. Production jitter should be random, but a test that cannot reproduce its delay is testing less than it appears to test. Add a contract test against a controlled endpoint that returns 429, then a successful response, and verify that only one terminal delivery is recorded.
During rollout, start with shadow metrics rather than changing paging behavior immediately. Measure the fraction of attempts receiving 429, the p95 age at terminal disposition, duplicate delivery count, queue depth, and the percentage of events missing a correlation ID. These measurements expose whether a retry policy improves evidence quality or merely creates more records.
A compact decision rule is enough for operators: retry only when the failure is plausibly temporary and the operation is safe to repeat; defer according to server guidance; stop at the age or attempt boundary; retain one immutable record per attempt. The objective is not maximum delivery persistence. It is a reconstruction trail that remains truthful when the destination is busy.
Top comments (0)