Short answer: when a push, email, or SMS provider returns 429 for user reminders, keep the subscriber delivery in a durable queue, apply channel-specific backoff, and redrive it only while its stable idempotency key and expiry make another request safe.
In a healthtech system, the periodic cleanup must run outside the web request that created or changed a reminder. Its first duty is less obvious than deleting old rows: it must preserve any delivery that is queued, in backoff, or awaiting controlled redrive. A provider's 429 Too Many Requests is an admission-control signal, not evidence that an email or SMS is permanently undeliverable. If the cleanup job mistakes "old" for "finished," it can erase the only durable evidence that a subscriber is still owed a reminder.
This design chooses at-least-once processing with explicit deduplication over an exactly-once claim. Exactly once isn't a property a queue can manufacture across a database and an external messaging provider. The operational target is narrower: no reminder is silently dropped, duplicates are bounded by an idempotency contract, and a channel-wide rate limit doesn't consume the latency budget of every other channel.
Security and retention policy define deletion authority
The diagnostic signature is a rising 429 response count accompanied by retryable queue depth and increasing oldest-message age. Throughput may still look healthy in aggregate, which is why a single global graph is weak evidence: email can be saturated while push remains below its limit, or one SMS account can be constrained while another is not. Split the signals by provider account, channel, and response class, but keep subscriber identifiers out of metric labels. Those identifiers have high cardinality and, in a health context, don't belong in operational telemetry.
HTTP defines 429 Too Many Requests as a rate-limiting response, and the server may include Retry-After to indicate how long the client should wait. "May" matters. If the header is absent or malformed, the consumer needs a bounded local backoff policy; if it is present and valid, immediately retrying on a shorter timer defeats the provider's instruction and turns a temporary limit into queue contention.
Look at four timestamps for one opaque reminder ID: scheduled time, first-attempt time, latest-attempt time, and expiry. Then inspect the attempt outcome and the next eligible time. That trace distinguishes a scheduling delay from provider throttling and from a worker that is repeatedly leasing work before it is eligible. Don't start by increasing worker count. More consumers aimed at the same constrained provider can raise request pressure while making the queue-age graph look briefly active.
More workers won't fix that.
One short rule helps: 429 changes eligibility, not ownership. The worker records the attempt, computes next_attempt_at, and releases the item without acknowledging it as delivered. A 2xx response can move the record to delivered; a locally defined permanent outcome can move it to terminal; expiry moves it out of the live queue under a documented business rule. The periodic cleanup deletes only terminal records older than the organization's retention boundary, and it reports counts before it removes anything.
This is a governance boundary as much as a queue rule. The team that owns the delivery SLO must approve the terminal-state definition; the privacy or records owner sets retention; the platform team supplies leases, admission controls, and recovery tooling. Cleanup receives authority to delete only where those decisions overlap. That separation prevents an operator under backlog pressure from turning retention into an improvised queue-drain mechanism.
Cost follows recovery ownership
The buy-versus-build decision is mostly an ownership decision. A managed scheduler can remove control-plane maintenance; a durable workflow engine can make multi-step state visible; a self-hosted queue can preserve infrastructure control. None of those labels answers how an external provider deduplicates a request.
| Option | Useful boundary | Team obligation | Poor fit |
|---|---|---|---|
| Self-hosted database queue and workers | Queue schema, leases, and rate buckets remain under team control | Operate polling, failover, migrations, DLQ tooling, and on-call | A small team that cannot staff queue operations |
| Inngest | Event-triggered functions include documented retry and concurrency controls | Validate provider idempotency and map function behavior to the delivery SLO | Teams requiring all scheduling control state inside their own infrastructure |
| Temporal | Durable workflows and Activities model retries as workflow state | Operate or procure the service and learn workflow determinism constraints | A single, simple periodic job where workflow machinery adds unjustified load |
| Amazon EventBridge Scheduler | One-time and recurring schedules can target AWS services with retry and DLQ settings | Design downstream admission control and account for platform coupling | A portability requirement that excludes AWS-specific targets and policy |
Stick with a managed option when reduced control-plane on-call work outweighs platform coupling and its documented retry model matches the SLO. Stick with self-hosting when data placement, custom admission control, or portability warrants the operational burden. For a healthtech cleanup task, the selection review should require a restore test and a redrive drill, not a feature checklist; delivery guarantees become credible only after the team has observed lease expiry, duplicate suppression, and backlog recovery under controlled conditions.
Cost belongs in the capacity sheet, but it shouldn't lead the architecture. Model scheduled invocations, retained state, worker runtime, outbound requests, telemetry, and engineer on-call time under both normal volume and a provider-throttled backlog. Your mileage may vary because retry amplification depends on the actual limit signals and arrival pattern. The honest comparison records those assumptions next to the number.
How should a subscriber queue back off push, email, and SMS user reminders after 429?
Start with separate rate-control buckets for each provider account and channel. A global queue can remain the durable source of work, but admission must happen at the narrowest limit you can identify. This keeps an SMS cap from blocking email and push, and it gives the on-call engineer a concrete control to lower without redeploying the application. The bucket key is operational configuration; it isn't a subscriber attribute.
The state transition should be small enough to write on a whiteboard:
- Lease only a record whose
next_attempt_athas passed and whose expiry has not. - Acquire capacity from the matching provider-and-channel bucket.
- Send an opaque delivery key with the request when the provider supports idempotency. Otherwise, record the provider message ID and accept that a lost response can leave duplicate risk.
- On
429, parseRetry-After; when it is unavailable, use capped exponential backoff with jitter. Persist the next eligible time before releasing the lease. - Move exhausted or expired work to a dead-letter state with its attempt metadata. Redrive creates a new controlled attempt; it does not reset history or bypass admission control.
That last point is where many runbooks get dangerous. A DLQ is evidence, not overflow storage. Bulk redrive into the primary queue can recreate the same burst that caused throttling, so redrive needs its own rate, an upper bound on item age, and a dry-run count by channel. The catch is that at-least-once delivery is not suitable when the downstream action cannot tolerate a duplicate and offers no idempotency mechanism. In that case, keep the item for review or change the downstream contract; retry timing alone cannot solve the ambiguity.
Redrive stays paced.
Capacity planning begins with the promised reminder window, not CPU. Let B be eligible backlog, R the sustained admitted provider rate, and N the count of new reminders expected during recovery. The rough drain time is (B + N) / R, before accounting for further retries. Compare that estimate with the delivery SLO and expiry window. I'm not sure a provider's documented ceiling will equal the rate available to a particular account at a particular hour; a controlled load test and observed headers settle that question better than an optimistic constant.
Consider a recovery calculation rather than an incident claim: if the queue snapshot contains 9,000 eligible SMS reminders, the provider bucket is deliberately held at 40 requests per second, and another 3,000 reminders are expected during the drain, the optimistic floor is 300 seconds before retry overhead. That arithmetic is useful because it forces three decisions into the open. First, does five minutes fit the reminder-delivery SLO? Second, will any record expire during that window, in which case redrive would be wrong even though capacity exists? Third, can new work and recovery work share the 40-request budget without starving either lane? A sensible controller reserves capacity for new reminders, assigns the remainder to redrive, and recomputes from observed accepted requests rather than configured worker concurrency. If the SLO cannot tolerate the resulting drain time, the answer is not an unbounded consumer fleet — it is a pre-agreed degradation policy, more contracted provider capacity, or an earlier scheduling horizon.
Migration plan for moving delayed work out of web requests
Move this policy in two controlled steps: first make the queue record, rather than the web request, authoritative for attempt count and next eligibility; then replace any in-process retry timer with a worker lease. During the migration, one owner flag must select the old or new consumer for each record. Running both consumers against the same reminder is not a canary — it is an uncontrolled duplicate experiment.
The following program is deliberately local: its test server returns two 429 responses, one using delta seconds and one using an HTTP date, before accepting the reminder. It demonstrates parsing, capped fallback backoff, jitter, one stable delivery key, and a terminal attempt limit without embedding a commercial API route. Save it as main.go and run go run main.go.
package main
import (
"bytes"
"fmt"
"io"
"math/rand"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"time"
)
type Reminder struct {
ID string
Channel string
Attempts int
NextAttempt time.Time
ExpiresAt time.Time
}
func retryAfter(value string, now time.Time) (time.Duration, bool) {
value = strings.TrimSpace(value)
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second, true
}
if when, err := http.ParseTime(value); err == nil && when.After(now) {
return when.Sub(now), true
}
return 0, false
}
func fallbackDelay(attempt int, cap time.Duration) time.Duration {
base := time.Second
for i := 1; i < attempt && base < cap/2; i++ {
base *= 2
}
if base > cap {
base = cap
}
// Full jitter avoids synchronizing workers on one retry boundary.
return time.Duration(rand.Int63n(int64(base) + 1))
}
func deliver(client *http.Client, endpoint string, r *Reminder) (bool, error) {
if time.Now().After(r.ExpiresAt) {
return false, fmt.Errorf("reminder expired")
}
body := []byte(`{"template_id":"follow-up-24h","subscriber_ref":"sub_7f2"}`)
req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
if err != nil {
return false, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", r.ID)
resp, err := client.Do(req)
if err != nil {
return false, err
}
defer resp.Body.Close()
io.Copy(io.Discard, resp.Body)
r.Attempts++
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return true, nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return false, fmt.Errorf("terminal status %d", resp.StatusCode)
}
delay, ok := retryAfter(resp.Header.Get("Retry-After"), time.Now())
if !ok {
delay = fallbackDelay(r.Attempts, 30*time.Second)
}
r.NextAttempt = time.Now().Add(delay)
return false, nil
}
func main() {
responses := 0
provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
responses++
switch responses {
case 1:
w.Header().Set("Retry-After", "1")
w.WriteHeader(http.StatusTooManyRequests)
case 2:
w.Header().Set("Retry-After", time.Now().Add(time.Second).UTC().Format(http.TimeFormat))
w.WriteHeader(http.StatusTooManyRequests)
default:
w.WriteHeader(http.StatusAccepted)
}
}))
defer provider.Close()
r := &Reminder{
ID: "rem_20260814_7f2",
Channel: "sms",
ExpiresAt: time.Now().Add(24 * time.Hour),
}
for r.Attempts < 5 {
if wait := time.Until(r.NextAttempt); wait > 0 {
time.Sleep(wait)
}
delivered, err := deliver(provider.Client(), provider.URL, r)
if err != nil {
fmt.Println("terminal:", err)
return
}
if delivered {
fmt.Printf("delivered after %d attempts\n", r.Attempts)
return
}
fmt.Printf("attempt %d deferred until %s\n", r.Attempts, r.NextAttempt.Format(time.RFC3339))
}
fmt.Println("dead-letter: attempt limit reached")
}
Production code needs a transaction around lease state and next_attempt_at; an in-memory struct is only making the transition visible. The same rule applies in a Node.js worker. Keep retry policy in the queue consumer rather than scattering timers through request handlers, and use a database outbox if creating the reminder and publishing queue work must share a commit boundary. No web request should wait through the backoff.
There is another boundary worth stating plainly. An idempotency key helps only if the receiving provider defines and honors that contract. Reusing the reminder ID as a header does not, by itself, prove deduplication. Where that support is absent, store the uncertainty after a network interruption and avoid an automatic redrive that could send a second clinical reminder.
Evaluation gates for redrive and rollback
Before deployment, run a provider stub that returns 429 with delta-seconds, an HTTP date, a missing header, and a malformed header. Assert that no case creates a hot loop, that the delay cap is enforced, that the same idempotency key survives every attempt, and that an expired reminder cannot be sent. Then test two channels together: sustained SMS throttling must not increase push queue age. This is the isolation proof.
The release gate should compare p50 and p99 queue age, eligible backlog, attempt rate, 429 rate, DLQ ingress, and delivered-to-scheduled latency against explicit SLO thresholds. Alert on burn rate and oldest eligible work, not raw queue depth alone; a large future-scheduled queue can be healthy, while a small set of old reminders can represent a user-visible miss.
Redrive is a change, so give it a change budget. First count eligible DLQ records by channel and age, exclude expired and ambiguous deliveries, sample the stored reason, and start below the currently observed successful rate. Pause automatically if 429 ratio or queue-age burn exceeds the rollout threshold. A redrive record should retain its original reminder ID, attempt history, operator or automation reason, and timestamp.
Rollback is intentionally boring: set the new consumer's admission rate to zero, stop new leases, allow in-flight leases to expire, and return ownership to the previous consumer version. Do not delete queue rows, reset attempt counters, or run cleanup while ownership is uncertain. The cleanup scheduler can resume after the old consumer demonstrates stable queue age and the redrive lane remains closed.
Done means more than "the 429 graph fell." It means the oldest eligible reminder is within SLO, no channel is starved by another, DLQ ingress has returned to its expected band, and terminal cleanup removed only records that the retention rule permits. Keep that evidence with the change record. Next time, diagnosis starts from state rather than folklore.
References
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- https://www.rfc-editor.org/rfc/rfc9110.html#name-retry-after
- https://www.inngest.com/docs
- https://docs.temporal.io/activities
- https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html
- https://opentelemetry.io/docs/specs/semconv/messaging/messaging-spans/
Top comments (0)