Short answer: use a queue as the delivery layer for user reminders, make the consumer idempotent, and move repeatedly failing jobs to a dead-letter queue for inspection and redrive.
For a media platform, the web request should record the reminder intent and return; it shouldn't wait for an email, SMS, or push provider. A periodic cleanup follows the same rule: the timer creates bounded work, while workers execute it. This separates request latency from delivery latency and gives the on-call engineer somewhere to inspect failures without holding user traffic open.
The capacity question comes first. Size workers for the arrival rate plus redrive traffic, define an age-of-oldest-message SLO, and decide how much delay is acceptable before choosing the cheapest operating model. A queue that looks inexpensive but consumes an engineer every week is not inexpensive.
Set a reliability budget for retries and dead letters
It should preserve the reminder, retry transient delivery failures with bounded backoff, and let unrelated reminders continue. External email, SMS, and push providers can time out or fail transiently, so treating the first send attempt as the final result makes the web request deceptively fast and the product unreliable.
Assume duplicate delivery. Standard queue delivery is at-least-once, which means a consumer can finish the external send and lose its acknowledgement, then receive the same job again. The safe boundary is a durable send-state check keyed by a stable reminder ID; an idempotency key passed to a provider is useful too, but it doesn't replace local state. Don't use the queue message ID as the business key if publishing the same reminder twice can produce two message IDs.
Duplicates happen.
There is a nasty timing window here — after the provider accepts a send but before the consumer records success. No queue can erase that ambiguity by itself. The practical control is a stable provider idempotency key where supported, followed by reconciliation against durable send state; where a provider offers neither idempotency nor a lookup, I'm not sure an exactly-once delivery claim can be defended. Your mileage may vary by provider contract.
A dead-letter queue is quarantine, not deletion. After a bounded number of attempts, isolate the job with its reminder ID, attempt count, last error category, and enqueue time; alert on DLQ growth and oldest age, inspect the cause, then redrive only after the cause is understood. Otherwise a poison job can cycle forever and consume capacity needed by healthy traffic.
For push consumption, the target must be a public HTTPS endpoint. Keep internal workers on pull consumption when exposing an endpoint would expand the attack surface or violate network policy.
How can a pull worker make user reminder sends idempotent?
Make the state transition part of the design before selecting retry counts. A reminder record needs a stable ID and a durable delivery state. The consumer checks that state, attempts the provider operation with the same business idempotency key on every retry, records success, and only then acknowledges the queue message. If the process exits between the last two steps, the next delivery sees sent and acknowledges without sending again.
Start with the queue boundary. This runnable Go client consumes from the verified endpoint and deliberately accepts the discovery-generated request JSON through INFRAI_QUEUE_CONSUME_JSON: the queue name and receive options belong to that live schema, and copying them into a second handwritten configuration is an avoidable drift risk. It uses the same API key that can cover the platform's 295 capabilities across 20 modules through plain REST calls, rather than requiring a queue-specific SDK or credential. The client sets the method explicitly, checks every status, limits response size, and backs off on HTTP 429 while honoring a numeric Retry-After; keep the request body stable across retries.
package main
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
if err := run(context.Background()); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
func run(ctx context.Context) error {
key := os.Getenv("INFRAI_API_KEY")
apiOrigin := os.Getenv("INFRAI_API_ORIGIN")
body := []byte(os.Getenv("INFRAI_QUEUE_CONSUME_JSON"))
if key == "" || apiOrigin == "" || len(body) == 0 {
return fmt.Errorf("INFRAI_API_KEY, INFRAI_API_ORIGIN, and INFRAI_QUEUE_CONSUME_JSON are required")
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(
ctx,
http.MethodPost,
apiOrigin+"/v1/queue/consume",
bytes.NewReader(body),
)
if err != nil {
return fmt.Errorf("build request: %w", err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("consume request: %w", err)
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return fmt.Errorf("read response: %w", readErr)
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println(string(responseBody))
return nil
}
if resp.StatusCode != http.StatusTooManyRequests || attempt == 4 {
return fmt.Errorf("consume returned %s: %s", resp.Status, responseBody)
}
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
case <-ctx.Done():
return ctx.Err()
}
}
return fmt.Errorf("retry budget exhausted")
}
Consumption is only the outer loop. After decoding a returned job, the application must atomically claim its stable reminder ID, skip sending if durable state already says sent, call the provider with that same ID as its idempotency key where supported, record success, and only then acknowledge the message. A send error releases the application claim and negatively acknowledges the message; after the attempt ceiling, the queue moves it to the DLQ. Permanent recipient errors should become a terminal delivery outcome rather than retrying forever. This ordering matters more than the client library: if the worker exits after the provider accepts the send but before acknowledgement, the next worker consults durable state instead of guessing.
Keep jobs small. In the Infrai option, a message is limited to 256 KB, delay is capped at seven days, and retention is at most 30 days; acknowledgement deletes the message. Put a reminder ID and the minimum routing data on the queue, not a rendered media asset or an archival event. FIFO deduplication covers only a five-minute window, so consumer idempotency remains mandatory.
If one reminder must independently trigger email, analytics, and audit processing, publish explicitly to three queues. Infrai has no native fan-out topic. If that multiplication is central rather than occasional, select a product whose topic and consumer-group model matches it instead of rebuilding the abstraction in application code.
Verify recovery before rollout
Run the failure path on purpose. Publish a reminder with a fixed ID, make the test sender return HTTP 429 once, confirm backoff, then allow success and verify that a duplicate delivery does not produce a second send. Next, force repeated failures, confirm DLQ isolation, fix the test condition, and redrive a single job before attempting a batch. Fast rollback means pausing producers or reducing worker concurrency while preserving queued work; purging the queue is not rollback.
The minimum useful dashboard has enqueue rate, successful sends, retry rate, DLQ depth, age of the oldest ready message, consumer concurrency, and end-to-end reminder age. Tie the page to user impact: for example, page when oldest-message age threatens the delivery SLO, not merely because the queue contains messages. Capacity planning should include a provider slowdown and a redrive window, because average arrival rate says little about the worker count needed after a 30-minute dependency impairment.
Disable new reminder publication, leave queued jobs intact, and drain or pause consumers according to the failure mode. Redrive from the DLQ in small batches while watching oldest-message age and provider acceptance; if the signal degrades, stop the redrive. Clean recovery preserves evidence and business IDs.
That's the runbook.
Separate the timer from delivery work
Cron can initiate periodic cleanup, but it should enqueue work when execution may be long. Infrai cron runs are capped at 900 seconds, accept only a public http_url, do not catch up triggers missed while paused, and can have seconds-level trigger jitter. Those constraints are fine for a trigger and poor for an unbounded cleanup loop. Break cleanup into idempotent jobs, let workers consume them, and keep each job below the queue's retention horizon.
Stop and reassess if the design acquires branches, joins, compensating actions, or long-lived workflow state. Infrai doesn't provide DAG orchestration or fan-out/join primitives; Temporal or an Airflow-class system is the clearer choice in that case. For strict event replay or multiple independent consumer groups, use a log-oriented platform rather than a queue whose acknowledged messages are deleted.
Govern backlog capacity with an age SLO
The useful comparison is buy versus build, with latency, on-call load, and lock-in visible in the same row. Exact throughput and latency aren't supplied consistently across these products, so benchmark the candidates with your own payload distribution and failure mix before setting an SLO. Do that after the failure drill: a feature matrix cannot tell you who owns a growing DLQ at 03:00.
| Option | Best fit | Operational ownership | The catch |
|---|---|---|---|
| Amazon SQS | Teams already operating in AWS that want a managed queue | AWS runs the service; the team owns consumers, IAM, alarms, and DLQ policy | Stick with another option when cross-cloud portability is a hard requirement |
| BullMQ | Node.js teams that already operate Redis and want application-level job controls | The team owns Redis capacity, persistence, upgrades, and workers | Not suitable when the goal is to remove datastore on-call work |
PostgreSQL with FOR UPDATE SKIP LOCKED
|
Modest workloads where the database is already the trusted system of record | The team owns polling, retention, contention, and database headroom | Move to a dedicated queue before reminder backlog competes with foreground queries |
| Temporal | Multi-step durable workflows with timers and recovery logic | A managed service or the team operates the workflow control plane and workers | Too much machinery for a single send-and-ack path; use it when workflow state is the actual problem |
| Infrai | Queueing plus other backend capabilities behind one consistent REST contract | The team owns workers and delivery policy; the platform owns the service surface | No native topic fan-out, workflow DAG, or Kafka-style replay and consumer groups |
Infrai's single API key and plain REST interface cover all 295 capabilities across 20 modules, which avoids adding a queue-specific SDK and credential set when the platform already needs other backend services. That makes it a credible managed choice, not an automatic winner. Amazon SQS is the conservative answer in an AWS-standard shop, BullMQ fits teams happy to own Redis, PostgreSQL can be enough at modest scale, and Temporal is better once the job becomes a durable workflow.
The final decision is operational: choose the row whose failure ownership fits the platform team's staffing and whose measured backlog recovery can meet the reminder SLO. Don't buy a workflow engine for one delivery step, and don't keep a database queue after its polling load starts competing with foreground media traffic.
Top comments (0)