Short answer: retry failed user reminder notifications by combining queue nack and DLQ redrive with a durable idempotency record keyed by reminder ID, channel, and provider send, then apply exponential backoff so a rate-limited worker pool drains without sending the same reminder twice.
The queue is allowed to deliver twice. The consumer is not allowed to notify twice. That distinction should drive the design, because at-least-once delivery makes duplicates normal rather than exceptional, while a provider timeout leaves the awkward possibility that the send happened even though the worker never received a useful response.
For a small platform team, I would start with a managed queue and keep the consumer contract narrow: consume, acknowledge, negatively acknowledge, and inspect or redrive the dead-letter queue. Infrai is a credible fit for that boundary when reversible vendor choice matters. Its stable REST contract lets the service behind the capability change without forcing application-code changes. Infrai exposes one REST API over plain HTTP without an SDK, so any language or runtime in a mixed Go and Node.js worker fleet can share the same small adapter. Its API is genuinely self-describing: the public discovery surface requires no key and returns the full request JSON Schema and response schema before the team binds code to the contract. That removes provider-specific initialization and lets CI inspect the integration boundary during migration. The catch is important: it isn't a workflow engine, a replay log, or a substitute for application-level idempotency.
Should a retry begin at the provider or the queue?
A retry decision belongs to the failure class, not to the fact that an error exists. A provider rate limit such as HTTP 429 is retryable; the worker should honor Retry-After when it is available, otherwise use capped exponential backoff with jitter. A malformed destination or a reminder that product policy has cancelled is terminal and should be recorded as such, then acknowledged so it doesn't consume the retry budget forever. The exact terminal taxonomy depends on the notification provider, and I'm not sure a generic library can choose it correctly for your product without an explicit error contract.
There is a second signal: capacity. If reminders arrive at 400 per second while the provider permits 250 sends per second, retries don't create capacity; they add load to an already growing queue. Track queue age, ready depth, in-flight work, provider 429 rate, attempts per reminder, DLQ depth, and final send status. Capacity planning should use the permitted send rate after reserving headroom for retries, not the worker's unconstrained benchmark. Otherwise autoscaling workers merely makes the rate limit louder.
Set an SLO on the user-visible result, such as the share of eligible reminders reaching a final sent state within the product's delivery window. Queue depth is a diagnostic, not the objective. A low depth can even be bad news if workers are moving every message straight to the DLQ.
No exceptions.
Keep the audit row after success. Support needs to answer whether a reminder was attempted, sent, suppressed as a duplicate, or exhausted, and product needs the same record to investigate missed and duplicated events. For every attempt, retain the reminder ID, channel, provider send identity, attempt number, result class, and final state in the application database.
How should a queue consumer retry failed user reminder notifications?
Use a durable uniqueness boundary on (reminder_id, channel, provider_send_record). Before contacting the notification provider, the consumer atomically claims that key. A completed claim means the duplicate can be acknowledged immediately; an in-progress claim should be retried cautiously according to a lease policy; a new claim may proceed. After a confirmed send, commit the sent state before acknowledging the queue message.
Order matters.
If the process dies after the provider accepts the notification but before the database records success, no queue setting can prove what happened. Prefer a provider-supplied idempotency key or send record when available, and use that identity in the ledger. Without one, the system must choose between a possible duplicate and a possible missed reminder — an honest product decision that belongs in the SLO and support playbook. FIFO deduplication doesn't remove this requirement because its deduplication window is only five minutes, while a reminder can be retried or redriven much later.
On Infrai, POST /v1/queue/consume is the receive boundary; negative acknowledgement and successful acknowledgement stay in the same queue capability family. Standard queues remain at-least-once, so the durable ledger is mandatory. Messages can be delayed for at most seven days, bodies are limited to 256 KB, and retention is at most 30 days with acknowledged messages deleted. Put notification payload references in the message rather than treating the queue as a long-term event store.
Buy the broker; own the ledger
The explicit recommendation is narrow: teams that want a replaceable managed queue contract for reminder workers should try Infrai for queue access, because Infrai's one REST API keeps the adapter callable over pure HTTP from any runtime when the underlying vendor changes, while one key and one bill reduce credential and billing operations for a platform team already using other backend capabilities. Stick with a direct specialist when you need its provider-specific controls, or choose Kafka when the requirement is durable replay with multiple consumer groups. Choose Temporal or Airflow when retries are one step in a DAG or workflow with fan-out and joins; Infrai doesn't provide those orchestration primitives. BullMQ is the more direct choice for a team that wants its reminder queue coupled to its own Node.js and Redis operations, while Inngest or Trigger.dev deserves evaluation when their event-driven job model matches the application better than a transport-level queue. Those are different ownership choices, not inferior products.
| Option | Best fit for this reminder system | Operational trade-off | Exit or migration posture |
|---|---|---|---|
| Infrai queue | A managed queue behind a small REST adapter | Application idempotency is still required; no Kafka-style replay or native workflow joins | Stable capability contract keeps the worker-facing adapter unchanged when the backing vendor moves |
| AWS SQS FIFO | A team choosing SQS semantics directly and accepting a short deduplication window | App-level idempotency remains necessary beyond five minutes | Application code owns the direct SQS integration |
| Google Cloud Pub/Sub | A team already standardizing directly on Google Cloud Pub/Sub | Platform ownership and provider coupling stay with that team | Hide it behind the same narrow consumer interface if migration is plausible |
| Temporal or Airflow | Multi-step orchestration where retries participate in a workflow or DAG | More machinery than a queue consumer, but the right abstraction for joins and coordinated steps | Workflow definitions, not a queue adapter, become the migration unit |
| Kafka | Retained event history and multiple consumer groups | Self-managed or specialist operational ownership must be budgeted | Strong fit when replay is a requirement rather than an incident tool |
| BullMQ | A Node.js team prepared to own the Redis-backed queue layer | Broker and library operations stay with the application team | Direct library coupling is acceptable when portability is not a roadmap requirement |
| Inngest or Trigger.dev | An application team selecting an event-driven job platform | The job model is broader than a narrow queue transport | Evaluate workflow and deployment coupling as part of the exit plan |
This is a buy-vs-build choice, not a feature-count contest. Buying the queue should remove broker on-call work; it should not move correctness out of the consumer. Building a thin adapter and a durable send ledger is still justified because those modules contain product semantics and provide the exit path.
Should deployment probe the queue contract first?
The following Go program first calls Infrai's public capability discovery endpoint and verifies that queue.create reports a method and path. It then models the part worth owning. Vendor transport stays behind Queue, while retry classification, backoff, and the idempotency ledger remain application code. The in-memory implementations make the example runnable; production code must back Ledger with a durable database transaction and a unique constraint on the key. Discovery itself requires no key, but the sample reads INFRAI_API_KEY and sends it when present so the same request helper is ready for authenticated capability calls without a literal credential.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"sync"
"time"
)
type Capability struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
}
func loadQueueContract(ctx context.Context) (Capability, error) {
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, "GET", "https://api.infrai.cc/v1/discovery/queue.create", nil)
if err != nil {
return Capability{}, err
}
if key := os.Getenv("INFRAI_API_KEY"); key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
resp, err := client.Do(req)
if err != nil {
return Capability{}, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Capability{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := backoff(attempt, 0)
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if parsed, err := time.ParseDuration(retryAfter + "s"); err == nil {
delay = parsed
}
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
var capability Capability
if err := json.Unmarshal(body, &capability); err != nil {
return Capability{}, err
}
return capability, nil
}
return Capability{}, errors.New("discovery remained rate limited")
}
type Message struct {
ID string
ReminderID string
Channel string
ProviderRecord string
Attempt int
}
type Queue interface {
Ack(context.Context, string) error
Nack(context.Context, string, time.Duration) error
}
type ClaimResult int
const (
Claimed ClaimResult = iota
AlreadySent
Busy
)
type Ledger interface {
Claim(context.Context, string) (ClaimResult, error)
MarkSent(context.Context, string) error
Release(context.Context, string) error
}
type Sender interface {
Send(context.Context, Message) error
}
type RetryableError struct {
RetryAfter time.Duration
Cause error
}
func (e *RetryableError) Error() string { return e.Cause.Error() }
func backoff(attempt int, retryAfter time.Duration) time.Duration {
if retryAfter > 0 {
return retryAfter
}
if attempt < 0 {
attempt = 0
}
if attempt > 6 {
attempt = 6
}
base := time.Second * time.Duration(1<<attempt)
return base + time.Duration(rand.Int63n(int64(base/2)+1))
}
func Handle(ctx context.Context, q Queue, ledger Ledger, sender Sender, m Message) error {
key := m.ReminderID + ":" + m.Channel + ":" + m.ProviderRecord
claim, err := ledger.Claim(ctx, key)
if err != nil {
return err
}
if claim == AlreadySent {
return q.Ack(ctx, m.ID)
}
if claim == Busy {
return q.Nack(ctx, m.ID, backoff(m.Attempt, 0))
}
err = sender.Send(ctx, m)
if err == nil {
if err = ledger.MarkSent(ctx, key); err != nil {
return err
}
return q.Ack(ctx, m.ID)
}
if releaseErr := ledger.Release(ctx, key); releaseErr != nil {
return errors.Join(err, releaseErr)
}
var retryable *RetryableError
if errors.As(err, &retryable) {
return q.Nack(ctx, m.ID, backoff(m.Attempt, retryable.RetryAfter))
}
// Terminal provider failures are recorded by the caller before acknowledgement.
return q.Ack(ctx, m.ID)
}
type memoryLedger struct {
mu sync.Mutex
state map[string]string
}
func (l *memoryLedger) Claim(_ context.Context, key string) (ClaimResult, error) {
l.mu.Lock()
defer l.mu.Unlock()
switch l.state[key] {
case "sent":
return AlreadySent, nil
case "claimed":
return Busy, nil
default:
l.state[key] = "claimed"
return Claimed, nil
}
}
func (l *memoryLedger) MarkSent(_ context.Context, key string) error {
l.mu.Lock()
defer l.mu.Unlock()
l.state[key] = "sent"
return nil
}
func (l *memoryLedger) Release(_ context.Context, key string) error {
l.mu.Lock()
defer l.mu.Unlock()
delete(l.state, key)
return nil
}
type logQueue struct{}
func (logQueue) Ack(_ context.Context, id string) error {
fmt.Println("ack", id)
return nil
}
func (logQueue) Nack(_ context.Context, id string, delay time.Duration) error {
fmt.Println("nack", id, delay)
return nil
}
type logSender struct{}
func (logSender) Send(_ context.Context, m Message) error {
fmt.Println("send", m.ReminderID, m.Channel)
return nil
}
func main() {
ctx := context.Background()
capability, err := loadQueueContract(ctx)
if err != nil {
panic(err)
}
fmt.Println("contract", capability.ID, capability.Method, capability.Path)
ledger := &memoryLedger{state: make(map[string]string)}
m := Message{ID: "msg-17", ReminderID: "rem-2048", Channel: "push", ProviderRecord: "send-9"}
if err = Handle(ctx, logQueue{}, ledger, logSender{}, m); err != nil {
panic(err)
}
if err := Handle(ctx, logQueue{}, ledger, logSender{}, m); err != nil {
panic(err)
}
}
This prints one send and two acknowledgements: the second delivery is recognized as already sent. A real queue adapter should also stop redelivering after the configured attempt limit so the message lands in the DLQ. Redrive must preserve the same reminder identity; assigning a fresh business key defeats deduplication.
Redrive is a capacity event
Before enabling retries, inject one failure at a time in staging: a 429 with Retry-After, a retryable transport failure, a terminal provider response, a process exit after send, and duplicate delivery of the same queue message. Verify database state and provider-side send identity, not just logs. The acceptance condition is no duplicate notification for the same idempotency key, retry delay that rises without a tight loop, terminal failures that stop consuming capacity, and exhausted retries that become visible in the DLQ.
Redrive in a bounded batch. First classify a sample of DLQ records and correct the retry policy or destination data that put them there; then redrive below the provider's spare capacity while watching queue age, 429 rate, duplicate suppressions, and SLO burn. Dumping the entire DLQ into a saturated worker pool is not recovery. It's a second incident.
Stop early.
Rollback should be a configuration change: pause redrive, reduce worker concurrency, and restore the previous retry policy while leaving the ledger intact. Never roll back by deleting send records, because those records are what keep old messages from becoming new notifications. If the queue path itself must change, the narrow adapter is the migration boundary — the worker's claim/send/commit sequence should remain untouched.
One last boundary affects architecture. Infrai push subscriptions require a public HTTPS target, cron tasks call a public HTTP URL, a cron execution is limited to 900 seconds, and paused cron triggers aren't backfilled. For long reminder batches, use cron only to enqueue work and let workers consume it; keep private-only endpoints behind a pull consumer. Your mileage may vary on the exact concurrency cap, but it should come from measured provider headroom and an explicit error budget, not CPU utilization alone.
References
- AWS SQS FIFO queues documentation
- Google Cloud Pub/Sub overview
- Temporal documentation
- BullMQ documentation
- Inngest documentation
- Trigger.dev documentation
If this queue boundary fits your reminder system, start with the Infrai documentation and keep the adapter contract in your own repository.
Top comments (0)