Short answer: treat the renewal deadline as an expiry policy, keep a durable send record before retrying, and redrive only a bounded, classified batch from the DLQ. Exponential backoff is useful, but it is not what prevents a customer from receiving the same reminder twice.
This is the runbook I would use for a media service that sends a reminder before a subscription renews. The queue may deliver a message more than once. A worker may lose its lease after the notification provider accepted the request. A late redrive may be technically successful and still miss the business deadline. Those are three different incidents, and the consumer needs a separate decision for each one.
The first question on a page should be: “Did we send this reminder, or do we only know that we tried?” Logs alone cannot answer that reliably.
It failed.
Choose a deadline policy before delivery
Put a stable reminder ID, the renewal deadline, the recipient channel, and an expiry timestamp in the message. The expiry is an application rule: a reminder sent after the renewal event is no longer useful, even if the broker is happy to deliver it. Do not let a generic queue retention setting decide that business question.
The consumer should also have a durable send ledger keyed by something like (reminder_id, channel, template_version). Its useful states are small: pending, sending, sent, expired, and terminal. A unique constraint on that key is more valuable than a counter in process memory, because a restart must not turn attempt seven into attempt one.
Claim the ledger row with a lease before calling the provider. On a duplicate delivery, a final sent, expired, or terminal row is an acknowledgement, not another send. If the row is sending and its lease is still live, defer the delivery. If the lease expired, reconcile the provider request ID before making a new request.
One more guard matters for renewal reminders: the producer should snapshot the intended deadline and template version. If a customer changes plans while a message is waiting, the worker should consult the subscription state and apply an explicit policy. Silently sending an old plan notice is a data correctness problem, not a retry problem.
How should a Node.js queue consumer handle retries, idempotency, and DLQ redrive?
Classify the failure before choosing an action. A temporary timeout, a rate limit, or a dependency refusal can be retried. An invalid address, an unsubscribed recipient, or an expired renewal deadline should be recorded as terminal or expired and acknowledged. Retrying a poison message forever is just a slow outage.
For retryable work, use a persisted attempt number, exponential backoff with a cap, and jitter. A practical shape is min(cap, base * 2^(attempt-1)); the actual delay should also respect a provider Retry-After value when one exists. The handler should return control to the queue rather than sleeping while holding a worker slot. The exact cap is a service policy, not a universal constant.
The Node.js part of the design is the adapter boundary. The state transition belongs in a repository transaction, while the broker-specific acknowledge and negative-acknowledge operations belong in the queue adapter. The following Go example makes the transitions explicit without pretending that an in-memory map is a production database.
package main
import (
"errors"
"fmt"
"time"
)
var errTemporary = errors.New("temporary provider response")
type LedgerRow struct {
Status string
Attempts int
}
type Store struct {
Rows map[string]LedgerRow
}
func backoff(attempt int) time.Duration {
if attempt < 1 {
attempt = 1
}
delay := time.Second * time.Duration(1<<uint(attempt-1))
if delay > 30*time.Second {
return 30 * time.Second
}
return delay
}
func consume(store *Store, key string, send func() error) (ack bool, retryAfter time.Duration) {
row := store.Rows[key]
if row.Status == "sent" || row.Status == "expired" || row.Status == "terminal" {
return true, 0
}
row.Attempts++
store.Rows[key] = row
if err := send(); err == nil {
row.Status = "sent"
store.Rows[key] = row
return true, 0
} else if errors.Is(err, errTemporary) && row.Attempts < 5 {
return false, backoff(row.Attempts)
}
row.Status = "terminal"
store.Rows[key] = row
return true, 0
}
func main() {
store := &Store{Rows: map[string]LedgerRow{}}
calls := 0
send := func() error {
calls++
if calls < 3 {
return errTemporary
}
fmt.Println("sent renewal reminder")
return nil
}
for {
ack, wait := consume(store, "renewal-1042:email:v3", send)
fmt.Println("ack", ack, "retry_after", wait)
if ack {
break
}
}
ack, _ := consume(store, "renewal-1042:email:v3", send)
fmt.Println("duplicate_ack", ack, "provider_calls", calls)
}
The important behavior is at the end: the duplicate delivery acknowledges the message without calling send again. Real code needs a database uniqueness constraint, an atomic claim, a lease, and a provider-facing idempotency key where the provider supports one. The example does not solve the crash window by itself; it shows where that policy lives.
The expensive failure mode looks like this. The consumer claims renewal-1042, sends the provider request, and loses its lease before it records sent. The broker redelivers the message. A naive consumer sees pending and sends again. A better one keeps the provider request ID, checks the provider's request status when available, and reuses the same idempotency identity. If the result is still ambiguous, the runbook should prefer a documented reconciliation path over inventing a second send. That reconciliation needs a deadline check too: a provider lookup that takes ten minutes can turn a still-useful reminder into a late reminder, so the operator has to decide whether the customer's interest in avoiding a duplicate is greater than the value of sending after the renewal cutoff. Record that decision against the reminder, with the reason and the person or automation that made it. Otherwise the next redrive will re-open the same ambiguity and the next engineer will assume that a missing sent row means no message was delivered.
I've seen the retry counter become fiction when it lived only in a worker process. That is a small implementation choice with a very visible customer consequence. Your mileage may vary across notification providers; I'm not sure every SDK exposes a useful idempotency key, so the ledger still needs to be authoritative for the application decision.
Measure the gap between attempts and outcomes
A dead-letter queue is a holding area, not a repair mechanism. Before redrive, group messages by failure class and inspect a sample from each group. Confirm that the renewal deadline has not passed, that the recipient is still eligible, and that the ledger does not already contain a final send. Then fix the input or dependency policy and redrive a bounded batch with the original reminder ID intact. Measure the gap between delivery attempts and provider outcomes; a queue metric that says “acknowledged” cannot prove that a notification was accepted.
The redrive command should be observable and reversible at the batch level. Record who started it, the source and destination, the selected failure class, the batch size, and the resulting ledger states. Stop redrive if the duplicate-send rate, provider throttling, or expiry rate moves in the wrong direction. Do not delete the DLQ to make a dashboard green.
Start small.
A useful alert distinguishes “messages waiting near expiry” from “messages failing repeatedly.” The first is a deadline risk; the second is a dependency or data-quality problem. Both may page the team, but they need different responders and different rollback actions.
Govern DLQ redrive as a change
| Design choice | Useful when | Trade-off |
|---|---|---|
| Standard at-least-once queue | Delivery and consumer scaling matter more than ordering | Application-level deduplication remains mandatory. |
| FIFO-style ordering and broker deduplication | A bounded deduplication window and message-group ordering fit the workflow | Broker deduplication does not cover a reminder that waits beyond that window or a provider-side ambiguity. |
| Managed publish/subscribe delivery | The team wants decoupled producers and consumers | Delivery attempts, expiry, and the send ledger are still application responsibilities. |
| Workflow orchestration | The reminder has multiple timers, approvals, or compensating steps | The workflow model adds operational and developer overhead for a single notification. |
There is no universally correct queue choice. Stick with a plain queue when the workflow is one deadline check followed by one provider call. Choose workflow orchestration when the business process has several durable steps. Choose a broker with stronger ordering or replay semantics when those are explicit requirements, not because the word “FIFO” sounds safer.
The cost of the design is operational complexity: a ledger, lease cleanup, metrics, and a redrive tool. That is not suitable when the notification is disposable and duplicate delivery has no user impact; a simpler best-effort path may be enough. For renewal reminders, the customer-facing consequence makes the extra state worth carrying.
Verify the runbook before enabling production redrive
Test a duplicate delivery, a process restart after provider acceptance, a rate limit with Retry-After, an invalid destination, an expired deadline, and a message that reaches the DLQ. Verify that the same reminder ID produces one final ledger row, that attempts survive a restart, and that redrive cannot bypass a final row.
For rollback, pause the consumer or stop the redrive batch first. Preserve the ledger and DLQ for inspection. After the classifier or dependency policy is corrected, resume with a small sample and watch send outcomes, age, expiry, provider responses, and duplicate suppression. The on-call query should accept a reminder ID and show channel, template version, deadline, status, attempts, last error class, lease expiry, and provider request ID.
That query is the handoff between an alert and a safe action. Without it, an operator is forced to guess whether “retrying” means “not sent” or “possibly sent.” Guessing is how a missed reminder becomes a duplicate reminder.
Top comments (0)