MagnusNilsson2124

Posted on Oct 25

Implement Delayed Reminders Using a Message Queue (Payment Recovery)

#scheduling #messagequeues #backend #reliability

For a B2B SaaS product that reconciles payments every night, the least complex delayed-reminder design is a durable message with a due timestamp, an idempotency key, and a retry record kept outside the queue. The queue wakes work up; it does not become the payment ledger.

That is the whole contract.

Short answer: publish a reminder job with due_at, persist the job state, and let a worker claim only due jobs. A worker must acknowledge the queue message after the state transition is durable, and the reminder sender must tolerate the same job arriving more than once. That shape works for a user reminder and for the operational reminder that follows an incomplete payment reconciliation.

The important measurement is not the queue's advertised delay feature. It is the retained state you need to explain a missed reminder at 02:00, after the provider has returned a partial result and an operator is asleep.

What does the bill actually contain?

In this system, the dominant operating cost is usually retained work and repeated delivery attempts, not the timestamp attached to one small message. A nightly run might inspect thousands of payment records, create one follow-up reminder per exception, and keep the evidence required to reconcile or suppress each reminder. The exact volume depends on the account and provider; there is no honest universal multiplier.

That distinction changes the design. Keep the compact job envelope in the queue: a stable job ID, account ID, due time, attempt count, and a reference to the reconciliation result. Keep the larger audit payload in durable storage. Do not copy provider responses into every retry message.

A useful retention policy separates three things: the active job, the delivery attempt log, and the reconciliation evidence. Active jobs need enough history to recover. Attempt logs can be compact and expire according to the compliance policy. Evidence may need a longer retention period because a support engineer has to explain why a customer was reminded. In practice, I would define those periods before shipping the worker, because changing retention after a compliance review is harder than adding one field to the job record. The retention setting is a policy decision, not a queue default, and it should be reviewed with whoever owns payment records and notification consent.

I once assumed the queue was the natural place to preserve the whole story. That made recovery look convenient until the story had several retries and a provider response had changed shape. The queue then carried too much mutable context. The correction was simple: store facts once, reference them from messages, and make the state machine explicit.

You deliberately stop retaining duplicate message bodies. The cost is that a recovery tool must fetch the referenced record. That is a good trade when the alternative is an opaque pile of copied payloads that disagree during an incident.

How should I implement delayed reminders using a message queue?

No. A delayed message is a wake-up signal. The source of truth is the job row or document that records whether the reminder is pending, claimed, sent, suppressed, or failed. This is the boundary that makes an overnight reconciliation recoverable.

For each reminder, I use a state transition with a unique key such as account_id:reconciliation_id:exception_id:reminder_kind. A retry can publish the same key again. The sender checks the state before sending, and the write that records the send result is conditional on the key still being pending. The exact database syntax varies, but the invariant does not: one logical reminder may have many attempts and one accepted business outcome.

The queue acknowledgement belongs after the worker has completed the durable transition it is responsible for. RabbitMQ's acknowledgement guidance describes this relationship clearly: acknowledgements tell the broker when a delivery has been processed, while a rejected or requeued delivery can be attempted again. A crash before acknowledgement can therefore produce a duplicate delivery. That is expected behavior, not an edge case to hide.

A minimal job shape looks like this:

from dataclasses import dataclass
from datetime import datetime

@dataclass(frozen=True)
class ReminderJob:
    job_id: str
    account_id: str
    reconciliation_id: str
    due_at: datetime
    idempotency_key: str
    evidence_ref: str

In a Node.js service, the same fields can be represented by a typed object or schema. The field names matter more than the language. due_at must be interpreted consistently, preferably as an absolute UTC instant, and the consumer must validate it before doing delivery work.

Why does operational recovery outrank a clever timer?

Because the nightly job already has a recovery problem. A provider can answer late, return a partial page, or report a transaction that cannot yet be matched. The reminder should say what is known and what remains open; it should not force the worker to reconstruct yesterday's provider response from a transient queue payload.

The worker loop has four decisions:

Load the durable job by ID and discard a message whose state is already sent or suppressed.
If the job is early, defer it without increasing the delivery-attempt count.
Claim the job with a short lease, then re-check the reconciliation state before sending.
Record success, suppression, or a retryable failure before acknowledging the queue delivery.

The early-message branch matters. Queue systems differ in how they implement delayed delivery, and clocks drift. A queue message arriving a few seconds early should not cause an accidental reminder. A message arriving late should still be eligible if the job is pending. This is why due_at belongs in the job record even when the queue offers a delay option.

Here is the decision logic in compact Python; the same transitions can sit behind a Node.js consumer.

def handle_delivery(message, store, sender, clock, queue):
    job = store.get_job(message["job_id"])
    if job is None or job.state in {"sent", "suppressed"}:
        queue.ack(message)
        return

    if clock.now() < job.due_at:
        queue.defer(message, until=job.due_at)
        return

    if not store.claim(job.id, lease_seconds=60):
        queue.ack(message)
        return

    current = store.get_job(job.id)
    if current.state != "pending":
        store.release(job.id)
        queue.ack(message)
        return

    try:
        result = sender.send(
            account_id=current.account_id,
            idempotency_key=current.idempotency_key,
            evidence_ref=current.evidence_ref,
        )
    except RetryableDeliveryError as error:
        store.record_retry(job.id, str(error), next_due_at=clock.backoff(job.id))
        queue.ack(message)
        return

    store.mark_sent(job.id, provider_reference=result.reference)
    queue.ack(message)

The example acknowledges a retryable delivery after recording the next due time and republishing or scheduling the next wake-up according to the queue adapter. That policy must be transactional in the real service: if the database commit succeeds but the publish fails, a periodic sweeper should find the pending job; if the publish succeeds but the process dies before the acknowledgement, the duplicate delivery should be harmless.

Do not use an in-memory timer as the recovery mechanism. A process restart, deployment, or autoscaling event can erase it. An in-memory timer is fine as a latency optimization after durable state exists; it is a poor record of an obligation.

This design is a poor fit for a reminder that must be delivered at an exact second, because queue wake-up and worker scheduling introduce jitter. It is also unnecessary for a small, single-process workflow with no recovery requirement. In those cases, a durable scheduler or a database-backed polling loop may be easier to operate. The trade-off is operational complexity: a queue adds acknowledgement, retry, and poison-message policy. That limitation is acceptable when work can be retried, distributed, and recovered after the nightly reconciliation has finished.

Where should retention stop?

The useful stopping point is after the business decision is durable and the evidence is queryable. Keeping every queue delivery forever gives a comforting trail but makes operational queries noisy. Retain a compact attempt record with the job ID, timestamp, result class, and correlation ID. Retain the reconciliation evidence under the policy that applies to payment records. Remove or redact message content that is not needed for support, audit, or compliance.

This also reduces notification risk. A reminder worker should not log full payment data, one-time credentials, or message bodies by default. It needs enough structured data to answer: which account, which reconciliation, which exception, what decision, and what happened next. That is the same information an operator needs at 02:00, without turning logs into a second datastore.

For email and SMS, deliverability belongs in the failure model. A successful handoff to a provider is not proof that a user saw the reminder. Track accepted, rejected, bounced, and suppressed outcomes when the channel exposes them. Respect consent, quiet hours, regional rules, and unsubscribe state before sending. A retry of a temporary transport error must not bypass a permanent suppression.

The same rule applies to OTP-like reminders even when the message is not an authentication code: never treat a duplicate send as harmless by default. It can confuse users, trigger provider filtering, or create a support ticket that the reconciliation data cannot explain.

A practical recovery test

Test the transitions that operators will actually meet. Freeze the clock around the due time. Deliver the same queue message twice. Kill the worker after the sender accepts but before the acknowledgement. Make the provider return a timeout, then a permanent rejection. Run the nightly reconciliation twice. Restart the process with pending jobs in durable storage.

The expected result is not zero duplicate deliveries at the transport layer. The expected result is one business decision per idempotency key, a visible attempt history, and a pending item that a sweeper can recover when the wake-up message disappears.

A small operational dashboard should expose pending count by due-time bucket, oldest pending age, retryable failures, permanent failures, and the difference between accepted and delivered notifications. Alert on age and state transitions rather than raw queue depth alone. A queue can be shallow while one stuck job is already past its customer-facing deadline.

For the B2B SaaS reconciliation case, the decision rule is narrow: use the queue for wake-up and concurrency, durable state for truth, and a sweeper for recovery. The retention change is to keep references and decision evidence, not duplicated payloads. That combination gives the nightly process a way to explain itself after a crash without turning the scheduler into a payment system.

DEV Community