Short answer: At-least-once delivery protects user reminders from silent loss, but standard queues can redeliver work, so reliable fintech reminders need durable idempotency keys, explicit duplicate suppression, and an SLO that names the acceptable delay.
A reminder system should prefer a late reminder to a silently missing one, but at-least-once delivery means the worker must assume the same reminder can arrive again. The least complex design that meets that contract is a durable reminder record, an idempotency key derived from that record, and a send operation whose result is recorded separately from queue acknowledgement.
That distinction matters in fintech. A duplicate payment reminder is embarrassing; a duplicate message about a repayment date can create a support incident, and a missing message can create a compliance question. I treat delivery guarantees as an explicit product and SLO decision, not as a property that appears automatically because a queue is durable.
The alert page is already late
The page arrives when the background worker pool is full, the oldest reminder is 11 minutes past its scheduled time, and the queue depth is still rising. The first instinct is to add workers. That can reduce age for a while, but it does not answer the more dangerous question: how many reminders have been delivered twice while workers retry ambiguous failures? The queue is innocent.
Work backwards from the signal. A useful alert should combine queue age, available worker capacity, and the count of reminders whose delivery attempt has exceeded its deadline. Queue depth alone is a poor proxy for user impact: ten thousand future reminders may be harmless, while a small queue containing reminders due now is an incident. In a fintech reminder pool, I would split the dashboard by due-time bucket, channel, and attempt number, then compare the oldest due item with the age target in the SLO; that extra dimensionality matters because a healthy aggregate can hide one channel that is timing out, one cohort that is retrying, or one worker shard whose lease clock is wrong.
The earlier instrumentation change is to emit one event for each state transition: scheduled, claimed, send-accepted, acknowledged, and expired. Attach the reminder ID, delivery-attempt number, scheduled timestamp, and a redacted account reference. Keep the event schema stable enough to aggregate by due-time bucket without putting payment data into logs.
I would page on the age of due work and the error budget burn rate, then investigate duplicate suppression as a separate measure. A threshold set too low has a cost: paging on a short burst trains the on-call to ignore the signal, while a threshold set too high lets a reminder miss the user’s useful window. The correct threshold comes from the reminder’s business deadline and the SLO, not from a round number that looks tidy on a dashboard.
Why does at-least-once delivery create duplicate reminders?
At-least-once delivery generally means a queue may redeliver a message when acknowledgement is lost, a lease expires, a worker crashes, or the sender cannot determine whether the downstream operation succeeded. The queue is protecting against loss. It cannot, by itself, know whether an external notification provider accepted the message before the worker disappeared.
That creates an unavoidable ambiguous interval:
type Reminder struct {
ID string
DueAt time.Time
Channel string
Payload []byte
}
type DeliveryState struct {
ReminderID string
Key string
Attempt int
Status string
}
If the worker acknowledges before sending, a crash can lose the reminder. If it sends before acknowledging, a crash can cause a second send. A transaction covering both the queue acknowledgement and an external side effect is usually unavailable, so exactly-once is not a default recovery mode; it is a larger system design with narrower guarantees than its label suggests.
The practical contract is therefore: retries are allowed, duplicate effects are controlled, and the user-facing operation is idempotent where the channel permits it. Store a unique key such as reminder_id + delivery_type + logical_due_time\. A retry must reuse that key. Generating a new key on every attempt defeats deduplication while making the logs look healthy.
Duplicates are normal.
What should the worker do when the send result is ambiguous?
The worker should claim a reminder with a lease, perform the send using the stable idempotency key, persist the observed result, and acknowledge only after the durable state is updated. The lease is a concurrency boundary, not proof that the work will run once. A crashed worker can leave a lease to expire, after which another worker may claim the same record.
The send adapter should expose the uncertainty instead of collapsing every failure into false\. A timeout may mean the provider rejected the request, or it may mean the provider accepted it and the response was lost. The retry policy needs a bounded backoff, a dead-letter path for poison messages, and an operator-visible reason for each terminal state.
package reminder
import (
"context"
"time"
)
type Sender interface {
Send(ctx context.Context, key string, payload []byte) (accepted bool, outcomeKnown bool, err error)
}
func Deliver(ctx context.Context, store Store, sender Sender, r Reminder) error {
key := r.ID + ":" + r.Channel
if done, err := store.AlreadyDelivered(ctx, key); err != nil {
return err
} else if done {
return store.MarkAcknowledged(ctx, r.ID)
}
accepted, known, err := sender.Send(ctx, key, r.Payload)
if err != nil && known {
return store.RecordFailure(ctx, key, err)
}
if !accepted && known {
return store.RecordFailure(ctx, key, err)
}
if err := store.RecordDelivered(ctx, key); err != nil {
return err
}
return store.MarkAcknowledged(ctx, r.ID)
}
The important property here is not the interface spelling. It is that the duplicate decision is made from durable state and a stable key, while the queue remains responsible for redelivery. The storage operation must also be concurrency-safe: two workers can pass a read check at the same time unless the insert or state transition is protected by a unique constraint or equivalent atomic operation.
Capacity planning starts with due-time traffic
A rate-limited worker pool needs two capacity plans. The first is ordinary throughput: reminders per second, average processing time, and available worker time. The second is retry capacity: the work created by timeouts, expired leases, and downstream throttling. A pool sized only for the nominal rate can spend its entire budget replaying uncertain sends during an incident.
For a rough model, let D\ be reminders due in the busiest minute, T\ the average worker-seconds per attempt, and R\ the expected attempts per reminder during the planning window. Required worker-seconds are approximately D * T * R\. That is a planning estimate, not a benchmark. Measure the distribution of processing time and the tail of attempts before setting the pool limit.
Keep a reserve for retries. When the downstream channel is rate-limited, reducing concurrency can improve completion because it prevents the queue from converting throttling into a larger retry storm. The right response is usually a combination of backoff, jitter, and a clear maximum age for due work, with a separate path for reminders that can no longer meet their user-facing deadline.
This is where I put the SLO boundary: “99.9% of reminders are accepted by the channel within five minutes of due time” is testable; “the queue is reliable” is not. The SLO should say what accepted means, how duplicates are counted, and what happens to reminders after the deadline.
Choosing the guarantee without fooling the dashboard
At-most-once delivery reduces duplicates by discarding work after a single attempt, but it accepts silent loss. At-least-once delivery reduces silent loss by retrying, but requires idempotency and duplicate observability. A business may choose different contracts for different reminder classes: a mandatory notice can tolerate a duplicate more readily than an optional nudge, provided the message is written to make a repeated delivery understandable.
| Decision | Operational consequence | Evidence to collect |
|---|---|---|
| Retry until accepted or expired | More delivery coverage, more duplicate risk | Due age, attempt count, terminal reason |
| Deduplicate by stable reminder key | Fewer repeated effects, more state to maintain | Suppression count, key collisions, state lag |
| Treat timeout as unknown | Safer against loss, requires reconciliation | Unknown outcomes and later confirmations |
| Scale workers during a burst | Lower queue age, higher downstream pressure | Worker saturation and throttle responses |
Do not make the dashboard report only successful sends. Report due reminders, expired reminders, duplicate suppressions, unknown outcomes, and acknowledgement lag. A duplicate that was correctly suppressed is a successful reliability behavior, not a failure hidden by a zero-error metric.
The cron schedule itself is not a delivery guarantee. The Linux crontab(5)\ documentation describes a five-field time specification and notes that jobs scheduled during a daylight-saving-time transition can be skipped or run twice. That is enough reason to make the scheduler create durable due records and let the queue handle execution; the scheduler should not be the only copy of the work.
The limitation is deliberate.
At-least-once delivery cannot guarantee that an external channel will apply an effect exactly once when the response is ambiguous. If the channel cannot accept a stable idempotency key and the business cannot tolerate a duplicate, this design is the wrong contract; choose an approval or reconciliation workflow that can keep the effect pending until its outcome is known.
Top comments (0)