DEV Community

HoldenFox8476
HoldenFox8476

Posted on

Python Queues: How At-Least-Once Delivery Impacts Reliability of Logistics User Reminders

#python #queues #logistics #distributedsystems

Short answer: at-least-once delivery makes a reminder system safer against lost work, but it makes duplicate work an expected outcome. For a logistics worker pool, I treat the queue as a delivery attempt, not as proof that a notification was sent. The durable reminder record owns identity, the notification boundary enforces idempotency, and draining workers stop accepting new work before they finish in-flight attempts.

That decision favors a bounded increase in storage and coordination over a duplicate SMS or email. It also keeps the latency-versus-cost choice visible: a strict deduplication check adds a small synchronous step, while retrying without one can create customer-facing noise and repeated delivery charges.

I have fought rate limits, spam filters, and OTP delivery gaps. The recurring lesson is that the queue is not the delivery receipt; it is the place where the next attempt becomes possible.

Duplicates happen.

The boundary matters.

The decision record

The system has four invariants. A scheduled reminder has one stable identity. A delivery attempt can be repeated. A successful send is recorded before the worker acknowledges the queue message. A worker that is draining does not take another message after its stop deadline.

The failure boundaries matter more than the happy path. A process can send a message and crash before acknowledging it. A queue can redeliver a message whose acknowledgement was delayed. A provider can accept a request while the client times out. A scheduler can create the same logical reminder twice if its run is retried. None of these cases can be made safe by assuming that a queue message is unique.

Option Duplicate risk Lost-work risk Latency and cost trade-off
Acknowledge before sending High High after a crash Low per-attempt overhead, poor reliability
Send, then acknowledge, with no dedupe High Low until retention expires Fast to implement, repeated notifications are expensive
Send through an idempotency record, then acknowledge Bounded by the record boundary Low, with retry policy One durable lookup and write per logical reminder

The third option is the useful baseline. It does not promise exactly-once behavior across every external provider. It gives the application a clear place to decide what a repeated attempt means.

How do at-least-once queues impact delivery reliability for user reminders?

Draining is often described as an operational detail. In a rate-limited logistics service, it is part of correctness. Suppose a pool has 24 workers and the downstream notification endpoint allows only a fixed number of requests per time window. During deployment, the pool must finish work already claimed without starting an unbounded tail of new work. Otherwise the shutdown deadline turns ordinary retries into lost reminders or abrupt duplicate attempts after a restart.

A draining worker should therefore have two clocks: a lease or visibility deadline for the queue message, and a shutdown deadline for the process. The worker renews the first only while it can still finish or safely abandon the attempt. The second is a hard application boundary. When it expires, the message must become eligible for another attempt, and the attempt counter must make that handoff observable. In practice, I write these deadlines into the worker state and logs together. Otherwise an on-call engineer sees a rising retry count without knowing whether the cause was downstream throttling, a deployment deadline, or a lease that expired while a provider request was still unresolved. That ambiguity is where teams start tuning concurrency blindly, which can increase queue age and notification duplication at the same time.

This is where at-least-once semantics help. If acknowledgement happens only after the application has recorded the outcome, a crash usually leaves work available for another attempt. The cost is that the next worker may see work that the previous worker already submitted. That is the expected duplicate window, not a rare theoretical defect.

The reminder identity must survive retries. I use a logical key made from the business reminder, recipient, and scheduled occurrence. A queue message ID is unsuitable because a retry may have a new message envelope. A timestamp captured at each attempt is also unsuitable because it changes the identity while the work is being retried.

The critical path

The following Python sketch keeps the important ordering explicit. claim_once represents a durable insert with a uniqueness constraint on dedupe_key; its result must distinguish a new claim from an existing completed claim. The notification adapter is intentionally generic.

from dataclasses import dataclass
from datetime import datetime


@dataclass(frozen=True)
class ReminderJob:
    reminder_id: str
    recipient_id: str
    scheduled_for: datetime
    payload: dict[str, str]


def handle(job: ReminderJob, store, notifier, queue_message) -> None:
    dedupe_key = (
        f"{job.reminder_id}:{job.recipient_id}:"
        f"{job.scheduled_for.isoformat()}"
    )

    claim = store.claim_once(dedupe_key, job.payload)
    if claim.already_completed:
        queue_message.ack()
        return

    try:
        receipt = notifier.send(
            recipient_id=job.recipient_id,
            payload=job.payload,
            idempotency_key=dedupe_key,
        )
    except Exception:
        store.release_claim(dedupe_key)
        raise

    store.mark_completed(dedupe_key, receipt)
    queue_message.ack()
Enter fullscreen mode Exit fullscreen mode

The ordering is deliberate. A claim that is marked completed before the external send can suppress a real notification after a crash. A queue acknowledgement before mark_completed can lose the logical reminder. The durable record needs a state such as claimed, completed, or retryable, plus attempt metadata and timestamps. A stale claimed state needs an explicit recovery rule; otherwise a worker crash can strand the reminder indefinitely.

There is a hard limitation here. If the provider accepts the notification and the process dies before mark_completed, the next attempt cannot know from the queue alone whether the first request succeeded. A provider-side idempotency key, a queryable delivery receipt, or a reconciliation process can narrow that uncertainty. Without one of those boundaries, the application must choose between a possible duplicate and a possible omission. That is a product decision disguised as an exception handler.

What should the queue and scheduler each guarantee?

The scheduler should create a stable logical occurrence and make creation idempotent. The queue should provide retry, visibility control, and acknowledgement semantics. The worker should enforce the delivery state machine. Mixing these responsibilities causes subtle failures: a scheduler retry creates two occurrences, or a worker treats a successful queue acknowledgement as proof of external delivery.

Cron is useful for invoking a periodic reconciliation command, but it does not turn a schedule into a durable delivery log. The Linux crontab(5) manual describes schedules and the command execution environment; the reminder service still needs durable state for occurrence identity, retry status, and completion. A five-minute reconciliation job can find overdue records, but it should enqueue the same logical key rather than create a new one.

For rate-limited pools, backoff belongs to the retry policy, not to the identity. A retry at 10:02 and a retry at 10:07 are attempts for the same scheduled occurrence. Keep the attempt count, last error class, next eligible time, and provider receipt separate from the dedupe key. This lets operations distinguish throttling from an invalid recipient and prevents a permanent error from consuming the whole pool.

During a drain, I would emit at least these measurements: claimed jobs, completed jobs, retryable failures, expired claims, duplicate claims, queue age, and time spent waiting for the rate limit. The useful alert is not just worker count. A pool can be full while queue age rises because every worker is waiting on a downstream limit.

The rejected option and its valid use

The rejected design is “acknowledge first, then send.” It looks attractive during shutdown because the queue empties quickly and the process has less work to finish. It creates the wrong failure boundary: a crash between acknowledgement and send removes the only durable retry signal. For user reminders, that is an omission that may be invisible until someone asks why a delivery never arrived.

That pattern still has a valid use. It can be appropriate for best-effort telemetry where a dropped event is acceptable and a duplicate is more harmful than a gap. It is a poor default for time-sensitive logistics notifications, where an occasional duplicate can be explained and reconciled but a missing reminder can cause a missed handoff or appointment.

The other tempting shortcut is a process-local set of completed IDs. It handles repeated messages on one worker, then fails during a restart, a second replica, or a deployment drain. Memory is fast; it is not a shared record of delivery. The durable store is the cost of making the decision survive the events the queue is designed to tolerate.

A practical rule for choosing the boundary

Choose the narrowest idempotency boundary that matches the actual side effect. If the side effect is an email or SMS request, use the provider's supported idempotency mechanism when one exists and store the resulting receipt. If it is an internal notification service, make the receiving endpoint accept a stable key and record it under a uniqueness constraint. If neither side can deduplicate, use reconciliation and make the ambiguity visible to operators instead of claiming exactly-once delivery.

The key test is simple: after a worker crashes at every line in the critical path, can another worker determine whether it should send, retry, or wait for reconciliation? If the answer depends on a queue message ID or process memory, the design is not ready for an at-least-once queue.

For a logistics pool, I would accept the extra durable write and keep the drain deadline explicit. It buys a controlled retry path, measurable ambiguity, and a decision that remains correct when rate limits, deployments, and delayed acknowledgements overlap. Reliability here is not the absence of duplicates. It is the ability to make a duplicate or omission an understood, bounded outcome.

References

Top comments (0)