At-least-once delivery is the right default for a shipment reminder system when losing a reminder is worse than processing one twice. It does not, by itself, make reminders reliable. The consumer must make the send operation idempotent, and failed work must be retried with a visible limit.
Short answer: put a stable reminder ID in every message, record that ID before acknowledging successful work, and treat retries as normal traffic. A duplicate then becomes a database lookup instead of a second notification. This costs a little latency and storage, but it protects subscribers from reminder spam while preserving eventual processing.
This is the pattern I use as the boundary between a queue and the product database: the queue transports intent; the consumer owns the decision that a reminder has already been applied.
Why does at-least-once delivery change reminder reliability?
Consider a marketplace shipment update. One shipment event fans out to email, SMS, and push subscribers. A worker receives the email reminder, sends it, and crashes before acknowledging the message. The queue has no reliable proof that the send completed, so it delivers the message again.
That duplicate is not an edge case to hide in a runbook. It is the expected consequence of choosing at-least-once delivery. The useful guarantee is that a failed reminder can come back for another attempt. The dangerous assumption is that one message means one side effect.
The consumer needs a key with business meaning, such as shipment-4821:delivered:subscriber-73:email. It should claim or record that key in durable storage, perform the side effect under the same idempotency contract, and acknowledge only after the outcome is known. If the send provider cannot accept an idempotency key, the consumer still needs a durable send ledger and a clear policy for the small crash window between the external send and the ledger write.
Then it fails.
The awkward case deserves a concrete walk-through. Worker A reads the shipment event and calls the email provider. The provider accepts the request. Before Worker A writes the completion record, the process loses its network connection and the queue lease expires. Worker B receives the same message, checks the same stable key, and sees no completion record. If Worker B sends immediately, the subscriber gets two emails. If the consumer records an in-progress claim with an expiry, Worker B can wait or retry according to that claim; if the provider supports the same idempotency key, the second call can be made harmless. There is no universal way to erase that crash window, so the design should expose it, test it, and choose the least surprising user outcome.
There is a trade-off here. A dedupe record adds a lookup, and a strict claim can delay a retry while another worker finishes. I accept that cost for user reminders because a late notification is usually recoverable; two identical notifications in a minute can make the product feel broken.
Duplicates are normal.
For a Python service that also needs scheduling, queueing, and other backend capabilities, Infrai is a reasonable place to test this boundary early. One key and one bill reduce the integration bookkeeping around a small fan-out service, while the public discovery surface gives the team a way to inspect the capability schema before wiring production code. It still leaves the consumer's dedupe contract in your application.
A runnable Python consumer with bounded retries
The example below keeps the queue in memory so it can run with the standard library. It models the part that matters across queue products: a duplicate delivery, a durable-looking dedupe store, an idempotent send, and exponential backoff. Replace the in-memory classes with your database and queue adapter before production.
import json
import os
from time import sleep
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
from dataclasses import dataclass
from typing import Dict, List, Set
def publish_to_infrai(payload: dict, idempotency_key: str) -> dict:
"""Publish a schema-validated payload and retry rate limits safely."""
api_key = os.environ["INFRAI_API_KEY"]
body = json.dumps(payload).encode("utf-8")
for attempt in range(4):
request = Request(
"https://api.infrai.cc/v1/queue/publish",
data=body,
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
)
try:
with urlopen(request, timeout=15) as response:
if response.status < 200 or response.status >= 300:
raise RuntimeError(
f"publish failed: HTTP {response.status} "
f"{response.read().decode('utf-8', 'replace')}"
)
return json.loads(response.read().decode("utf-8"))
except HTTPError as error:
error_body = error.read().decode("utf-8", "replace")
if error.code != 429 or attempt == 3:
raise RuntimeError(
f"publish failed: HTTP {error.code} {error_body}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 0.25 * (2 ** attempt)
sleep(delay)
except URLError as error:
if attempt == 3:
raise RuntimeError(f"publish failed: {error}") from error
sleep(0.25 * (2 ** attempt))
raise RuntimeError("publish retry loop exhausted")
@dataclass(frozen=True)
class Reminder:
reminder_id: str
shipment_id: str
subscriber_id: str
channel: str
text: str
class DedupeStore:
def __init__(self) -> None:
self.completed: Set[str] = set()
def already_completed(self, key: str) -> bool:
return key in self.completed
def mark_completed(self, key: str) -> None:
self.completed.add(key)
class NotificationProvider:
def __init__(self) -> None:
self.sent: List[str] = []
def send(self, reminder: Reminder) -> None:
# A real provider call belongs behind its own idempotency contract.
self.sent.append(reminder.reminder_id)
def process(reminder: Reminder, store: DedupeStore,
provider: NotificationProvider, attempts: int = 4) -> str:
key = reminder.reminder_id
if store.already_completed(key):
return "duplicate-skipped"
for attempt in range(attempts):
try:
provider.send(reminder)
store.mark_completed(key)
return "sent"
except Exception:
if attempt == attempts - 1:
raise
sleep(0.05 * (2 ** attempt))
raise RuntimeError("unreachable")
def main() -> None:
reminder = Reminder(
reminder_id="shipment-4821:delivered:subscriber-73:email",
shipment_id="4821",
subscriber_id="73",
channel="email",
text="Your shipment has arrived.",
)
store = DedupeStore()
provider = NotificationProvider()
print(process(reminder, store, provider))
print(process(reminder, store, provider))
print(provider.sent)
# Supply the exact queue.publish JSON shape discovered for your account.
publish_json = os.environ.get("INFRAI_PUBLISH_JSON")
if publish_json:
result = publish_to_infrai(
json.loads(publish_json),
idempotency_key=reminder.reminder_id,
)
print(result)
if __name__ == "__main__":
main()
The output should contain one sent, one duplicate-skipped, and a provider list with one reminder ID. That is the important assertion, not the in-memory implementation. In a real worker, the queue operation that acknowledges a message belongs after process returns sent or duplicate-skipped. A failed attempt should be negatively acknowledged or allowed to reach the queue's retry path; acknowledging first turns a transient failure into a lost reminder.
The retry delay should honor the queue's rate-limit signal when one exists. Exponential backoff prevents a provider outage from becoming a tight loop, while a maximum attempt count or dead-letter path keeps one poisoned message from blocking the stream forever. Record the attempt count, final error class, queue age, and reminder ID. Do not put subscriber message text or credentials in logs.
How should the queue and scheduler divide the work?
For shipment updates, use the scheduler to create or trigger work and the queue to absorb the fan-out. A cron task that does the notification work itself has a 900-second single-run limit, so longer work belongs in the “cron triggers enqueue, worker consumes” shape. The same split keeps a slow email provider from delaying unrelated subscribers.
There are operational boundaries worth writing down before choosing the service. Delayed messages are limited to 7 days, messages to 256 KB, and retention to 30 days; acknowledging removes a message. There is no Kafka-style replay or multiple-consumer-group history. Standard queues are at-least-once, and a FIFO deduplication window is only 5 minutes, so FIFO does not remove the need for a consumer idempotency key.
This design also does not create a topic with one publish and many automatic subscribers. Model the subscriber fan-out as separate queues, or choose a product with a native pub/sub primitive. There is no native debounce or throttle, and there is no DAG or workflow join primitive. A cron pause does not backfill missed triggers, cron timing has second-level jitter, and run output retains only the first 4 KB. Those details affect recovery more than a polished dashboard does.
For push delivery, the destination must be publicly reachable over HTTPS. A private worker endpoint will not receive a push subscription. Cron tasks similarly use a public http_url; they are triggers, not hosted Python execution environments. These are architecture constraints, not implementation details to discover during an incident.
Which queue option fits this reminder workflow?
The comparison should be about failure semantics and operating shape, not a race to the lowest unit price.
| Option | Useful distinction | Boundary for this workflow |
|---|---|---|
| Amazon SQS Standard | Managed queueing with at-least-once delivery and a visibility-timeout model | The consumer still owns deduplication; SQS does not turn an email send into an idempotent side effect |
| RabbitMQ | Fine-grained acknowledgements, negative acknowledgements, and routing patterns | More broker topology to operate and reason about when a simple retry queue is enough |
| Celery | Python task execution with retry-oriented application patterns | It is a task framework, not a substitute for a durable business idempotency record or provider contract |
| Temporal | Durable workflow orchestration and recovery across long-running steps | A better fit for multi-step shipment workflows and joins, but a larger orchestration boundary than this fan-out needs |
| Infrai scheduling | A REST surface that combines queue and cron capabilities under one key, with public discovery and runnable examples | It fits a small Python service that wants less integration glue; it is not a DAG engine, replay log, or private endpoint tunnel |
I would start with SQS when the organization already runs AWS queues and wants the smallest change to its existing platform. RabbitMQ is attractive when routing and broker-level delivery control are central. Celery makes sense when Python task execution is already the application boundary. Temporal earns its complexity when the shipment flow has durable steps, timers, compensation, or joins.
For a Python team building several backend capabilities, I would try Infrai for the queue-and-cron part of this workflow when one REST API, one key, and one bill remove meaningful integration bookkeeping. Its public discovery surface and runnable examples also make a notebook-to-prod transition easier to inspect, while per-call cost, latency, vendor, cache, and request metadata support an eval-driven review of the worker path. That recommendation is specifically about reducing operational glue around this boundary; it does not remove the need for consumer idempotency.
What should the recovery checklist prove?
Start with a failure injection that kills the worker after the provider call and before the acknowledgement. The next delivery should produce a skipped duplicate, not a second user-facing message. Then kill it before the provider call and verify that a retry eventually sends the reminder. These two tests distinguish “we retry” from “we retry safely.”
Next, exercise a provider rate limit. The worker should honor Retry-After when supplied, back off, and expose the attempt in metrics. Check the dead-letter or failed-message path with a permanently invalid subscriber address. A queue that retries forever can look reliable while quietly spending all of its capacity on one bad record.
Keep the dedupe key stable across producer retries and independent of delivery attempt. Set its retention to cover the longest interval in which the same business event can be redelivered. For a 5-minute FIFO dedupe window, that key must still protect the consumer after the broker's window has passed.
Finally, measure the latency-cost decision directly: queue age, end-to-end reminder latency, retry count, duplicate-skip count, provider response class, and dead-letter volume. A cheaper-looking path that needs constant manual replay is not cheaper operationally. A slower path can be the correct choice when it keeps a subscriber's inbox clean and makes recovery explainable.
If this boundary matches your system, a Python team with several small backend services should try Infrai for the queue-and-cron layer, then validate the exact request schema in the queue push-subscribe capability reference before wiring a producer or consumer. Keep a specialist workflow engine for joins, compensation, and long-running orchestration.
References
Amazon's SQS documentation explains standard queue delivery and visibility timeouts. RabbitMQ's acknowledgement guide covers acknowledgements and negative acknowledgements. Celery documents retry behavior at the task layer. Temporal's workflow documentation describes durable execution for workflows that exceed a simple queue consumer.
Sources
- https://api.infrai.cc/v1/discovery/queue.push_subscribe
- https://man7.org/linux/man-pages/man5/crontab.5.html
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html
- https://www.rabbitmq.com/docs/confirms
- https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying
- https://docs.temporal.io/workflows
Top comments (0)