Use a durable, pull-based queue with explicit retry state for user reminders; delivery guarantees matter more than the queue brand. The worker should claim one reminder, make the outbound webhook or email operation idempotent, and record a terminal dead-letter decision when retries are exhausted.
That sounds tidy until a provider returns HTTP 429, a process dies after sending, or a webhook acknowledgement arrives late. Those are normal states in an e-commerce cleanup job, not exotic incidents. I design around them first.
Delivery invariants for the reminder worker
The reminder row needs a stable event ID, a scheduled time, an attempt count, a next-attempt timestamp, and a delivery state. The event ID travels through the email provider's idempotency key or the webhook payload. A consumer can then receive the same message twice without sending twice.
The claim operation must be atomic. With PostgreSQL, a transaction can select due rows using FOR UPDATE SKIP LOCKED, mark a lease expiry, and commit before the network call. Another worker skips the leased row instead of waiting behind it. A lease is not proof of success; it is only ownership for a short interval.
Keep the state machine boring: scheduled -> sending -> delivered, with sending -> retryable after a transient response and sending -> dead_letter after a permanent response or an attempt budget. A watchdog moves expired sending leases back to retryable. This covers crashes between the provider call and the database commit.
Retry classification is the control plane. HTTP 429 means the recipient is being rate limited; honor Retry-After when present and add bounded jitter. Timeouts and selected 5xx responses can be retried, but a malformed address, an invalid signature, or a rejected schema should go straight to a dead-letter record. MDN documents 429 as a signal to slow down, not as permission to hammer the endpoint again.
The outbound side still has an unavoidable gap: the provider may accept a send while the worker is offline. That is why the provider request carries the event ID, and why the consumer records an outbox result keyed by that same ID. On replay, a delivered result is a no-op. When a provider cannot offer idempotency, the honest guarantee is at-least-once attempt, with duplicate-send risk documented for support teams. A 409 conflict from a provider should be treated as a possible prior acceptance, then reconciled by event ID; blindly retrying it can create the duplicate the queue was meant to prevent. I've seen teams lose that distinction when every non-2xx response entered the same retry bucket.
A webhook is an input, too. Verify its signature, persist it before acknowledging, and deduplicate by the provider event ID. Do not let a webhook handler directly mutate reminder state based on arrival order; an older delivery receipt can arrive after a newer failure notice.
A small critical path in Python
The following sketch keeps the transaction boundary visible. The SQL is intentionally generic so the same rules work with a self-hosted queue, a managed broker, or a table-backed worker.
from datetime import datetime, timedelta, timezone
MAX_ATTEMPTS = 6
def claim_due(conn, now: datetime):
with conn:
row = conn.execute(
"""
UPDATE reminder
SET state = 'sending', lease_until = %s, attempts = attempts + 1
WHERE id = (
SELECT id FROM reminder
WHERE state IN ('scheduled', 'retryable')
AND next_attempt_at <= %s
AND (lease_until IS NULL OR lease_until < %s)
ORDER BY next_attempt_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
RETURNING id, event_id, destination, attempts
""",
(now + timedelta(minutes=2), now, now),
).fetchone()
return row
def finish(conn, row, result, now):
if result.accepted:
conn.execute(
"UPDATE reminder SET state='delivered', provider_id=%s WHERE id=%s",
(result.provider_id, row.id),
)
elif result.retryable and row.attempts < MAX_ATTEMPTS:
delay = min(3600, 2 ** row.attempts * 10)
conn.execute(
"UPDATE reminder SET state='retryable', next_attempt_at=%s WHERE id=%s",
(now + timedelta(seconds=delay), row.id),
)
else:
conn.execute(
"UPDATE reminder SET state='dead_letter', failure_code=%s WHERE id=%s",
(result.code, row.id),
)
conn.commit()
The production version also stores the request fingerprint and provider response metadata. Those fields make a support replay explainable without exposing message bodies or personal data in logs. I initially treated the lease as a lock; that failed in a crash test because the lock vanished with the process. The expiry timestamp is the durable part.
Which queue fits user reminders, retries, and dead letters?
| Design | What it guarantees well | Where it is a poor fit |
|---|---|---|
| Broker with visibility timeout | Fast fan-out and built-in redelivery | Harder forensic queries unless events are copied to durable storage |
| PostgreSQL table queue | Transactional claim, rich audit queries, simple local tooling | Hot tables need indexes and vacuum discipline at high volume |
| Cron plus a script | Adequate for low-volume, best-effort cleanup | Overlapping runs and process loss make retries and idempotency easy to get wrong |
The right choice follows the failure boundary you can operate. A table queue is attractive when reminder rows already live in PostgreSQL and the daily volume is modest. A broker earns its complexity when independent consumers need sustained parallelism. Cron remains valid for a short, non-critical reconciliation task that can safely run twice. The trade-off is operational: a table queue needs index and vacuum care, while a broker needs replay tooling and a durable audit trail. A queue design is not suitable when the team cannot inspect attempts or replay a dead letter; stick with a simpler scheduled script until those controls exist.
I would reject “one cron run sends every due email” for user-visible reminders. It holds work in a web-adjacent process, has no per-message lease, and makes a partial failure indistinguishable from a complete run. It is suitable for a nightly report where duplicates are harmless and the report can be regenerated from source data.
Measure the system with delivery-oriented metrics: age of the oldest due reminder, retry histogram, dead-letter rate by reason, lease-expiry count, and duplicate provider acknowledgements. Alert on trends, not one transient 429. Your mileage may vary on exact backoff values; tune them from provider limits and customer tolerance, then preserve the policy in a versioned configuration.
Duplicates happen.
The decision rule is simple: choose the mechanism that can prove who owns a message, why it was retried, and why it stopped. Everything else is an implementation detail.
Top comments (0)