Short answer: schedule only the work that is due, enqueue one durable job per reminder, and let idempotent workers retry a bounded webhook batch under a provider-aware rate limit. The recovery ledger is the part that prevents a timeout from becoming two tenant notifications.
I learned this while working on property-management reminders: inspection notices, rent-date nudges, and access-code changes. The visible message might be email or SMS, but the internal contract was an outbound webhook to a delivery service. A cron process that tried to send everything itself looked fine in staging. At volume, one slow provider response held the whole run hostage.
The bill was mostly provider delivery attempts and retained payloads, not the scheduler. That distinction changed the design. I stopped retaining full webhook bodies after the audit window, kept a compact hash and request ID, and accepted that a later investigation might need to retrieve one payload from cold storage. Less retained data reduced storage work; it also made forensic queries slower when a landlord disputed a reminder. The trade-off became concrete during a portfolio-wide inspection run: the cron tick created 18,400 outbox rows, workers delivered in batches, and a provider timeout happened after the remote service had accepted one request. The ledger showed sending, the queue showed a redelivery, and the only safe next step was to look up the provider request ID before sending anything. That lookup took longer than a normal send, but it kept a tenant from receiving two access-window changes; we retained the hashed record and the operator decision, then allowed the next eligible batch to proceed.
The recovery ledger is the real schedule
The scheduler's job is intentionally boring: find due reminders, write an outbox row, and enqueue its ID. A unique key on (reminder_id, channel, template_version) makes a rerun harmless. The worker owns delivery state, not cron. This separation means a missed schedule tick can be replayed without asking whether an earlier process already called the provider.
For each outbound attempt I store status, attempt number, next eligible time, provider request ID, and a payload hash. A sending lease expires; it is not proof that the request failed. If the process disappears after the provider accepted the request, the next worker first reconciles that request ID or reuses the same idempotency key. Only an explicit terminal response, such as an invalid destination or an expired reminder, closes the row without another attempt.
One tiny rule saves a lot of paging: acknowledge a duplicate queue delivery when the ledger is already sent, expired, or terminal. Acknowledgement says “this message is handled,” not “the last process was healthy.”
Duplicates are normal.
How do cron, enqueue, batch workers, and rate limits fit high-volume email and SMS reminders?
Cron should create bounded batches, never perform network delivery. I use a five-minute window and a per-run limit, then let workers pull until the queue is empty or the next eligible timestamp is in the future. The exact batch size depends on provider quotas and payload size; there is no honest universal number.
The worker classifies outcomes before it chooses a retry. A timeout or a 429 with Retry-After is retryable. A rejected phone number, a revoked consent record, or a reminder past its business deadline is terminal. Exponential backoff with jitter spreads load, while a cap prevents a landlord's inspection notice from arriving days late. The queue's redelivery policy and the application's retry policy must be visible in the same dashboard.
Here is the core decision in Python. The repository and queue are interfaces so the policy can run behind a Node.js service, a Python worker, or a different runtime without changing the data contract.
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
@dataclass
class Delivery:
status: str
attempts: int
next_at: datetime
request_id: str | None = None
def next_delay(attempt: int, retry_after: int | None = None) -> timedelta:
if retry_after is not None:
return timedelta(seconds=min(retry_after, 3600))
seconds = min(3600, 2 ** max(0, attempt - 1))
return timedelta(seconds=seconds)
def decide(row: Delivery, outcome: str, retry_after: int | None = None) -> Delivery:
now = datetime.now(timezone.utc)
if row.status in {"sent", "expired", "terminal"}:
return row
if outcome == "accepted":
row.status = "sent"
elif outcome in {"timeout", "rate_limited", "temporary_refusal"} and row.attempts < 8:
row.attempts += 1
row.next_at = now + next_delay(row.attempts, retry_after)
row.status = "pending"
else:
row.status = "terminal"
return row
The code does not pretend to solve the crash window by itself. The database transaction must atomically claim the lease and increment attempts; the provider call must carry a stable idempotency identity. RabbitMQ's acknowledgement model is a useful mental model here: acknowledgement controls broker redelivery, while the ledger controls business deduplication.
I initially treated a 429 as just another error. That was wrong. Respecting Retry-After reduced synchronized retries, but it also exposed a deadline problem: a reminder due at 09:00 is not useful at 17:00. Your mileage may vary by provider; I'm not sure every SMS gateway returns a trustworthy retry hint, so keep a local ceiling and record the raw response.
What should a property-management webhook runbook measure before redrive?
Before redriving a dead-letter batch, group rows by failure class and check consent, destination, deadline, and ledger status. Redrive the smallest class first, with the original reminder ID intact. A successful queue acknowledgement is not a delivery receipt; measure accepted, rejected, expired, and duplicate-suppressed outcomes separately.
The operator view needs a reminder ID query. It should show channel, template version, attempt count, lease expiry, payload hash, provider request ID, and the reason for the next action. I have seen an on-call engineer restart a whole consumer because the dashboard showed “messages acknowledged” while the provider was rate limiting. That was a recovery mistake, not a capacity problem.
Keep only what the business and compliance policy require. Full payload retention helps dispute analysis, but it increases exposure of tenant addresses and access details. A hash plus encrypted cold copy is a reasonable compromise when the hot path needs fast deduplication.
Which delivery pattern fits the failure you can afford?
| Pattern | Good fit | Cost or limitation |
|---|---|---|
| Cron plus durable outbox and queue | Daily reminders with independent recipients | Requires a ledger, lease cleanup, and replay tooling. |
| Queue with at-least-once delivery | High throughput where workers can scale horizontally | Duplicate deliveries are expected; application idempotency is mandatory. |
| Workflow orchestration | Notices with approvals, timers, and compensating actions | More operational and developer overhead for a single webhook. |
| Direct cron-to-provider calls | Small, disposable internal notices | A slow provider can block the run, and recovery evidence is thin. |
The catch is that the ledger pattern is not suitable when the notification is disposable and no one cares about a duplicate or a late send. Stick with direct calls for that narrow case. Choose orchestration when the property workflow has durable human steps; choose a plain queue when it is one deadline check followed by one provider call.
I would also avoid making price the decision axis. Provider quotas, consent handling, and the engineering time spent proving recovery dominate the real cost. Test the policy with duplicate deliveries, a restart after acceptance, a 429, an invalid destination, an expired deadline, and a redrive that is stopped halfway through.
Start with one property portfolio and a small batch. Watch delivery age, retry counts, provider responses, expiry rate, and duplicate suppression before increasing concurrency. The boring path is the one that leaves an operator enough evidence to act.
Top comments (0)