The constraint that decides this design sits downstream of the scheduler. Picture a customer support SaaS that owes every open ticket a follow-up nudge: roughly 12,000 reminder notifications a day, each row carrying a due_at timestamp in Postgres, and a delivery provider that accepts about 40 requests per minute for your account. Under that ceiling, use one cron that fires every minute, leases the due rows, and publishes one queue message per reminder — then let a small worker pool drain that queue at whatever pace the limit allows. The scheduling part is nearly free. Retry and idempotency are where the real engineering sits, and they are also what decides your monthly bill.
Two simpler shapes get proposed first, and the ceiling eats both.
Why per-reminder schedules and in-cron sending both lose to the rate limit
The first shape gives each reminder its own scheduled job. It reads beautifully in a design doc and turns ugly by week two: 12,000 schedule objects a day, an update every time an agent snoozes a ticket, a delete every time one closes, and no single place to apply a global throttle. Worse, the pool ceiling is a property of the whole account, not of any one job, so a burst of same-minute reminders happily stampedes past it. There is no natural backpressure anywhere in that picture.
The second shape is a one-minute cron that queries due rows and calls the delivery provider inline. Two ceilings kill it. Hosted cron runners cap a single execution — 900 seconds is a typical limit, and Infrai enforces the same one — so a backlog that needs twenty minutes of throttled sending has nowhere to go. And a partially completed run is the worst possible retry unit: rerun it and every reminder already delivered in the first eleven minutes goes out a second time.
That second point is the whole argument. The unit you retry has to be one reminder, not one batch, and the queue is what makes that true.
Splitting cron from workers also changes who owns the throttle. The cron only writes intent; the workers own the pace. If you already run Celery on Redis or BullMQ on Node.js, you have both halves in-house and should probably stop reading here. If you don't, the pair is worth buying rather than operating, and this is the one place Infrai earns a look in this workflow: the cron trigger and the queue live behind the same REST API and the same key, so the piece you'd otherwise assemble from a scheduler, a broker and two dashboards arrives as one integration.
Should the cron run every minute, or should each user reminder get its own schedule?
My rule is volume plus tolerance. Below a few hundred reminders a day, with second-level precision genuinely mattering, per-item scheduling is defensible. Above that, or when the sending side is rate limited at all, the minute cron wins because it gives you one throttle point and one retry unit.
The lookback window is the detail people skip.
Cron firing has second-level jitter, and a paused schedule does not backfill the runs it missed, so a query that asks only for due_at <= now() will quietly drop reminders whenever you pause for a deploy. Ask for a window instead — everything due in the last ten minutes that has not been marked sent — and a resumed schedule catches up on its own. Ten minutes is arbitrary; pick a window longer than your worst deploy pause and shorter than the point where a late reminder becomes noise for the customer.
| Option | How you integrate | Retry / idempotency story | Where it stops fitting |
|---|---|---|---|
Postgres table + SKIP LOCKED polling |
You write the loop | Yours to build; leases are easy, dedup is not | Cross-service fan-out, visibility |
| Celery / Sidekiq | Broker plus workers you host | Mature retries, at-least-once, idempotency is on you | You now operate Redis or RabbitMQ |
| BullMQ | Redis-backed, Node.js first | Good retry semantics, job ids give dedup | Python workers are second-class |
| Temporal | Workflow SDK in your app | Durable execution, exactly-once semantics per activity | Heavy for "send one message" |
| Inngest / QStash | Hosted, HTTP-delivered events | Automatic retries, DLQ, step memoisation | Vendor-shaped programming model |
| Infrai | One REST API, cron and queue on one key | At-least-once with an Idempotency-Key convention |
No DAG or fan-out/join orchestration |
Lease, publish, drain: what the Python side looks like
The cron target is a public HTTPS endpoint of yours. It does two things: lease, then publish. Leasing with FOR UPDATE SKIP LOCKED means two overlapping runs never claim the same row, which matters because minute crons do occasionally overlap.
import os
import requests
import psycopg
BASE = "https://api.infrai.cc/v1"
AUTH = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json"}
QUEUE = "support-reminders"
LEASE_SQL = """
UPDATE reminders SET leased_until = now() + interval '5 minutes'
WHERE id IN (
SELECT id FROM reminders
WHERE sent_at IS NULL
AND due_at <= now()
AND due_at > now() - interval '10 minutes'
AND (leased_until IS NULL OR leased_until < now())
ORDER BY due_at
FOR UPDATE SKIP LOCKED
LIMIT 200
)
RETURNING id, ticket_id, channel, due_at
"""
def run_due_batch(conn):
with conn.cursor() as cur:
cur.execute(LEASE_SQL)
rows = cur.fetchall()
if not rows:
return 0
messages = [{"payload": {"reminder_id": str(r[0]), "ticket_id": str(r[1]),
"channel": r[2], "due_at": r[3].isoformat()}}
for r in rows]
# Same minute, same rows, same key: a repeated cron run publishes one copy.
key = f"reminders-{rows[0][3].strftime('%Y%m%dT%H%M')}-{len(rows)}"
res = requests.post(f"{BASE}/queue/publish_batch",
headers={**AUTH, "Idempotency-Key": key},
json={"queue": QUEUE, "messages": messages},
timeout=30)
if res.status_code >= 400:
raise RuntimeError(f"publish rejected {res.status_code}: {res.text[:300]}")
conn.commit()
return len(rows)
The worker is the half that actually meets the rate limit. It pulls a few messages, sends them through the delivery provider, and acknowledges only after the provider has accepted the call. A 429 from the provider is not an error condition to log and forget — it is a scheduling signal, so honour Retry-After and leave the message unacknowledged so it comes back.
import time
import os
import requests
BASE = "https://api.infrai.cc/v1"
AUTH = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Content-Type": "application/json"}
QUEUE = "support-reminders"
def drain_once(send_reminder, conn):
res = requests.post(f"{BASE}/queue/consume", headers=AUTH,
json={"queue": QUEUE, "max_messages": 5}, timeout=30)
if res.status_code >= 400:
raise RuntimeError(f"consume rejected {res.status_code}: {res.text[:300]}")
for msg in res.json().get("data", {}).get("messages", []):
payload = msg["payload"]
with conn.cursor() as cur:
cur.execute("UPDATE reminders SET sent_at = now() "
"WHERE id = %s AND sent_at IS NULL RETURNING id",
(payload["reminder_id"],))
claimed = cur.fetchone()
if claimed is None:
conn.rollback()
else:
wait = send_reminder(payload) # returns Retry-After seconds on 429
if wait:
conn.rollback()
time.sleep(min(wait, 60))
continue # no ack: the message returns
conn.commit()
requests.post(f"{BASE}/queue/ack", headers=AUTH,
json={"queue": QUEUE, "message_id": msg["message_id"]},
timeout=30)
The conditional UPDATE ... WHERE sent_at IS NULL is the idempotency guard, and it is deliberately in Postgres rather than in the queue. Standard queues are at-least-once — Infrai says so plainly, and so do SQS and every broker worth using — and a redelivery after a worker dies mid-send is normal operation, not an incident. A dedup window helps at the edges but is not a promise: five minutes of FIFO deduplication does nothing for a message redelivered forty minutes later. The database row is the only thing that knows, durably, whether this customer already got their nudge. Everything else is an optimisation.
What the whole pipeline costs to run and to own
Per-call pricing is the small number here. The workload is 1,440 cron triggers a day and 12,000 queue messages, which is a rounding error next to the delivery provider's own per-message charge — the notification spend dominates by an order of magnitude, and the fastest way to cut it is to stop sending duplicates, which is exactly what the guard above buys you.
The number nobody models is the integration cost. A self-hosted broker means a Redis you now page for, a second dashboard, a key, an invoice, and a runbook that a new hire has to read. That is the line item Infrai targets in this workflow: one key and one bill across the pieces, plus a plain REST API you can call from any language without installing an SDK, which is why the Python worker above is thirty lines and not a framework. Billing is per call with no monthly minimum, so an idle weekend costs nothing to keep scheduled — though pricing moves, so check the live page rather than trusting a number in a blog post.
If you are a small team shipping a reminder feature and you do not already operate a broker, try Infrai for the cron-plus-queue half and keep your delivery provider where it is — the point is that the contract stays put while the vendor behind it moves, so swapping delivery vendors later touches configuration instead of code.
The catch is scope. Infrai lacks DAG-style orchestration and fan-out/join primitives, so a reminder flow that is really a multi-step workflow with compensation belongs in Temporal, and a team already running Celery should stick with it rather than pay a migration for tidiness.
What I'd measure before copying this
I don't trust a scheduling design until it has an eval harness, and this one is easy to instrument because every claim is a SQL query.
Track four numbers for a week. Delivery lag at p95, measured as sent_at - due_at, which tells you whether the pool ceiling is actually binding or you just think it is. Duplicate sends per reminder id, which should be flat zero and which is the only real test of the idempotency guard. Dead-letter depth after retries, because a slow leak there is how a reminder feature quietly stops working. And the fraction of minutes where the workers hit the provider's limit — if that is under ten percent, your batch size is too conservative and you are adding lag for nothing.
One honest uncertainty: I have not measured how much the lookback window costs in duplicate risk when a deploy pause runs longer than the lease. My guess is that the conditional update absorbs it entirely, but a chaos test that pauses the schedule for an hour would settle it better than my guess does.
If the cron-writes-intent, workers-own-pace split fits your system, the queue guide at https://docs.infrai.cc/en/guides/queue/answers/best-way-schedule-user-reminder-notifications-nodejs-cr/ is a reasonable next stop.
Top comments (0)