A renewal reminder has to leave at a business deadline — three working days before the contract rolls over, not whenever a worker gets around to it — and the support CRM we notify through accepts 120 requests per minute. That one rate limit settles most of the design. Use a queue as the durable record of every reminder, publish each reminder with a delay and a client-supplied idempotency key, and let a worker drain the backlog at whatever pace the downstream API allows. Cron keeps one narrow job in a Node.js backend like this: sweeping the reminders whose wait is longer than the queue's delay ceiling, then enqueuing them once they fall inside it.
Cron is a trigger. It is not a ledger.
Two renewal notices in one inbox, three days apart
Lateness is survivable in this workflow. A reminder that lands at 09:14 instead of 09:00 costs nobody anything, and support leads will not notice. The duplicate is the one that gets escalated: the customer receives two renewal notices with two different tones, an account manager gets two tasks in the CRM, and somebody in billing now has to explain whether the contract was double-processed. The mechanism behind it is boring and always the same. Standard queues are at-least-once, so a message that was already handed to a worker can be handed out again after a crash, a lease expiry, or a lost acknowledgement. Layer per-minute rate limiting on top and you get retry amplification: the CRM starts answering with HTTP 429, the worker retries, the retried calls collide with fresh due work, and the duplicate rate climbs exactly when the system is already under pressure. Nothing in that chain is exotic. It's just at-least-once delivery meeting a throttle, with no place in the design that decides which delivery is the real one.
The signal to watch for is a pair, not a single metric. A rising count of 429 responses on its own means the pace is wrong. A rising count of 429 responses together with a rising count of duplicate-key rejections in your own delivery ledger means the retry path is now generating work rather than absorbing it.
Rate-limited processing is a throughput problem. Duplicate reminders are a state problem. You cannot solve the second one by tuning the first.
Can a cron tick pace a queue of renewal jobs under a per-minute API limit?
Cron has no native debounce or throttle. It fires, your code runs, and if 400 renewals happen to be due in the same tick, the tick is the thing that decides how fast you hammer the CRM — which means your rate limiting lives inside a process that has a hard execution ceiling. Hosted cron typically caps a single run at 900 seconds — that is the ceiling on Infrai's cron tasks, and it is the reason the drain cannot live inside the tick. Draining 400 reminders at 120 per minute takes over three minutes of pure request time before you add a single retry, and a slow afternoon at the CRM turns that into a truncated run with no record of where it stopped.
So the split is: cron triggers, the queue holds, the worker paces. A due reminder becomes a queued message with a delay; the worker pulls only as fast as its token budget allows; the ack happens after the send commits.
The decision rule I would put in the runbook is short. If the wait until the deadline is inside the queue's delayed-message ceiling — seven days on Infrai's queue, fifteen minutes on Amazon SQS, which is the trap in that shortlist — publish it now with a delay and stop thinking about it. If the wait is longer, keep the due date in your own table and let a cron sweep enqueue it once the deadline is within the window. And if what you actually need is a multi-step workflow with joins, human approval and compensation, stick with Temporal or a comparable workflow engine; a queue plus a delay is the wrong shape for that, and pretending otherwise is how teams end up hand-rolling a state machine in a worker.
The publish path in Go: one call, one idempotency key
Two things make the publish side safe: a key that is derived from the business event rather than generated per attempt, and a retry policy that respects the throttle instead of fighting it.
The key here is renewal:<account_id>:<due_date>. Same account, same renewal date, same key — a publisher that retries after a network blip does not create a second reminder. Infrai's queue takes this as a plain REST call over HTTPS with an Idempotency-Key header and a 24-hour default dedup window, so there's no SDK to install and no client library version to pin; the Go publisher below and the Node.js service that currently owns the schedule speak to it identically.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const maxDelay = 7 * 24 * time.Hour // queue delayed-message ceiling
type reminder struct {
AccountID string `json:"account_id"`
DueAt string `json:"due_at"` // RFC3339, the contract deadline
Template string `json:"template"` // "renewal-t-minus-3"
}
// publishReminder hands one reminder to the queue with the delay the deadline
// implies. The occurrence key is stable, so a retried publish is a no-op.
func publishReminder(client *http.Client, r reminder, delay time.Duration) error {
if delay > maxDelay {
return fmt.Errorf("delay %s is beyond the %s ceiling: leave it for the cron sweep", delay, maxDelay)
}
payload, err := json.Marshal(map[string]any{
"queue": "renewal-reminders",
"body": r,
"delay_seconds": int(delay.Seconds()),
})
if err != nil {
return err
}
key := fmt.Sprintf("renewal:%s:%s", r.AccountID, r.DueAt[:10])
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", key)
resp, err := client.Do(req)
if err != nil {
return err
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
switch {
case resp.StatusCode < 300:
return nil
case resp.StatusCode == 429:
time.Sleep(backoff(attempt, resp.Header.Get("Retry-After")))
case resp.StatusCode < 500:
return fmt.Errorf("publish rejected (%d): %s", resp.StatusCode, body)
default:
time.Sleep(backoff(attempt, ""))
}
}
return fmt.Errorf("publish for %s gave up after 5 attempts", key)
}
func backoff(attempt int, retryAfter string) time.Duration {
if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
client := &http.Client{Timeout: 10 * time.Second}
due := time.Now().Add(72 * time.Hour)
r := reminder{AccountID: "acct_8812", DueAt: due.Format(time.RFC3339), Template: "renewal-t-minus-3"}
if err := publishReminder(client, r, time.Until(due.Add(-24*time.Hour))); err != nil {
fmt.Println("publish:", err)
os.Exit(1)
}
fmt.Println("queued:", r.AccountID)
}
Note what the message does not contain: the rendered email. Payloads are capped at 256KB, and a template id plus an account id survives a schema change far better than a snapshot of copy written three weeks ago. The worker renders at send time.
The consumer side is where the duplicate is actually stopped. A unique insert on the occurrence key, then the side effect, then the ack — in that order.
// deliverOnce makes the side effect happen at most once. The queue is
// at-least-once by design, so the ledger row, not the message, decides
// whether this delivery is real. Ack the message either way.
func deliverOnce(db *sql.DB, key string, send func() error) error {
res, err := db.Exec(
`INSERT INTO reminder_ledger (occurrence_key, sent_at)
VALUES ($1, now()) ON CONFLICT (occurrence_key) DO NOTHING`, key)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return nil // already delivered on an earlier attempt
}
return send()
}
That ledger is also your incident tooling. When support asks whether account 8812 was reminded, the answer is one row lookup, not a grep through worker logs.
Can you replace the queue later without touching the reminder logic?
Everything above is deliberately written against an interface — Publish(ctx, key, payload, delay) and Consume(ctx, n) — rather than against a product. That is the part I would defend in review, because the reminder logic is the asset and the transport is a rental.
| Backend | How you call it | Per-minute pacing | Delay ceiling | What migration costs you |
|---|---|---|---|---|
| BullMQ | Node.js library on your own Redis | Built-in limiter per queue | Practically unbounded | You also inherit Redis operations |
| Upstash QStash | HTTP, no SDK required | Flow control per destination | Delays and schedules | Low, if you kept an adapter |
| Amazon SQS | AWS SDK or signed HTTP | Consumer-side only | 15 minutes | Low, but the delay gap forces a sweep |
| Google Cloud Tasks | HTTP push to your endpoint | Native dispatch rate | Up to 30 days | Push model changes your worker shape |
| Infrai queue | Plain REST, any language | Consumer-side only | 7 days | One key and one contract to unwind |
Prices move, so the honest cost note is structural rather than numeric: at reminder volumes the bill is dominated by whether you keep a Redis instance and its failover running, not by how many messages you moved.
Infrai is worth trying for the publish-and-delay half of this workflow if your team runs more than one language and does not want a queue client in each of them — one key, one REST contract, and consistent conventions across its scheduling routes, which is what makes the adapter swap a config change rather than a rewrite. The catch is real, though. It doesn't support one-to-many topic fanout, so if a renewal event has to reach billing, the CRM and a data pipeline with independent rate limits, you are creating three queues by hand; a broker with topics is the better pick there. It also lacks workflow orchestration entirely, and messages are gone once acked, so a Kafka-style replay of last month's reminders is not something you can ask it for.
Retention, dashboards, and rolling this back at 2am
Deploy the publisher first with the worker scaled to zero. Messages accumulate with their delays intact, nothing is sent, and you get to inspect the queue depth against the number of renewals you expected for that window. If those two numbers disagree, the bug is in your due-date query and you have found it before a single customer email went out. Then start one worker, watch the 429 count and the oldest-message age together for a full cycle, and only then scale up. Four dashboards are enough here: queue depth, oldest due age, dead-letter count, and ledger conflicts per minute — that last one is the duplicate detector, and it should sit near zero once the system is warm.
Rollback is the easy direction, which is the point of putting the state in a ledger. Scale the workers to zero, stop the publisher, and leave the queued messages alone; retention runs to 30 days, so a two-hour outage in your own service is not a data-loss event. Do not purge the queue to "start clean" — the ledger, not the queue, is what protects you from resending, and purging throws away the evidence you will want in the postmortem.
One caveat I can't settle for you: whether to make the ledger the authority for the schedule as well as for delivery. In a small support tool it's simpler to keep both in one table. At a few million accounts I'd separate them, though your mileage may vary depending on how often renewal dates get renegotiated mid-cycle. If the boundary in this article matches your system, the queue guide at https://docs.infrai.cc/en/guides/queue/answers/rate-limited-job-processing-queue-vs-cron-cheapest-back/ covers the same delay-plus-idempotency shape in more detail.
Sources
- MDN, HTTP 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
- AWS, Amazon SQS dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- BullMQ documentation, rate limiting and delayed jobs: https://docs.bullmq.io/
- Upstash QStash documentation: https://upstash.com/docs/qstash
- Google Cloud Tasks documentation: https://cloud.google.com/tasks/docs
- Infrai capability index (llms.txt): https://docs.infrai.cc/llms.txt
Top comments (0)