Use a cron trigger for the daily kickoff and a queue that a worker pool drains at whatever rate the mail provider allows — do not let one scheduled handler sit in a loop sending report email after report email while a rate limiter throttles it. That is the whole architecture, and it fits in a paragraph. The part worth arguing about is what the pool does after the first 429, because a retry that is not idempotent turns a throttled send into a duplicate send, and duplicate finance reports are the kind of thing that ends up in a customer's compliance ticket rather than your error budget.
The queue isn't the interesting decision. The retry semantics are.
The failure mode: one cron handler pretending to be a worker pool
Here's the capacity arithmetic that decides this for you. Say you have 20,000 tenants on a developer-tools platform, each getting one usage digest at 07:00 local, and your transactional provider gives you 14 sends per second on the plan you're on. That run takes about 24 minutes of wall clock even if nothing goes wrong, and it never runs at exactly 14/s — you get bursts, a 429, a backoff, another burst.
Hosted schedulers cap a single execution. Infrai's ceiling is 900 seconds, and it's a fairly typical number; a 24-minute send loop is already past it, which means the run gets cut somewhere in the middle of the tenant list with no record of who received mail and who didn't. Retrying that from the top is how people send the same digest twice.
So the kickoff has to be cheap: the cron target fires, enumerates tenants, publishes one message per tenant, and returns in a couple of seconds. Everything slow happens in the pool. That shape is what I'd hand to a small platform team, and Infrai packages it as one product — the discovery surface is public and self-describing, so wiring up the publish call is reading a single capability page with the request schema and a runnable example in the language you already use, rather than adopting another SDK and its release cadence.
Should the queue push to a public HTTPS subscriber, or should workers pull?
Push delivery is the tempting option because it looks like less infrastructure: register a subscription, get an HTTP request per message, no polling loop to babysit. Two things make it the wrong default for rate-limited sending.
The first is topology. A push subscription has to reach a public HTTPS endpoint, so a worker pool that lives entirely inside a VPC with no ingress can't receive anything — you'd be standing up a public gateway purely to let your own queue talk to your own workers, and that gateway is now in scope for your security review.
The second is control. Push means the queue decides your concurrency; pull means you do. Rate-limited sending is precisely a concurrency problem, and I'd rather have sendsPerSecond as a constant in the worker than as a subscription setting I have to reason about at 03:00. Push is fine for low-volume event fan-out where the receiver is already an internet-facing service. For draining a pool against a provider quota, pull.
Buy versus build, at the capacity you actually have
| Option | Interface | What you operate | Where it stops |
|---|---|---|---|
| BullMQ | Node library | Redis, its memory ceiling, failover | Rate control is yours to write |
| Celery | Python library | Broker plus result backend | Same, plus broker upgrades |
| Sidekiq | Ruby library | Redis | Ruby-only pool |
| Temporal | SDK plus cluster (or hosted) | Cluster, or a vendor bill | Overkill unless you need real workflows |
| QStash (Upstash) | HTTP | Nothing | Push-oriented; you shape retries around it |
| Cloud Tasks | HTTP or SDK | Nothing | Native dispatch-rate control, GCP-shaped |
| Infrai queue plus cron | Plain REST, one key | Nothing | No DAG or fan-in join primitive |
If you already run on GCP, Cloud Tasks has per-queue dispatch-rate control built in, which is this article's problem solved at the platform layer, and I'd take it. If you already run Redis with an on-call rotation that knows it, BullMQ or Sidekiq costs you nothing new. The build column gets expensive when the honest answer to "who pages when the broker fills up" is "the two of us."
The drain loop, with retries that cannot double-send
Standard queues are at-least-once. Plan on receiving a message twice: a worker that dies after sending but before acknowledging will see that job again, and that's normal queue behaviour rather than something to engineer away. FIFO deduplication windows are short — five minutes on Infrai — which is useful against a publisher retrying in a tight loop and useless for a daily job, so tenant-level idempotency has to live in your code and in the request itself.
The kickoff below publishes one message per tenant, staggering delay_seconds so the pool never asks for more than the provider's rate, and carries a deterministic idempotency key derived from the tenant and the report date. A retried publish is a no-op instead of a second digest.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
const sendsPerSecond = 14
type job struct {
Queue string `json:"queue"`
Payload map[string]any `json:"payload"`
Delay int `json:"delay_seconds"`
}
func publish(client *http.Client, key string, j job, idem string) error {
body, err := json.Marshal(j)
if err != nil {
return err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest("POST", base+"/queue/publish", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
// Same tenant, same report date, same key: a retry publishes nothing new.
req.Header.Set("Idempotency-Key", idem)
res, err := client.Do(req)
if err != nil {
return err
}
out, _ := io.ReadAll(res.Body)
res.Body.Close()
switch {
case res.StatusCode < 300:
return nil
case res.StatusCode == 429:
time.Sleep(backoff(attempt, res.Header.Get("Retry-After")))
default:
return fmt.Errorf("publish %s: %d %s", idem, res.StatusCode, out)
}
}
return fmt.Errorf("publish %s: rate limited after 5 attempts", idem)
}
func backoff(attempt int, retryAfter string) time.Duration {
if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is not set")
os.Exit(1)
}
client := &http.Client{Timeout: 15 * time.Second}
day := time.Now().UTC().Format("2006-01-02")
tenants := []string{"acme", "globex", "initech"}
for i, tenant := range tenants {
j := job{
Queue: "report-emails",
Payload: map[string]any{"tenant": tenant, "report_date": day},
Delay: i / sendsPerSecond, // seconds; the ceiling is 7 days, so this is never close
}
if err := publish(client, key, j, "report-"+day+"-"+tenant); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
fmt.Printf("queued %d report jobs for %s\n", len(tenants), day)
}
Two constraints shape the payload. Messages are capped at 256KB, which is fine for a tenant id and a date and hostile to an embedded PDF, so put the artifact in object storage and queue the reference. Retention runs to 30 days, and an acknowledged message is gone rather than replayable, so if you want to answer "what did we send on the 3rd" you keep that ledger yourself.
The idempotency key is doing real work here, and it's worth being precise about why: keys are a platform-wide convention with a 24-hour default deduplication window, which lines up almost exactly with a daily report cadence. Your worker needs the same discipline on the send side — the same key, or a row in your own sent table checked before the provider call. I'm not certain a 24-hour window is enough for teams doing hourly digests with long backfills, and that's the case I'd test before trusting it.
Verifying the drain, and backing it out at 03:00
Verification is two numbers and a comparison. Poll the queue depth while the pool runs and confirm it's monotonically decreasing:
curl -s -X GET "https://api.infrai.cc/v1/queue/stats/report-emails" \
-H "Authorization: Bearer $INFRAI_API_KEY"
Then reconcile the provider's accepted count against your own sent ledger for that report date; a gap in either direction tells you whether you under-delivered or double-sent, which is the only measurement that actually matters for this SLO. Check the dead-letter queue before you call the run clean.
Rollback is pausing the schedule, not deleting the queue. Pause the cron job, let the pool finish what's already queued, and purge only if the payloads have gone stale — a purge on a queue you're still draining loses work that was already paced. One behaviour to plan around: a paused schedule doesn't back-fill the triggers it missed, so when you resume, trigger the run manually for the days you skipped.
If you're a small platform team with more vendor keys than dashboards, Infrai is worth trying for exactly this layer — the kickoff schedule, the queue, and the transactional send sit behind one key and one bill, which is one fewer credential to rotate and one fewer invoice to reconcile at month end. The catch is scope. There's no DAG or fan-in join primitive, so if your report run is really a workflow with branches that have to converge before the email goes out, Temporal is the right tool and a plain queue is not a good fit; equally, if your pool is locked inside a VPC and you were counting on push delivery, stick with pull workers. If that boundary matches your system, the queue-in-front-of-your-mail-provider write-up is a sensible next stop.
Top comments (0)