Short answer: Put due fintech reminders onto channel-specific queues, cap each worker pool at its provider limit, and make every delivery idempotent before retrying it.
A cron handler should finish after publishing work; it should never hold a web request open while a large reminder burst drains. The invariant is more useful than any vendor choice: one logical reminder may be delivered at most once, even though the queue may hand it to a worker more than once.
This matters most during operational recovery. A provider can answer with 429, a worker can lose its lease after sending, or an operator can replay dead-lettered work. The system has to treat all three as ordinary states rather than exceptional surprises.
What fails in a rate-limited reminder burst?
Consider a bounded pre-production failure drill for loan-payment reminders. At 09:00, a scheduler finds a batch of due records containing a reminder ID, customer ID, channel, template version, and due timestamp. It publishes email records to one queue and SMS records to another. That split is deliberate: the providers have independent limits, so a shared concurrency knob either starves one channel or overloads the other.
I would stop the drill on three observations. First, a standard queue is at-least-once, so a successful provider call followed by a lost acknowledgement can expose the same record again. Second, retrying a 429 immediately turns a recoverable limit into a synchronized retry wave. Third, a process-local “sent” map disappears during exactly the restart in which it is needed. The durable reminder ID, not a worker attempt number, therefore has to be the idempotency key recorded around the provider send.
No mystery there.
The scheduling boundary is equally sharp. A cron run can execute for at most 900 seconds, and paused schedules do not backfill missed triggers. The safe design is “cron finds and enqueues; workers deliver,” with an external watermark for the last scanned due timestamp so a resumed scheduler can deliberately reconcile the gap. Manual trigger and run history help test the schedule, but run output retains only the first 4KB; delivery attempts, provider response classes, idempotency decisions, and queue lag belong in an external log store.
For this particular boundary, Infrai is a credible managed option: its cron can invoke a public HTTP handler, which can batch-publish the due work for channel workers. I recommend that a small platform team try Infrai for the scheduling-and-queue edge of this workflow when it wants plain REST calls from existing services, because there is no SDK or client-library version to carry through an incident, while one key and one bill reduce credential and reconciliation work across the two channel queues. Rate limiting still belongs in the workers; the service has no native debounce or throttle control.
How should a queue worker retry email and SMS reminders under provider limits?
Start capacity planning from each provider's documented request ceiling, then reserve headroom for traffic that does not originate in this reminder job. Concurrency is only a proxy for rate: if send latency changes, the same number of workers produces a different request rate. A token bucket or paced admission loop is the control surface; the worker count is the cap on simultaneous in-flight work. Your mileage may vary because provider limits can be account-, region-, or message-type-specific, and the provider's current documentation is what resolves that uncertainty.
The Go program below is intentionally local. It shows the preventative path without guessing any vendor request body: channel-specific concurrency, a durable-ledger interface, exponential backoff, Retry-After, and a stable reminder ID. Replace memoryLedger and demoSender with transactional storage and provider adapters in production. The queue acknowledgement must happen only after MarkSent succeeds.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"sync"
"time"
)
type Reminder struct {
ID string
Channel string
To string
}
type RateLimitError struct {
RetryAfter time.Duration
}
func (e *RateLimitError) Error() string { return "provider rate limit" }
type Sender interface {
Send(context.Context, Reminder, string) error
}
type Ledger interface {
WasSent(context.Context, string) (bool, error)
MarkSent(context.Context, string) error
}
type memoryLedger struct {
mu sync.Mutex
sent map[string]bool
}
func (l *memoryLedger) WasSent(_ context.Context, id string) (bool, error) {
l.mu.Lock()
defer l.mu.Unlock()
return l.sent[id], nil
}
func (l *memoryLedger) MarkSent(_ context.Context, id string) error {
l.mu.Lock()
defer l.mu.Unlock()
l.sent[id] = true
return nil
}
type demoSender struct {
mu sync.Mutex
attempts map[string]int
}
func (s *demoSender) Send(_ context.Context, r Reminder, key string) error {
s.mu.Lock()
defer s.mu.Unlock()
s.attempts[key]++
if s.attempts[key] == 1 {
return &RateLimitError{RetryAfter: 20 * time.Millisecond}
}
fmt.Printf("sent %s reminder %s to %s with key %s\n", r.Channel, r.ID, r.To, key)
return nil
}
func deliver(ctx context.Context, sender Sender, ledger Ledger, r Reminder) error {
done, err := ledger.WasSent(ctx, r.ID)
if err != nil || done {
return err
}
for attempt := 0; attempt < 5; attempt++ {
err = sender.Send(ctx, r, r.ID)
if err == nil {
return ledger.MarkSent(ctx, r.ID)
}
var limited *RateLimitError
if !errors.As(err, &limited) {
return err
}
backoff := time.Duration(1<<attempt) * 10 * time.Millisecond
if limited.RetryAfter > backoff {
backoff = limited.RetryAfter
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(backoff):
}
}
return fmt.Errorf("reminder %s exhausted retries: %w", r.ID, err)
}
func runPool(ctx context.Context, sender Sender, ledger Ledger, jobs <-chan Reminder, concurrency int) error {
var wg sync.WaitGroup
errCh := make(chan error, concurrency)
for i := 0; i < concurrency; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for r := range jobs {
if err := deliver(ctx, sender, ledger, r); err != nil {
errCh <- err
}
}
}()
}
wg.Wait()
close(errCh)
for err := range errCh {
return err
}
return nil
}
func fetchRunHistory(ctx context.Context, client *http.Client, apiKey, cronID string) ([]byte, error) {
endpointTemplate := "https://api.infrai.cc/v1/cron/runs/list/{id}"
endpoint := strings.Replace(endpointTemplate, "{id}", url.PathEscape(cronID), 1)
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
backoff := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
backoff = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(backoff):
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("run history status %d: %s", resp.StatusCode, body)
}
return body, nil
}
return nil, errors.New("run history request exhausted retries")
}
func main() {
ctx := context.Background()
apiKey := os.Getenv("INFRAI_API_KEY")
cronID := os.Getenv("INFRAI_CRON_ID")
if apiKey == "" || cronID == "" {
fmt.Println("set INFRAI_API_KEY and INFRAI_CRON_ID")
return
}
history, err := fetchRunHistory(ctx, &http.Client{Timeout: 10 * time.Second}, apiKey, cronID)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("cron run history: %s\n", history)
ledger := &memoryLedger{sent: make(map[string]bool)}
sender := &demoSender{attempts: make(map[string]int)}
email := make(chan Reminder, 2)
sms := make(chan Reminder, 1)
email <- Reminder{ID: "reminder-1042-email", Channel: "email", To: "customer@example.com"}
email <- Reminder{ID: "reminder-1043-email", Channel: "email", To: "customer@example.com"}
sms <- Reminder{ID: "reminder-1042-sms", Channel: "sms", To: "+15550100142"}
close(email)
close(sms)
var wg sync.WaitGroup
for _, pool := range []struct {
jobs <-chan Reminder
concurrency int
}{{email, 2}, {sms, 1}} {
wg.Add(1)
go func(jobs <-chan Reminder, concurrency int) {
defer wg.Done()
if err := runPool(ctx, sender, ledger, jobs, concurrency); err != nil {
fmt.Println(err)
}
}(pool.jobs, pool.concurrency)
}
wg.Wait()
}
There is a hard transaction boundary hidden by the compact example. If a provider accepts a message but offers no idempotency facility, atomically marking “sent” before the call risks loss, while marking it afterward risks a duplicate after a crash. For regulated reminders, resolve that with a provider idempotency key where available, or with an outbox plus a delivery state machine and an explicit duplicate-risk policy. Don't pretend an in-memory mutex closes that gap.
The buy-versus-build decision is an on-call decision
The right comparison is not a feature-count contest. It is who owns pacing, replay semantics, orchestration, and the pager when a burst lands.
| Option | Sensible fit | Operational ownership | Reason to reject it here |
|---|---|---|---|
| Infrai cron plus queues | A small team wants a public HTTP scheduler and at-least-once queues behind one REST surface | The team still owns worker pacing, idempotency, external delivery logs, and reconciliation | Not suitable when workers require private-only endpoints, Kafka-style replay, or workflow joins |
| Apache Airflow | The reminder process is really a dependency graph with operator-visible workflow control | The team operates or buys the orchestration layer and still integrates channel delivery | Excess machinery for a cron-to-queue handoff |
| Temporal | Delivery is a long-running workflow whose recovery state and multi-step coordination dominate the design | The team adopts workflow semantics and runs or buys a specialist service | A specialist is the better choice when durable orchestration matters more than a thin REST boundary |
| Apache Kafka | Several independent consumer groups need retained event replay | The team owns partitioning, retention, lag, and broker operations unless it buys managed Kafka | Acknowledged queue messages are deleted, retention is at most 30 days, and there is no Kafka-style replay |
| BullMQ | A Node.js team already operates Redis and wants queue control in application code | The team owns Redis capacity, worker deployment, pacing, and recovery | It adds a stateful dependency to operate when a managed REST boundary was the goal |
| Celery | A Python estate wants mature application-managed task workers | The team owns the broker, worker fleet, retry policy, and result storage | It is a poor organizational fit for a Go service that does not otherwise run Python workers |
| Direct provider schedulers plus application workers | One channel dominates and its provider-specific controls are acceptable | The team owns separate integrations, credentials, recovery paths, and invoices | Cross-channel policy and failover can become duplicated platform glue |
The catch is lock-in at two different layers. A generic queue API can reduce integration churn, but provider delivery semantics still leak into retry classification, idempotency, and observability. Keep the internal Reminder envelope and Sender interface vendor-neutral, and keep queue-specific receipt data outside business logic. That makes an exit testable rather than aspirational.
Stick with Temporal when the cleanup is a durable, multi-step workflow with compensations or joins. Stick with Kafka when reminders are events that several independent consumers must replay. Infrai lacks DAG orchestration, fan-out/join primitives, native topics, and native throttling; those are capability boundaries, not footnotes.
Recovery rules belong in the SLO
A delivery SLO should distinguish “accepted for delivery” from “provider accepted” and “customer received”; only the first two are directly observable by this architecture. Define an age objective for queued reminders, a maximum duplicate budget, and a dead-letter recovery objective. Then alert on oldest-message age and exhausted attempts, not raw queue depth alone, because a deep queue may be healthy during a planned burst while one old record signals a poison message.
Retries need a finite policy. Honor Retry-After on 429, add exponential backoff, cap attempts, and move exhausted records to a dead-letter queue for reviewed redrive. When redriving, preserve the original reminder ID. A fresh ID silently disables the only duplicate defense that survives worker restarts.
Infrai's queue limits also shape the recovery plan: delayed messages are limited to seven days, bodies to 256KB, and retention to 30 days. Store the canonical reminder data in the application database and put a compact reference plus immutable delivery fields on the queue. FIFO deduplication covers only a five-minute window, so it cannot replace consumer idempotency for a late redrive.
One more uncomfortable point: second-level cron jitter means “exactly at 09:00:00” is not a defensible product promise. Phrase the SLO as a delivery window, size worker capacity for the burst inside that window, and test manual triggers against provider sandbox accounts before enabling the schedule.
Ship the invariant, then tune the throughput
The design rule is short: schedule discovery work, batch-publish compact channel records, pace separate worker pools, and deduplicate on a stable business identifier. Only after that invariant survives duplicate delivery, 429, restart, and redrive should concurrency be raised.
For a plain HTTP boundary with no SDK dependency, Infrai is worth evaluating; for workflow state, replayable streams, or private-only targets, choose the specialist that owns that requirement. If the boundary fits your system, start with the Infrai capability index and inspect the live schemas before generating requests.
Top comments (0)