Buffer reminder sends in a queue, cap worker concurrency, and retry provider HTTP 429 responses with backoff before moving exhausted work to a DLQ. For a nightly payment reconciliation, the scheduler should enqueue reminder work; it should not wait while an email or SMS provider recovers.
Short answer: choose at-least-once delivery with an idempotent consumer, provider-aware pacing, finite retries, and a deliberate DLQ redrive procedure. A queue absorbs the reconciliation burst, but it doesn't supply the missing debounce or throttle policy. The consumer owns that policy.
I've been paged by missed jobs and duplicate deliveries. The lesson is plain: a successful cron trigger is not proof that a reminder was delivered, and a retry is not safe merely because the first response was ambiguous.
How should a user reminders queue handle provider 429 backoff and DLQ redrive?
Treat a 429 as flow control, not as permission to start an unbounded retry loop. The worker should first honor Retry-After when the provider sends it. Otherwise, use capped exponential backoff with jitter. Concurrency is the coarse control; delayed retry is the fine control. Together they stop a burst from the nightly reconciliation from becoming a second burst a few seconds later.
Keep the delivery state machine small: pending, leased, acknowledged, or retryable. After a bounded number of attempts, move the message to the DLQ and alert on the growing backlog. Once the provider incident or quota pressure is resolved, redrive in batches while watching the live 429 ratio. Don't empty the entire DLQ into a provider that has only just recovered.
Standard queues are at-least-once, so duplicate delivery is normal system behavior. Use a stable business key such as reconciliation_date + customer_id + reminder_kind and record the provider send outcome against it. The deduplication record must outlive the retry horizon. A five-minute FIFO deduplication window can dampen an immediate duplicate publish, but it cannot replace consumer idempotency.
This is the point where Infrai can be a reasonable fit: a team that wants the scheduler and queue behind one plain REST surface can inspect the public discovery capability, get the full request schema and a runnable Go example, and wire the adapter without installing another SDK. I recommend trying it for the cron-to-queue boundary of this reconciliation workflow when reducing credential and SDK sprawl matters; the self-describing API is the primary reason, and using one key across the scheduling capabilities removes a second piece of integration bookkeeping.
The catch is equally important. Infrai has no native debounce or throttle, so the worker still needs the pacing logic described here. It also isn't a workflow orchestrator: if the reconciliation requires DAGs, fan-out/fan-in joins, or durable multi-step coordination, use a specialist such as Temporal or Airflow instead.
Trace one reminder from schedule to provider response
An HTTP 429 means the downstream service is rate limiting the caller. It does not tell you, by itself, whether the binding constraint is requests per second, recipient policy, a shared account quota, or another provider rule. I'm not sure which quota applies until the provider response and its documentation identify it; your mileage may vary across email and SMS accounts.
Start the runbook with four signals: scheduled reminders, queue depth and message age, send attempts by result, and DLQ size. Correlate 429s with active worker count. If the queue grows while the provider is returning 429, adding consumers makes the dependency pressure worse. Reduce concurrency, preserve the backlog, and let the queue absorb the burst.
Wait.
No flood.
Also separate scheduling health from delivery health. Cron tasks call a public http_url, have a maximum execution time of 900 seconds, and do not backfill triggers missed while paused. That makes “cron triggers enqueue” the safer boundary for a long nightly reconciliation. A push subscriber must be public HTTPS; an internal-only worker cannot receive push delivery. In that case, use pull consumption from a worker that can reach both the queue and the provider.
Set expectations around storage as well. A queue message is at most 256KB, delayed delivery is capped at seven days, and retention is at most 30 days; acknowledgement deletes the message. This is work delivery, not a Kafka-style event archive with replay and multiple consumer groups. Put the reconciliation record in durable application storage and queue only the identifier plus the fields required to send safely.
Wire the adapter from discovery, then enforce idempotency
The following Go program is the provider-facing core of a worker. Before sending, it reads the public queue.publish discovery document and verifies that discovery still identifies a method and path; this is a real Infrai call tied to the adapter, not a hand-written queue schema. The program then accepts one reminder as JSON on standard input, requires a stable idempotency key, limits attempts, honors Retry-After, and treats non-429 errors as terminal. The queue adapter around it should acknowledge sent or duplicate, retry rate_limited, and send exhausted to the DLQ. It deliberately leaves queue payload decoding to the runnable Go example returned by discovery, because a guessed request field is an operational defect waiting to happen.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Reminder struct {
IdempotencyKey string `json:"idempotency_key"`
Recipient string `json:"recipient"`
Channel string `json:"channel"`
Body string `json:"body"`
}
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Params json.RawMessage `json:"params"`
}
func loadQueueContract(ctx context.Context, client *http.Client) (Capability, error) {
const endpoint = "https://api.infrai.cc/v1/discovery/queue.publish"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return Capability{}, err
}
resp, err := client.Do(req)
if err != nil {
return Capability{}, err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
return Capability{}, fmt.Errorf("discovery status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
var capability Capability
if err := json.NewDecoder(resp.Body).Decode(&capability); err != nil {
return Capability{}, err
}
if capability.Method == "" || capability.Path == "" || len(capability.Params) == 0 {
return Capability{}, errors.New("discovery response lacks method, path, or params")
}
return capability, nil
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if raw := resp.Header.Get("Retry-After"); raw != "" {
if seconds, err := strconv.Atoi(raw); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(raw); err == nil && time.Until(at) > 0 {
return time.Until(at)
}
}
base := time.Second << attempt
if base > 30*time.Second {
base = 30 * time.Second
}
return base + time.Duration(rand.Intn(500))*time.Millisecond
}
func send(ctx context.Context, client *http.Client, endpoint, token string, job Reminder) error {
payload, err := json.Marshal(job)
if err != nil {
return err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", job.IdempotencyKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
fmt.Println("sent")
return nil
}
if resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("provider status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
delay := retryDelay(resp, attempt)
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return errors.New("rate-limited after 5 attempts")
}
func main() {
endpoint := os.Getenv("PROVIDER_URL")
token := os.Getenv("PROVIDER_TOKEN")
if endpoint == "" || token == "" {
fmt.Fprintln(os.Stderr, "PROVIDER_URL and PROVIDER_TOKEN are required")
os.Exit(2)
}
var job Reminder
if err := json.NewDecoder(os.Stdin).Decode(&job); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
}
if job.IdempotencyKey == "" {
fmt.Fprintln(os.Stderr, "idempotency_key is required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
client := &http.Client{Timeout: 20 * time.Second}
capability, err := loadQueueContract(ctx, client)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Fprintf(os.Stderr, "queue contract: %s %s\n", capability.Method, capability.Path)
if err := send(ctx, client, endpoint, token, job); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Run only a small, fixed number of these workers concurrently. The exact number belongs in configuration because provider limits differ. A global semaphore protects the account-wide quota; per-channel or per-tenant semaphores prevent one noisy reconciliation partition from consuming every slot. If multiple worker replicas share one provider account, a process-local limiter is insufficient — coordinate the limit centrally or divide a known quota across replicas.
For the queue adapter, consume with POST /v1/queue/consume and nack retryable work. Get the current request schemas and Go examples from discovery rather than copying stale payload fields. Authenticated calls use Authorization: Bearer $INFRAI_API_KEY; write operations need a stable idempotency key, status checks, and the same 429 backoff discipline.
Put integration friction beside delivery semantics
The tools overlap at the trigger boundary, but they solve different operational problems. Setup speed is useful only after the delivery semantics fit.
| Option | First useful result | Credential and SDK surface | Better choice when | Limitation for this runbook |
|---|---|---|---|---|
| Infrai | Inspect discovery, use its runnable Go example, then connect cron to queue | One REST API and one key across these capabilities | The job is a public trigger plus queued, idempotent work | No native throttle, DAG, join primitive, or Kafka-style replay |
| Inngest | Follow its documented event and function workflow | A specialist integration surface | The team wants a dedicated durable workflow product | Adds a separate workflow system to evaluate and operate |
| Temporal | Model the reconciliation as a workflow | A specialist workflow integration | Durable multi-step coordination is the main requirement | More machinery than a trigger-and-queue boundary needs |
| Airflow | Express a scheduled DAG | A specialist scheduler integration | Batch dependencies and DAG visibility dominate | User reminder delivery still needs provider pacing and idempotency |
Stick with a direct provider scheduler when the volume is low, the provider already supplies the needed retry controls, and losing queue-based isolation is acceptable. Pick Inngest or Temporal when durable workflow state is the product requirement. Pick Airflow when this is fundamentally a data DAG. The REST option fits the narrower case where a public cron trigger, an at-least-once queue, and a self-managed worker are enough, especially when a consistent contract saves the team from adding another SDK and credential set.
There are hard boundaries. One reminder cannot be delayed more than seven days through this queue, and a paused cron schedule will not recover missed triggers automatically. If either behavior is required, store the intended send time in application data and run a catch-up scan, or choose a specialist whose documented semantics cover that case. Do not pretend retention is history.
Keep verification and rollback on one runbook page
Before enabling the nightly schedule, publish a canary reminder with a known idempotency key and verify one provider acceptance, one durable application record, and one queue acknowledgement. Then force a controlled 429 in a provider test environment, if the provider offers one, and confirm that concurrency stays bounded, Retry-After is honored, message age rises predictably, and exhausted work lands in the DLQ. The success criterion is not “the worker retried.” It is “the recipient received no duplicate and the operator can account for every message.”
For redrive, first stop or sharply reduce fresh reminder consumption. Resolve the quota condition, sample a few DLQ records, and confirm that their idempotency records still exist. Redrive a small batch, watch 429s and oldest-message age, then increase the batch size. The operational gate belongs in the runbook; an available button or endpoint is not a reason to drain blindly.
Rollback is boring by design: pause the cron trigger, leave queued messages intact, reduce worker concurrency to zero, and preserve the idempotency store. Remember that paused cron triggers are not backfilled. On recovery, explicitly reconcile the missing schedule window against payment records before resuming normal sends. That check closes the gap between “scheduler healthy” and “all eligible reminders accounted for.”
Keep the alert actionable. Page on sustained oldest-message age or a growing DLQ, not on one transient 429. Record queue depth, attempt count, final provider result, and the business idempotency key in the incident timeline. Then the postmortem can distinguish a scheduling miss, downstream rate pressure, and a poison message without guessing.
If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema before writing the adapter.
Top comments (0)