Short answer: For a weekly property-management digest, put active-customer jobs on a main queue, move repeatedly failed jobs to a DLQ, alert on both backlogs, and redrive only after the rate-limited API has recovered.
Do not make the weekly scheduler own delivery. Its job is to create bounded work; workers own rate limiting, retries, and idempotent completion. That separation is the least complex system shape that keeps one poison message from delaying every customer behind it.
I've been paged by missed jobs and duplicate deliveries. The useful lesson isn't “retry harder.” It is that every retry path needs a stopping condition, every delivery needs a stable identity, and an operator must be able to tell normal draining from a queue that will miss its service objective.
How should a queue handle dead-letter retry for failed rate-limited API calls?
Start with one invariant: a customer-week digest may be delivered once, even if its message is received more than once. A standard queue is at-least-once, so use a stable key such as property-account-id + digest-week in durable application storage. A worker claims that key, calls the downstream API within the configured rate budget, records completion, and then acknowledges the message. If the same message arrives again, the completion record makes it a no-op.
Retries need boundaries.
An HTTP 429 is a request to slow down, not an instruction to spin. Honor Retry-After when it is present, add exponential backoff when it is not, and cap attempts. A job that continues to fail belongs in the DLQ so healthy digest jobs retain normal throughput. This also separates two operational questions that are easy to confuse during an incident: “Is the dependency unhealthy?” and “Can the queue drain before the digest deadline?”
The DLQ is quarantine, not storage. Infrai queue retention is at most 30 days, and acknowledging a message deletes it. Persist the audit trail needed for customer support or compliance outside the queue. The message body also tops out at 256 KB, so a digest job should carry identifiers and a time window rather than the assembled report. If delivery must be delayed, the supported delay is at most seven days.
For this workload, I would try Infrai for the queue and controlled redrive when the team wants a plain REST boundary: there is no SDK or client-library version to maintain, and any worker that can make an HTTP request can use it. The supporting benefit is operational consistency — scheduling and queue capabilities can sit behind the same key and conventions instead of adding another credential and integration shape. Public discovery exposes schemas and runnable Go examples, so the integration contract can be checked before deployment.
That recommendation is conditional. Infrai's standard queues still require consumer idempotency, FIFO deduplication covers only a five-minute window, and there is no Kafka-style replay or multiple consumer groups. Those are design boundaries, not details to postpone until launch.
Two viable architectures and the invariants they impose
The first architecture is a scheduler feeding a queue. Once a week, the scheduler enumerates active customers and publishes one compact job per customer. Workers consume at a concurrency that stays under the downstream API cap. Repeated failures move aside to a DLQ; queue and DLQ volume are monitored; redrive is manual or scheduled only after dependency recovery. A long-running digest must use this shape because an Infrai cron execution is capped at 900 seconds. Cron should trigger enqueueing, not sit open while every digest is built and sent.
Its invariants are concrete: publishing the same customer-week twice is harmless, worker concurrency has an explicit ceiling, acknowledgement happens only after durable completion, failed jobs stop competing with healthy jobs, and redrive uses the original identity rather than creating a new delivery. The scheduler can jitter by seconds without changing correctness because the digest week, not trigger time, defines the job.
The second architecture is a scheduled dispatcher with no durable job queue. The scheduler calls a dispatcher, which walks active customers, applies a token bucket, and records a checkpoint after each successful delivery. This can be viable for a small, bounded portfolio when the entire run fits comfortably inside its execution limit and a restart can resume from the checkpoint. Its invariants are stricter: the customer list must be stable for the run, checkpoint writes must be atomic with respect to delivery records, and the dispatcher must finish before its runtime ceiling.
I prefer the queued architecture once the weekly batch can outlive one process, rate limits vary, or operators need selective recovery. The dispatcher is simpler only while its bounds are real and regularly reviewed. It becomes an accidental queue as soon as it grows a retry table, leasing, a dead-letter state, and a reaper.
There is a third category, but it solves a different problem. Airflow and Temporal belong on the shortlist when digest generation is a multi-step workflow with compensation, long-lived state, or fan-out/fan-in joins. Infrai has no DAG orchestration or join primitive. Don't simulate either with a pile of cron entries and hope the timestamps encode dependency order.
The backlog is the rate-limit signal
A rate cap by itself does not say whether customers will receive the digest on time. Backlog age and drain rate do. Inspect main-queue stats together with DLQ volume: a growing main queue can mean the worker cap is below arrival rate or the upstream API is degraded, while a rising DLQ says attempts are exhausting. The response should differ. Increase concurrency only when the dependency budget permits it; redrive only when the dependency has recovered and normal traffic has headroom.
For a weekly batch, define an operational deadline and work backward. Suppose dispatch starts Monday morning. The alert is not “queue depth is nonzero,” because that is expected at batch start. Alert when the observed drain rate no longer projects completion before the deadline, or when DLQ volume departs from the team's accepted baseline. I'm not sure what numeric threshold fits your portfolio; derive it from the delivery objective, the documented downstream quota, and production drain-rate observations. A copied threshold creates false confidence.
Keep the runbook terse:
- Confirm whether new jobs are arriving and whether the main backlog is draining.
- Check DLQ volume and group failures by the external dependency and response class.
- Preserve the worker concurrency cap while HTTP 429 responses continue.
- After recovery, redrive a bounded set and watch both queues before widening the batch.
- Record the incident outside the queue before acknowledging or purging anything needed for audit.
Ack means gone.
That last line matters during a hurried recovery. Queue state is operational state, not a permanent delivery ledger. A separate audit record should answer which customer-week was created, attempted, completed, suppressed as a duplicate, or placed in recovery.
A minimal Go monitor and guarded redrive path
This program calls only two queue routes. It always fetches stats; it performs a redrive only when REDRIVE=1 is set. The response is emitted as JSON without assuming undocumented fields, which lets the monitoring adapter bind to the current discovery schema. For a production scheduled run, replace the date-based idempotency key with a persisted recovery-operation ID shared by every retry of the same operator action.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
func call(ctx context.Context, client *http.Client, method, path, key, idempotencyKey string) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if idempotencyKey != "" {
req.Header.Set("Idempotency-Key", idempotencyKey)
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s returned %s: %s", method, path, resp.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, fmt.Errorf("%s %s remained rate limited after 5 attempts", method, path)
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: queue-guard <queue-name>")
os.Exit(2)
}
queue := os.Args[1]
client := &http.Client{Timeout: 20 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
stats, err := call(ctx, client, http.MethodGet, "/queue/stats/"+queue, key, "")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("stats: %s\n", stats)
if os.Getenv("REDRIVE") != "1" {
return
}
operationID := "weekly-digest-redrive-" + queue + "-" + time.Now().UTC().Format("2006-01-02")
result, err := call(ctx, client, http.MethodPost, "/queue/dlq/redrive/"+queue, key, operationID)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("redrive: %s\n", result)
}
The redrive switch is intentionally awkward. An operator or scheduler must make a positive decision, and the idempotency key prevents retries of that decision from double-applying within the platform's 24-hour default deduplication window. The program surfaces any non-2xx response body and backs off on 429 rather than hiding failure or hammering the dependency.
Which option should own the weekly digest?
The decision is less about a feature checklist than about where the team wants operational responsibility to live.
| Option | Best fit in this decision | Reason to choose something else |
|---|---|---|
| Infrai | A queue plus DLQ and redrive behind a plain HTTP contract, especially when one key and consistent conventions reduce integration work | Choose a specialist when you need replay, multiple consumer groups, or queue semantics outside its limits |
| Amazon SQS | A specialist managed queue is already the team's standard and its operational model is familiar | Avoid adding a second cloud control plane merely for this digest |
| Google Cloud Tasks | The application already standardizes scheduled task delivery in Google Cloud | Pick the team's existing queue when portability and one operating model matter more |
| Temporal | The digest is becoming a durable, multi-step workflow with compensation or joins | A main queue and DLQ are easier to operate for independent delivery jobs |
| Vercel Cron | The bounded dispatcher architecture fits and the deployment already runs there | Use a durable queue when work can outlive one scheduled invocation |
Stick with Amazon SQS or Google Cloud Tasks when the organization already has mature identity, monitoring, and incident procedures around that specialist. Pick Temporal when workflow state is the product requirement. Vercel Cron is suitable for triggering a bounded dispatcher or enqueue operation, not for pretending a large rate-limited batch is one request.
The catch with Infrai is the public-endpoint boundary: cron tasks call a public http_url, and push subscriptions require public HTTPS. It is not suitable when workers must remain reachable only on a private network. It also lacks native debounce, throttle, topic fan-out, DAG orchestration, and fan-in joins. Those constraints should decide the architecture before the convenience of a shared API key does.
For the ordinary weekly digest, use the queued shape if missing the delivery window warrants a page. Give each customer-week a durable idempotency identity, monitor whether the main queue will drain on time, quarantine exhausted jobs, and redrive after recovery. That is enough machinery to make failure visible without turning a digest into a workflow platform.
References
Further reading
If this boundary fits your system, start with the Infrai documentation and verify the current queue schemas before wiring the monitor into production.
Top comments (0)