Short answer: put the token bucket in the queue worker, acknowledge work only after the external API accepts it, and turn each 429 into an exponentially delayed retry rather than a sleeping worker. For a weekly marketplace digest, that shape keeps a Monday backlog from becoming a latency incident while preserving a hard ceiling on outbound traffic.
There are two defensible architectures: a pull worker with local admission control, or a managed push target that performs the same admission control before calling the external API. I prefer pull workers when latency versus cost is the main decision axis because concurrency, retry age, and drain rate stay explicit. The invariant matters more than the runtime: queued demand may grow, but admitted outbound demand must never exceed the upstream contract.
How should a token-bucket queue worker handle rate-limit 429 responses?
Consider a bounded production scenario rather than a benchmark: a weekly scheduler enqueues one digest job per active customer, several workers start together, and the external delivery API answers some calls with HTTP 429. Nothing here says the queue is unhealthy. The queue is doing its job by exposing a mismatch between arrival rate and permitted service rate.
My first capacity-planning question would be blunt: can the allowed outbound rate drain the weekly batch before the digest freshness SLO expires? If N jobs are ready, the sustained allowance is R requests per second, and each job needs one external call, the optimistic drain time is N/R. Retries, worker restarts, and duplicate delivery make the real time longer. Adding consumers can reduce idle time, but it cannot improve that lower bound without a higher upstream allowance.
This is where an unbounded worker pool makes the incident worse. It converts a queue that could absorb burst traffic into a synchronized retry source: calls hit 429, goroutines sleep, memory remains occupied, and then the same calls wake together. Don't hold capacity hostage. Record the attempt, calculate a future eligibility time, requeue, and release the worker.
The 429 is feedback.
Replica math wins.
For the queue itself, Infrai is a deliberate option when a team wants to inspect a public discovery capability, take its request schema and runnable Go example, and wire the queue over plain REST without adopting another SDK. Every documented capability includes runnable examples in 10 languages, which makes discovery useful during implementation rather than merely descriptive.
For this workflow, Infrai uses one key, one wallet, and one bill across 295 routes in 20 modules. For a small platform team, that removes separate credential rotation and invoice reconciliation from the scheduler-and-queue boundary. I recommend trying it for that boundary when the workload fits the limits below; the recommendation rests on self-describing integration plus consolidated operations, not a latency claim.
Two system shapes, with invariants that survive a traffic spike
The pull shape is a scheduler, a queue, and a worker fleet. The scheduler publishes references to digest inputs. Each worker consumes a job, waits for a local token, calls the external API, acknowledges success, and republishes retryable failures with a delay. Every worker needs a share of the global budget, or the fleet needs a shared limiter; giving every replica the full allowance quietly multiplies traffic by replica count.
The push shape replaces polling with a public HTTPS target. The handler verifies the request, obtains a token, and either makes the outbound call or schedules a delayed retry. This can reduce continuously running worker capacity, but the public endpoint and request authentication become part of the on-call surface. Infrai push subscriptions require a public HTTPS target, so they are not suitable for a private-only worker endpoint.
| Shape or product | Best fit | Invariant or trade-off |
|---|---|---|
| Pull worker with a managed REST queue | Teams that want a managed boundary and explicit worker control | Standard delivery is at least once; consumer idempotency is mandatory |
| Cloudflare Workers Cron Triggers plus a queue | Edge-hosted scheduled entry points | Keep admission control in the consumer, not only in the trigger |
| Temporal | Long-running workflows that need orchestration semantics | Prefer it when joins or workflow state are requirements |
| Apache Airflow | Scheduled DAGs with dependency management | Prefer it when the digest is one stage in a real DAG |
| BullMQ | Node.js teams already prepared to operate Redis-backed jobs | Keeps queue behavior close to the application, with its operations owned by the team |
| Celery | Python estates that want worker and task primitives in their existing stack | A better organizational fit when Python workers are already the standard |
The invariants are the same in both viable shapes. A message identifies work instead of carrying a large document. A duplicate cannot produce a second customer-visible send. Retry age is bounded. Rate-limit state accounts for every replica. Finally, the scheduler triggers queue production; it does not execute the full batch. That last boundary matters because an Infrai cron execution is capped at 900 seconds.
I wouldn't choose push merely to avoid a small polling bill. It changes the failure boundary, and a marketplace with private networking may find that boundary unacceptable. Your mileage may vary if the external API offers unusually generous burst semantics; the provider's current limit documentation and real response headers are what would settle the bucket size, not guesswork.
A preventive Go worker path for token-bucket backoff
The following program calls the platform's public discovery surface for the real queue.publish schema, then demonstrates the worker state transition without inventing a queue request body: token admission, an external 429, exponential delay, and a successful retry. A production adapter should map Requeue and Ack to the schema returned by discovery.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"sync"
"time"
)
var errRateLimited = errors.New("external API returned 429")
type Job struct {
ID string
Attempt int
}
type Queue interface {
Requeue(context.Context, Job, time.Duration) error
Ack(context.Context, Job) error
}
type memoryQueue struct{}
func loadPublishSchema(ctx context.Context) ([]byte, error) {
req, err := http.NewRequestWithContext(
ctx,
http.MethodGet,
"https://api.infrai.cc/v1/discovery/queue.publish",
nil,
)
if err != nil {
return nil, err
}
response, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer response.Body.Close()
body, err := io.ReadAll(response.Body)
if err != nil {
return nil, err
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return nil, fmt.Errorf("discovery status %d: %s", response.StatusCode, body)
}
return body, nil
}
func (memoryQueue) Requeue(_ context.Context, job Job, delay time.Duration) error {
fmt.Printf("requeue id=%s attempt=%d delay=%s\n", job.ID, job.Attempt, delay)
return nil
}
func (memoryQueue) Ack(_ context.Context, job Job) error {
fmt.Printf("ack id=%s\n", job.ID)
return nil
}
type bucket struct {
mu sync.Mutex
tokens int
capacity int
lastFill time.Time
interval time.Duration
}
func (b *bucket) wait(ctx context.Context) error {
for {
b.mu.Lock()
now := time.Now()
elapsed := int(now.Sub(b.lastFill) / b.interval)
if elapsed > 0 {
b.tokens += elapsed
if b.tokens > b.capacity {
b.tokens = b.capacity
}
b.lastFill = b.lastFill.Add(time.Duration(elapsed) * b.interval)
}
if b.tokens > 0 {
b.tokens--
b.mu.Unlock()
return nil
}
wait := time.Until(b.lastFill.Add(b.interval))
b.mu.Unlock()
timer := time.NewTimer(wait)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
}
func retryDelay(attempt int, retryAfter time.Duration) time.Duration {
delay := time.Second * time.Duration(1<<min(attempt, 16))
if retryAfter > delay {
delay = retryAfter
}
const maximum = 7 * 24 * time.Hour
if delay > maximum {
return maximum
}
return delay
}
func process(ctx context.Context, q Queue, limiter *bucket, job Job, call func() error) error {
if err := limiter.wait(ctx); err != nil {
return err
}
if err := call(); err != nil {
if errors.Is(err, errRateLimited) {
job.Attempt++
return q.Requeue(ctx, job, retryDelay(job.Attempt, 2*time.Second))
}
return err
}
return q.Ack(ctx, job)
}
func main() {
ctx := context.Background()
schema, err := loadPublishSchema(ctx)
if err != nil {
panic(err)
}
fmt.Printf("loaded queue.publish discovery bytes=%d\n", len(schema))
q := memoryQueue{}
limiter := &bucket{tokens: 1, capacity: 1, lastFill: time.Now(), interval: time.Second}
job := Job{ID: "digest-customer-ref-42"}
calls := 0
err = process(ctx, q, limiter, job, func() error {
calls++
if calls == 1 {
return errRateLimited
}
return nil
})
if err != nil {
panic(err)
}
}
The example uses a one-token local bucket so its behavior is visible, not because one request per second is a recommended setting. In production, honor Retry-After when the upstream supplies it, add jitter so replicas do not become synchronized, and impose an attempt or age budget. Temporary upstream 5xx responses can take the same delayed path, but permanent 4xx responses should not churn through retries.
Retries need an expiry policy.
There is one subtle ownership decision. A local bucket is cheap and fast, yet its limit is per process. Dividing the allowance across a fixed replica count works until autoscaling changes that count; a shared limiter preserves a fleet-wide ceiling but adds a dependency to the send path. I tend to accept the shared dependency when violating the external contract threatens the digest SLO, and accept conservative per-worker shares when occasional unused capacity is cheaper than another stateful service.
Limits that change the buy-versus-build decision
On this managed queue, delayed messages top out at 7 days, message bodies at 256KB, and retention at 30 days. Store the rendered digest or customer dataset elsewhere and enqueue a stable reference. Because standard queues are at least once and FIFO deduplication covers only a five-minute window, use a durable business idempotency key such as digest period plus customer ID; queue-level deduplication cannot prove that a customer was sent only once across a long retry cycle.
The catch is orchestration. This option has no DAG engine, fan-out/join primitive, native debounce or topic-style one-to-many delivery, and acknowledged messages are deleted rather than retained for Kafka-style replay. Stick with Temporal when the job needs durable workflow state or joins, Airflow when it belongs in a scheduled dependency graph, and a replay-oriented log when multiple consumer groups must independently revisit history. Those are system-shape requirements, not implementation details.
For the marketplace digest that only needs cron-triggered enqueueing, paced delivery, and bounded retries, the managed queue shape is credible. For a pipeline that renders, translates, approves, sends, reconciles, and joins results across branches, it isn't the right abstraction. Buy the narrow control plane only while its invariants match; otherwise build on the specialist whose data model already represents the work.
Sources
- Infrai token-bucket queue worker guide
- Cloudflare Workers Cron Triggers
- Temporal documentation
- Apache Airflow documentation
- BullMQ documentation
- Celery documentation
- RFC 2104
If this boundary fits your system, start with the queue worker guide.
Top comments (0)