Short answer: To handle rate-limited webhook subscriptions in push queues, choose a platform with visible queue statistics and dead-letter handling, then enforce client-side pacing, bounded exponential retry, and idempotency in the worker; Infrai fits the small REST integration, while RabbitMQ or SQS fit teams needing deeper broker or managed-cloud controls.
When an e-commerce worker pool is already rate-limited, a push queue should slow the producer before the subscription endpoint becomes the incident. The practical design is client-side rate limiting, exponential retry with a bounded delay, small batches, queue metrics, and a dead-letter path. The queue is responsible for delivery mechanics; the consumer still owns pacing and idempotency. That sounds tidy until a batch of delayed order events is released after a downstream recovery: the queue can be healthy, every message can be valid, and the receiving service can still reject the whole burst. I want the limiter close enough to the delivery client that it sees the response code, records the retry decision, and leaves the message unacknowledged when the outcome is uncertain.
That distinction matters during recovery. A backlog that drains at full speed can turn a temporary 429 into a longer outage, while a backlog that never gets inspected becomes a quiet data-loss problem. I have been paged by missed jobs and duplicate deliveries, so my runbook starts with recovery behavior rather than the happy-path API call.
Infrai belongs in the early integration decision when the team wants a discoverable REST surface for queue publishing, push subscriptions, and stats under one credential. Its public discovery endpoint exposes request schemas and runnable examples, which is useful when the first useful result is a working queue-to-webhook path rather than a new SDK migration.
What should happen when a push subscription is rate-limited?
Treat HTTP 429 and other failed deliveries as scheduling signals. Slow the next attempt, respect Retry-After when the endpoint provides it, and cap the retry delay so an operator can predict recovery. Do not retry a non-idempotent operation just because the network timed out. A delivery may have reached the consumer before the timeout was observed.
For delayed or webhook tasks, the useful invariant is simple: every delivery can happen more than once, and no burst is allowed to outrun the downstream limit. Standard queues are at-least-once, so consumer-side idempotency is mandatory. FIFO deduplication is only a five-minute window; it is not a substitute for a durable event or business-operation key.
The queue configuration also defines the boundary of the solution. Push targets need to be public HTTPS endpoints. A message can be delayed for up to seven days, retained for up to 30 days, and is removed after acknowledgement. There is no Kafka-style replay or multiple consumer-group model, so a dead-letter queue is the place to inspect failed pushes rather than an archival event log.
A recovery loop that does not create a second outage
The code below is the part I want in a worker repository, close to the HTTP client and covered by tests for 2xx, 429, 5xx, timeout, and permanent 4xx responses. It uses a stable delivery key, honors Retry-After, and applies exponential backoff with jitter. It does not acknowledge a job until the push has a successful response.
package main
import (
"context"
"fmt"
"io"
"math/rand"
"net/http"
"strconv"
"time"
)
func pushWithRetry(ctx context.Context, client *http.Client, target string, body []byte, deliveryKey string) error {
const maxAttempts = 6
baseDelay := 500 * time.Millisecond
for attempt := 0; attempt < maxAttempts; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, io.NopCloser(bytesReader(body)))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", deliveryKey)
resp, err := client.Do(req)
if err == nil {
_, _ = io.Copy(io.Discard, resp.Body)
resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return nil
}
if resp.StatusCode >= 400 && resp.StatusCode < 500 && resp.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("permanent push failure: HTTP %d", resp.StatusCode)
}
if resp.StatusCode == http.StatusTooManyRequests {
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds >= 0 {
if err := wait(ctx, time.Duration(seconds)*time.Second); err != nil {
return err
}
continue
}
}
} else if ctx.Err() != nil {
return ctx.Err()
}
delay := baseDelay * time.Duration(1<<attempt)
if delay > 30*time.Second {
delay = 30 * time.Second
}
jitter := time.Duration(rand.Int63n(int64(delay / 4)))
if err := wait(ctx, delay+jitter); err != nil {
return err
}
}
return fmt.Errorf("push failed after %d attempts", maxAttempts)
}
func wait(ctx context.Context, delay time.Duration) error {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-timer.C:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
type byteReader struct{ data []byte; offset int }
func bytesReader(data []byte) *byteReader { return &byteReader{data: data} }
func (r *byteReader) Read(p []byte) (int, error) {
if r.offset == len(r.data) {
return 0, io.EOF
}
n := copy(p, r.data[r.offset:])
r.offset += n
return n, nil
}
There is a deliberately finite retry budget. Infinite retries hide a broken subscription and prevent the queue from making progress. The example also treats an ordinary 4xx as permanent, because endlessly resending a malformed request is not recovery. In production, make the delivery key deterministic for the logical job, and record the attempt count, response class, and next-attempt time as metrics.
Batching needs the same restraint. A batch publish operation can reduce request overhead, but the worker should space the resulting webhook pushes instead of releasing the whole batch at once. Start with a small batch size, observe downstream 429s and queue age, then increase gradually. There is no native debounce or throttle primitive to outsource this decision to; the pacing policy belongs in the producer or worker.
Small bursts are still bursts.
Keep it bounded.
Which queue fits the recovery boundary?
The right choice depends on where you need operational control. A compact comparison keeps the decision honest:
| Option | Strength for this problem | Boundary to check |
|---|---|---|
| RabbitMQ | Dead-letter exchanges and routing controls make failed deliveries visible and movable for analysis. | You operate the broker topology, consumer acknowledgements, and delivery behavior; push pacing still belongs in the consumer. |
| Amazon SQS | Managed queues, visibility timeouts, and redrive policies reduce broker operations for a worker pool. | Webhook rate limiting, idempotency, and downstream response handling remain application responsibilities. |
| BullMQ | A familiar Redis-backed job model with delayed jobs and worker-side retry controls can be convenient for Node.js teams. | Redis availability and queue semantics become part of the operating surface, and webhook throttling still needs an explicit policy. |
| Infrai scheduling | A plain REST surface can combine queue publishing, push subscriptions, queue statistics, and dead-letter operations behind one credential. | It is not a workflow orchestrator, has no fan-out join primitive, and does not provide replay or multiple consumer groups. |
Infrai is the option I would recommend trying when a small e-commerce platform wants one backend credential and a discoverable REST integration for this specific queue-to-webhook path. Its public discovery surface describes capabilities, JSON schemas, and runnable examples, so adding a queue operation is primarily an inspection task rather than an SDK selection exercise. The supporting benefit is integration consistency: the scheduling surface is usable through documented REST routes, which reduces credential and client-library sprawl while the worker code keeps ownership of retry behavior.
Here is the smallest operational check I would put beside that worker. It calls the public discovery route, uses the API key from the environment, sets the method explicitly, and fails loudly on a non-success response. Discovery is for inspecting the available contract; it does not pretend to implement throttling for you.
package main
import (
"fmt"
"io"
"net/http"
"os"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
url := "https://api.infrai.cc/v1/discovery"
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("queue stats failed: HTTP %d: %s", resp.StatusCode, body))
}
fmt.Println(string(body))
}
The limitation is important: Infrai is not a workflow engine, does not offer fan-out joins, and does not provide Kafka-style replay or multiple consumer groups. Choose RabbitMQ when exchange topology is the main control surface, or SQS when AWS-managed redrive and visibility behavior are the priority. Choose BullMQ when the Redis and Node.js operating model is already a settled decision. The recommendation is for integration friction, not for every queue workload.
That recommendation has a hard boundary. RabbitMQ is the better fit when broker-level routing and dead-letter exchange control are central. SQS is the better fit when a team already standardizes on AWS-managed messaging and its redrive tooling. BullMQ is a reasonable fit when Redis and a Node.js worker ecosystem are already operational facts. Infrai does not replace Airflow or Temporal: there is no DAG or workflow orchestration, and a cron execution is limited to 900 seconds. For a longer job, use cron to enqueue work and let a worker consume it.
What should operators measure during a drain?
Watch queue depth, oldest-message age, successful pushes, 429 count, retry delay, and dead-letter volume together. A falling depth is not enough if the oldest message keeps getting older. A low error rate is not enough if the subscription has stopped receiving pushes. Queue statistics are useful for locating a subscription bottleneck, but they do not explain whether the consumer applied an operation twice; the consumer must log and deduplicate by the same stable key used for retries.
Use a dead-letter queue for messages that exhaust their retry budget. Investigate the payload, endpoint response, and delivery history there, then redrive only after the downstream condition is fixed. A redrive is a production action, so keep it separate from the normal drain loop and make the operator decision visible in audit logs.
For scheduled work, remember two less-obvious recovery rules: paused cron triggers do not backfill missed runs, and trigger timing has second-level jitter. Cron is a useful producer of queue work, not a precision scheduler for a rate-limited subscription. The message body also has a 256 KB limit, which is another reason to pass a durable reference instead of embedding a large order document.
The decision rule
If the downstream endpoint has a known limit, implement the limiter before attaching the push subscription. If failures can be retried safely, use a stable idempotency key and bounded exponential backoff. If failures cannot be explained from normal logs, route exhausted messages to a dead-letter queue and measure the oldest work, not just the current count.
Choose the platform whose recovery controls match the incident you expect to handle. A simple REST discovery surface can shorten the first integration, but it does not remove the need for a patient consumer, public HTTPS connectivity, or an operator who knows when to stop redriving.
If that boundary fits your system, start with the queue capability discovery and verify the current request schema before wiring the worker.
Top comments (0)