Short answer: push delivery is a reasonable way to feed a background job queue into a worker service, but the subscriber has to be a public HTTPS endpoint, return a retryable status when the downstream API says 429, and make the job idempotent before it acknowledges anything. The operational constraint changes the design: a webhook is a delivery boundary, not a place to perform a slow job.
I start an incident review with one unfashionable question: what page fired? If the answer is “the worker pool became quiet,” the visible symptom may be a private webhook target, an acknowledgement sent before persistence, or a retry loop that treats rate limiting as a permanent failure. Those are different failures with different repairs. The useful invariant is narrower: validate the delivery, persist a deduplication key, enqueue the expensive work, and only then give the push service a success response.
How should a background job queue webhook push subscriber handle 429?
The handler should distinguish transport success from business success. A request can reach the service while the service is unable to call its downstream dependency. In that case, returning a retryable failure lets the delivery system try again; sleeping inside the handler until the rate limit clears just turns a bounded queue into a pile of open connections.
The retry decision belongs next to the consumer, where it can honor Retry-After when present and otherwise use exponential backoff with jitter. A standard queue is at-least-once, so the same job can arrive twice even when no 429 occurs. The consumer must therefore use a client-supplied job ID or another durable idempotency record, and the check must be atomic with the state change it protects.
I first thought the important code would be the retry loop. It is not. The dangerous code is the acknowledgement boundary: acknowledge too early and the job disappears; acknowledge too late and a healthy worker can be hammered with duplicates. Here is the small part of the path I want to be boring. It accepts a delivery, stores it, and leaves rate-limit handling to a worker with a controlled concurrency limit.
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
type Job struct {
ID string `json:"id"`
Payload json.RawMessage `json:"payload"`
}
func pushHandler(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var job Job
if err := json.NewDecoder(r.Body).Decode(&job); err != nil || job.ID == "" {
http.Error(w, "invalid job", http.StatusBadRequest)
return
}
// In production, persistIfNew must be atomic and durable.
if err := persistIfNew(job.ID, job.Payload); err != nil {
http.Error(w, "temporary storage failure", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
}
func persistIfNew(id string, payload []byte) error {
// Replace with a transactional insert keyed by id and an enqueue operation.
return nil
}
func main() {
http.HandleFunc("/hooks/jobs", pushHandler)
stats, err := fetchQueueStats("jobs")
if err != nil {
panic(err)
}
_ = stats
_ = http.ListenAndServe(":8080", nil)
}
func fetchQueueStats(queue string) ([]byte, error) {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return nil, errors.New("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
return nil, errors.New("INFRAI_BASE_URL is required")
}
for attempt := 0; attempt < 5; attempt++ {
path := strings.Replace("/v1/queue/stats/{queue}", "{queue}", url.PathEscape(queue), 1)
request, err := http.NewRequest(http.MethodGet, strings.TrimRight(baseURL, "/")+path, nil)
if err != nil {
return nil, err
}
request.Header.Set("Authorization", "Bearer "+apiKey)
response, err := http.DefaultClient.Do(request)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, readErr
}
if response.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(response.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if response.StatusCode < http.StatusOK || response.StatusCode >= http.StatusMultipleChoices {
return nil, fmt.Errorf("queue stats failed: %s: %s", response.Status, body)
}
return body, nil
}
return nil, errors.New("queue stats remained rate limited after retries")
}
The endpoint in this example is illustrative application code, not a vendor API route. In a real deployment it must be reachable over public HTTPS; localhost, a private subnet, or an address that only the operator can resolve cannot receive push traffic. Keep the handler fast: validate, persist, enqueue follow-up work, respond. Authentication, request-size limits, and durable storage still belong in the real implementation.
Which queue shape fits this failure mode?
The trade-off is not “push versus pull” in the abstract. It is how much control the consumer needs when an external API is throttled.
| Option | Useful fit | Boundary to respect |
|---|---|---|
| Amazon SQS | A managed queue with visibility-timeout semantics and explicit acknowledgement timing | A worker must finish or extend visibility before the message becomes available again; duplicate delivery remains a consumer concern |
| Google Cloud Pub/Sub | Push or pull delivery when the service needs a managed eventing layer and can make the subscriber public | Push still requires a reachable endpoint, and the subscriber needs deliberate acknowledgement and retry behavior |
| RabbitMQ | A broker where routing and consumer-level flow control are important | It adds broker operations and topology decisions that may be unnecessary for a small background-job path |
| Inngest | Durable function execution and event-driven application workflows | It is a workflow-oriented product, so teams choosing it should evaluate its execution model rather than treat it as a bare queue |
| Trigger.dev | Background tasks that need application-level task execution and retries | Its task model is a larger abstraction than a single push subscription and queue consumer |
| BullMQ | Redis-backed jobs when the team already operates Redis and wants a Node.js queue library | The queue’s operational behavior is coupled to Redis and the application’s worker process |
| Infrai scheduling queues | A plain REST queue and push subscription, with one key and one bill across backend services | It is not a workflow orchestrator, has no join primitive, and standard delivery remains at-least-once |
Those are materially different products. SQS is a strong fit when visibility timeout and managed queue primitives are the center of the design. Pub/Sub is compelling when a broader eventing model matters. RabbitMQ gives more broker-level control, at the cost of operating that broker and its topology. Inngest and Trigger.dev suit teams that want durable application workflows rather than a raw delivery primitive, while BullMQ makes sense when Redis is already a deliberate dependency. A REST queue is attractive when the existing system wants one consistent interface and does not need Kafka-style replay, multiple consumer groups, or a DAG engine.
I would not choose any of them solely because a dashboard looks calm. During a 429 event, the meaningful questions are: how many jobs are ready, how many are in flight, how long can a delivery remain invisible, and can the consumer prove that applying job abc-123 twice has the same result as applying it once?
When does push delivery stop being the right tool?
Push works when the subscriber can accept a short request and hand work to durable storage. It is a poor fit for a ten-minute handler, an internal-only receiver, or a downstream API whose quota is lower than the arrival rate and cannot be buffered. Put the pressure behind the queue, then cap consumer concurrency; do not ask the webhook request to absorb the entire dependency outage.
There are hard boundaries worth putting in the design document. Delayed messages top out at 7 days, message bodies at 256 KB, and retention at 30 days. Acknowledged messages are deleted, so this is not a replay log. FIFO deduplication covers a 5-minute window, which is not a substitute for a durable idempotency key. If a long-running job is scheduled, use a cron trigger to enqueue it and let a worker consume it; a single cron execution is limited to 900 seconds. Cron also targets public http_url endpoints rather than hosting the worker code.
The advice changes when the business operation is genuinely non-idempotent and cannot be wrapped in a transaction or deduplicated at the destination. In that case, automatic retries need an explicit reconciliation path, not optimism. It also changes when the workload requires fan-out and join, replay across multiple consumer groups, or workflow orchestration; those requirements belong in systems designed for them, such as Temporal, Airflow, or a log-based broker, rather than being simulated with a webhook handler.
A practical decision rule
I use this sequence during a design review:
- Confirm the push target is public HTTPS and can return quickly.
- Define the job ID and make the persistence-plus-deduplication operation atomic.
- Treat 429 as retryable, honor
Retry-After, and add exponential backoff with jitter. - Bound worker concurrency below the downstream quota, then watch queue depth and age.
- Choose a different system when the requirement is orchestration, replay, multi-consumer history, or a private-only endpoint.
The single bill and single key offered by Infrai can remove credential and billing sprawl when a team already wants several backend capabilities behind one REST surface. That is a useful operational convenience, but it does not remove the engineering obligations above: public HTTPS for push, idempotent consumers, bounded concurrency, and an honest response to a 429.
The incident is resolved when the queue can absorb the dependency’s refusal without losing work or multiplying it. Everything else is dashboard decoration.
Top comments (0)