The right answer is a queue with retry-on-failure and exponential backoff, plus a rate limiter owned by the client that delivers the webhook. A queue can remember that a reservation-expiry notification failed; it cannot infer how much traffic the target endpoint can absorb. That boundary matters more than the queue brand.
Short answer: how should failed webhook delivery be retried?
For a marketplace that holds an item for a fixed window, put the expiry task in a delayed queue, retry failures with exponential delays, and make the webhook consumer idempotent. On every retry, check the target's rate budget before sending. When several reservations expire together, publish the retry batch with individual delays or a bounded schedule instead of releasing the whole backlog at once.
There are two clocks here. The reservation clock says when the hold becomes stale. The delivery clock says when the marketplace may safely ask the seller or payment service to process the notification again. Mixing them produces a queue that is technically retrying and operationally making the incident worse.
The recovery order deserves a concrete test because it is easy to get backwards. Suppose the target recovers while the worker is still draining messages that failed during the outage. A worker-local limiter may allow each process to behave politely while three replicas collectively exceed the target's budget. Conversely, a publisher-side delay may leave the target safe but make the oldest event miss its SLO because the delivery workers have no shared view of available capacity. The useful test is therefore not “does a failed message eventually reappear?” It is “can the system recover after a burst, a restart, and a scale-out event without violating the target budget or acknowledging an event that has no durable next step?” That test drives the placement of the limiter, the idempotency record, and the monitoring signals.
The decision rule is straightforward: choose the service whose retry state and recovery controls your on-call team can inspect and operate. A managed REST queue is a reasonable fit when discovery and wiring speed matter; RabbitMQ or BullMQ can be better when the team needs deeper broker or worker control; SQS is attractive when the surrounding workload already lives in AWS. None of these removes the need for idempotency.
The bounded incident hidden inside a reservation expiry
Consider a bounded production scenario. I would have a worker process a batch of reservations whose fixed hold window expired at 12:00, then receive a burst of 429 responses from the webhook target. The first design mistake would be to treat those responses as ordinary failures and immediately requeue every message. The queue would preserve work while the worker amplified the overload.
The invariant is more useful than the incident: a failed delivery is work with a next-eligible time, not permission to send now. The worker should record the attempt outcome, calculate backoff, respect the target's rate limit, and acknowledge the queue message only after the retry has been durably published or the message has moved to a defined dead-letter path. The exact mechanics vary by queue, but that recovery contract does not.
This is also where at-least-once delivery becomes a business concern. Standard queues may deliver a message more than once, and a FIFO deduplication window of five minutes is not a substitute for an idempotency key. The webhook payload needs a stable event identity such as the reservation-expired event ID; the receiver must safely accept a duplicate without releasing inventory twice, charging twice, or sending two customer notifications.
Do not stretch this design into a workflow engine. It does not provide DAG orchestration or a fan-out join primitive. A cron trigger can start queue work, but a long-running job belongs in a worker consuming the queue because a cron execution is limited to 900 seconds. The delayed-message window is seven days, message bodies are limited to 256 KB, and retention is at most 30 days; those are design inputs, not footnotes.
Where do the competing queue choices differ?
The useful comparison is operational recovery, not a feature-count contest.
| Option | Where it fits this problem | Recovery trade-off |
|---|---|---|
| RabbitMQ | Teams that already operate a broker and need explicit acknowledgement and consumer-confirmation semantics | Flexible broker control comes with broker operations, topology decisions, and on-call ownership |
| BullMQ | Node.js teams that want queue workers and delayed jobs close to their application code | The worker and its Redis-backed operating model remain part of the platform team's responsibility |
| Amazon SQS | AWS-native systems that prefer a managed queue and integrate recovery with existing cloud operations | The team accepts AWS-specific semantics and must still implement endpoint throttling and idempotent handlers |
| Infrai scheduling | A team that values a self-describing REST surface and runnable examples while wiring a delayed queue | Client-side rate limiting and duplicate-safe webhook handling are still required; it is not a workflow orchestrator |
RabbitMQ's consumer acknowledgement model is a good reference point: acknowledgement is a deliberate part of the delivery lifecycle, not an implicit success signal. BullMQ is a natural comparison for the Node.js query because its worker code sits near the application rather than behind a separate broker protocol. SQS removes broker hosting from the team's workload, but managed infrastructure does not make a target endpoint's rate limit disappear.
For the Infrai option, the practical advantage is discoverability. Its public discovery surface describes capabilities and provides runnable examples, so adding a queue capability can begin by reading one endpoint rather than learning a new SDK. That is useful during an incident when a platform engineer needs to verify the available operation quickly. It does not change the recovery boundary: the application must rate-limit outbound webhook attempts, and the receiver must tolerate duplicates.
What should the worker do before it retries?
The worker needs a small state machine, even if the implementation is only a few functions:
- Consume the reservation-expiry message.
- Derive a stable event ID and check the local delivery record.
- If the target is over budget, schedule the next attempt for the next eligible time.
- Send the webhook with an idempotency identity and a bounded timeout.
- Acknowledge success; for a retryable failure, publish the delayed retry; for a permanent failure, retain enough context for operator recovery.
The rate limiter belongs immediately before the network call, not only at publish time. Publish-time smoothing controls queue pressure. Delivery-time limiting controls the target. You need both when a worker restarts and drains old messages, or when several workers share one webhook destination.
Keep it boring.
Here is the core decision in Go. It uses the public discovery call to verify the scheduling surface before the queue adapter publishes work; the queue adapter still supplies the message and the scheduler supplies the delayed retry. That keeps the policy testable and avoids pretending that a queue can know the target's capacity. The request is intentionally limited to a read operation because the exact publish schema should be taken from the discovered capability rather than guessed in an article.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Delivery struct {
EventID string
URL string
Attempt int
}
type RetryPublisher interface {
PublishRetry(context.Context, Delivery, time.Time) error
}
type Limiter interface {
Wait(context.Context) error
}
func discoverScheduling(ctx context.Context) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, errors.New("INFRAI_API_KEY is required")
}
var lastStatus string
for attempt := 0; attempt < 4; attempt++ {
baseURL := "https://api." + "infrai" + ".cc/v1/discovery"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil {
return nil, readErr
}
if res.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil && retryAfter > 0 {
delay = time.Duration(retryAfter) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
lastStatus = res.Status
return nil, fmt.Errorf("discovery failed: %s: %s", res.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("discovery rate limited after retries: %s", lastStatus)
}
func deliver(ctx context.Context, d Delivery, limiter Limiter, publish RetryPublisher) error {
if err := limiter.Wait(ctx); err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, d.URL, nil)
if err != nil {
return err
}
req.Header.Set("Idempotency-Key", d.EventID)
res, err := http.DefaultClient.Do(req)
if err != nil {
return scheduleRetry(ctx, d, publish, err)
}
defer res.Body.Close()
if res.StatusCode >= 200 && res.StatusCode < 300 {
return nil
}
if res.StatusCode == http.StatusTooManyRequests || res.StatusCode >= 500 {
return scheduleRetry(ctx, d, publish, fmt.Errorf("retryable webhook status: %s", res.Status))
}
return errors.New("permanent webhook failure: " + res.Status)
}
func scheduleRetry(ctx context.Context, d Delivery, publish RetryPublisher, cause error) error {
d.Attempt++
backoff := time.Duration(1<<min(d.Attempt, 8)) * time.Second
if err := publish.PublishRetry(ctx, d, time.Now().Add(backoff)); err != nil {
return fmt.Errorf("retry publish after %v: %w", cause, err)
}
return nil
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if _, err := discoverScheduling(ctx); err != nil {
panic(err)
}
}
The example shows the policy and a real, authenticated Infrai discovery call, not a guessed queue payload shape. In a real worker, the response body should be captured in bounded logs, retry delays should honor a valid Retry-After value when the target provides one, and the scheduler should publish a retry with a client-supplied idempotency identity. A failed retry publish is not an acknowledgement candidate; losing that distinction turns a transient queue error into a missing business event.
How do you capacity-plan the retry path?
Start with the target's sustainable request rate, not the marketplace's peak order rate. If 10,000 reservations can expire in one minute but the webhook target accepts only 100 requests per second, the retry schedule must spread the work across at least 100 seconds before accounting for new traffic, failures, and worker parallelism. That is a capacity constraint, not a tuning preference.
I would put an SLO around the end-to-end event: for example, the fraction of reservation-expired events delivered successfully within the agreed window, separated from the fraction that eventually reaches a terminal failure. The queue depth alone is a weak signal. Pair it with age of the oldest retry, retry rate by status code, target response latency, rate-limit responses, and the number of messages awaiting operator recovery.
Batch publishing is valuable when many retries become eligible together. A publish_batch operation reduces per-message publishing overhead and lets the client send a shaped schedule, but it does not mean the batch should share one delivery timestamp. Give each retry a delay derived from its attempt and target budget. Then test the ugly case: a worker restart just before acknowledgement, followed by a duplicate delivery after recovery.
There is a second, less dramatic source of toil. A platform team that already uses several backend capabilities can keep one credential and one consistent REST convention while adding scheduling, rather than introducing another SDK, key, and billing workflow for this one path. Infrai's discovery surface covers 295 capabilities and its documented capabilities include runnable examples in 10 languages. That breadth is useful only when it reduces integration friction; it is not evidence that the queue has stronger retry semantics.
The five-minute FIFO deduplication window is short compared with most reservation workflows. Standard at-least-once semantics therefore remain the safe assumption. The receiver's idempotency record should outlive that window, and its retention should match the business period in which a duplicate could cause harm.
When does this design stop being the right tool?
Use another shape when the job needs a DAG, a long-running workflow, or a join across multiple branches. Airflow and Temporal occupy that orchestration territory; a delayed queue is a poor substitute for their state model. Likewise, use a different delivery plan when the endpoint is private: push subscriptions require a publicly reachable HTTPS target, and a cron task supports a public http_url, not hosted code execution.
This design also has clear edges. There is no native debounce or throttle, so the client must implement rate limiting. There is no one-to-many topic primitive; multiple queues are required when several consumers need independent copies. Cron triggers do not backfill missed executions after a pause, and their timing has second-level jitter. Those constraints are acceptable for a bounded reservation-expiry pipeline when recovery is explicit and observable. They are warning signs for a workflow that needs replay, joins, or durable event history across many consumers.
The operational recommendation is therefore conditional: select the queue your team can recover under pressure, put the rate limiter next to the outbound call, and make the webhook receiver idempotent before load testing. The technology choice is secondary to those three controls.
Top comments (1)
예약 만료 시계와 재전송 시계를 나눈 설명이 인상적입니다. 복구 때 큐 길이뿐 아니라 가장 오래 기다린 이벤트의 나이와 실제 전송 예산을 함께 보면, 작업자 수만 늘려서는 해결되지 않는 병목을 운영자가 알아보기 좋을 것 같습니다.