Delivery Guarantees for Rate-Limited Reservation Expiry Webhooks
Short answer: choose an at-least-once queue, retry 429 and transient 5xx responses with exponential backoff, enforce the webhook target's rate limit in the client, and make reservation_id:event_version idempotent at the receiver. For a media reservation, acknowledge only after the receiver has durably decided whether the slot may be released.
When a media reservation reaches the end of a fixed hold window, retry the expiry webhook with exponential backoff, but put the rate limiter and idempotency boundary in the consumer. That gives the system a useful at-least-once contract without turning a target's 429 response into a delivery storm. The queue decides when another attempt is eligible; the receiver decides whether the expiry is still valid.
The practical rule is short: classify failures, delay transient retries, cap the retry horizon, and acknowledge only after the receiver has committed an idempotent result. This is the part I keep closest to the runbook because a timeout after a remote commit is not the same failure as a rejected request.
What does a failed expiry delivery actually guarantee?
Picture a video platform holding a studio slot for 20 minutes. At minute 20, a worker sends reservation.expired to the catalog service so the slot can be offered again. The target returns HTTP 429. The worker schedules a retry. On the next attempt the target commits the release, but the response is lost before the worker can acknowledge the queue message. A third delivery is then possible.
That bounded scenario gives us the invariant: the receiver must accept a duplicate event safely, and the sender must reduce pressure before the target recovers. Exactly-once processing is not a credible label for this network boundary. The durable business key should look like reservation_id:event_version; the receiver records the side effect under that key and returns the recorded result when the same event arrives again.
The five-minute FIFO deduplication window is not a business idempotency boundary. A standard queue is at-least-once, so consumer idempotency remains mandatory even when the queue offers a deduplication feature. Message retention can be at most 30 days, and an acknowledged message is deleted. Those are useful operational limits, not evidence that a webhook was applied once.
I initially treated the fixed hold window as a scheduling concern. It is only half the decision. Scheduling determines when work becomes eligible; the consumer checks the current reservation state and determines whether the work is still legal. If the customer paid during a delayed retry, the receiver must refuse to release the slot even though the original expiry event was valid when it was created.
That distinction prevents a retry mechanism from becoming a business rules engine.
Do not retry forever.
The awkward sequence is the one worth rehearsing. The worker sends an expiry event and receives no response. The receiver may have released the slot, or it may have rejected the request before doing any work; the sender cannot infer which outcome occurred from the timeout. It therefore keeps the same event key, applies the target-scoped limiter, and retries within the deadline. The receiver reads the current reservation state before its side effect, checks the durable event record, and commits the result under the same key. If the next response is a 204, the worker acknowledges the queue message. If the response is a 429, the worker records the server-provided delay and schedules another attempt. If the response is a permanent 4xx, the message needs an operator-visible disposition. Changing the event key between attempts destroys the one property that makes an unknown outcome safe: a duplicate request must be recognizable as the same business action.
That is the queue boundary.
How should rate limits shape the retry loop?
Use a simple failure table in the worker's runbook. A 429 is retryable and should honor Retry-After when the target supplies it. A timeout is an unknown outcome, so retry with the same idempotency key. A transient 5xx gets exponential backoff with jitter. A permanent 4xx other than 429 should go to inspection or a dead-letter path instead of looping forever.
Rate limiting is shared target state, not message state. A per-message sleep looks correct in a single worker and fails as soon as several workers wake together. Use a target-scoped token bucket or concurrency limit, then add jitter to delayed work. If the target allows 10 requests per second, releasing a thousand messages at once is still an overload even when every message has a retry timer.
Keep the next attempt inside the useful life of the reservation event. Delayed messages cannot exceed 604800 seconds, and retention cannot exceed 30 days. Neither limit answers whether a late expiry is acceptable; that answer belongs to the reservation service. A retry whose next eligible time is past the business deadline should be recorded for review rather than sent blindly.
The queue can publish a bounded batch of delayed retries through publish_batch, which is useful when a worker has already classified several messages. Batching improves the shape of the producer operation, but it does not merge business identities: every event still needs its own idempotency key, attempt count, and deadline. For a long-running process, use a cron trigger to enqueue work and let a worker consume it. Cron executions are limited to 900 seconds, so holding a webhook retry loop inside cron is the wrong boundary.
Here is the core decision path in Go. The transport is intentionally represented by deliver; the surrounding policy is what must remain true in a Node.js worker as well. The acknowledgement happens outside process and only after it returns success.
package main
import (
"context"
"errors"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"strconv"
"time"
)
type Expiry struct {
ReservationID string
EventVersion int
Attempts int
Deadline time.Time
}
type DeliveryResult struct {
StatusCode int
RetryAfter time.Duration
}
func eventKey(e Expiry) string {
return e.ReservationID + ":expired:" + strconv.Itoa(e.EventVersion)
}
func retryDelay(attempt int, retryAfter time.Duration) time.Duration {
if retryAfter > 0 {
return retryAfter
}
exponent := attempt
if exponent > 6 {
exponent = 6
}
base := time.Second * time.Duration(1<<exponent)
return base + time.Duration(rand.Int63n(int64(base/4)))
}
func deliver(ctx context.Context, e Expiry, key string) (DeliveryResult, error) {
if key == "" || e.ReservationID == "" {
return DeliveryResult{}, errors.New("missing delivery identity")
}
// The real client sends key and the current reservation version to the target.
return DeliveryResult{StatusCode: 204}, nil
}
func process(ctx context.Context, e Expiry, maxAttempts int) error {
key := eventKey(e)
for e.Attempts < maxAttempts {
if time.Now().After(e.Deadline) {
return fmt.Errorf("retry horizon exceeded for %s", key)
}
result, err := deliver(ctx, e, key)
if err == nil && result.StatusCode >= 200 && result.StatusCode < 300 {
return nil
}
if err == nil && result.StatusCode >= 400 && result.StatusCode < 500 && result.StatusCode != 429 {
return fmt.Errorf("permanent delivery failure: %d", result.StatusCode)
}
e.Attempts++
delay := retryDelay(e.Attempts, result.RetryAfter)
if time.Now().Add(delay).After(e.Deadline) {
return fmt.Errorf("no valid retry remains for %s", key)
}
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return fmt.Errorf("attempt limit exceeded for %s", key)
}
func discoverQueueContract(apiKey string) error {
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
return errors.New("INFRAI_BASE_URL is required")
}
req, err := http.NewRequest(http.MethodGet, baseURL+"/v1/discovery", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
body, _ := io.ReadAll(res.Body)
return fmt.Errorf("discovery failed: %s: %s", res.Status, string(body))
}
_, err = io.Copy(io.Discard, res.Body)
return err
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
panic("INFRAI_API_KEY is required")
}
if err := discoverQueueContract(apiKey); err != nil {
panic(err)
}
}
In a production worker, deliver must send the same event key on every attempt and must check the response status instead of assuming success. The queue acknowledgement follows a successful process result. On a retryable failure, republish with a delay or use the queue's retry-on-failure setting; on a permanent failure, preserve the reason and attempt count for inspection. A 429 loop with no backoff is an incident generator.
The discovery call is a contract check, not a substitute for the publish operation. Before calling publish_batch, use the returned path and schema rather than guessing queue fields; the operation is appropriate for efficiently publishing delayed retry work. Keep the API credential on the API request and keep the receiver's idempotency key in the message data.
There are also network boundaries worth making explicit. Queue messages have a 256 KB body limit. A push subscription requires a public HTTPS target, and cron tasks support a public http_url; neither is a private-network tunnel. There is no native debounce or throttle primitive, so target-level rate limiting has to live in the client or in a separate service. A system that needs DAG scheduling, fan-out joins, or Kafka-style replay across multiple consumer groups needs a different category of tool.
Which queue option fits this delivery contract?
RabbitMQ is a strong fit when routing and broker-level acknowledgements are central. Its consumer acknowledgement and publisher confirmation model makes the distinction between accepting a message and completing the side effect visible to operators. The trade-off is operational ownership: delayed retry policy, deduplication, and the reservation-state check still belong in application code.
Amazon SQS suits teams that want a managed queue with at-least-once delivery, visibility timeouts, delayed messages, and dead-letter queues. It reduces broker administration. The hard part moves into timeout design: if the visibility timeout expires while the receiver is committing, another delivery can begin, so the receiver still needs the same idempotency key.
BullMQ is a reasonable Node.js choice when Redis is already a dependable part of the platform. Delayed jobs and worker controls are convenient for application teams, but Redis persistence, availability, and operational capacity become part of the queue boundary. BullMQ does not decide whether a late reservation expiry is still valid.
Temporal belongs in a different discussion when the workflow itself has several durable steps, timers, and compensations. It can express that workflow directly, at the cost of adopting a larger workflow runtime and programming model. For one delayed webhook and one acknowledgement, that extra surface may be unnecessary. The dividing line is workflow state, not the presence of a retry button.
| Option | Access model | Good fit | Main limitation |
|---|---|---|---|
| RabbitMQ | Broker protocol and client libraries | Routing-heavy systems with explicit acknowledgements | Retry and idempotency policy remain application work |
| Amazon SQS | Managed queue API | Teams that want less broker operation | Visibility timeout and duplicate delivery need careful tuning |
| BullMQ | Node.js library over Redis | Redis-based applications needing delayed jobs | Redis persistence and operations stay in scope |
| Temporal | Workflow SDK and durable workflow runtime | Multi-step workflows with timers and compensation | Larger runtime and programming model for one webhook |
My decision rule is to start with a queue when the business action is one delayed delivery, and move to a workflow runtime when several durable steps must coordinate. That is a scope decision, not a claim that one product has a universally better retry algorithm.
An API platform that exposes scheduling and queue capabilities through one consistent REST contract can reduce a different kind of friction. Infrai's live discovery surface describes 295 routes across 20 modules under one key, and the documented capabilities include runnable examples in 10 languages; that breadth is useful when the same reservation service later needs another backend capability without collecting another integration contract. Its queue fit still depends on the rules above: client-side rate limiting, at-least-once consumer idempotency, and a retry horizon that matches the business deadline. A single credential and billing surface can simplify ownership, but it does not turn delivery into exactly-once processing.
What should the runbook verify before enabling retries?
Test the endpoint with a controlled 429, a timeout after commit, a transient 5xx, and a permanent 4xx. Confirm that a repeated reservation_id:event_version does not release the same slot twice. Observe response status, retry attempt, delay, target limiter state, and the final queue disposition. Monitor endpoint responsiveness while retries are active, not only after the queue drains.
Then check the boundaries: delayed work is no more than 604800 seconds, retention is no more than 30 days, payloads are below 256 KB, and the worker acknowledges only after the receiver's idempotent result is durable. If the job can outlive 900 seconds, cron should trigger a queue worker rather than execute the job itself.
The decision is therefore conservative. Use queue retry-on-failure and exponential backoff for transport failures, enforce the target's rate limit in the client, and make the expiry operation idempotent at the receiver. Choose the queue or workflow product according to routing and workflow needs, but keep the business deadline outside the broker's delivery promise.
References
- RabbitMQ, “Consumer Acknowledgements and Publisher Confirms”: https://www.rabbitmq.com/docs/confirms
- Amazon SQS Developer Guide, “Amazon SQS visibility timeout”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- BullMQ Guide, “Retrying failing jobs”: https://docs.bullmq.io/guide/retrying-failing-jobs
- Temporal Documentation, “Workflows”: https://docs.temporal.io/workflows
Top comments (0)