Short answer: retry failed webhook deliveries from a durable ledger, gate each attempt through a destination-level rate limiter, and reuse one idempotency key across every replay.
When a media worker pool is being drained, a failed webhook should become a durable delivery record, not an immediate retry loop. The practical choice is to combine a bounded retry schedule, a per-destination rate limiter, and an idempotency ledger that survives worker shutdown. That arrangement lets the pool stop accepting work while already-recorded attempts finish predictably.
The important distinction is between redelivery and replay. A queue can make a job visible again; it cannot prove that the remote system did not accept the previous HTTP request. I design the boundary as at-least-once transport with an exactly-once mindset: every attempt may happen again, while the business effect is committed once by an idempotent consumer. Auditability is the mechanism that makes that claim testable.
The queue is not the ledger.
What changes when the pool must drain?
A normal retry policy can hide an operational contradiction. If workers are asked to drain, but each failed callback immediately creates another runnable callback, the queue never reaches zero. A media pipeline makes this visible during a large ingest: thumbnail completion events, caption updates, and publication notifications can all target the same partner endpoint, while that endpoint advertises a small request budget.
I use three states for a delivery record: ready, in_flight, and settled. The record contains a stable event ID, destination ID, attempt count, next eligible time, and the last response class. A worker claims one record with a lease, sends at most one request, and writes the outcome before the lease expires. The lease is a recovery aid, not proof of exactly-once execution.
A drain begins by stopping new claims, then allowing already-claimed records to reach a short deadline. Records whose lease expires return to ready with their original event ID. This is more controllable than terminating processes and trusting a broker-specific redelivery detail, because the ledger preserves the reason for each transition and gives reconciliation something concrete to inspect.
Consider a batch of 240 caption-complete events for one destination when the pool enters drain mode. If 20 workers each reclaim five expired records, the destination can see a burst even though the queue itself is behaving correctly. The drain controller should therefore mark the generation as closed to new claims, let the destination limiter admit only its available tokens, and leave records beyond the deadline in ready with their next eligible timestamp intact. A worker that wakes after the deadline must release its lease without sending; otherwise the operational meaning of drain depends on process timing. That small rule also gives the next generation a clean starting point: it inherits durable work, not partially inferred work from a previous process lifetime.
Drain is a state transition.
How do I retry failed webhook delivery from a message queue?
They answer different questions. Retry asks when another attempt is eligible. Throttling asks whether the destination may receive it now. Idempotency asks what happens if the previous attempt actually succeeded. Collapsing these into one counter produces misleading dashboards and unsafe recovery.
For a transient response such as a timeout or a 5xx, I use exponential backoff with jitter and a hard attempt ceiling. A Retry-After value can move the next eligible time later, but it should not bypass the destination limiter. For a permanent 4xx, malformed payload, or an explicitly rejected signature, the record settles as failed and enters a review path rather than consuming the same retry budget forever.
The limiter belongs to the destination, not the worker. A pool of 20 workers must not accidentally turn a destination limit of 5 requests per second into 20 simultaneous requests. A token bucket or equivalent admission counter can enforce the rate, while a separate concurrency bound protects the connection pool. Rate and concurrency are related; they are not interchangeable.
The idempotency key should be derived from the immutable media event, for example media-asset-1842:published:v3, rather than from an attempt UUID. The receiver should persist that key with the applied result and return the same semantic result for a duplicate. If the receiver cannot offer that contract, the sender must treat a timeout after request transmission as an ambiguous outcome and reconcile by querying a read model or a partner-provided event status. Blind replay is not a correctness strategy.
I initially treated the retry count as the main safety control. The harder case is a timeout after the receiver commits but before the sender sees the response; the count says nothing useful there, while the event key still gives reconciliation a stable handle.
Here is the decision core in Go. It deliberately keeps transport, admission, and persistence separate, so a test can force a timeout between the request and the ledger write without pretending that a network call is transactional.
type Delivery struct {
EventID string
DestinationID string
Attempt int
NextEligible time.Time
State string
}
func nextAttempt(d Delivery, now time.Time, retryAfter time.Duration) Delivery {
const maxBackoff = 30 * time.Minute
backoff := time.Second * time.Duration(1<<min(d.Attempt, 10))
if backoff > maxBackoff {
backoff = maxBackoff
}
if retryAfter > backoff {
backoff = retryAfter
}
d.Attempt++
d.NextEligible = now.Add(backoff)
d.State = "ready"
return d
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
The production version needs jitter, durable compare-and-swap on the lease, and a transaction around the state transition. Those details matter because two workers can observe the same eligible record. The code is a policy sketch, not a claim that a queue acknowledgement and a remote HTTP side effect form one atomic operation.
Which queue behavior is relevant, and which is not?
Queue products expose different building blocks, but the design question stays stable: where are ownership, delay, acknowledgement, and deduplication recorded? Acknowledgement-based brokers such as RabbitMQ let a consumer explicitly acknowledge or negatively acknowledge deliveries; that is useful for recovering an unprocessed message, but it does not know whether a remote webhook was accepted before the consumer failed. Its publisher confirms and consumer acknowledgements solve broker delivery boundaries, not external side-effect atomicity.
BullMQ places delayed jobs and retry attempts in a Redis-backed job model, which is convenient when the application already owns that storage boundary. Its retry configuration still needs a destination-level limiter and an idempotency record, because a delayed job is not a receipt from the remote endpoint.
Amazon SQS offers visibility timeouts and redrive policies, which provide useful lease and poison-message controls. Those controls likewise do not turn an HTTP callback into an exactly-once operation; the application must persist the event key and settle its own outcome. These are objective boundaries, not a ranking. Pick the primitive that fits the existing operational model, then keep the delivery ledger explicit instead of outsourcing the business guarantee to queue terminology.
This design is a poor fit for a tiny, non-critical callback where a lost notification is acceptable; a durable ledger and reconciliation loop add operational weight. It is also the wrong place to promise exactly-once effects when the receiving system cannot store an idempotency key. In those cases, accept at-least-once delivery explicitly and document the duplicate-handling trade-off.
What should the drain and recovery runbook verify?
First, freeze new claims and record the drain generation. Workers from an older generation must finish or release their leases; otherwise a late process can keep producing attempts after the operator believes the drain is complete. Second, continue destination admission for already-claimed records until the deadline, but stop scheduling retries whose next eligible time exceeds that deadline. Those records remain durable and can be resumed by the next generation.
I would measure four separate quantities: eligible records, leased records, settled records, and ambiguous outcomes. A single queue-depth graph hides the most dangerous class. An ambiguous outcome means the request may have reached the receiver, so the recovery job should reconcile the event key before sending again. Log the event ID, destination ID, attempt number, limiter decision, response class, and ledger transition; exclude payload secrets and authorization headers.
The tests should include a 429 with Retry-After, a timeout after bytes leave the process, two workers claiming one record, a lease expiring during drain, and a duplicate event arriving with a new transport message ID. I would also run a property test that repeatedly crashes between each pair of state writes. The invariant is modest and defensible: no settled success is silently forgotten, and no retry bypasses the destination's admission rule.
A compact rollout path
Start by writing delivery records beside the existing job submission path, with the immutable event ID as the uniqueness constraint. Shadow the limiter and compare its decisions with current traffic before enforcing it. Then enable bounded retries for one destination class, measure ambiguous outcomes, and add reconciliation before raising concurrency.
During a drain, prefer a visible backlog over an unbounded retry storm. The backlog is recoverable state; duplicate external effects are often not. A queue can schedule work, but the ledger, receiver contract, and audit trail define what delivery means.
Sources
- RabbitMQ, “Consumer Acknowledgements and Publisher Confirms”: https://www.rabbitmq.com/docs/confirms
- Amazon Web Services, “Amazon SQS visibility timeout”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/visibility-timeout.html
- Amazon Web Services, “Amazon SQS dead-letter queues”: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
- BullMQ, “Retrying failing jobs”: https://docs.bullmq.io/guide/retrying-failing-jobs
- BullMQ, “Rate limiting”: https://docs.bullmq.io/guide/rate-limiting
- RFC 9110, “HTTP Semantics”: https://www.rfc-editor.org/rfc/rfc9110
Top comments (0)