Short answer: for a queue handling failed jobs, choose retry scheduling that respects the downstream rate limit, then make the database-held operation ID the authority for the business effect. A queue can postpone a webhook and limit pressure on a downstream service; it cannot make an uncertain acknowledgment become exactly once.
That distinction matters in a marketplace. A seller may receive an order-created webhook, return a timeout after accepting it, and receive the same delivery again when the worker retries. The correct outcome is one durable business effect, an auditable record of every attempt, and a response that can be reconstructed after an operator redrives the work. The transport is allowed to deliver at least once. The effect is not allowed to be applied twice.
A ledger checkpoint for every outbound webhook
Start with ownership. The queue owns delivery and visibility. The scheduler owns when a retry becomes eligible. A limiter owns admission to the constrained endpoint. The database owns the operation status, the idempotency key, and the audit trail. Keeping those responsibilities separate makes an incident easier to reason about, because an increase in attempts does not masquerade as an increase in accepted marketplace events.
The publisher should create one operation ID at the authorization boundary and put that ID in every later attempt. Do not mint a new key inside the retry loop. I treat HTTP 429 as a scheduling decision, not as a reason to discard the job: respect the provider's retry signal when present, apply bounded exponential backoff with jitter, and keep the next eligible time durable so a process restart cannot reset the pressure.
Short key. Long audit trail.
The worker should load the authoritative operation, insert an attempt record, and commit the business mutation and its resulting status in one transaction where the storage model permits it. A unique constraint on the operation ID is useful evidence, but it is not the whole protocol: the handler must also distinguish a committed success from an unknown outcome after a network timeout. On a later delivery, it returns the recorded result or routes the operation to reconciliation rather than performing a second effect. In the difficult case, the first worker receives a timeout after the remote marketplace has accepted the webhook, the queue makes the job visible again, a second worker reads the same operation ID, and the database must be able to say either "already complete" or "unknown and awaiting reconciliation" without guessing. That state transition is longer than the retry policy because it carries the evidence an operator needs to decide safely.
The failure drill that exposes false success
Standard queue delivery is generally at least once, so duplicates are normal input. FIFO ordering can preserve an ordering relationship and can suppress duplicates for a bounded deduplication interval, but a later redrive or a retry outside that interval still needs application-level idempotency. A short transport window cannot represent a marketplace obligation that may remain auditable for years.
Delay limits matter too. If a queue can delay a message only for a bounded period, persist next_attempt_at in the job table and let a small scheduler enqueue due operations. Keep the queued payload small: an operation ID, event type, schema version, and lookup key are usually safer than copying mutable order details into every retry. The durable record is the place to retrieve the current state.
The dead-letter queue (DLQ) is a holding area for decisions, not a trash can. After retryable failures are exhausted, an operator or policy engine should be able to see the last response, attempt count, next action, and reconciliation status. Redrive only after checking the downstream condition and the authoritative operation record. Otherwise, a successful redrive metric may hide a second request that was correctly rejected by the business invariant.
This is also why queue retention and compliance retention are separate policies. A queue may retain work for a limited operational window, while an audit record may need a longer retention period under the marketplace's legal and financial obligations. The exact period belongs in the applicable retention policy and counsel's review; it should not be inferred from the queue setting.
Queue selection is an ownership decision
The useful comparison is not which product promises the most retries. It is which responsibility remains with the application team after the transport accepts a message.
| Option | Where it fits | Responsibility that remains | Poor fit when |
|---|---|---|---|
| SQS Standard | Managed, at-least-once work where duplicate delivery is an explicit consumer concern | Durable idempotency, concurrency control, and DLQ review | A documented ordering invariant is mandatory |
| SQS FIFO | Work with a genuine ordering relationship and bounded transport deduplication needs | Idempotency for delayed retries and operator redrives | FIFO is being used as a substitute for a business key |
| Cloud Tasks | HTTP-oriented delivery to a task target with scheduled attempts | Handler correctness, endpoint availability, and effect reconciliation | The target cannot use the required HTTP task model |
| Redis BullMQ | Teams that already choose Redis and want queue behavior close to application code | Redis persistence and operations, worker recovery, and business deduplication | Reducing stateful infrastructure ownership is the priority |
None of these rows changes the decision rule. Use the simplest transport that meets the delivery, ordering, delay, and operational recovery requirements, and spend the design effort on the state transition that makes a repeated webhook harmless. I don't count a queue's successful acknowledgment as proof that the business effect happened: it proves only that the transport accepted the consumer's response. If the workload needs a DAG, fan-in joins, or independent replayable consumer groups, evaluate a workflow engine or event log instead of stretching a retry queue into a different abstraction.
The catch is operational ownership. A managed service can reduce the queue processes a team runs, while a Redis-based design may offer closer control over worker behavior at the cost of operating the data store and its persistence policy. Stick with the option that matches the team's incident response capacity when the webhook endpoint is rate-limited for several hours, not the option that looks shortest in a diagram.
What should a queue do when failed jobs meet rate limiting?
The following consumer sketch makes the ordering explicit. It records the attempt, checks the durable operation, and acknowledges only after the transaction has made a final decision. The repository methods are intentionally generic; their contract, rather than a queue-specific API, is the part that protects the business effect.
package webhook
import (
"context"
"errors"
"time"
)
var ErrRetryable = errors.New("retryable delivery")
type Delivery struct {
OperationID string
Attempt int
}
type Operation struct {
OperationID string
Completed bool
Result string
}
type Repository interface {
WithTransaction(ctx context.Context, fn func(context.Context) error) error
GetOperation(ctx context.Context, operationID string) (Operation, error)
RecordAttempt(ctx context.Context, delivery Delivery, outcome string) error
ApplyEffect(ctx context.Context, operationID string) (string, error)
}
func Handle(ctx context.Context, repo Repository, delivery Delivery) error {
return repo.WithTransaction(ctx, func(tx context.Context) error {
op, err := repo.GetOperation(tx, delivery.OperationID)
if err != nil {
return ErrRetryable
}
if op.Completed {
return repo.RecordAttempt(tx, delivery, "duplicate_returned")
}
result, err := repo.ApplyEffect(tx, op.OperationID)
if err != nil {
if !isRetryable(err) {
return repo.RecordAttempt(tx, delivery, "permanent_failure")
}
return ErrRetryable
}
if err := repo.RecordAttempt(tx, delivery, "accepted:"+result); err != nil {
return ErrRetryable
}
return nil
})
}
func isRetryable(err error) bool {
return errors.Is(err, context.DeadlineExceeded) || errors.Is(err, ErrRetryable)
}
var _ = time.Second
The important property is transactional coordination between the operation state and the effect. In a real implementation, ApplyEffect must enforce the same operation key at the database boundary; an in-memory map or a queue-provided deduplication flag is not an audit-grade substitute. The example also leaves an uncomfortable case visible: if the downstream effect is external to the transaction and its outcome is unknown, the system needs a reconciliation state and a documented inquiry path. Silently retrying is not a correctness strategy.
Roll out recovery with reconciliation
Begin with one low-risk event type. Add a unique operation constraint, attempt records, a DLQ review view, and metrics that separate delivered attempts from accepted operations. Alert on age of the oldest retry, rate-limit responses, reconciliation backlog, and the difference between attempt count and business-effect count.
Then test the cases that make optimistic designs fail: a timeout after commit, two workers receiving the same operation, an approved redrive after the transport deduplication window, a process restart during backoff, and a downstream limit that remains active across a deployment. The expected result is one effect, several auditable attempts, and a stable operation result.
Roll out with a feature flag or a single queue partition where the platform permits it. Review the audit record before increasing concurrency. Your mileage may vary on the best scheduler boundary; it depends on the downstream contract, ordering invariant, and retention obligations. The decision should be reversible, observable, and explicit about who owns a job that cannot yet be proved safe to retry.
Top comments (0)