Short answer: Use Redis when you need application-controlled retry timing and can own recovery; use AWS SQS when managed visibility and redelivery semantics reduce operational work. In both cases, failed game webhooks need idempotency keys because a timeout can follow an accepted request.
A failed game webhook should be retried, but a retry queue cannot make an outbound request exactly once. The useful design is to make delivery at least once, make the receiver idempotent, and choose a retry window that fits the latency players can tolerate. Redis can provide a fast, operator-controlled queue. Amazon SQS can provide durable queue semantics with managed visibility timeouts. Neither removes the need for a delivery key, a dead-letter path, and an alert that distinguishes delayed work from lost work.
The decision is therefore not "which queue is better?" It is: how much delay can this event accept, where should retry state live, and what happens when the worker dies after the receiver has accepted the request?
The page that fires after the damage
The page arrives as a webhook delivery alert: the retry backlog for a game event is above its threshold, and the oldest item is 11 minutes old. A worker has received a match-result notification, called a partner endpoint, and timed out waiting for a response. It retries. The partner later records the first request and then records the retry as a second event.
That sequence is normal for at-least-once delivery. A timeout is not proof that the remote side did nothing. Treating it as proof is how a harmless network delay becomes a duplicate reward, duplicate notification, or repeated account action.
The runbook starts with the delivery key, not with a queue command. The worker logs the event ID, attempt number, scheduled retry time, response class, and elapsed request time. The receiver stores the event ID before applying a side effect, or uses an equivalent idempotency record with a defined retention period. A repeated event then becomes an acknowledged no-op.
The earlier signal is usually less dramatic: queue age rises while request latency rises, or the retry count distribution shifts toward its maximum. I want both signals. A count of pending messages says how much work exists; age says how long a player or partner has been waiting. Alerting on count alone pages during a normal burst. Alerting on age alone can miss a rapidly growing queue whose oldest item has not crossed the limit yet.
False positives have a cost. If a low threshold pages every time a tournament starts, the on-call learns to ignore it. If the threshold is high enough to avoid that noise but no longer matches the product's delivery promise, the first useful signal is a partner complaint. The threshold needs a stated latency budget and a replay test, not a convenient round number.
Should a failed game webhook use a Redis queue or AWS SQS?
A retry record needs more than a payload. At minimum, it needs a stable event ID, destination, attempt number, next-attempt time, and an expiration or maximum-attempt policy. Keep the original payload or a durable reference to it; rebuilding a request from mutable game state during an incident is a quiet source of inconsistency.
Redis is a reasonable fit when the system needs low scheduling latency and the team is prepared to define durability, claiming, delayed visibility, and recovery behavior around it. A sorted set can represent due work, while a separate record holds the body and delivery metadata. That arrangement is flexible, but the pieces must be recovered together. A worker that removes a due ID before persisting its claim can lose work on a crash; a worker that leaves it visible can process it twice. The queue is not the idempotency boundary.
SQS offers a different boundary: a message can become invisible for a visibility timeout while a worker attempts delivery, and it can become visible again if the worker does not finish. This reduces queue machinery in the application, but it does not turn an HTTP call into a transaction. The timeout must exceed the normal processing time with room for jitter, and the worker must renew it for unusually slow work.
RabbitMQ's dead-letter exchange model makes another trade-off visible: rejected or expired messages can be routed to a different exchange for inspection or later handling. That is useful for separating poison messages from transient failures, while the exact retry schedule still belongs in the application or broker topology. These are different operational shapes, not interchangeable labels.
| Concern | Redis-backed retry state | SQS-style visibility | RabbitMQ dead-letter path |
|---|---|---|---|
| Fast due-work selection | Application-defined | Queue semantics plus delay controls | Queue and exchange topology |
| Crash after remote acceptance | Duplicate delivery is possible | Duplicate delivery is possible | Duplicate delivery is possible |
| Poison-message isolation | Build a quarantine path | Use a dead-letter queue policy | Route through a dead-letter exchange |
| Main design burden | Recovery and atomic claiming | Timeout and redelivery tuning | Topology and retry policy |
The table is not a scorecard. It tells me where the runbook will be longest.
How do you test the dangerous interval?
The dangerous interval is between the remote server accepting a request and the worker recording success. Kill the worker there. Delay the response. Return a 500 once, then a 200. Drop the connection after the request body is sent. Each case should produce the same business result after replay.
A minimal Go worker should make the delivery key explicit and classify failures before scheduling another attempt:
\`go
package retry
import (
"bytes"
"context"
"fmt"
"net/http"
"strconv"
"time"
)
type Delivery struct {
ID string
Payload []byte
Attempt int
}
func send(ctx context.Context, client *http.Client, endpoint string, d Delivery) (time.Duration, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(d.Payload))
if err != nil {
return 0, err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", d.ID)
req.Header.Set("X-Delivery-Attempt", strconv.Itoa(d.Attempt))
started := time.Now()
resp, err := client.Do(req)
if err != nil {
return time.Since(started), err
}
defer resp.Body.Close()
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return time.Since(started), nil
}
if resp.StatusCode >= 400 && resp.StatusCode < 500 && resp.StatusCode != http.StatusTooManyRequests {
return time.Since(started), fmt.Errorf("non-retryable status: %s", resp.Status)
}
return time.Since(started), fmt.Errorf("retryable status: %s", resp.Status)
}
`\
The snippet is deliberately incomplete as queue code. It also needs a client timeout, a bounded body, a durable acknowledgement, and a retry schedule with jitter. The important contract is visible: every attempt carries the same delivery key, while the attempt number remains observable. The receiver must enforce the key; a header by itself does nothing.
For a gaming workload, latency and cost pull in opposite directions. Short delays can keep a player-facing status current, but they create more calls during a partner outage. Long delays reduce pressure on the remote endpoint, but the queue retains work longer and the product may show stale state. Separate event classes instead of forcing one schedule onto every webhook. A match result may deserve a shorter first retry than an archival analytics notification.
Turning the alert into a decision rule
Start with a written delivery objective: the percentage of accepted events that should reach a responsive partner within a selected window. The exact target belongs to the product contract, not to the queue vendor. Measure it from event creation through acknowledged delivery, and record the age of exhausted or quarantined work separately.
Then choose based on operational ownership. If the team needs custom due-time selection, local testing, and direct control over retry records, Redis may fit, provided persistence and recovery are tested. If the team values a managed visibility model and accepts tuning timeouts and redelivery policies, SQS may fit. If routing failures through exchanges and a dead-letter path matches existing operations, RabbitMQ may fit. None is a duplicate-prevention mechanism.
During an incident, operators need actions that are reversible and scoped: pause one destination, extend a retry window, quarantine one event class, or replay a bounded time range. Avoid a global replay button. It turns a partner outage into a traffic burst and makes the original failure harder to reconstruct.
The postmortem question is not only why the request failed. Ask why the system could not tell the difference between unknown outcome, confirmed rejection, and accepted-but-unobserved outcome. That distinction determines whether retrying is safe, and it is the part a queue cannot infer from an HTTP timeout.
Further reading
- Redis sorted sets: https://redis.io/docs/latest/develop/data-types/sorted-sets/
- Amazon SQS visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- RabbitMQ dead letter exchanges: https://www.rabbitmq.com/docs/dlx
- HTTP Semantics, status codes and request processing: https://httpwg.org/specs/rfc9110.html
Top comments (0)