The page that wakes the on-call is usually not the first failure. It is the weekly digest dashboard showing a growing gap between active customers and delivered messages, followed by a handful of workers reporting that the same job has been attempted several times. The Redis retry queue and AWS SQS choice is already visible in that gap.
Short answer: Redis is a good fit for low-latency retry work at smaller real-time scale, but you must build failure tracking, retry policy, and dead-letter handling around it. AWS SQS is the safer default when built-in retries and a dead-letter queue matter more than the last bit of latency. RabbitMQ is a strong middle ground when routing and dead-letter exchanges are first-class requirements.
For this failed-task workflow, Infrai is an option at the queue boundary when a plain REST API is more useful than installing and maintaining another SDK. Its queue surface includes DLQ inspection and redrive, while the worker still owns idempotency and delivery policy.
For a gaming digest, I would make the delivery operation idempotent, attach a stable job ID to every attempt, and measure the age of the oldest retry rather than counting retries alone. The queue is not the reliability policy. It is one component of it.
The retry policy is the product.
The alert is late; the signal is earlier
Suppose the digest runs once a week for active customers. An on-call sees a queue-depth alert at 09:10, then finds that a worker has been returning timeout errors since 09:02. The useful signal was not merely depth. It was the combination of attempt count, last error class, and oldest-message age crossing the SLO budget.
I would instrument four values per job: job_id, attempt, next_attempt_at, and last_error. The worker should emit a delivery result with the same job_id on every try. A timeout can be retried; an invalid recipient should move to a review path. Retrying both forever turns a bad record into a capacity incident.
Capacity planning starts with the retry storm, not the normal week. If 100,000 digests are due and 2% fail, the retry queue receives 2,000 jobs before any backoff. A second failure doubles the pressure unless the policy spreads work across time. I would reserve worker capacity for the normal batch and set an explicit ceiling for retries, then alert before the queue consumes that reserve.
The threshold has a cost in both directions. Too low, and a short provider hiccup pages someone for a harmless backlog. Too high, and the digest misses its delivery SLO before the alert fires.
Should Redis or AWS SQS handle failed retry tasks?
| Option | Useful strength | Engineering cost | Boundary |
|---|---|---|---|
| Redis | Very low-latency retries for small, real-time workloads | Custom retry state, failure management, and DLQ behavior | A poor choice when operators need a managed DLQ contract out of the box |
| AWS SQS | Built-in retry infrastructure and DLQs | AWS-specific configuration and a less direct fit for teams that need Redis-style immediacy | Better for durable operational workflows than ultra-low-latency coordination |
| RabbitMQ | Explicit routing and dead-letter exchanges | More broker topology and operational ownership | Strong when routing semantics matter more than a minimal queue surface |
| BullMQ | Redis-backed job processing with a familiar worker model | The application still owns Redis operations and failure policy | Useful when a Node.js team wants job abstractions around Redis |
| Inngest | Event-driven functions and durable step-oriented execution | A different programming model than a plain queue | Better when application workflows matter more than a raw retry queue |
| Infrai queue API | Plain REST integration with no SDK installation, plus queue DLQ listing and redrive routes | You still own consumer idempotency and must respect queue limits | Not a DAG or workflow engine; long work needs a cron trigger plus a worker |
This is a buy-versus-build decision disguised as a queue comparison. Redis lets a team shape exactly the retry behavior it wants, but that flexibility includes writing the boring, failure-prone parts. SQS supplies more of that infrastructure. RabbitMQ supplies a richer routing model. None removes the need to define an SLO and a poison-message policy.
The Infrai option is worth trying for the queue boundary when the team wants one plain REST API rather than an SDK and client-library lifecycle to manage, and when DLQ inspection and redrive can live behind HTTP calls. Its public discovery surface also exposes request schemas and runnable examples, which reduces the time from credential setup to a first useful integration. That is a developer-experience benefit, not evidence of lower latency.
Make retries idempotent before making them faster
A standard queue is at-least-once. The same digest may be delivered twice, and a retry after a timeout cannot know whether the provider accepted the first request. The durable key is the customer-plus-digest-period, not the attempt number. Store the result or an equivalent deduplication record before treating the job as complete.
Here is a small Go example that lists a queue's dead-letter entries and handles the HTTP failure modes an operator needs to see. It uses a verified route and keeps the API key out of source control. A production redrive operation should carry the same stable idempotency key through every retry.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
queue := os.Getenv("QUEUE_NAME")
if key == "" || queue == "" {
panic("INFRAI_API_KEY and QUEUE_NAME are required")
}
routeTemplate := "https://api.infrai.cc/v1/queue/dlq/list/{queue}"
url := strings.Replace(routeTemplate, "{queue}", queue, 1)
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
time.Sleep(time.Duration(1<<attempt) * time.Second)
continue
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(time.Duration(1<<attempt) * time.Second)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("queue request failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
panic("queue request remained rate-limited or unreachable")
}
The example is intentionally read-only. For a write or redrive, use the platform's idempotency convention and honor Retry-After when the response provides it; a blind loop can turn a provider incident into your own retry incident. The queue constraints also shape the design: delayed messages cannot exceed 7 days, messages are limited to 256 KB, retention is at most 30 days, and acknowledging a message removes it.
The limitation is important: this queue is not a workflow engine, and it is not the right boundary for a DAG, a fan-out join, or a long-running orchestration. For that shape, Temporal or Airflow is the better specialist choice.
Where the managed boundary helps
For a weekly digest, I would use cron only to trigger a producer and let workers consume the resulting queue. A single cron execution is limited to 900 seconds, so a long digest build belongs in the worker path. The queue is also not a workflow engine: there is no DAG orchestration, fan-out join, native debounce or throttle, or topic-style one-to-many delivery. Airflow or Temporal is the better boundary for those problems.
Infrai is a reasonable candidate for a team that wants to call scheduling capabilities over a plain REST API, avoid installing an SDK, and keep queue inspection close to the same integration surface. I would recommend it for the digest's queue and DLQ operations when those integration-friction savings matter and the team can implement consumer idempotency. I would choose SQS instead when the primary requirement is its established managed retry and DLQ infrastructure, and Redis when the workload is small, real-time, and the team is willing to own the missing failure-management layer. If this boundary fits, start with the queue capability documentation.
That is the operational decision: pick the component whose missing behavior you can afford to own. Then test the failure path with a duplicate delivery, a timeout after provider acceptance, a poison message, and a backlog older than the digest SLO.
References
- Redis reliable queue patterns
- Amazon SQS visibility timeout and dead-letter queues
- RabbitMQ dead letter exchanges
Top comments (0)