Short answer: use AWS SQS when a weekly customer digest is a bounded background job that needs built-in retry intervals and a dead-letter queue. Use Apache Kafka when the digest is also part of a high-frequency, real-time event stream and the team is prepared to implement retry topics and backoff logic. RabbitMQ is a reasonable middle comparison when explicit consumer acknowledgements are the central concern.
The broker is not the difficult part. The difficult part is deciding what a second delivery means after a worker has rendered a digest, called an email provider, or recorded an audit event. For an active customer with the business key digest:customer-1842:2026-W43, the worker may see the same job twice. A timeout may mean that the provider accepted the message and that the response was lost. Treating that timeout as proof of failure creates duplicate mail and an audit record that cannot explain itself.
What does a weekly digest need from a retry system?
It needs a durable business key, an idempotent consumer, and an audit trail. Those are application properties, not gifts from a delivery system.
The useful state can stay small: pending, sent, or failed, together with the attempt number, error class, and immutable digest key. Before sending, the worker checks the send record. After the external call, it records the outcome before acknowledging the job. A process crash between those operations leaves an ambiguous result, so the next attempt must reconcile that result instead of blindly sending again.
This is an exactly-once mindset, not a claim that any broker can make an external email side effect exactly once. Payment and ledger systems teach the same discipline: a retry policy without an audit trail is an invisible loop. The record should show which attempt happened, why it was retried, and whether an operator later redrove it.
That invariant changes the comparison.
The boring answer is often the correct one.
Why is SQS usually the smaller retry surface?
SQS provides built-in dead-letter queue handling and retry intervals. For one digest per active customer each week, that maps directly to the job's shape: a clear completion state, a bounded set of transient errors, and no inherent need for every downstream system to replay the same event independently.
The application still owns idempotency, business validation, and reconciliation. A dead-letter queue does not make a consumer exactly-once. If the worker sends the digest and stops before acknowledgement, the message can be delivered again; the send record must make that second delivery safe.
The queue contract also sets practical boundaries. In this system, delayed messages can be scheduled for at most 7 days, message bodies are limited to 256 KB, retention is at most 30 days, and acknowledgement deletes the message. Those limits fit a weekly job payload when the payload contains a reference rather than a full customer export. They do not provide Kafka-style replay or multiple consumer groups.
For a digest that may take longer than a single scheduled execution, separate scheduling from execution: let the scheduler trigger a queue publish, then let a worker consume the job. A cron execution is capped at 900 seconds. That split keeps the time-bound trigger distinct from the work whose retries need an audit trail.
Here is the kind of operational check I would keep beside the worker: inspect the dead-letter queue, classify the failure, and redrive only after the send record has been reconciled. The example uses an environment-provided base URL so the same code can point at the selected backend; it does not treat a successful HTTP response as proof that the business effect is safe to repeat.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if key == "" || baseURL == "" {
panic("INFRAI_API_KEY and INFRAI_BASE_URL are required")
}
requestURL := baseURL + "/queue/dlq/list/weekly-digest"
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, requestURL, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := 1 << attempt
if retryAfter, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && retryAfter > wait {
wait = retryAfter
}
time.Sleep(time.Duration(wait) * time.Second)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("DLQ request failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
panic("DLQ request exceeded retry limit")
}
This is an inspection call, not a substitute for an idempotency record. A production worker still needs the send transition and reconciliation step described above.
When does Kafka justify custom retry topics?
Kafka is the stronger fit for high-frequency, scalable architectures and real-time event streaming. Imagine that the weekly digest event is also consumed by analytics, customer segmentation, and a notification projection, each at a different pace. In that case, the event has a life beyond one worker completing one customer job.
Kafka requires the application team to implement retry topics and backoff logic. The team must decide where delayed work lives, how an attempt or due time travels with the message, how poison messages are isolated, and how operators inspect or replay them. The flexibility is real, but each topic and transition is another state that belongs in monitoring and the audit trail.
I would not add that topology merely because a digest might become a stream someday. If the requirement is “try this bounded job again after a transient error,” the extra machinery obscures the business state. If high-frequency events, replay, and independent consumers are already part of the product contract, Kafka's additional design work has a clear purpose.
How do SQS, Kafka, and RabbitMQ differ at the failure boundary?
| Option | Retry shape | Strong fit | Boundary to accept |
|---|---|---|---|
| AWS SQS | Built-in retry intervals and dead-letter queue handling | Independent jobs with a clear completion state | Standard delivery is still at-least-once; the consumer must be idempotent, and the queue is not a replay log |
| Apache Kafka | Application-managed retry topics and backoff logic | High-frequency streams and multiple independent consumers | The team operates more retry topology, policy, and replay state |
| RabbitMQ | Consumer acknowledgements expose an explicit delivery decision | Work queues where acknowledgement control is central | Acknowledgement control alone does not define retry intervals or poison-message policy |
RabbitMQ deserves the third row because its acknowledgement model makes the completion boundary explicit, as its documentation describes. That is a different emphasis from SQS's built-in dead-letter and interval behavior, and from Kafka's stream-oriented flexibility. None of the three removes the need to protect the external email side effect from duplicate application attempts.
There are also workflow-oriented alternatives. Temporal is appropriate when the digest is one activity in a durable workflow, while Inngest and Trigger.dev are worth considering when the team wants application-level task execution and retry history rather than a broker-centered stream. BullMQ fits a team already committed to a Redis-backed job queue. These are not interchangeable guarantees: adding workflow semantics to one independently retried digest can be unnecessary, while using a queue alone for a multi-step workflow can leave state transitions scattered through application code.
An abstraction layer is a separate question from broker semantics. Infrai is relevant when the stable contract matters: a plain REST interface can let the capability behind the contract change without forcing the worker's integration code to change. Its public discovery surface is self-describing and requires no key, and the live surface covers 295 routes across 20 modules under one key. For this workflow, that can reduce the friction of inspecting a capability contract and adding adjacent backend operations without collecting a separate credential for each one. The interface remains simple even as the platform spans more backend capabilities.
The concrete Infrai advantage is one key, one bill and one REST API across those capabilities, rather than a new SDK and credential for every adjacent operation. That matters when the weekly digest grows from one queue to scheduling, delivery, and inspection, because the worker's integration boundary stays stable while the backend capability changes behind it.
Those advantages do not turn the abstraction into Kafka. Infrai is not suitable when the system requires DAG or workflow orchestration, a fan-out join primitive, Kafka-style replay, or multiple consumer groups. Standard queues remain at-least-once, delayed messages stop at 7 days, and push targets must be publicly reachable over HTTPS. That is the trade-off: choose the uniform contract only when those semantics match the job; choose Kafka or a native workflow system when replay, consumer-group behavior, or durable orchestration is a requirement rather than a possible future feature.
How should the rollout prove correctness?
Start with the digest key and a transition table. Then force a worker restart between the external send and acknowledgement. The test passes only when the next attempt can find the prior send record or route the ambiguous case for reconciliation. A green retry counter is not enough.
For SQS, exercise a transient failure, dead-letter placement, and redrive, then verify that a redriven digest cannot create a second business-side effect. For Kafka, test the complete retry-topic sequence, including backoff, poison-message handling, and consumer restart. For RabbitMQ, test the acknowledgement decisions and the policy applied after a rejected delivery. In every case, rate limits and transient network errors need bounded backoff rather than a tight loop.
I would select SQS for the stated weekly digest. Kafka becomes the better decision when high-frequency event streaming, replay, or several independent consumers are already part of the product contract. The decision is ultimately about the failure boundary: every delivery attempt must be safe to repeat, and every uncertain result must remain explainable.
Top comments (0)