KasimirBerg5341

Posted on Oct 25

Google Cloud Pub/Sub Dead Letter Queues vs AWS SQS DLQ — Weekly Edtech Digest Retries

#python #messaging #deadletterqueues #edtech

Short answer: choose the dead-letter design that makes delivery evidence easiest to prove, not the one with the most attractive retry vocabulary. For a weekly digest sent to active students, both Google Cloud Pub/Sub and Amazon SQS can support at-least-once processing with a dead-letter destination. Neither destination turns an ambiguous handoff into exactly-once email delivery. The design still needs an idempotency key, a durable delivery record, and an operator-visible reason for every quarantine.

The important distinction is operational. Pub/Sub forwards an undeliverable message from a subscription to a dead-letter topic after an approximate number of delivery attempts, with a configurable range of 5 to 100. SQS moves a message after its receive count reaches maxReceiveCount. Those controls look similar in a diagram, but they measure different observations and create different proof obligations.

What guarantee does a weekly digest actually need?

Start with the business event, not the queue. A digest job represents a particular customer, reporting period, and content revision. I would derive one stable key from those values and write a delivery record before acknowledging the queue message. The mail provider is then an effect of that record, not the source of truth.

from dataclasses import dataclass


@dataclass(frozen=True)
class DigestJob:
    customer_id: str
    period: str
    revision: str

    @property
    def idempotency_key(self) -> str:
        return f"digest:{self.customer_id}:{self.period}:{self.revision}"


def should_send(job: DigestJob, delivery_store) -> bool:
    status = delivery_store.status(job.idempotency_key)
    return status not in {"sent", "sending"}

That record must distinguish queued, sending, sent, and failed, with timestamps and the last error class. A worker crash after the email API accepts the request but before the acknowledgement is the classic duplicate boundary. A DLQ can preserve the job for inspection; it cannot retract the first email.

Do not confuse quarantine with deletion.

For this workload, I would define the guarantee as: every eligible digest is either sent once as observed by the delivery ledger, or remains recoverable with enough evidence to retry safely. That is a narrower and more honest contract than “exactly once.”

Why do Pub/Sub and SQS DLQs behave differently?

The comparison belongs after the contract because the settings are not interchangeable. Pub/Sub uses a dead-letter topic attached to a subscription. The service forwards messages that exceed the configured delivery-attempt threshold, and the forwarding path depends on the required service-agent permissions. Google documents the delivery-attempt count as approximate, so it is a trigger for quarantine rather than a precise execution counter.

SQS uses a redrive policy on the source queue, with maxReceiveCount controlling when a message is moved to its dead-letter queue. The source queue and DLQ must satisfy the service's queue-type and regional constraints. A visibility-timeout expiration makes a message visible again, so a receive count records receive cycles, not necessarily completed worker attempts. Retention behavior also matters: for standard queues, the original enqueue timestamp is retained for expiration accounting after movement; FIFO queues have different timestamp behavior.

The storage-minded question is therefore: which timestamp and counter will an on-call engineer trust during a digest incident? If the answer is “the number in the console,” the design is incomplete. Persist the application attempt, the broker metadata you receive, and the transition that caused quarantine.

That is the limit.

Concern	Pub/Sub dead-letter topic	SQS dead-letter queue	Design consequence
Quarantine trigger	Approximate delivery-attempt threshold	`maxReceiveCount` on receive/redrive behavior	Treat both as safety valves, not business truth
Destination shape	Topic consumed through another subscription	Queue read directly by a consumer	Keep replay tooling behind an internal interface
Duplicate boundary	Forwarding and acknowledgement remain separate effects	Visibility expiry and deletion remain separate effects	Make the digest ledger idempotent
Evidence	Delivery metadata plus application logs	Receive metadata plus application logs	Store a correlation key outside the broker

Azure Service Bus uses a dead-letter subqueue and a delivery-count threshold, which is another useful comparison: a third implementation still does not remove the need for an application-level ledger. The names differ; the failure boundary does not.

Which failures should be retried, and which should be quarantined?

A retry policy should classify errors before it counts them. A temporary timeout from the content store can be retried with bounded exponential backoff. A malformed recipient address should go to quarantine without consuming a week of repeated attempts. A permission error needs an alert and a short retry window, because blindly replaying it only produces more identical failures.

The classifier should be deterministic and recorded with the job. Keep the broker callback small: fetch the job, apply the decision, and make the acknowledgement or visibility change explicit.

from enum import Enum


class Action(Enum):
    RETRY = "retry"
    QUARANTINE = "quarantine"
    MARK_SENT = "mark_sent"


def classify(error: Exception | None) -> Action:
    if error is None:
        return Action.MARK_SENT
    if isinstance(error, (TimeoutError, ConnectionError)):
        return Action.RETRY
    if isinstance(error, ValueError):
        return Action.QUARANTINE
    return Action.QUARANTINE

This is intentionally conservative. An unknown exception is not evidence that a retry is safe. It may be a programming defect, a schema mismatch, or a side effect that already succeeded. The DLQ should receive enough context to answer three questions without reconstructing the entire week: what digest was intended, what was attempted, and why did the worker stop?

Do not use the broker's dead-letter payload as your only archive. Message retention, destination retention, and replay permissions are operational settings, not a permanent audit strategy. The delivery ledger and immutable application logs should have an owner and a retention policy that matches the promises made to customers.

How should the comparison change the architecture?

I would hide the broker behind four operations: publish, receive, acknowledge, and quarantine. The adapter can translate a Pub/Sub topic or an SQS queue into those operations, while the digest service owns idempotency, error classification, and replay authorization. That boundary is more valuable than pretending the two services expose the same semantics.

A replay command should accept a digest key or a reviewed batch, create a new attempt record, and refuse to send a job already marked sent unless an operator explicitly supplies a new revision. Replaying an entire dead-letter destination by default is a poor fit for customer mail: one poison message can be joined by many valid messages, and a bulk action makes the blast radius hard to inspect.

The trade-off is deliberate: a reviewed replay is slower than a bulk redrive, but it gives the operator a chance to separate an expired recipient record from a transient outage. A bulk redrive is useful only when the failure class is known, the downstream side effect is idempotent, and the team has a bounded way to stop it. Without those conditions, the queue's convenience is purchased with an unmeasured customer-visible risk.

For observability, measure the states that explain customer impact: age of the oldest queued digest, number of jobs by failure class, quarantine rate, time from sending to sent, and duplicate-prevention decisions. Alerting only on queue depth misses a worker that is acknowledging messages before it writes the ledger.

The same test matrix should run against both adapters. Inject a timeout before the external send, after the external send, before acknowledgement, and after acknowledgement. Then verify that a replay cannot create a second send for the same idempotency key. Test malformed payloads separately. Also test permissions for the dead-letter destination, because a configured destination that the service cannot publish to is a configuration failure, not a recoverable customer job.

The most useful test is the uncomfortable one: let the worker lose its process after the downstream mail request returns success, then let the broker deliver the same job again. The expected result is not “the message was eventually acknowledged.” The expected result is a single delivery-ledger transition, one recorded external request, and a second worker that declines to send because the idempotency key is already resolved. If the implementation cannot observe that boundary, a green retry dashboard says very little about customer-visible duplicates. This test also exposes a common storage mistake: treating a sending row as proof that the external request failed, when it may instead mean that the process disappeared after the request succeeded.

That boundary deserves its own runbook entry. Name the job key, the ledger row, the broker message identifier, and the external request identifier separately; they answer different questions during replay. The broker identifier tells you what was redelivered, while the idempotency key tells you whether the business effect was already claimed. The external request identifier is the evidence to ask the mail system about. Collapsing those values into one log field makes a duplicate investigation look tidy while removing the links needed to prove what happened.

A restrained rollout for delivery guarantees

Roll out the weekly digest path in stages. First, write delivery records and observe the existing worker without changing its send policy. Next, enable bounded retries for a single customer cohort and inspect the failure taxonomy. Then enable quarantine and replay only after an operator can locate a job, read its evidence, and make a reversible decision.

The broker choice is easy to revisit while the queue adapter is isolated. The delivery contract is expensive to revisit after customers have received duplicates or missed a digest. Make that contract the stable part: at-least-once job processing, idempotent effects, explicit quarantine, and evidence that survives a broker migration.

DEV Community