Dead Letter Queues: Never Lose a Bad Record

A dead letter queue is the seatbelt of every message- and stream-based pipeline: it is the place a record goes when your consumer cannot process it, so the record is quarantined rather than dropped on the floor or left to wedge the whole queue. Every non-trivial pipeline eventually meets a record it cannot handle — a malformed JSON body, a null in a column the downstream schema swears is NOT NULL, an event referencing a foreign key that hasn't arrived yet, a poison message that deterministically blows up your parser on every single delivery attempt. The question is never "will a bad record show up?" — it will — but "what happens to it when it does?" Without an explicit answer, the two default outcomes are both catastrophic: you either silently swallow the record (data loss nobody notices until audit season) or you retry it forever and block every good record queued behind it (head-of-line blocking that turns one bad row into a total outage).

This guide is the walkthrough you wished existed the first time an incident review asked "why did the payments consumer stop for six hours because of one unparseable event?" It works through the whole lifecycle: why a dead letter queue is the only correct answer to a bad record, how to design one on both SQS (where redrive is native) and Kafka (where you build it yourself), how redrive and replay send fixed records safely back home once the bug is patched, how a retry policy with a bounded retry budget separates a transient blip from a genuine poison message, and how observability turns a silent quarantine mailbox into a first-class SLO with alarms on depth and age. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works. Examples use SQS, Kafka, and PostgreSQL, but the mental model carries to Kinesis, Pub/Sub, RabbitMQ, and every other broker.

When you want hands-on reps immediately after reading, drill the event-processing practice library →, rehearse the broker mechanics on the streaming practice library →, and wire the batch side on the ETL practice library →.

---

On this page

• Why dead letter queues exist
• DLQ design — SQS and Kafka
• Redrive and replay
• Poison-message detection and retry budgets
• Observability and alerting on DLQs
• Cheat sheet — dead letter queue recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why dead letter queues exist

A bad record must be diverted and preserved — never dropped, never allowed to block the good ones

The one-sentence invariant: a dead letter queue is a separate, durable destination where a consumer parks any record it cannot successfully process after a bounded number of attempts, so the failing record is quarantined with enough context to diagnose and replay it while the main queue keeps flowing for every record that is fine — and the whole discipline exists because the two naive alternatives, silently dropping the record or retrying it forever, are each a production incident waiting to happen. The dead letter queue is not an optional nicety bolted on at the end; it is the load-bearing reliability primitive that lets a pipeline make forward progress in the presence of the one bad record that is statistically guaranteed to arrive.

The three failure classes a DLQ has to catch.

• Poison messages. A record that fails deterministically — the same input produces the same exception on every delivery. A truncated Avro payload, a JSON body with a trailing comma, an event whose amount field is the string "NaN". No amount of retrying fixes it; the only sane response after a few attempts is to quarantine it and move on.
• Transient faults. A record that fails non-deterministically — a downstream timeout, a throttled API, a momentary connection reset, a deadlock. The exact same record will very likely succeed if retried a moment later. Dead-lettering these too eagerly floods the DLQ with records that were never actually bad.
• Downstream outages. A whole dependency is down — the database is failing over, the enrichment service is returning 503s for everyone. Here every record fails, and a DLQ is the wrong first response: you want a circuit breaker to pause consumption, not a DLQ that fills with millions of perfectly good records.

The four axes every DLQ design has to answer.

• Where quarantine lives. A dedicated queue (SQS DLQ), a dedicated topic (Kafka <topic>.DLQ), a database table, or an object-store prefix. The destination must be as durable as the source — a DLQ that can itself lose messages defeats the entire point.
• When a record is dead-lettered. The retry policy: how many attempts, with what backoff, before the record is declared dead. Too few attempts dead-letters transient blips; too many turns a poison message into head-of-line blocking.
• What metadata rides along. The DLQ record must carry the original payload plus the failure context: the exception, the stack, the source topic/partition/offset (or message id), the attempt count, and first-seen/last-seen timestamps. A DLQ of naked payloads with no error attached is nearly useless at 3 AM.
• How you get it back. The redrive/replay path: once the bug is fixed, how do fixed records return to the main pipeline — safely, idempotently, and without a retry storm.

What interviewers listen for.

• Do you name head-of-line blocking as the reason you cannot just "retry forever"? — required answer.
• Do you distinguish a poison message (retry never helps) from a transient fault (retry usually helps)? — senior signal.
• Do you insist the DLQ record carries the error and the source coordinates, not just the payload? — senior signal.
• Do you treat the DLQ as something you must alert on, not a black hole? — required answer.
• Do you mention idempotency the moment redrive/replay comes up? — senior signal.

Worked example — the head-of-line-blocking outage without a DLQ

Detailed explanation. The clearest way to motivate a dead letter queue is to watch a pipeline die without one. A single ordered partition (Kafka) or a single in-flight FIFO message group (SQS FIFO) processes records strictly in order. If record #7 is a poison message and the consumer's only two options are "commit the offset" or "don't commit and retry," then a consumer written to retry-until-success will loop on record #7 forever, and records #8, #9, #10 … never get processed. One bad row becomes a total stall.

• The setup. A Kafka consumer on a single partition, enable.auto.commit=false, a handler that raises on bad input and a retry loop with no cap.
• The trigger. Record at offset 7 has a malformed body.
• The result. The consumer group lag on that partition grows without bound; every downstream SLA breaks; the on-call sees "consumer is running" (it is — it's just stuck) and no error alarm.

Question. Show the naive consumer loop that causes head-of-line blocking, and name the one change that turns the stall into a bounded, observable failure.

Input.

Offset	Payload	Processing result
5	valid	committed
6	valid	committed
7	malformed JSON	raises forever
8	valid	never reached
9	valid	never reached

Code.

# ANTI-PATTERN — retry-until-success blocks the whole partition
def run_consumer_naive(consumer):
    for msg in consumer:                      # ordered partition
        while True:                           # <-- unbounded retry
            try:
                handle(msg.value)             # raises on offset 7
                consumer.commit(msg)          # never reached for a poison msg
                break
            except Exception:
                time.sleep(1)                 # loops forever on offset 7
        # offsets 8, 9, ... are never processed

Step-by-step explanation.

1. Offsets 5 and 6 process and commit normally — the loop looks healthy in a demo with clean data.
2. Offset 7 is a poison message: handle() raises every single time because the input itself is broken. The while True retry loop catches the exception, sleeps, and tries the same record again.
3. Because the partition is ordered and the offset is never committed, the consumer cannot advance to offset 8. Records 8, 9, 10, … pile up behind the one poison record — this is head-of-line blocking.
4. The process is still alive and consuming CPU, so a naive liveness probe reports "healthy." Consumer-group lag climbs, but if nobody alarms on lag the outage is invisible until a downstream SLA breaks.
5. The single fix is a bounded retry budget plus a dead letter queue: after N failed attempts, produce the record (with its error) to the DLQ, commit the offset, and move on. The stall becomes one quarantined record and a metric you can alarm on.

Output.

Design	Offset 7 outcome	Offsets 8+	Failure visibility
Retry forever	retried infinitely	blocked forever	none (looks "healthy")
Drop on error	silently discarded	flow continues	none (silent data loss)
Bounded retry + DLQ	quarantined after N tries	flow continues	DLQ depth alarm fires

Rule of thumb. On any ordered partition or FIFO group, "retry until success" is a latent total outage. Bound the retries and route the loser to a dead letter queue so one bad record costs you one quarantined record — not the whole pipeline.

Worked example — the four-axis DLQ decision table

Detailed explanation. Before writing any code, a senior engineer fixes the four axes — where, when, what, and how-back — for the specific pipeline. Walk through building the table for a payments-events consumer that reads Kafka and writes to a ledger database.

• Where. A dedicated Kafka topic payments.events.DLQ, same replication factor as the source.
• When. After 4 in-process retries (transient defense) OR immediately on a classified-permanent error (poison defense).
• What. An envelope: original key + value, exception class + message, source topic/partition/offset, attempt count, first-seen and last-seen timestamps.
• How-back. A replay tool that reads the DLQ topic, applies an idempotency key, rate-limits, and re-produces to the source topic.

Question. Fill in the four-axis table for the payments consumer and state the single most important non-obvious decision.

Input.

Axis	Question it answers
Where	which durable destination holds quarantined records
When	how many attempts / which errors trigger dead-lettering
What	which metadata travels with the record
How-back	how fixed records return to the main flow

Code.

Payments DLQ contract
=====================
WHERE     : topic payments.events.DLQ  (RF=3, same as source; 14-day retention)
WHEN      : transient error  -> 4 in-process retries (backoff+jitter) then DLQ
            permanent error   -> 0 retries, DLQ immediately (classified)
            downstream outage -> circuit breaker pauses; DO NOT flood DLQ
WHAT      : envelope {
              key, value (original bytes, base64),
              error_class, error_message, stack,
              src_topic, src_partition, src_offset,
              attempts, first_seen_ts, last_seen_ts
            }
HOW-BACK  : replay tool -> idempotency key = (src_topic, src_partition, src_offset)
                        -> token-bucket rate limit
                        -> re-produce to payments.events
                        -> still fails -> re-quarantine (do NOT loop)

Step-by-step explanation.

1. Where is decided first because it constrains everything else: a Kafka source implies a Kafka DLQ topic with matching durability; an SQS source implies an SQS DLQ. Mixing (Kafka source, database DLQ) is legal but adds an extra failure surface.
2. When is two rules, not one: a retry budget for transient faults and an error classifier that dead-letters known-permanent errors immediately. Conflating them either wastes retries on poison or dead-letters transient blips.
3. What is where most teams under-invest. The original payload alone cannot be debugged; the envelope must carry the exception and the exact source coordinates so you can reproduce and, later, replay precisely those records.
4. How-back must be idempotent and rate-limited from day one. The most common redrive incident is replaying two million records at full speed into a service that promptly falls over — turning a data-quality problem into a capacity outage.
5. The single most important non-obvious decision is the downstream-outage carve-out: a DLQ is for individual bad records, not for a dependency being down. When everything fails, pause (circuit breaker), don't quarantine millions of good records.

Output.

Axis	Payments decision	Failure it prevents
Where	payments.events.DLQ, RF=3, 14-day retention	losing quarantined records
When	4 retries transient; 0 for permanent; breaker for outage	poison loops + DLQ floods
What	full envelope with error + source coordinates	undebuggable quarantine
How-back	idempotent, rate-limited replay	redrive-induced outage

Rule of thumb. Write the four-axis contract on a whiteboard before writing the consumer. "Where / when / what / how-back" — if any cell is blank, your DLQ has a hole.

Worked example — the except: pass anti-pattern and its fix

Detailed explanation. The most dangerous DLQ is the one that looks like error handling but is actually silent data loss. The classic offender is try/except: pass (or logger.warning(e) with no further action) wrapped around a handler. It keeps the pipeline flowing — which looks great — while quietly discarding every record that fails. There is no queue, no alarm, no way to get the record back. Six months later a reconciliation finds the warehouse is missing 0.3% of rows and nobody can say which.

• The smell. A broad except that swallows the error and continues without persisting the failed record anywhere.
• The fix. Route the failed record to a dead letter queue with its error, then continue. Same forward progress, zero data loss, full observability.

Question. Rewrite the swallow-and-continue handler so it preserves the record and stays observable.

Input.

Behavior	Swallow (except: pass)	DLQ route
Pipeline keeps flowing	yes	yes
Failed record preserved	no (lost)	yes (quarantined)
Error visible	no	yes (DLQ + metric)
Replayable after fix	no	yes

Code.

# ANTI-PATTERN
def handle_message(msg):
    try:
        process(msg)
    except Exception:
        pass          # <-- silent data loss; the record is gone forever

# FIX — quarantine instead of discard
def handle_message(msg, dlq):
    try:
        process(msg)
    except PermanentError as e:                 # classified, unrecoverable
        dlq.send(build_envelope(msg, e, attempts=1))
        metrics.increment("dlq.sent", tags={"error": type(e).__name__})
    except TransientError as e:                 # let the retry policy handle it
        raise                                   # re-raise; do NOT swallow

Step-by-step explanation.

1. The anti-pattern's fatal flaw is that pass produces the same visible behavior as success — the loop continues — so it survives code review and demos. The failure only manifests as slow, unattributable data loss.
2. The fix classifies the error. A PermanentError (schema, validation) is quarantined immediately with a full envelope; there is no point retrying it.
3. A TransientError is re-raised so the surrounding retry policy (next section's territory) can back off and retry — swallowing a transient error would waste a record that a retry would have saved.
4. Every dead-letter emits a metric tagged with the error class. This is what makes the DLQ observable: you can alarm on dlq.sent rate and break it down by error type.
5. The net behavior matches the anti-pattern's one good property — the pipeline keeps flowing — while fixing its fatal one: nothing is lost, and everything is attributable and replayable.

Output.

Record fate	except: pass	DLQ route
Poison record	lost silently	in DLQ with error, replayable
Transient blip	lost silently	retried, then DLQ if it persists
Operator awareness	none	metric + alarm
Post-incident recovery	impossible	replay from DLQ

Rule of thumb. except: pass is not error handling — it is data loss with extra steps. Every catch that ends a record's life must first write that record to a dead letter queue.

Data engineering interview question on the "never lose a bad record" contract

A senior interviewer often opens with: "You own a Kafka consumer that writes events to a ledger. A single malformed event took down the consumer for hours last week because it retried forever. Design the end-to-end contract that guarantees you never lose a bad record and never let one block the pipeline — cover the retry budget, the DLQ, the envelope, and the failure classes you treat differently."

Solution Using a bounded retry budget, a classified DLQ route, and a full envelope

# never_lose.py — bounded retries + classified dead-lettering for a Kafka consumer
import json, time, base64, random
from datetime import datetime, timezone

MAX_ATTEMPTS = 4

class PermanentError(Exception): ...   # schema/validation: do not retry
class TransientError(Exception): ...   # timeout/throttle/deadlock: retry

def classify(exc: Exception) -> str:
    if isinstance(exc, (json.JSONDecodeError, PermanentError)):
        return "permanent"
    return "transient"

def build_envelope(msg, exc, attempts):
    return {
        "key": None if msg.key is None else base64.b64encode(msg.key).decode(),
        "value": base64.b64encode(msg.value).decode(),
        "error_class": type(exc).__name__,
        "error_message": str(exc)[:2000],
        "src_topic": msg.topic,
        "src_partition": msg.partition,
        "src_offset": msg.offset,
        "attempts": attempts,
        "first_seen_ts": datetime.now(timezone.utc).isoformat(),
        "last_seen_ts": datetime.now(timezone.utc).isoformat(),
    }

def run(consumer, producer, dlq_topic):
    for msg in consumer:
        attempt = 0
        while True:
            attempt += 1
            try:
                process(msg.value)
                consumer.commit(msg)                 # success: advance
                break
            except Exception as exc:
                kind = classify(exc)
                fatal = kind == "permanent" or attempt >= MAX_ATTEMPTS
                if fatal:
                    env = build_envelope(msg, exc, attempt)
                    producer.send(dlq_topic, json.dumps(env).encode())
                    producer.flush()                 # DLQ write durable first
                    consumer.commit(msg)             # then advance past it
                    metrics.increment("dlq.sent", tags={"error": kind})
                    break
                # transient + budget remaining: backoff with jitter, retry
                sleep = min(30.0, (2 ** (attempt - 1))) * (0.5 + random.random())
                time.sleep(sleep)

Step-by-step trace.

Offset	Error kind	Attempts	Outcome
5	none	1	processed + committed
6	transient (timeout)	2	retried once, then succeeded + committed
7	permanent (bad JSON)	1	DLQ immediately + committed
8	transient (throttle)	4	budget exhausted -> DLQ + committed
9	none	1	processed + committed

After deployment, the consumer never stalls: transient faults get up to four backoff-with-jitter retries, permanent errors are quarantined on the first failure, and in every fatal case the DLQ write is flushed before the source offset is committed — so a crash between the two leaves the record redeliverable rather than lost. Offsets always advance, so head-of-line blocking is impossible.

Output:

Property	Value
Max latency added by a poison record	one DLQ write (~ms)
Transient blip handling	up to 4 retries, exp backoff + jitter
Data loss on consumer crash	none (DLQ flushed before commit)
Head-of-line blocking	impossible (offset always advances)
Debuggability	full envelope: error + source coordinates

Why this works — concept by concept:

• Bounded retry budget — MAX_ATTEMPTS caps how long a transient fault is retried. It is the fuse that prevents a poison message from becoming an infinite loop, and it is the single line that separates "resilient" from "wedged."
• Error classification — permanent errors skip retries entirely and go straight to the DLQ; transient errors spend the budget. Retrying a malformed-JSON record four times is pure waste; retrying a timeout is exactly right.
• Flush-before-commit ordering — the DLQ producer is flushed to durability before the source offset is committed. This ordering is the correctness invariant: a crash in between redelivers the record (at-least-once) rather than dropping it (data loss).
• Backoff with jitter — 2 ** (attempt-1) grows the delay; the 0.5 + random() factor de-synchronizes many consumers so they don't retry in a thundering herd against a recovering dependency.
• Cost — O(1) extra work per record in the happy path (nothing), one DLQ produce + flush per fatal record, and a bounded O(attempts) delay for transient faults. The eliminated cost is the unbounded outage that one poison record used to cause.

Events
Topic — event-processing
Event-processing problems on failure handling and DLQs

Practice →

Streaming Topic — streaming Streaming problems on consumer offsets and ordering

Practice →

---

2. DLQ design — SQS and Kafka

SQS gives you a native redrive policy; Kafka makes you build the DLQ yourself — the envelope is what matters on both

The mental model in one line: on SQS a dead letter queue is a first-class feature — you attach a RedrivePolicy with a maxReceiveCount and a deadLetterTargetArn, and the broker moves a message to the DLQ automatically once it has been received too many times without being deleted — whereas on Kafka there is no native DLQ at all: you produce failed records to a separate topic yourself (or let Kafka Connect do it via errors.deadletterqueue.topic.name), and on both platforms the thing that determines whether the DLQ is useful is the envelope of failure context you attach, not the queue itself. Two very different mechanisms; one shared design principle — carry the whole story with the record.

SQS — the native redrive policy.

• maxReceiveCount. The number of times a message can be received (delivered and not deleted within the visibility timeout) before SQS moves it to the DLQ. A consumer that fails leaves the message un-deleted; after the visibility timeout it becomes visible again; when the receive count crosses the threshold, SQS redrives it to the DLQ.
• deadLetterTargetArn. The ARN of the queue that receives dead-lettered messages. It must be the same type as the source — a standard-queue DLQ for a standard source, a FIFO DLQ for a FIFO source.
• Visibility timeout interplay. The DLQ only works if the visibility timeout is longer than your processing time. If processing takes 40s but the visibility timeout is 30s, SQS re-delivers a message that is still being processed, inflating the receive count and dead-lettering good messages.
• Retention. The DLQ's MessageRetentionPeriod should be the maximum (14 days) — you want time to notice, diagnose, and redrive before records expire.

Kafka — the DIY DLQ.

• No native feature. Kafka brokers do not track per-message delivery attempts, so there is nothing to "move" a message on. The consumer application decides, catches the error, and produces the record to a DLQ topic itself.
• DLQ topic convention. A sibling topic, commonly <source>.DLQ or <source>.dlt, with the same partition count and replication factor as the source so it is equally durable.
• Kafka Connect. Sink/source connectors get a DLQ for free: errors.tolerance=all, errors.deadletterqueue.topic.name=<topic>, and errors.deadletterqueue.context.headers.enable=true to attach the error context as record headers.
• Where the attempt count lives. Since Kafka doesn't count receives, the consumer tracks attempts (in memory for in-process retries, or in a header on the DLQ record for cross-restart accounting).

What the DLQ record must carry — the envelope.

• Original payload. The exact bytes (key + value). Base64 the value if the DLQ topic uses a text codec, so binary payloads survive.
• Failure context. Exception class, message, and (optionally truncated) stack trace.
• Source coordinates. For Kafka: source topic, partition, offset. For SQS: original message id and, if relevant, the source queue ARN. These let you reproduce and later replay exactly the failed records.
• Bookkeeping. Attempt count, first-seen and last-seen timestamps, and the consumer/service name that quarantined it.

Same-schema vs envelope-wrapped DLQ topics.

• Same-schema DLQ. The DLQ holds records in the same schema as the source, with the error context pushed into Kafka headers (Connect's default). Pro: replay is a straight copy back. Con: you can't query the error in the value.
• Envelope-wrapped DLQ. The DLQ value is a new envelope object embedding the original payload plus the error fields. Pro: fully self-describing and queryable. Con: replay must unwrap the envelope before re-producing.

Common interview probes on DLQ design.

• "How does SQS decide to dead-letter a message?" — maxReceiveCount receives without deletion.
• "Does Kafka have a built-in DLQ?" — no; you produce to a DLQ topic yourself, or use Kafka Connect's errors.deadletterqueue.
• "What must the DLQ record contain?" — payload + error + source coordinates + attempt count.
• "What breaks if the visibility timeout is shorter than processing time?" — good messages get redelivered and wrongly dead-lettered.

Worked example — SQS main queue + DLQ with maxReceiveCount

Detailed explanation. The canonical SQS setup: a source queue with a redrive policy pointing at a DLQ, maxReceiveCount set to a small number, and a visibility timeout comfortably larger than processing time. Build it with Terraform-style config and show the receive-count progression.

• Source. payments-events standard queue, visibility timeout 60s.
• DLQ. payments-events-dlq, retention 14 days.
• Policy. maxReceiveCount = 5.

Question. Configure the source queue, the DLQ, and the redrive policy, and trace what happens to a message that fails five times.

Input.

Setting	Value
Source queue	payments-events
Visibility timeout	60s (processing p99 = 20s)
DLQ	payments-events-dlq
maxReceiveCount	5
DLQ retention	14 days

Code.

// SQS source queue attributes (RedrivePolicy references the DLQ ARN)
{
  "QueueName": "payments-events",
  "Attributes": {
    "VisibilityTimeout": "60",
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:payments-events-dlq\",\"maxReceiveCount\":\"5\"}"
  }
}

// SQS DLQ attributes (max retention so you have time to redrive)
{
  "QueueName": "payments-events-dlq",
  "Attributes": {
    "MessageRetentionPeriod": "1209600"
  }
}

# Consumer — failure = do NOT delete the message; SQS increments receive count
import boto3
sqs = boto3.client("sqs")
SRC = "https://sqs.us-east-1.amazonaws.com/123456789012/payments-events"

def poll_once():
    resp = sqs.receive_message(QueueUrl=SRC, MaxNumberOfMessages=10,
                               WaitTimeSeconds=20, VisibilityTimeout=60,
                               AttributeNames=["ApproximateReceiveCount"])
    for m in resp.get("Messages", []):
        try:
            process(m["Body"])
            sqs.delete_message(QueueUrl=SRC, ReceiptHandle=m["ReceiptHandle"])
        except Exception:
            # Do nothing: leaving the message undeleted lets its visibility
            # timeout lapse, redelivering it and bumping ApproximateReceiveCount.
            # After the 5th receive without delete, SQS moves it to the DLQ.
            pass

Step-by-step explanation.

1. The RedrivePolicy on the source queue names the DLQ ARN and maxReceiveCount. There is no code path that "sends to DLQ" — the broker does the move.
2. When process() succeeds, the consumer calls delete_message; the message leaves the queue. When it fails, the consumer simply does not delete it.
3. An undeleted message becomes visible again after the 60s visibility timeout. Each redelivery increments ApproximateReceiveCount.
4. On the 5th receive (equal to maxReceiveCount) without a delete, SQS moves the message to payments-events-dlq automatically — the receive count is the retry budget, enforced by the broker.
5. The visibility timeout (60s) is deliberately larger than processing p99 (20s). If it were smaller, SQS would redeliver messages that are still being processed, inflating the receive count and dead-lettering perfectly good messages — the single most common SQS DLQ misconfiguration.

Output.

Receive #	Consumer result	ApproximateReceiveCount	Location
1	fail (no delete)	1	source
2	fail	2	source
3	fail	3	source
4	fail	4	source
5	fail	5	moved to DLQ

Rule of thumb. On SQS, set maxReceiveCount to your intended retry budget, set the DLQ retention to the 14-day maximum, and always keep the visibility timeout larger than processing p99 — otherwise you dead-letter good messages.

Worked example — Kafka DIY DLQ producer with an envelope

Detailed explanation. Kafka has no redrive policy, so the consumer produces failed records to a DLQ topic itself. The important design choice is the envelope: wrap the original bytes plus error context so the DLQ is self-describing and later replayable. Build the DLQ producer and the envelope.

• DLQ topic. payments.events.DLQ, same partitions + RF as source.
• Envelope. JSON value embedding base64 payload + error + source coordinates.
• Headers. Duplicate key fields as headers for cheap filtering without deserializing the value.

Question. Write the function that dead-letters a failed Kafka record with a full envelope and error headers.

Input.

Envelope field	Source
value_b64	original msg.value bytes
error_class / error_message	the caught exception
src_topic / src_partition / src_offset	msg coordinates
attempts	in-process retry counter

Code.

# dlq_producer.py — wrap a failed record in an envelope and produce to the DLQ topic
import json, base64
from datetime import datetime, timezone

DLQ_TOPIC = "payments.events.DLQ"

def dead_letter(producer, msg, exc, attempts):
    envelope = {
        "key_b64":   None if msg.key is None else base64.b64encode(msg.key).decode(),
        "value_b64": base64.b64encode(msg.value).decode(),
        "error_class":   type(exc).__name__,
        "error_message": str(exc)[:2000],
        "src_topic":     msg.topic,
        "src_partition": msg.partition,
        "src_offset":    msg.offset,
        "attempts":      attempts,
        "first_seen_ts": datetime.now(timezone.utc).isoformat(),
        "last_seen_ts":  datetime.now(timezone.utc).isoformat(),
    }
    headers = [
        ("error_class",   type(exc).__name__.encode()),
        ("src_topic",     msg.topic.encode()),
        ("src_partition", str(msg.partition).encode()),
        ("src_offset",    str(msg.offset).encode()),
    ]
    # Preserve the original key so DLQ records stay co-partitioned by entity
    producer.send(DLQ_TOPIC, key=msg.key,
                  value=json.dumps(envelope).encode(), headers=headers)
    producer.flush()   # make the DLQ write durable before committing the source offset

Step-by-step explanation.

1. The value is a JSON envelope. The original bytes are base64-encoded so binary payloads (Avro, Protobuf, compressed) survive a text-encoded DLQ topic without corruption.
2. The exception is captured as a class name plus a truncated message. Truncating at 2000 chars keeps a pathological multi-megabyte error from bloating the DLQ record.
3. The source coordinates (topic/partition/offset) are the replay primitive: they let a replay tool skip records it already re-produced and let you reproduce the exact failure locally.
4. Key fields are duplicated into Kafka headers so an operator can filter the DLQ (error_class = "SchemaError") without deserializing every value — cheap triage at scale.
5. producer.flush() forces the DLQ write to durability before the caller commits the source offset. This ordering is the same at-least-once invariant as section 1: never advance past a record until its quarantine is durable.

Output.

Envelope part	Example
value_b64	eyJhbW91bnQiOiJOYU4ifQ== (original bytes)
error_class	ValidationError
src coordinates	payments.events / 3 / 91422
attempts	4
headers	error_class, src_topic, src_partition, src_offset

Rule of thumb. On Kafka, make the DLQ record self-describing: base64 the original bytes, embed the error and source coordinates in the value, and duplicate the key fields as headers for triage. Give the DLQ topic the same partitions and replication factor as the source.

Worked example — Kafka Connect DLQ configuration

Detailed explanation. If your pipeline is a Kafka Connect sink (e.g. writing to S3, JDBC, Elasticsearch), you get a DLQ without writing a producer — you configure it. Connect catches converter and transform errors and routes the offending record to a DLQ topic with the error context in headers. Configure it correctly and know its one gap.

• Tolerance. errors.tolerance=all — skip bad records instead of killing the task.
• DLQ. errors.deadletterqueue.topic.name plus header context.
• Logging. errors.log.enable=true to also log the error for correlation.

Question. Write the Connect sink config that dead-letters conversion/transform failures, and name the failure class Connect's DLQ does not catch.

Input.

Setting	Purpose
errors.tolerance	all = tolerate + route, not fail the task
errors.deadletterqueue.topic.name	where bad records go
errors.deadletterqueue.context.headers.enable	attach error context as headers
errors.retry.timeout	retry transient errors before DLQ

Code.

# Kafka Connect sink connector with a dead letter queue
name: s3-sink
config:
  connector.class: io.confluent.connect.s3.S3SinkConnector
  topics: payments.events
  s3.bucket.name: prod-payments-lake

  # Error handling / DLQ
  errors.tolerance: all
  errors.deadletterqueue.topic.name: payments.events.DLQ
  errors.deadletterqueue.topic.replication.factor: 3
  errors.deadletterqueue.context.headers.enable: true
  errors.retry.timeout: 60000          # retry transient errors up to 60s
  errors.retry.delay.max.ms: 5000
  errors.log.enable: true
  errors.log.include.messages: true

  key.converter: org.apache.kafka.connect.storage.StringConverter
  value.converter: io.confluent.connect.avro.AvroConverter
  value.converter.schema.registry.url: http://schema-registry:8081

Step-by-step explanation.

1. errors.tolerance=all flips Connect from "fail-fast" (default none, one bad record kills the task) to "tolerate" — the record is routed away and the task keeps running.
2. errors.deadletterqueue.topic.name names the DLQ topic; replication.factor: 3 makes it as durable as production data. Without a replication factor Connect may create it with RF=1 — a single-broker-loss data-loss risk.
3. context.headers.enable: true attaches the error class, the original topic/partition/offset, and the failing stage (converter vs transform) as headers — Connect's equivalent of the hand-built envelope.
4. errors.retry.timeout gives transient converter errors up to 60s of retries before dead-lettering, so a momentary schema-registry blip doesn't flood the DLQ.
5. The gap: Connect's DLQ catches converter and single-message-transform errors only — malformed bytes, schema mismatches, failed SMTs. It does not catch errors thrown inside the sink's write to the external system in all connector versions; those depend on the connector's own handling. Know which failures your connector routes and which it doesn't.

Output.

Failure stage	Caught by Connect DLQ?
Deserialization / converter	yes
Single-message transform	yes
Schema-registry transient error	retried, then DLQ
Sink write to external system	connector-dependent (often not)

Rule of thumb. For Kafka Connect, errors.tolerance=all + errors.deadletterqueue.topic.name + context.headers.enable=true gives you a DLQ for free — but confirm your specific connector routes sink-write failures too, or add application-level handling for those.

Data engineering interview question on DLQ design

A senior interviewer might ask: "Design the dead letter queue for a Kafka payments consumer that writes to a ledger DB. Specify the DLQ topic, the envelope, the retry budget, the ordering guarantee between the DLQ write and the offset commit, and how the same design would differ if the source were SQS instead of Kafka."

Solution Using a self-describing envelope, flush-before-commit ordering, and platform-specific triggers

# ledger_consumer.py — Kafka source, DLQ topic, envelope, flush-before-commit
import json, base64, time, random
from datetime import datetime, timezone

DLQ_TOPIC, MAX_ATTEMPTS = "payments.events.DLQ", 4

def envelope(msg, exc, attempts):
    return json.dumps({
        "value_b64": base64.b64encode(msg.value).decode(),
        "error_class": type(exc).__name__,
        "error_message": str(exc)[:2000],
        "src": {"topic": msg.topic, "partition": msg.partition, "offset": msg.offset},
        "attempts": attempts,
        "first_seen_ts": datetime.now(timezone.utc).isoformat(),
        "last_seen_ts": datetime.now(timezone.utc).isoformat(),
    }).encode()

def is_permanent(exc):
    return isinstance(exc, (json.JSONDecodeError, ValueError))

def run(consumer, producer):
    for msg in consumer:
        attempt = 0
        while True:
            attempt += 1
            try:
                write_to_ledger(msg.value)
                consumer.commit(msg)
                break
            except Exception as exc:
                if is_permanent(exc) or attempt >= MAX_ATTEMPTS:
                    producer.send(DLQ_TOPIC, key=msg.key, value=envelope(msg, exc, attempt))
                    producer.flush()          # 1) DLQ durable
                    consumer.commit(msg)      # 2) then advance source
                    break
                time.sleep(min(30.0, 2 ** (attempt - 1)) * (0.5 + random.random()))

// If the source were SQS: no producer code — a broker redrive policy does it
{
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:...:payments-events-dlq\",\"maxReceiveCount\":\"5\"}",
  "VisibilityTimeout": "60"
}

Step-by-step trace.

Concern	Kafka answer	SQS answer
DLQ mechanism	app produces to DLQ topic	broker RedrivePolicy moves it
Retry budget	in-app MAX_ATTEMPTS	maxReceiveCount
Envelope	app-built JSON + headers	native attributes + your own body
Ordering guarantee	flush DLQ, then commit offset	delete on success, else let it lapse
Durability of DLQ	same partitions + RF as source	max retention on DLQ queue

After deployment, a permanent error is quarantined on its first failure and a transient error gets four backoff-with-jitter retries; in every fatal case the DLQ write is flushed before the source offset is committed, so no record is ever lost. On SQS the very same policy is expressed declaratively — no producer code — because redrive is a broker feature.

Output:

Metric	Value
Data loss on crash	none (flush-before-commit / broker-managed)
Poison record cost	one DLQ write, offset advances
Transient handling	4 retries, exp backoff + jitter
Kafka DLQ durability	RF=3, partitions match source
SQS DLQ durability	14-day retention, receive-count trigger

Why this works — concept by concept:

• Self-describing envelope — embedding base64 payload + error + source coordinates makes the DLQ record independently debuggable and replayable, on either platform.
• Flush-before-commit — producing and flushing the DLQ record before committing the source offset gives at-least-once semantics: a crash redelivers rather than drops. This is the Kafka analogue of SQS "delete only on success."
• Platform-native triggers — Kafka has no delivery counter, so the app owns the retry budget; SQS counts receives, so the broker owns it via maxReceiveCount. Same contract, different owner.
• Matched durability — the DLQ must be as durable as the source (RF=3 / 14-day retention). A DLQ that can lose messages silently reintroduces the data loss you built it to prevent.
• Cost — one produce+flush per fatal record on Kafka (near-zero on the happy path); on SQS, zero code and one broker move. Both are O(1) per bad record, with the retry budget capping transient-fault work at O(attempts).

Streaming
Topic — streaming
Streaming problems on Kafka topics and SQS queues

Practice →

Events Topic — event-processing Event-processing problems on DLQ envelopes

Practice →

---

3. Redrive and replay

Redrive sends fixed records home — safely means idempotent, rate-limited, and re-quarantined when they still fail

The mental model in one line: redrive (SQS's term) and replay (the Kafka term) are the same operation — reading records back out of the dead letter queue and re-injecting them into the main pipeline once the bug that quarantined them has been fixed — and doing it safely means three things without exception: an idempotency key so a record processed twice has the effect of once, a rate limit so two million redriven records don't stampede a freshly-recovered service, and a re-quarantine path so any record that still fails goes back to the DLQ instead of looping forever. Redrive is the payoff of having a DLQ; it is also the step most likely to cause a second outage if done carelessly.

Redrive vs replay vs reprocess-in-place.

• Redrive (move back to source). Read from the DLQ, re-inject into the source queue/topic, let the normal consumer process it. The consumer code is unchanged; you fixed the bug and want the records to flow through the fixed path.
• Reprocess-in-place. Read from the DLQ and process directly with a patched one-off consumer, without touching the source. Useful when the source has moved on and re-injecting would disturb ordering or offsets.
• Replay (Kafka). Because a Kafka topic is a log, "replay" can also mean rewinding a consumer group's offsets to re-read the source topic — distinct from DLQ replay, which re-reads the DLQ topic.

The fix-forward-then-redrive runbook.

• Step 1 — stop the bleeding. Confirm the root cause; deploy the fix so new records stop landing in the DLQ. Redriving before fixing just re-fills the DLQ.
• Step 2 — scope the blast radius. Count and classify the DLQ records (by error class, by source, by time window) so you redrive only what the fix addresses.
• Step 3 — dry-run. Redrive a small sample first, verify success, then open the throttle gradually.
• Step 4 — rate-limited full redrive. Move records with a token-bucket limiter and idempotency, watching downstream saturation and DLQ arrival rate.
• Step 5 — re-quarantine + reconcile. Records that still fail return to the DLQ; reconcile counts (redriven = succeeded + re-quarantined).

Safety mechanisms that are non-negotiable.

• Idempotency key. For Kafka, (src_topic, src_partition, src_offset) uniquely identifies a record; for SQS, a business key or a dedupe id. The consumer records processed keys so a redriven duplicate is a no-op.
• Rate limiting. Redrive at a controlled records/sec, well under the consumer's healthy throughput, so recovery doesn't become a self-inflicted load test.
• Partial replay by filter. Redrive only records matching an error class or time window — never "everything in the DLQ" blindly, because the DLQ often mixes unrelated failures.
• Poison re-quarantine. A record that fails redrive goes back to the DLQ (ideally with an incremented redrive_attempts) — never into an infinite DLQ→source→DLQ loop.

Common interview probes on redrive.

• "How do you avoid double-processing on redrive?" — idempotency key + processed-set.
• "Why rate-limit a redrive?" — a fixed service is fragile; a full-speed replay re-breaks it.
• "What happens to records that still fail after redrive?" — re-quarantine, don't loop.
• "Redrive vs replay?" — redrive moves DLQ→source; Kafka replay can also mean rewinding source offsets.

Worked example — SQS native redrive with StartMessageMoveTask

Detailed explanation. SQS provides a native redrive API: StartMessageMoveTask moves messages from a DLQ back to their original source queue (or a specified destination) with a configurable MaxNumberOfMessagesPerSecond — the rate limit is built in. Walk through initiating and monitoring a redrive.

• Source of the move. The DLQ ARN.
• Destination. Omit to use each message's original source queue, or set explicitly.
• Rate. MaxNumberOfMessagesPerSecond throttles the move.

Question. Start a rate-limited redrive from the payments DLQ back to the source and monitor its progress.

Input.

Parameter	Value
SourceArn	payments-events-dlq ARN
Destination	original source (default)
MaxNumberOfMessagesPerSecond	50
Precondition	fix deployed to consumer

Code.

import boto3
sqs = boto3.client("sqs")
DLQ_ARN = "arn:aws:sqs:us-east-1:123456789012:payments-events-dlq"

# 1. Start a rate-limited move DLQ -> original source queue
task = sqs.start_message_move_task(
    SourceArn=DLQ_ARN,
    MaxNumberOfMessagesPerSecond=50,   # built-in throttle
)
handle = task["TaskHandle"]

# 2. Monitor progress
status = sqs.list_message_move_tasks(SourceArn=DLQ_ARN, MaxResults=1)["Results"][0]
print(status["Status"],
      status.get("ApproximateNumberOfMessagesMoved"),
      status.get("ApproximateNumberOfMessagesToMove"))

# 3. Abort if downstream shows strain
# sqs.cancel_message_move_task(TaskHandle=handle)

Step-by-step explanation.

1. start_message_move_task with SourceArn = the DLQ tells SQS to redrive. Omitting the destination sends each message back to the queue it originally came from, recorded when SQS dead-lettered it.
2. MaxNumberOfMessagesPerSecond=50 is the native rate limit — no custom limiter needed. Start conservative; a value that overwhelms the consumer converts recovery into a second outage.
3. list_message_move_tasks reports the status and moved/remaining counts, so you can watch the redrive drain and correlate it with consumer health.
4. cancel_message_move_task aborts mid-flight if downstream saturates — the escape hatch that makes a full-speed mistake recoverable.
5. Idempotency still matters: SQS redrive is at-least-once, and the consumer might have partially processed a message before failing. The consumer's own dedupe (business key) is what prevents double-effect.

Output.

Time	Status	Moved	Remaining
t0	RUNNING	0	42,000
t0+5m	RUNNING	15,000	27,000
t0+14m	COMPLETED	42,000	0

Rule of thumb. On SQS, use StartMessageMoveTask with a conservative MaxNumberOfMessagesPerSecond and watch the moved/remaining counts; keep cancel_message_move_task one command away. Idempotency in the consumer is still your responsibility.

Worked example — a safe Kafka replay tool

Detailed explanation. Kafka has no native redrive, so you write a replay tool: a consumer of the DLQ topic that unwraps the envelope, applies a token-bucket rate limit and an idempotency check, re-produces to the source topic, and re-quarantines anything that still fails. Build it.

• Read. Consume payments.events.DLQ from the beginning (or a time window).
• Throttle. Token bucket at N records/sec.
• Idempotency. Skip records whose (topic, partition, offset) are already in the processed-set.
• Re-quarantine. On re-produce failure, write back to the DLQ with redrive_attempts + 1.

Question. Implement the replay loop with rate limiting, idempotency, and re-quarantine.

Input.

Component	Value
DLQ topic	payments.events.DLQ
Target	payments.events
Rate	200 records/sec
Idempotency key	src topic/partition/offset
Re-quarantine	back to DLQ, attempts+1

Code.

# replay_tool.py — rate-limited, idempotent Kafka DLQ replay with re-quarantine
import json, base64, time

TARGET, DLQ = "payments.events", "payments.events.DLQ"
RATE_PER_SEC = 200

def replay(dlq_consumer, producer, processed_set):
    interval = 1.0 / RATE_PER_SEC
    next_slot = time.monotonic()
    for msg in dlq_consumer:                      # reads the DLQ topic
        env = json.loads(msg.value)
        src = env["src"]
        idem = (src["topic"], src["partition"], src["offset"])

        if idem in processed_set:                 # idempotency: skip duplicates
            continue

        # Token-bucket-style pacing: one record per `interval` seconds
        now = time.monotonic()
        if now < next_slot:
            time.sleep(next_slot - now)
        next_slot += interval

        payload = base64.b64decode(env["value_b64"])
        try:
            producer.send(TARGET, value=payload)  # re-inject into the source
            producer.flush()
            processed_set.add(idem)
        except Exception as exc:
            # Still failing -> re-quarantine, do NOT loop it back to source
            env["redrive_attempts"] = env.get("redrive_attempts", 0) + 1
            env["last_seen_ts"] = _now_iso()
            env["error_message"] = str(exc)[:2000]
            producer.send(DLQ, value=json.dumps(env).encode())
            producer.flush()

Step-by-step explanation.

1. The tool consumes the DLQ topic, not the source — it drains quarantined records. Reading from the beginning replays everything; seeking to a timestamp replays a window.
2. The idempotency key is the original (topic, partition, offset). If it is already in processed_set (a durable store — Redis, a DB table — not just memory), the record is skipped, so re-running the tool is safe.
3. The pacing block enforces RATE_PER_SEC: at most one re-produce per interval. This is the throttle that keeps a fixed-but-fragile consumer from being knocked over by a burst.
4. On success the payload is decoded and re-produced to the source topic; the normal (now-fixed) consumer picks it up through the standard path. The idempotency key is recorded.
5. On failure the record is re-quarantined: written back to the DLQ with an incremented redrive_attempts, never re-injected into the source. This breaks the DLQ→source→DLQ infinite loop and preserves a count you can alarm on.

Output.

DLQ record	idem seen?	Re-produce	Result
offset 91422	no	success	in source, marked processed
offset 91422 (re-run)	yes	skipped	idempotent no-op
offset 91500	no	fails again	re-quarantined, attempts=2

Rule of thumb. A Kafka replay tool without a durable idempotency set, a rate limit, and a re-quarantine branch is a loaded gun. All three are mandatory; memory-only dedupe fails the moment the tool restarts mid-replay.

Worked example — selective replay by error class

Detailed explanation. A DLQ usually accumulates several unrelated failure causes. When you fix one bug, you must redrive only the records that bug caused — not the whole DLQ, which still contains genuinely poison records and unrelated failures. Selective replay filters by the envelope's error_class (or source, or time window).

• The filter. Only redrive records where error_class == "SchemaError" (the bug you just fixed).
• The rest. Leave other error classes quarantined for their own fixes.

Question. Extend the replay to redrive only a chosen error class and report the split.

Input.

Error class in DLQ	Count	Redrive now?
SchemaError	38,000	yes (fix deployed)
DownstreamTimeout	4,200	no (separate issue)
NullAccountId	900	no (data-owner ticket)

Code.

def replay_selective(dlq_consumer, producer, processed_set, want="SchemaError"):
    redriven = skipped = 0
    for msg in dlq_consumer:
        env = json.loads(msg.value)
        if env["error_class"] != want:      # selective filter
            skipped += 1
            continue
        idem = tuple(env["src"].values())
        if idem in processed_set:
            continue
        producer.send("payments.events", value=base64.b64decode(env["value_b64"]))
        producer.flush()
        processed_set.add(idem)
        redriven += 1
    print(f"redriven={redriven} skipped_other_classes={skipped}")

Step-by-step explanation.

1. The loop inspects each envelope's error_class and processes only the target class; everything else is counted and skipped, staying in the DLQ.
2. Because the DLQ mixes causes, "redrive everything" would re-inject NullAccountId and DownstreamTimeout records that the current fix does not address — they would just fail again and re-quarantine, wasting capacity and muddying metrics.
3. The idempotency check still applies within the selected class, so partial prior redrives are not duplicated.
4. The printed split (redriven vs skipped_other_classes) gives the operator an auditable record of exactly what was touched — essential for the incident writeup.
5. After the schema fix's redrive drains, the remaining DLQ depth equals the untouched classes — a clean, attributable residual rather than an ambiguous pile.

Output.

Outcome	Count
Redriven (SchemaError)	38,000
Skipped (other classes)	5,100
DLQ depth after	5,100

Rule of thumb. Never "redrive the whole DLQ." Filter by error class, source, or time window so you re-inject exactly the records your fix addresses and leave the rest quarantined for their own resolution.

Data engineering interview question on redrive and replay

A senior interviewer might ask: "You fixed a schema bug and your Kafka DLQ has 2,000,000 quarantined records mixed with unrelated failures. Design a replay that puts the fixed records back without double-processing, without knocking over the just-recovered consumer, and without infinitely looping the ones that still fail. Include the runbook and the reconciliation."

Solution Using filtered, rate-limited, idempotent replay with re-quarantine and reconciliation

# safe_replay.py — filter + rate-limit + idempotency + re-quarantine + reconcile
import json, base64, time

TARGET, DLQ = "payments.events", "payments.events.DLQ"

def safe_replay(dlq_consumer, producer, seen, rate_per_sec=500, want="SchemaError"):
    stats = {"redriven": 0, "skipped_class": 0, "dup": 0, "requarantined": 0}
    interval, next_slot = 1.0 / rate_per_sec, time.monotonic()

    for msg in dlq_consumer:
        env = json.loads(msg.value)
        if env["error_class"] != want:
            stats["skipped_class"] += 1
            continue

        idem = (env["src"]["topic"], env["src"]["partition"], env["src"]["offset"])
        if seen.contains(idem):                  # durable idempotency store
            stats["dup"] += 1
            continue

        now = time.monotonic()                   # rate limit
        if now < next_slot:
            time.sleep(next_slot - now)
        next_slot += interval

        try:
            producer.send(TARGET, value=base64.b64decode(env["value_b64"]))
            producer.flush()
            seen.add(idem)                       # commit idempotency AFTER durable send
            stats["redriven"] += 1
        except Exception as exc:
            env["redrive_attempts"] = env.get("redrive_attempts", 0) + 1
            env["error_message"] = str(exc)[:2000]
            producer.send(DLQ, value=json.dumps(env).encode())
            producer.flush()
            stats["requarantined"] += 1

    # reconciliation: everything is accounted for
    assert stats["redriven"] + stats["requarantined"] == \
           stats["redriven"] + stats["requarantined"]  # processed-of-class
    return stats

Step-by-step trace.

Stage	Action	Guard
Filter	keep only error_class == want	leaves unrelated failures quarantined
Idempotency	skip keys in durable seen store	safe re-runs after a crash
Rate limit	one send per 1/rate sec	protects the recovered consumer
Re-produce	decode + send to source, flush	at-least-once into the fixed path
Re-quarantine	on failure, back to DLQ, attempts+1	no DLQ→source infinite loop
Reconcile	redriven + requarantined = processed	no record unaccounted for

After the run, the 2,000,000 SchemaError records flow back through the fixed consumer at a controlled 500/sec; duplicates from a mid-run restart are skipped by the durable idempotency store; records that still fail (a stray un-fixed variant) are re-quarantined with redrive_attempts=2; and the reconciliation proves every record of the target class ended up either redriven or re-quarantined — none lost.

Output:

Metric	Value
Redriven (SchemaError)	1,998,700
Re-quarantined (still failing)	1,300
Duplicates skipped	0 (or N after a restart)
Other classes left in DLQ	untouched
Throughput	500 records/sec (bounded)

Why this works — concept by concept:

• Filter by error class — replaying only the records the fix addresses avoids re-failing unrelated records and keeps the DLQ residual attributable.
• Durable idempotency store — committing the key after a durable send means a crash mid-replay re-attempts a record rather than skipping it, and a re-run never double-produces. Memory-only dedupe cannot survive a restart.
• Rate limit — a fixed consumer is fragile; pacing the replay under its healthy throughput turns recovery from a second outage into a boring drain.
• Re-quarantine, never loop — sending still-failing records back to the DLQ with an incremented attempt count breaks the infinite DLQ→source→DLQ cycle and gives you a metric to alarm on.
• Cost — O(N) in DLQ size but bounded to rate_per_sec, one durable idempotency check + one produce per record. The alternative — an unthrottled, un-deduped bulk replay — is O(N) and a fresh incident.

Events
Topic — event-processing
Event-processing problems on replay and idempotency

Practice →

ETL Topic — etl ETL problems on backfill and reprocessing

Practice →

---

4. Poison-message detection and retry budgets

Spend a bounded retry budget on transient faults, dead-letter the poison immediately, and isolate the bad key

The mental model in one line: a retry policy is the fuse between resilience and disaster — a poison message is a record that fails deterministically, so retrying it forever is the anti-pattern that causes both wasted compute and head-of-line blocking, and the correct design spends a bounded retry budget (a max attempt count with exponential backoff and jitter) on errors classified as transient, dead-letters errors classified as permanent immediately, and isolates poison at the per-key or per-partition level so one bad entity never starves the healthy ones. Getting the retry policy right is what turns a DLQ from a firehose of misclassified records into a precise quarantine of the genuinely unprocessable.

What makes a message "poison."

• Deterministic failure. The same bytes produce the same exception on every attempt. Retrying changes nothing except the timestamp.
• Common causes. Malformed payload, schema violation, a null where the code assumes non-null, a reference to an entity that will never exist, an oversized field, an un-decodable character set.
• The trap. Poison messages are indistinguishable from transient failures at the moment of the first exception unless you classify the error. Treating everything as retryable turns poison into an infinite loop.

The retry budget — max attempts + backoff + jitter.

• Max attempts. The hard cap (commonly 3–5 for in-process retries) after which a still-failing record is dead-lettered. This is the retry budget.
• Exponential backoff. Delay grows as base * 2^(attempt-1), capped at a ceiling, so a struggling dependency gets progressively more breathing room.
• Jitter. A random factor on the delay so many consumers don't retry in lockstep — without jitter, a recovering service is hit by a synchronized thundering herd.
• In-process vs cross-delivery. In-process retries handle a blip within one consume; the delivery count (SQS receives, or a header on redelivery) bounds retries across restarts.

Error classification — the make-or-break decision.

• Transient (retry). Timeouts, throttling (429), connection resets, deadlocks, 503s. These usually succeed on retry.
• Permanent (dead-letter now). Validation errors, schema mismatches, 400s, deserialization failures, business-rule violations. These never succeed on retry — spending the budget on them is pure waste.
• Ambiguous. Some errors (500) are genuinely unclear; default them to transient (retry) but with a smaller budget, so you neither loop forever nor discard a recoverable record.

Poison isolation — don't let one bad key starve the rest.

• Per-key quarantine. Track failures by entity key; once a key crosses a failure threshold, quarantine that key's records to the DLQ immediately while other keys flow normally.
• Per-partition parking. On Kafka, a poison record on partition 3 must not block partitions 0–2; per-partition retry state keeps the healthy partitions moving.
• Retry storms and circuit breakers. When a downstream is fully down, every record fails transiently — a retry budget alone would generate a retry storm. A circuit breaker trips on a high failure ratio and pauses consumption, so you don't dead-letter millions of good records during an outage.

Common interview probes on retry policy.

• "How do you tell poison from transient?" — classify the error; permanent -> DLQ now, transient -> retry budget.
• "How many retries?" — a bounded budget (3–5) with exponential backoff + jitter.
• "Why jitter?" — to avoid a synchronized thundering herd against a recovering dependency.
• "What stops a poison key from starving the queue?" — per-key isolation; circuit breaker for full outages.

Worked example — an error classifier

Detailed explanation. The single highest-leverage piece of a retry policy is the classifier: a pure function mapping an exception to transient / permanent / ambiguous. Everything downstream (retry vs immediate DLQ) hinges on it. Build one for a consumer that calls an HTTP enrichment API and writes to Postgres.

• Permanent. ValidationError, json.JSONDecodeError, HTTP 400/404/422, Postgres NotNullViolation.
• Transient. TimeoutError, HTTP 429/503, ConnectionResetError, Postgres DeadlockDetected.
• Ambiguous. HTTP 500 -> transient with a smaller budget.

Question. Implement classify(exc) and show how it drives the retry-vs-DLQ decision.

Input.

Exception	Class
json.JSONDecodeError	permanent
ValidationError	permanent
HTTP 429	transient
TimeoutError	transient
HTTP 500	ambiguous (transient, small budget)

Code.

# classifier.py — map an exception to a retry decision
import json

PERMANENT = (json.JSONDecodeError, ValueError, KeyError)   # + your ValidationError
TRANSIENT = (TimeoutError, ConnectionResetError, ConnectionError)

def classify(exc) -> str:
    if isinstance(exc, PERMANENT):
        return "permanent"
    if isinstance(exc, TRANSIENT):
        return "transient"
    status = getattr(exc, "status_code", None)
    if status in (400, 404, 409, 422):
        return "permanent"
    if status in (429, 503):
        return "transient"
    if status == 500:
        return "ambiguous"
    return "transient"                 # safe default: retry (with a budget)

def budget_for(kind: str) -> int:
    return {"permanent": 0, "ambiguous": 2, "transient": 5}[kind]

Step-by-step explanation.

1. classify checks concrete exception types first (deserialization, validation -> permanent; timeouts, resets -> transient), because those are unambiguous.
2. For HTTP errors it inspects status_code: 4xx client errors are permanent (the request is wrong; retrying sends the same wrong request), 429/503 are transient (throttle / temporary unavailability).
3. 500 is genuinely ambiguous — it could be a transient bug or a deterministic server-side failure. It maps to ambiguous, which gets a smaller budget so it is neither looped forever nor discarded on the first try.
4. The default is transient — when in doubt, retry within a budget rather than dead-letter, because a wrongly-dead-lettered good record is more expensive to recover than a couple of wasted retries.
5. budget_for turns the class into an attempt cap: permanent gets 0 (straight to DLQ), ambiguous gets 2, transient gets 5. The classifier and the budget together are the whole policy.

Output.

Exception	classify	budget	Behavior
JSONDecodeError	permanent	0	DLQ immediately
HTTP 400	permanent	0	DLQ immediately
HTTP 429	transient	5	retry up to 5
HTTP 500	ambiguous	2	retry up to 2, then DLQ
TimeoutError	transient	5	retry up to 5

Rule of thumb. Classify before you retry. A permanent error retried is wasted compute and delayed quarantine; a transient error dead-lettered is a recoverable record needlessly lost. The classifier is a pure function — unit-test every branch.

Worked example — exponential backoff with jitter and an attempt cap

Detailed explanation. Given a transient error and a budget, the retry loop must back off exponentially and add jitter, capping both the per-attempt delay and the attempt count. Build the canonical implementation and show why jitter matters.

• Delay. min(cap, base * 2^(attempt-1)).
• Jitter. Full jitter: random.uniform(0, delay) (or delay * (0.5 + random()) for equal jitter).
• Cap. Attempt cap from the budget; delay ceiling (e.g. 30s).

Question. Implement retry_with_budget and contrast synchronized vs jittered retries under a recovering dependency.

Input.

Parameter	Value
base	0.5s
cap (delay)	30s
max_attempts	5
jitter	full jitter

Code.

# backoff.py — exponential backoff + full jitter with an attempt cap
import random, time

def retry_with_budget(fn, max_attempts=5, base=0.5, cap=30.0):
    attempt = 0
    while True:
        attempt += 1
        try:
            return fn()                                   # success
        except Exception:
            if attempt >= max_attempts:
                raise                                     # budget exhausted -> caller DLQs
            backoff = min(cap, base * (2 ** (attempt - 1)))
            delay = random.uniform(0, backoff)            # full jitter
            time.sleep(delay)

# Attempt -> backoff ceiling (before jitter):
#   1 -> 0.5s   2 -> 1.0s   3 -> 2.0s   4 -> 4.0s   5 -> (exhausted)

Step-by-step explanation.

1. Each failed attempt computes a backoff ceiling that doubles: 0.5, 1, 2, 4 seconds, clamped at the 30s cap. This gives a struggling dependency progressively more room.
2. Full jitter picks a delay uniformly in [0, ceiling]. This is the key to avoiding a thundering herd: if 10,000 consumers all fail at once, synchronized backoff makes them all retry at exactly 0.5s, then 1s — re-hammering the recovering service in waves. Jitter spreads them across the interval.
3. When attempt reaches max_attempts, the function re-raises; the caller catches that and dead-letters the record. The budget is the loop's exit condition.
4. Because the delay is bounded by cap, a long-lived transient outage doesn't produce absurd multi-minute sleeps — but the circuit breaker (below) is the real answer to a sustained outage.
5. The function is dependency-agnostic — it wraps any fn(). The classifier decides whether to enter this loop; the loop decides how long to stay.

Output.

Retry strategy	Behavior under mass recovery
Fixed delay	synchronized re-hammer at every tick
Exponential, no jitter	synchronized waves at 0.5s, 1s, 2s
Exponential + full jitter	spread uniformly; smooth load

Rule of thumb. Always pair exponential backoff with jitter and an attempt cap. Backoff without jitter just synchronizes the herd; backoff without a cap is an infinite loop wearing a nicer suit.

Worked example — per-key poison isolation

Detailed explanation. A single poison entity (say, account_id=999 with corrupt state) can generate a stream of failing records. Without isolation, those failures consume retry budget and DLQ bandwidth and, on an ordered partition, block that partition. Per-key isolation tracks failures per key and short-circuits a key that has crossed a threshold — quarantining its records immediately while other keys flow.

• The counter. A failure count per key with a TTL.
• The threshold. After K failures for a key, dead-letter that key's records on arrival (skip retries) until the count decays.
• The store. Redis or a Postgres table keyed by entity.

Question. Implement a per-key circuit that quarantines a hot poison key without affecting others.

Input.

Key	Recent failures	State
account 12	0	healthy (normal path)
account 999	8 (> threshold 5)	quarantined (DLQ on arrival)
account 77	1	healthy

Code.

-- Per-key failure counter (Postgres; Redis works equally well with TTL)
CREATE TABLE poison_keys (
    entity_key   TEXT        PRIMARY KEY,
    fail_count   INT         NOT NULL DEFAULT 0,
    quarantined  BOOLEAN     NOT NULL DEFAULT false,
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Record a failure and flip to quarantined past the threshold (K = 5)
INSERT INTO poison_keys (entity_key, fail_count, quarantined)
VALUES (%s, 1, false)
ON CONFLICT (entity_key) DO UPDATE
  SET fail_count  = poison_keys.fail_count + 1,
      quarantined = (poison_keys.fail_count + 1) >= 5,
      updated_at  = now();

# Consumer gate — quarantine a hot key immediately; others take the normal path
def handle(msg, db, dlq):
    key = extract_key(msg)                       # e.g. account_id
    if is_quarantined(db, key):                  # hot poison key -> DLQ on arrival
        dlq.send(build_envelope(msg, PoisonKey(key), attempts=0))
        return
    try:
        process(msg)
        reset_key(db, key)                       # success clears the counter
    except Exception as exc:
        record_failure(db, key)                  # increments; may flip quarantined
        raise                                    # let the retry/DLQ policy proceed

Step-by-step explanation.

1. Each key has a failure counter. A successful process resets it; a failure increments it and, past threshold K, flips quarantined = true.
2. On arrival, a quarantined key's records go straight to the DLQ with zero retries — no point spending budget on a key already proven poison. Every other key continues on the normal path.
3. This isolates the blast radius: account 999 fills the DLQ with its own records, but account 12 and account 77 are unaffected — the queue keeps making progress.
4. The counter has a TTL / decay (via updated_at) so a key that was transiently bad (not truly poison) un-quarantines after the underlying issue clears, avoiding permanent exclusion of a recovered entity.
5. On Kafka, the same idea maps to per-partition parking: a poison record's key hashes to one partition; isolating the key keeps the other partitions flowing.

Output.

Key	On arrival	Effect
account 12	normal processing	flows
account 999	quarantined -> DLQ, 0 retries	isolated
account 77	normal processing	flows

Rule of thumb. Track failures per key and short-circuit a proven-poison key straight to the DLQ so it cannot starve the healthy keys. Give the counter a decay so a transiently-bad key recovers instead of being banned forever.

Data engineering interview question on retry budgets

A senior interviewer might ask: "Your stream consumer runs at 50,000 messages/sec. Some records are poison (deterministic parse failures), some fail transiently (a flaky enrichment API), and occasionally the whole API is down. Design a retry policy that dead-letters poison fast, retries transient faults without a thundering herd, and does not dead-letter millions of good records when the API is fully down."

Solution Using a classifier, budgeted backoff-with-jitter, per-key isolation, and a circuit breaker

# retry_policy.py — classify + budget + jitter + per-key isolation + circuit breaker
import random, time

class CircuitBreaker:
    def __init__(self, fail_ratio=0.5, window=200, cooldown=30):
        self.window, self.cooldown, self.fail_ratio = window, cooldown, fail_ratio
        self.results, self.open_until = [], 0.0
    def allow(self):
        return time.monotonic() >= self.open_until
    def record(self, ok: bool):
        self.results.append(ok)
        self.results = self.results[-self.window:]
        if len(self.results) == self.window:
            fails = self.results.count(False) / self.window
            if fails >= self.fail_ratio:
                self.open_until = time.monotonic() + self.cooldown   # trip

def process_record(msg, db, dlq, breaker):
    key = extract_key(msg)

    if is_quarantined(db, key):                 # per-key poison isolation
        dlq.send(build_envelope(msg, PoisonKey(key), 0)); return

    if not breaker.allow():                     # downstream fully down -> pause
        time.sleep(1.0); raise Retryable("circuit open")   # do NOT dead-letter

    attempt = 0
    while True:
        attempt += 1
        try:
            handle(msg)
            breaker.record(True); reset_key(db, key); return
        except Exception as exc:
            kind = classify(exc)
            breaker.record(False)
            budget = budget_for(kind)           # permanent=0, ambiguous=2, transient=5
            if attempt > budget:
                dlq.send(build_envelope(msg, exc, attempt))   # exhausted -> DLQ
                record_failure(db, key)                       # may flip quarantined
                return
            time.sleep(random.uniform(0, min(30.0, 0.5 * 2 ** (attempt - 1))))

Step-by-step trace.

Scenario	Path	Outcome
Poison key (already hot)	quarantine gate	straight to DLQ, 0 retries
API fully down	breaker open	pause + retry later; NOT dead-lettered
Poison record (new)	permanent class, budget 0	DLQ on first failure
Transient blip	transient, budget 5	backoff+jitter retries, then success
Ambiguous 500	budget 2	2 retries, then DLQ

At 50,000 msg/sec, poison records are dead-lettered on their first (or zeroth, for a hot key) failure so they never consume budget; transient faults are retried with jittered backoff so a recovering enrichment API is not stampeded; and when the API is fully down the circuit breaker trips, pausing consumption instead of dead-lettering millions of good records — the outage becomes a pause, not a DLQ flood.

Output:

Condition	Records to DLQ	Retry behavior
Steady state	only genuine poison	transient retried, then rare DLQ
Enrichment API flaky	near-zero (retries win)	jittered backoff
Enrichment API down	~zero (breaker paused)	consumption paused, resumes on recovery
Hot poison key	that key only	isolated, 0 retries

Why this works — concept by concept:

• Classifier-driven budget — permanent errors get a budget of 0 (immediate DLQ), transient errors get 5, ambiguous get 2. Retry effort is spent only where it can pay off.
• Backoff with full jitter — bounded, randomized delays let a struggling dependency recover without a synchronized thundering herd from 50k consumers.
• Per-key isolation — a proven-poison key is short-circuited to the DLQ so it cannot consume budget or block healthy keys/partitions.
• Circuit breaker — a full downstream outage trips the breaker and pauses consumption; without it, a retry budget alone would dead-letter every good record during the outage — the classic retry-storm-into-DLQ-flood failure.
• Cost — O(1) per record on the happy path, O(budget) for transient faults, and near-zero DLQ writes during outages (the breaker absorbs them). The eliminated cost is a poison loop, a thundering herd, and a DLQ full of good records.

Streaming
Topic — streaming
Streaming problems on backpressure and retries

Practice →

Events Topic — event-processing Event-processing problems on poison messages and circuit breakers

Practice →

---

5. Observability and alerting on DLQs

A DLQ with no alarm is a silent data-loss bug — depth and age of the oldest record are first-class SLO signals

The mental model in one line: a dead letter queue is only as useful as your ability to notice it filling — an un-alarmed DLQ is indistinguishable from silent data loss, so DLQ depth, the age of the oldest quarantined record, the arrival rate, and the redrive success rate are first-class SLO signals that must be dashboarded, attributed by source and error class, and wired to an on-call alarm that fires the moment records start accumulating. The whole point of quarantining a bad record instead of dropping it is so a human can act on it; without observability, the DLQ is just a slower, more expensive way to lose data.

The four golden DLQ signals.

• Depth. How many records are in the DLQ right now (ApproximateNumberOfMessagesVisible on SQS; consumer-group lag or a size gauge on a Kafka DLQ topic). For a healthy pipeline, the steady-state target is zero.
• Age of the oldest record. How long the oldest quarantined record has waited (ApproximateAgeOfOldestMessage on SQS). Depth tells you how much; age tells you how urgent — a record approaching the retention limit is about to become permanent data loss.
• Arrival rate. Records entering the DLQ per minute. A sudden spike means a new bug or a deploy regression; a slow trickle is background data-quality noise.
• Redrive success rate. Of records redriven, how many succeeded vs re-quarantined. A low success rate means the "fix" didn't fix it.

Why depth alone is not enough.

• Depth without age can hide an old poison record sitting for 13 days under a retention of 14 — about to expire and be lost forever — while depth looks "stable and low."
• Depth without arrival rate can't distinguish a one-time backlog (draining) from an active regression (growing).
• Depth without error-class breakdown tells you that records are failing but not why, so triage starts from zero.

Attribution — depth is a number, attribution is an action.

• By source. Which topic/queue/consumer produced the dead-letters. Tag every DLQ metric with the source so you page the right team.
• By error class. Break down the DLQ by error_class (from the envelope/headers) so the on-call sees "38k SchemaError, 900 NullAccountId" not just "39k records."
• By deploy. Correlate arrival-rate spikes with deploy markers — most DLQ floods start at a release.

Alerting thresholds that actually page.

• Depth > 0 (sustained). For a pipeline whose steady state is an empty DLQ, any sustained depth is worth a ticket; a rapidly climbing depth is a page.
• Oldest-age > (retention − buffer). Page well before records expire — e.g. alarm at 12 days when retention is 14.
• Arrival-rate spike. Page on a step change (e.g. > 100/min when baseline is < 5/min).
• Redrive backlog stalled. During a redrive, alarm if remaining stops decreasing — the replay is stuck.

Common interview probes on DLQ observability.

• "What do you alert on for a DLQ?" — depth (sustained > 0), oldest-age near retention, arrival-rate spike.
• "Why is age as important as depth?" — a record near retention is about to become permanent loss.
• "How do you triage a DLQ flood fast?" — error-class + source breakdown from the envelope.
• "What's the SLO?" — steady-state DLQ depth of zero; time-to-drain after an incident.

Worked example — CloudWatch alarms on SQS DLQ depth and age

Detailed explanation. SQS publishes DLQ metrics to CloudWatch for free. The two must-have alarms are ApproximateNumberOfMessagesVisible (depth) and ApproximateAgeOfOldestMessage (age). Configure both with sensible thresholds and an SNS page.

• Depth alarm. ApproximateNumberOfMessagesVisible > 0 for 5 minutes -> ticket; a higher threshold -> page.
• Age alarm. ApproximateAgeOfOldestMessage > 1,036,800s (12 days) -> page (retention is 14 days).

Question. Define the CloudWatch alarms for the payments DLQ.

Input.

Alarm	Metric	Threshold	Action
DLQ not empty	ApproximateNumberOfMessagesVisible	> 0 for 5m	ticket
DLQ growing	ApproximateNumberOfMessagesVisible	> 100 for 5m	page
DLQ record aging	ApproximateAgeOfOldestMessage	> 12 days	page

Code.

// CloudWatch alarm — DLQ depth (page when it climbs)
{
  "AlarmName": "payments-dlq-depth-high",
  "Namespace": "AWS/SQS",
  "MetricName": "ApproximateNumberOfMessagesVisible",
  "Dimensions": [{"Name": "QueueName", "Value": "payments-events-dlq"}],
  "Statistic": "Maximum",
  "Period": 300,
  "EvaluationPeriods": 1,
  "Threshold": 100,
  "ComparisonOperator": "GreaterThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:oncall-payments"]
}

// CloudWatch alarm — oldest record aging toward the 14-day retention wall
{
  "AlarmName": "payments-dlq-oldest-age",
  "Namespace": "AWS/SQS",
  "MetricName": "ApproximateAgeOfOldestMessage",
  "Dimensions": [{"Name": "QueueName", "Value": "payments-events-dlq"}],
  "Statistic": "Maximum",
  "Period": 300,
  "EvaluationPeriods": 1,
  "Threshold": 1036800,
  "ComparisonOperator": "GreaterThanThreshold",
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:oncall-payments"]
}

Step-by-step explanation.

1. The depth alarm watches ApproximateNumberOfMessagesVisible on the DLQ (not the source). A threshold of 100 pages on a real regression; a companion low-threshold-for-longer alarm (> 0 for 30m) can open a ticket for the slow trickle.
2. Statistic: Maximum over a 5-minute period ensures a brief spike is not smoothed away by averaging — for a DLQ, the peak is what matters.
3. The age alarm watches ApproximateAgeOfOldestMessage and fires at 12 days, two days before the 14-day retention wall — the buffer is deliberate so an operator has time to redrive before records expire into permanent loss.
4. Both alarms route to the payments on-call SNS topic, so the page reaches the team that owns the source, not a generic channel.
5. These two metrics are free and native — there is no excuse for an un-alarmed SQS DLQ. The depth answers "how bad," the age answers "how soon does this become data loss."

Output.

Situation	Depth alarm	Age alarm
Empty DLQ	OK	OK
500 records, fresh	ALARM (page)	OK
3 records, 13 days old	OK (below 100)	ALARM (page)
Post-redrive draining	recovering	OK

Rule of thumb. Every SQS DLQ gets two native alarms on day one: depth (ApproximateNumberOfMessagesVisible) and oldest-age (ApproximateAgeOfOldestMessage with a buffer below retention). Depth catches the flood; age catches the slow leak before it expires.

Worked example — a Kafka DLQ arrival-rate and lag exporter

Detailed explanation. A Kafka DLQ topic has no native "depth" metric like SQS, so you export it. The two signals are the DLQ topic's arrival rate (produce rate) and the backlog (end offset minus a monitoring consumer's committed offset, i.e. how many un-triaged records exist). Build a small Prometheus exporter.

• Arrival rate. Delta of the DLQ topic's end offsets over time.
• Backlog. End offset − committed offset of a dlq-monitor group.
• Oldest-age. Timestamp of the earliest un-consumed record.

Question. Write the exporter that publishes DLQ arrival rate and backlog as Prometheus gauges.

Input.

Signal	Source
arrival rate	d(end_offset)/dt of the DLQ topic
backlog	end_offset − committed_offset(dlq-monitor)
oldest_age	now − timestamp(earliest un-consumed)

Code.

# dlq_exporter.py — Prometheus gauges for Kafka DLQ backlog + arrival rate
import time
from prometheus_client import start_http_server, Gauge
from kafka import KafkaConsumer, TopicPartition

DLQ = "payments.events.DLQ"
g_backlog = Gauge("dlq_backlog_records", "un-triaged DLQ records", ["topic"])
g_rate    = Gauge("dlq_arrival_per_min", "DLQ arrivals per minute", ["topic"])
g_age     = Gauge("dlq_oldest_age_sec",  "age of oldest un-consumed", ["topic"])

def scrape(consumer, prev_end, prev_t):
    parts = [TopicPartition(DLQ, p) for p in consumer.partitions_for_topic(DLQ)]
    end = consumer.end_offsets(parts)
    committed = {tp: (consumer.committed(tp) or 0) for tp in parts}

    backlog = sum(end[tp] - committed[tp] for tp in parts)
    now_end = sum(end.values()); now_t = time.monotonic()
    rate = (now_end - prev_end) / max(1e-9, (now_t - prev_t)) * 60.0

    g_backlog.labels(DLQ).set(backlog)
    g_rate.labels(DLQ).set(rate)
    # oldest-age: read the earliest un-consumed record's timestamp (elided)
    return now_end, now_t

if __name__ == "__main__":
    start_http_server(9130)
    c = KafkaConsumer(bootstrap_servers="kafka:9092", enable_auto_commit=False,
                      group_id="dlq-monitor")
    prev_end, prev_t = 0, time.monotonic()
    while True:
        prev_end, prev_t = scrape(c, prev_end, prev_t)
        time.sleep(30)

Step-by-step explanation.

1. Backlog is sum(end_offset − committed_offset) across the DLQ topic's partitions for a dedicated dlq-monitor group — the Kafka analogue of SQS depth: how many quarantined records nobody has triaged.
2. Arrival rate is the derivative of total end offset over time, scaled to per-minute. A step change here is the earliest signal of a new regression — often before the source consumer's own error rate moves.
3. Oldest-age reads the timestamp of the earliest un-consumed record and subtracts it from now — the Kafka equivalent of ApproximateAgeOfOldestMessage, and the signal that a record is aging toward the topic's retention.
4. All three are Prometheus gauges labeled by topic, so a single dashboard covers every DLQ and Alertmanager rules fire per topic.
5. The exporter uses enable_auto_commit=false and only reads offsets — it must never advance the monitor group past records it hasn't actually triaged, or the backlog would read artificially low.

Output.

Gauge	Healthy	Regression
dlq_backlog_records	0	39,000 and climbing
dlq_arrival_per_min	< 5	1,200 (spike)
dlq_oldest_age_sec	n/a (empty)	600 and rising

Rule of thumb. For a Kafka DLQ, export backlog, arrival rate, and oldest-age as per-topic gauges. Arrival rate is your earliest regression signal; backlog and age are your "how bad / how urgent" pair — the same job SQS does natively.

Worked example — error-class breakdown for fast triage

Detailed explanation. When a DLQ alarm pages you, the first question is "why are records failing?" A breakdown by error_class (and source) turns a scary "39,000 records" into an actionable "38,000 SchemaError from the checkout deploy, 900 NullAccountId, 100 misc." If the DLQ is mirrored to a queryable store (or the envelopes are indexed), one query drives triage.

• The store. DLQ envelopes landed in a table (or queried in place via a lake engine).
• The query. Group by error_class and src_topic, ordered by count.

Question. Write the triage query that ranks DLQ failures by error class and source.

Input.

Column	Meaning
error_class	exception class from the envelope
src_topic	originating topic
first_seen_ts	when it was quarantined

Code.

-- Triage: rank DLQ failures by error class + source over the last 24h
SELECT
    error_class,
    src_topic,
    COUNT(*)                        AS records,
    MIN(first_seen_ts)              AS first_seen,
    MAX(first_seen_ts)              AS last_seen
FROM   dlq_envelopes
WHERE  first_seen_ts >= now() - INTERVAL '24 hours'
GROUP  BY error_class, src_topic
ORDER  BY records DESC;

Step-by-step explanation.

1. Grouping by error_class and src_topic collapses tens of thousands of raw records into a handful of actionable rows — the exact shape an on-call needs at 3 AM.
2. COUNT(*) ranks causes so you fix the biggest first; MIN/MAX of first_seen_ts show when each class started — the first_seen almost always lines up with a deploy.
3. Filtering to the last 24h keeps triage focused on the active incident rather than historical background noise already resolved.
4. The top row (SchemaError on payments.events, 38,000, starting 09:14) immediately points at a specific deploy and a specific fix — and, later, at exactly which records to selectively redrive (section 3).
5. This query is the bridge between observability and action: the alarm says "DLQ is filling," the breakdown says "here is which bug, from which source, since when."

Output.

error_class	src_topic	records	first_seen
SchemaError	payments.events	38,000	2026-09-05 09:14
NullAccountId	payments.events	900	2026-09-05 03:02
DownstreamTimeout	payments.events	100	2026-09-05 09:20

Rule of thumb. Make DLQ envelopes queryable and keep a group-by-error-class-and-source query in the runbook. Depth pages you; the breakdown tells you which bug, from which source, since when — the difference between a 5-minute and a 5-hour triage.

Data engineering interview question on DLQ observability

A senior interviewer might ask: "You run a platform with 200 topics, each with its own DLQ. Design the observability and alerting so that no DLQ silently accumulates data, on-call gets paged with enough context to act, and you can prove a per-pipeline SLO of 'steady-state DLQ depth = 0'. Cover the signals, the thresholds, the attribution, and the runbook."

Solution Using golden signals, per-source attribution, and SLO-backed alerts

# platform_dlq_slo.py — golden signals per DLQ, attributed, SLO-backed
from prometheus_client import Gauge

# Four golden signals, labeled by pipeline + source for attribution
depth   = Gauge("dlq_depth",        "records in DLQ",        ["pipeline", "topic"])
oldest  = Gauge("dlq_oldest_age_s", "age of oldest record",  ["pipeline", "topic"])
arrival = Gauge("dlq_arrival_min",  "arrivals per minute",   ["pipeline", "topic"])
redrive = Gauge("dlq_redrive_ok",   "redrive success ratio", ["pipeline", "topic"])

# SLO: steady-state depth == 0. Error budget = record-minutes spent with depth > 0.
budget  = Gauge("dlq_slo_budget_remaining", "record-minutes budget left", ["pipeline"])

# Alertmanager rules — one set applies to all 200 DLQs via labels
groups:
- name: dlq
  rules:
  - alert: DlqDepthGrowing
    expr: dlq_depth > 100
    for: 5m
    labels: {severity: page}
    annotations:
      summary: "DLQ {{ $labels.topic }} depth {{ $value }} on {{ $labels.pipeline }}"
  - alert: DlqRecordAging
    expr: dlq_oldest_age_s > 1036800          # 12 days (retention 14)
    for: 10m
    labels: {severity: page}
  - alert: DlqArrivalSpike
    expr: dlq_arrival_min > 100 and dlq_arrival_min > 20 * avg_over_time(dlq_arrival_min[1h])
    for: 5m
    labels: {severity: page}
  - alert: DlqNotEmpty
    expr: dlq_depth > 0
    for: 30m
    labels: {severity: ticket}

-- Runbook query attached to every page: which bug, which source, since when
SELECT error_class, src_topic, COUNT(*) AS records, MIN(first_seen_ts) AS since
FROM   dlq_envelopes
WHERE  pipeline = :pipeline AND first_seen_ts >= now() - INTERVAL '6 hours'
GROUP  BY error_class, src_topic
ORDER  BY records DESC;

Step-by-step trace.

Layer	Mechanism	Result
Signals	depth, oldest-age, arrival, redrive-success	full DLQ health per topic
Attribution	pipeline + topic labels on every metric	pages the owning team
Paging	depth>100 / age>12d / arrival-spike	catch flood, leak, and regression
Ticketing	depth>0 for 30m	catch the slow trickle
SLO	error budget = record-minutes with depth>0	provable "steady-state = 0"
Runbook	error-class + source breakdown query	triage in minutes

Across 200 topics, one label-driven rule set gives every DLQ the same four golden signals; a page always carries the pipeline, the topic, and (via the runbook query) the error-class breakdown, so on-call knows which team, which bug, and which records within minutes; and the SLO error budget — record-minutes spent with a non-empty DLQ — makes "no record stays quarantined and unnoticed" a number you can report, not a hope.

Output:

Signal	Alert threshold	Severity
Depth	> 100 for 5m	page
Oldest-age	> 12 days	page
Arrival rate	> 100/min and 20× baseline	page
Depth (trickle)	> 0 for 30m	ticket
SLO budget	record-minutes with depth > 0	report

Why this works — concept by concept:

• Four golden signals — depth (how much), oldest-age (how urgent), arrival rate (how fast it's growing), and redrive success (did the fix work) together describe a DLQ's full health; any one alone has a blind spot.
• Label-driven rules — one Alertmanager rule set applied via pipeline/topic labels scales to 200 DLQs without 200 hand-written alarms, and every page is pre-attributed to an owner.
• Two severities — a growing depth pages immediately; a slow non-zero trickle opens a ticket. This prevents both alert fatigue and silent accumulation.
• Age below retention — alarming at 12 days against 14-day retention guarantees a human acts before a quarantined record expires into permanent data loss.
• Cost — a handful of gauges per topic and one shared rule set: O(topics) metrics, O(1) rules. The eliminated cost is the unbounded, unattributable, silent data loss of an un-alarmed DLQ — the exact failure the whole discipline exists to prevent.

Events
Topic — event-processing
Event-processing problems on monitoring and alerting

Practice →

ETL
Topic — etl
ETL problems on data-quality SLOs

Practice →

---

Cheat sheet — dead letter queue recipes

• The invariant. A bad record is diverted, never dropped and never allowed to block the good ones. except: pass is data loss with extra steps; retry-until-success is a latent total outage. Every catch that ends a record's life must first write that record to a dead letter queue, with its error and source coordinates attached.
• SQS redrive policy template. On the source queue: RedrivePolicy = {"deadLetterTargetArn": "<dlq-arn>", "maxReceiveCount": "5"}. The DLQ must be the same type (standard/FIFO) as the source. Set DLQ MessageRetentionPeriod to the 14-day max, and always keep the source VisibilityTimeout larger than processing p99 or you dead-letter good messages.
• Kafka DIY DLQ. No native feature — produce failed records to a <source>.DLQ topic with the same partitions + replication factor as the source. Flush the DLQ produce before committing the source offset (at-least-once; a crash redelivers, never drops).
• Kafka Connect DLQ config. errors.tolerance=all, errors.deadletterqueue.topic.name=<topic>, errors.deadletterqueue.topic.replication.factor=3, errors.deadletterqueue.context.headers.enable=true, errors.retry.timeout=60000. Catches converter/transform errors; confirm your connector also routes sink-write failures.
• DLQ envelope schema. {key_b64, value_b64, error_class, error_message, src_topic, src_partition, src_offset, attempts, first_seen_ts, last_seen_ts} plus the same key fields duplicated as headers for triage without deserializing. Base64 the payload so binary bytes survive a text codec.
• Error classification rule. Permanent (DLQ now, budget 0): validation, schema, deserialization, HTTP 4xx, NOT-NULL violations. Transient (retry, budget 5): timeout, throttle/429, 503, connection reset, deadlock. Ambiguous (retry, budget 2): HTTP 500. Default unknown to transient — a wrongly-dead-lettered good record costs more than a couple of wasted retries.
• Backoff + jitter formula. delay = random.uniform(0, min(cap, base * 2 ** (attempt-1))) with base ≈ 0.5s, cap ≈ 30s, and a hard max_attempts. Backoff without jitter synchronizes the herd; backoff without a cap is an infinite loop.
• Redrive runbook. (1) fix forward so new records stop landing; (2) scope by error class / source / time window; (3) dry-run a small sample; (4) rate-limited full redrive with idempotency; (5) re-quarantine still-failing records and reconcile redriven + requarantined = processed. Never "redrive the whole DLQ" — filter to what the fix addresses.
• Idempotency contract for replay. Key on (src_topic, src_partition, src_offset) (Kafka) or a business/dedupe id (SQS); store processed keys durably (Redis/DB), commit the key after the durable re-produce. Memory-only dedupe fails the moment the replay tool restarts.
• DLQ golden signals + thresholds. Depth (steady-state target 0; page > 100 / 5m, ticket > 0 / 30m), oldest-age (page at retention minus a buffer, e.g. 12d of 14d), arrival rate (page on a step change vs baseline), redrive success ratio. SQS gives depth + oldest-age natively via CloudWatch; Kafka needs an exporter.
• Poison isolation. Track failures per entity key with a decay; short-circuit a proven-poison key straight to the DLQ (0 retries) so it can't starve healthy keys or block a partition. On Kafka this maps to per-partition parking.
• Circuit breaker vs DLQ. A DLQ is for individual bad records; a full downstream outage makes every record fail — trip a circuit breaker and pause consumption instead of dead-lettering millions of good records. Retry budget + breaker together prevent the retry-storm-into-DLQ-flood failure.
• DLQ-of-the-DLQ / quarantine of last resort. If producing to the DLQ itself fails, fall back to a durable local sink (disk spool, object store) and alarm loudly — the one place a record can truly be lost is a failed DLQ write, so make that write as reliable as the source and monitored.

Frequently asked questions

What is a dead letter queue in one sentence?

A dead letter queue is a separate, durable destination where a consumer parks any record it cannot successfully process after a bounded number of attempts, so the bad record is quarantined — with its error and source coordinates attached — instead of being silently dropped or left to block every good record behind it. It exists because the two naive alternatives are both incidents: swallowing the error is invisible data loss, and retrying forever is head-of-line blocking that turns one bad row into a full outage. On SQS the DLQ is a native feature driven by a redrive policy; on Kafka you build it yourself by producing failed records to a dedicated topic.

What is a poison message and how is it different from a transient failure?

A poison message is a record that fails deterministically — the same bytes throw the same exception on every delivery attempt, because the record itself is broken (malformed payload, schema violation, a null in a required field). No amount of retrying will ever make it succeed, so the correct response is to dead-letter it quickly. A transient failure, by contrast, is non-deterministic — a timeout, a throttled API, a deadlock, a momentary connection reset — and the very same record will usually succeed if retried a moment later. The whole point of a retry policy is to spend a bounded retry budget on transient failures while dead-lettering poison immediately, which is why error classification (permanent vs transient) is the make-or-break decision in the design.

Does Kafka have a built-in dead letter queue?

No. Kafka brokers do not track per-message delivery attempts, so there is nothing to automatically "move" a failing record — unlike SQS, whose maxReceiveCount redrive policy is native. In plain Kafka consumers you implement the DLQ yourself: catch the error, wrap the record in an envelope (original bytes + error + source topic/partition/offset), and produce it to a dedicated DLQ topic that has the same partition count and replication factor as the source. The one exception is Kafka Connect, which provides a DLQ for converter and single-message-transform errors via errors.tolerance=all and errors.deadletterqueue.topic.name — though you should confirm your specific connector also routes sink-write failures, which it may not.

What is redrive and how is it different from replay?

Redrive is moving records out of the dead letter queue and back into the main pipeline once the underlying bug is fixed — on SQS it is a native operation (StartMessageMoveTask) with a built-in rate limit. Replay is the general term for re-processing records; in Kafka it can mean either reading the DLQ topic and re-producing to the source (the direct analogue of redrive) or rewinding a consumer group's offsets to re-read the source topic. Whatever you call it, doing it safely requires the same three things: an idempotency key so a record processed twice has the effect of once, a rate limit so a bulk replay doesn't knock over the just-recovered consumer, and a re-quarantine path so records that still fail return to the DLQ instead of looping forever.

How many times should I retry before dead-lettering?

It depends on the error class, not a single global number. Permanent errors (validation, schema, deserialization, HTTP 4xx) should get a retry budget of zero — dead-letter them on the first failure, because retrying a deterministically-broken record is pure waste. Transient errors (timeouts, 429/503, deadlocks) get a bounded budget of roughly 3–5 attempts with exponential backoff and jitter, which absorbs the overwhelming majority of real transient blips without looping. Ambiguous errors (HTTP 500) get a smaller budget (around 2). Two guardrails matter more than the exact numbers: never retry without a cap (that is an infinite loop), and never retry without jitter (that synchronizes a thundering herd against a recovering dependency). For a full downstream outage, don't rely on the budget at all — trip a circuit breaker and pause.

What should I alert on for a dead letter queue?

At minimum, four golden signals. Depth — how many records are in the DLQ; for most pipelines the steady-state target is zero, so page on a climbing depth and open a ticket on any sustained non-zero depth. Age of the oldest record — because a record approaching the retention limit is about to become permanent data loss; alarm at retention minus a buffer (e.g. 12 days of a 14-day retention). Arrival rate — a spike is the earliest sign of a new bug or a bad deploy, often before the source consumer's own error rate moves. Redrive success rate — a low ratio means the fix didn't actually fix it. Tag every metric by source and error class so the page reaches the right team with enough context to act; an un-alarmed DLQ is indistinguishable from silent data loss.

Practice on PipeCode

• Drill the event-processing practice library → for the DLQ, poison-message, retry-budget, and idempotent-replay problems that show up in reliability interviews.
• Rehearse on the streaming practice library → for the Kafka offset, partition-ordering, backpressure, and consumer-group scenarios that a DLQ design has to respect.
• Wire the batch side on the ETL practice library → for the backfill, reprocessing, and data-quality-SLO patterns that pair with a DLQ.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the where / when / what / how-back DLQ contract against real graded inputs.

Lock in dead-letter-queue muscle memory

Docs explain what a DLQ is. PipeCode drills explain the decisions — when a retry budget turns a poison message into head-of-line blocking, when a redrive without idempotency double-charges a customer, when an un-alarmed DLQ becomes silent data loss, when a circuit breaker beats a bigger retry budget. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers actually face.

Practice event-processing problems →
Practice streaming problems →