Originally published on kuryzhev.cloud
A Lambda function can have a dead letter queue configured in the console, pass every health check, and still lose events. The usual reason: the DLQ that got wired up isn't the mechanism that actually handles the failure mode hitting production. AWS uses "dead letter queue" loosely across three unrelated systems, and teams frequently configure one, confirm it exists in the console, and assume it covers everything. Which mechanism fires depends entirely on how the function is invoked, and that detail is easy to miss during setup.
There are three different "DLQs" — know which one you're configuring
The function-level DeadLetterConfig applies only to asynchronous (Event) invocations. It does nothing for synchronous calls or for event source mappings like SQS, Kinesis, or DynamoDB Streams — those rely on the event source mapping's OnFailure destination, a setting that lives separately from the function's own DLQ field.
For SQS-triggered Lambdas, neither of the above matters as much as the source queue's redrive policy. A function polling from SQS that keeps failing relies on maxReceiveCount configured on that queue to move the message to its DLQ — the function's own config plays no role. Watch out for this exact gap: a team sets a function-level DLQ, sees it in the console, and assumes SQS failures are covered, when the SQS trigger path never touches that field at all.
Use on-failure Destinations instead of legacy DLQ for async invokes
For asynchronous invocations — S3 events, SNS, EventBridge-triggered functions — AWS documents DestinationConfig.OnFailure as the current recommended path over the legacy function DLQ. A destination can point at SNS, SQS, EventBridge, or another Lambda, and unlike a plain DLQ it captures the full requestContext, the original payload, and the response or error details. A raw DLQ stores only the event, with none of that diagnostic context.
Destinations are also decoupled from the invocation path in a way legacy DLQs aren't: a failed DLQ write can cause the invocation itself to fail, while a destination failure doesn't block processing. Either way, the execution role needs explicit permission on the target — sqs:SendMessage or sns:Publish for a legacy DLQ, and the equivalent action for whatever the destination points at. This is where the confusion usually starts, because that permission requirement is real for legacy DLQs and destinations but doesn't apply at all to SQS source-queue redrive, covered next. Routing failures through EventBridge as the destination adds another option worth considering: rules can split failures by error type into separate targets instead of dumping everything into one undifferentiated queue. See the AWS Lambda asynchronous invocation documentation for current field names and constraints.
For SQS-triggered functions, fix the redrive policy — not the function's DLQ
When a function consumes from SQS, the redrive policy on that source queue determines failure handling, not anything set on the Lambda side. maxReceiveCount sets how many failed poll attempts happen before SQS moves the message to the configured DLQ, and that move happens entirely inside the SQS service — no Lambda execution role is involved, because there's no caller principal authorizing the send. That's a different model from the legacy DeadLetterConfig case above, and it's worth keeping the two straight when you're debugging why messages aren't showing up where you expect.
The redrive policy itself is set through the queue's attributes, and RedrivePolicy is a JSON-encoded string value, not a nested object. A minimal example, following the shape documented in the SQS dead-letter queue documentation:
{
"Attributes": {
"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:orders-dlq\",\"maxReceiveCount\":\"5\"}"
}
}Two things break this silently. First, check the DLQ's own message retention period — the default is 4 days, and for a queue meant to hold evidence of failures until someone investigates, that's rarely enough. Set retention to the 14-day maximum, or messages can expire before anyone notices the alarm fired. Second, the source queue and its DLQ need to be the same queue type (both standard or both FIFO) and live in the same account and region. Mismatches don't throw an error when you save the policy — they just fail redrive quietly, and the only way you find out is by watching ApproximateNumberOfMessagesVisible stay at zero when it shouldn't.
Turn on partial batch failure reporting to cut reprocessing cost
Without ReportBatchItemFailures enabled, one bad message inside a batch of ten causes Lambda to retry the entire batch on the next poll, not just the failing item. Nine already-processed messages get reprocessed, multiplying invocation count and, for handlers with side effects, multiplying duplicate writes or API calls.
Enabling it requires setting FunctionResponseTypes: [ReportBatchItemFailures] on the event source mapping and returning only the failed item identifiers from the handler:
# Lambda handler for an SQS-triggered function using partial batch failure reporting.
# Only the failed messages are retried by the event source mapping,
# instead of the whole batch — reduces duplicate processing and cost.
import json
def handler(event, context):
batch_item_failures = []
for record in event["Records"]:
message_id = record["messageId"]
try:
body = json.loads(record["body"])
process(body) # your business logic
except Exception as exc:
print(f"Failed to process {message_id}: {exc}")
batch_item_failures.append({"itemIdentifier": message_id})
# This shape requires FunctionResponseTypes: [ReportBatchItemFailures]
# on the event source mapping.
return {"batchItemFailures": batch_item_failures}
def process(body):
if not body.get("order_id"):
raise ValueError("missing order_id")
# ... normal processingDuring a partial outage — say a downstream dependency rejecting one in ten requests — this is the difference between retrying one message and retrying the whole batch every cycle. That shows up directly in invocation cost and in added latency for the nine healthy messages that shouldn't have been touched again.
Alarm on DLQ depth — a quiet DLQ is not a healthy DLQ
The metric to alarm on is ApproximateNumberOfMessagesVisible on the DLQ itself. Zero messages for weeks is ambiguous by design: it can mean everything's healthy, or it can mean the DLQ was never wired up correctly and nothing has ever landed there — which, in practical terms, is the same as having no failure handling at all.
Pair that alarm with a widget on ApproximateAgeOfOldestMessage to catch messages that got redirected but never replayed. A DLQ quietly accumulating stale entries for weeks is its own failure mode, distinct from an empty one. Route both alarms to the same on-call channel that watches the primary pipeline; a DLQ nobody monitors is just a more expensive way to lose the same data. This gap between "configured" and "actually monitored" shows up across most serverless observability setups — more on that pattern in kuryzhev.cloud's broader coverage of AWS operational patterns.
Design replay for idempotency, not just resubmission
A naive replay script that reads the DLQ and re-sends the same body to the original queue can duplicate side effects if the handler isn't idempotent — a payment charged twice, a notification sent twice. Before writing a replay script, check whether the handler already has an idempotency key check. If it doesn't, add one before building tooling around it, not after.
A second, less obvious problem: replay scripts that don't preserve MessageAttributes, or for FIFO queues, MessageGroupId and MessageDeduplicationId, can silently break ordering guarantees on rehydration. Keep replay as a deliberate, manually triggered script or Step Functions flow rather than an automatic loop. An automated retry-on-DLQ-arrival pattern can recreate the exact failure storm that produced the messages in the first place — this is one case where automation isn't the safer default.
Treat DLQ contents as sensitive data
DLQ messages usually contain the full original payload — order details, tokens, internal IDs, sometimes PII — sitting in a queue that gets far less security scrutiny than the primary datastore because it "feels like internal plumbing." That assumption doesn't hold up under an audit or a breach investigation.
Enable SSE-KMS on the DLQ and scope IAM policies so only the function's execution role and a narrowly defined replay role can read or write to it, not the team's default deployment role. Apply the same retention limits and access logging to the DLQ that apply to the primary data store, rather than accepting whatever the console defaults to.
Use this as a pre-deploy check before shipping any function with SQS, Kinesis, or async event sources attached. Most Lambda dead letter queue failures trace back to one of the rows below.
DLQ mechanism decision cheat sheet:
Trigger type | Configure DLQ/failure handling on...
---------------------------|--------------------------------------
Async invoke (SNS, S3 evt) | Function DestinationConfig.OnFailure
Sync invoke (API Gateway) | No DLQ — caller must handle retries
SQS event source mapping | Source queue's RedrivePolicy + DLQ
Kinesis/DynamoDB Streams | Event source mapping OnFailure destination
Before shipping, verify:
[ ] Legacy DeadLetterConfig or async Destination targets have execution role permission (sqs:SendMessage / sns:Publish)
[ ] DLQ retention set to 14 days, not default 4
[ ] CloudWatch alarm on ApproximateNumberOfMessagesVisible
[ ] ReportBatchItemFailures enabled for batch triggers
[ ] Replay path checks idempotency key before reprocessing
[ ] DLQ encrypted (SSE-KMS) and access restricted via IAM
Top comments (0)