EventBridge Retry Policy vs SQS DLQ: The Right Call in Production

#devops

Originally published on kuryzhev.cloud

Your EventBridge target has been silently dropping failed events into a DLQ with zero write permissions — for three weeks — and nobody noticed until a customer asked where their order went. We pulled up the CloudWatch metrics, saw FailedInvocations climbing, and then discovered the DLQ had exactly zero messages in it. Not because nothing failed. Because the resource policy on the queue never granted events.amazonaws.com permission to write to it. That's the moment I stopped treating EventBridge retry policy and DLQ setup as a checkbox and started treating it as an actual design decision.

When you face this choice

This comes up the moment you wire an EventBridge rule to a target that isn't 100% reliable — a Lambda that occasionally times out, a Step Functions execution that throttles under load, or an API destination hitting a flaky third-party endpoint. You need a resilience story before this ships to production, and you have two real paths: lean entirely on EventBridge's built-in per-target RetryPolicy and DeadLetterConfig, or put an SQS queue in front of your actual consumer and own the retry/backoff/redrive logic yourself.

Here's the part that trips people up: this decision is made per target, not per bus and not per rule. I've seen teams configure a beautiful DLQ on their first target, ship a second target on the same rule three sprints later, and assume the DLQ config "just applies." It doesn't. Every target gets its own DeadLetterConfig, and if you forget it, that target fails open — into nothing, with no alert, no trace, no evidence anything went wrong until someone downstream notices missing data.

So before you write a single line of Terraform, decide: is native retry + DLQ enough for this target, or do you need the control that only a buffer queue gives you? Don't default to whichever pattern you copy-pasted from the last project. They solve different problems.

Option A: Native EventBridge RetryPolicy + DLQ (pros/cons)

This is the zero-infrastructure option. You set maximum_retry_attempts and maximum_event_age_in_seconds on the target, point DeadLetterConfig.Arn at an SQS queue, and EventBridge handles retries with its own exponential backoff. No Lambda glue, no extra queue to monitor for the happy path, and it works the same whether your target is Lambda, Step Functions, or an API destination.

The catch: those two knobs are the only knobs. You don't control the shape of the backoff curve — only the ceiling. Default MaximumRetryAttempts is 185, and default MaximumEventAgeInSeconds is 86400 (24 hours). Most teams never touch these defaults and don't realize what that means in practice: EventBridge will keep retrying a failing target for a full day, and every one of those retries is a billed invocation downstream. During an outage, a Lambda hammered by 185 retry attempts across dozens of events can spike your invocation cost 100x almost overnight.

Watch out: setting MaximumRetryAttempts: 0 alone does not give you immediate DLQ routing. You still need to explicitly set MaximumEventAgeInSeconds, or the event stays "alive" for the default 24 hours even with zero retry attempts configured — a confusing half-state that looks like a bug but is documented behavior.

Debugging is the other weak point. The message that lands in your DLQ is the raw event, not an error trace. You get no built-in "why did this fail" — you're cross-referencing CloudWatch Logs and metrics like TargetErrorCount and DeadLetterInvocations manually, event by event.

Option B: SQS buffer queue as the real target (pros/cons)

Here you don't target the Lambda or Step Function directly — you target an SQS queue, and your actual consumer reads from that queue. This buys you real control: batching behavior, visibility timeout tuning, and backoff shaped by your Lambda's SQS trigger config, not EventBridge's fixed curve. Redrive is native and supported — set RedrivePolicy.maxReceiveCount and AWS will actually let you redrive messages back out of the DLQ using aws sqs start-message-move-task, GA since late 2023.

It also decouples delivery from processing. EventBridge-to-SQS delivery is near-instant and reliable, so any failure is isolated to the consumer side — easier to reason about, easier to load-test independently of the bus.

The cost is real, though. You're now provisioning, monitoring, and paying for an extra moving part. You own idempotency — SQS is at-least-once delivery, and your own retry logic on top of that can double-process events if your handler isn't idempotent. There's also a subtle latency hit on the happy path since every event now makes an extra hop.

Gotcha: don't assume SQS-as-target gives you ordering. Standard queues don't guarantee it. Switching to FIFO fixes ordering but caps you at 300 msg/s without batching — a real bottleneck if your bus expects high fan-out volume. I've watched a team switch to FIFO "for safety" and immediately create a backlog during a traffic spike that the standard queue would have absorbed fine.

Decision matrix

Use this to skip the re-reading and go straight to a decision.

Row                          Native RetryPolicy+DLQ        SQS Buffer Queue
Retry control granularity    Low — ceiling only            High — full backoff/visibility control
Ease of setup                High — one-liner target block Medium — extra queue + policy + trigger
Cost overhead                Low, but retry storms spike   Moderate, predictable
                              downstream invocation cost
Observability quality        Poor — raw event, no error    Good — CloudWatch + queue depth +
                              context in DLQ                 redrive task status
Replay/redrive capability     Manual only, no native tool   Native via start-message-move-task
Latency impact                None                          Small added hop, usually milliseconds
Best-fit target               Lambda w/ idempotent handler  Step Functions, batch consumers,
                                                              high-throughput fan-out
Team size / maturity          Small teams, low on-call      Needs someone who can own redrive
                              overhead — good default        runbooks and queue monitoring

That last row matters more than people admit. I've seen small teams adopt the SQS buffer pattern because it's "more correct," then quietly regret it six months later when nobody remembers how the redrive script works and the DLQ has 40,000 stale messages nobody wants to touch.

My pick

I default to native RetryPolicy + DLQ for Lambda targets with idempotent handlers and non-critical volume. It's good enough for roughly 80% of the pipelines I've built, and I stopped over-engineering this after watching a team spend two sprints building custom SQS redrive tooling for a target that failed maybe twice a month. Not every event pipeline needs a queue in front of it.

I switch to the SQS buffer pattern only when I need batch processing, something close to ordering guarantees, or a redrive workflow that an on-call engineer can actually run at 2am without writing custom Lambda glue first. If your target is Step Functions, remember EventBridge's RetryPolicy only covers its own delivery attempt — internal state machine failures need their own Retry/Catch blocks regardless of which pattern you pick.

My non-negotiable stance: never ship an EventBridge target to production without some DLQ configured, and never trust that it's working without checking the SQS resource policy explicitly. The number of teams that discover silently-dropped events during an incident postmortem — not before — is the entire reason I write posts like this. We've covered similar production gotchas in our DevOps notes on kuryzhev.cloud if you want more of these before-it-bites-you writeups.

Here's the Terraform for the native approach, DLQ policy included — this is the one-liner people forget to pair with an actual resource policy:


# EventBridge rule targeting Lambda, with native RetryPolicy + DeadLetterConfig (Option A)

resource "aws_sqs_queue" "eventbridge_dlq" {
  name                      = "orders-eventbridge-dlq"
  message_retention_seconds = 1209600  # 14 days — max retention, buys time to investigate
  kms_master_key_id         = "alias/aws/sqs"  # encrypt at rest, security baseline
}

# Explicit resource policy — without this, DLQ writes fail silently
resource "aws_sqs_queue_policy" "dlq_policy" {
  queue_url = aws_sqs_queue.eventbridge_dlq.id
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Sid       = "AllowEventBridgeSend",
      Effect    = "Allow",
      Principal = { Service = "events.amazonaws.com" },
      Action    = "sqs:SendMessage",
      Resource  = aws_sqs_queue.eventbridge_dlq.arn,
      Condition = {
        ArnEquals = { "aws:SourceArn" = aws_cloudwatch_event_rule.orders_created.arn }
      }
    }]
  })
}

resource "aws_cloudwatch_event_rule" "orders_created" {
  name          = "orders-created"
  event_pattern = jsonencode({ source = ["orders.service"] })
}

resource "aws_cloudwatch_event_target" "process_order" {
  rule      = aws_cloudwatch_event_rule.orders_created.name
  arn       = aws_lambda_function.process_order.arn
  target_id = "process-order-lambda"

  retry_policy {
    maximum_retry_attempts       = 3          # cap retries — don't let defaults hammer for 24h
    maximum_event_age_in_seconds = 3600       # give up after 1h, not the 86400s default
  }

  dead_letter_config {
    arn = aws_sqs_queue.eventbridge_dlq.arn   # this is per-TARGET, not per-rule
  }
}

And when messages do end up in the DLQ, here's how to inspect and redrive them without writing a custom Lambda — this uses the native SQS message-move task:


# Redriving a stuck DLQ back to the consumer queue (Option B pattern),
# using the native SQS message-move task — no custom Lambda needed.

# 1. Inspect what's sitting in the DLQ before redriving blindly
aws sqs get-queue-attributes \
  --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

# Example output:
# {
#   "Attributes": {
#     "ApproximateNumberOfMessages": "47"
#   }
# }

# 2. Start the move task — requires awscli v2 >= 2.13.0 / botocore >= 1.31.0
aws sqs start-message-move-task \
  --source-arn "$DLQ_ARN" \
  --destination-arn "$MAIN_QUEUE_ARN" \
  --max-number-of-messages-per-second 5   # throttle to avoid re-triggering the original failure spike

# 3. Track progress of the redrive
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN"

# Common gotcha: if the original failure was a bad payload (not transient),
# redriving just replays the same crash — check CloudWatch Logs error signature
# on a sample message BEFORE redriving 47 messages back into production.

Set CloudWatch alarms on InvocationsSentToDlq > 0 no matter which option you pick — EventBridge will never alert you on its own, and the whole point of the DLQ is to catch failures you'd otherwise never see. For the exact semantics of retry policies and dead-letter configs, the AWS EventBridge DLQ documentation and the SQS redrive policy docs are worth bookmarking before you ship either pattern.