Sascha Mirow-Stanitzok

Posted on Nov 15 Edited on Nov 15 Originally published at sascha-ms.de

Dead-letter queue with CDK: alarms and automatic redrive

#aws #devops #infrastructure #serverless

Setting up a dead-letter queue with CDK takes three lines: deadLetterQueue: { queue, maxReceiveCount }, done. The real work starts afterwards. Who notices when the DLQ fills up? What happens to a message that will never work? And who moves the others back once the billing API is running again after an hour-long outage? In many projects the honest answer is: somebody, at some point, by hand.

In my first article on the SQS dead-letter queue I built this with the Serverless Framework: queue, DLQ, alarm and a redrive script. Today we go one step further. You wrap everything in a reusable AWS CDK construct, separate transient from permanent errors and let the DLQ move back automatically on a schedule. To keep this from turning into an endless loop, the most important ingredient comes before the schedule: clean error classification in the consumer.

In short

A custom CDK construct bundles queue, DLQ, alarms and a scheduled redrive via EventBridge Scheduler, so every new queue gets the same safety nets. Automatic redrive is only safe, however, if the DLQ contains nothing but transient failures. That is why the consumer parks permanently broken messages in a separate queue; otherwise they circle until their retention period runs out.

What you will learn in this article

Wrap queue, DLQ and alarms in a custom CDK construct
Separate transient and permanent errors in the Lambda consumer
Schedule the DLQ redrive with EventBridge Scheduler and a universal target
Protect the construct with Vitest and template assertions
Recognize when automatic redrive does more harm than good

Why build a dead-letter queue with CDK as a construct?

In the AWS CDK, a construct is a building block that combines several resources into one unit. You call it with a few parameters and get a finished package back. That sounds like convenience, but for queues it is mostly a matter of discipline. The rules for a good DLQ are well known, and yet the third queue in a project suddenly lacks the alarm, or its DLQ keeps messages just as briefly as the source queue.

A construct for the dead-letter queue with CDK defines these rules once. I would rather write 150 lines of tested TypeScript than copy the same 40 lines of configuration five times. You can also tighten the rules centrally later, for example by adding a new alarm, and every queue gets it on the next deployment.

Our example is a fictional bike rental. Every return puts an event into an SQS queue, and a Lambda function sends the rental duration to a billing API. The construct is called QueueWithDlq and provides:

a source queue with a suitable visibility timeout and redrive policy,
a DLQ with 14 days of retention,
a parking queue for permanently broken messages,
three CloudWatch alarms, optionally with SNS notifications,
on request, a scheduled redrive via EventBridge Scheduler.

Separating transient and permanent errors

The classic DLQ makes no distinction between two very different situations. In the first case, the billing API is down for an hour. Every message is fine; it just arrived at the wrong time. In the second case, the message lacks a bikeId. That message will not work in three days either, no matter how often you try.

As long as both sit in the same DLQ, you cannot redrive automatically. A redrive would take the broken messages along; they fail five times, move back to the DLQ and wait for the next redrive. This is known as a “poison message loop”, and it costs Lambda invocations, clutters logs and hides real problems.

The solution lives in the consumer. For every error it decides whether a later attempt can change anything. If not, it sends the message to a parking queue and reports it as processed successfully, so SQS deletes it from the source queue. If yes, it reports it as failed via the partial batch response. That is a response from the Lambda function listing only the failed messages of a batch, so the successful ones are not retried.

The consumer decides: retry via the DLQ or park right away.

How do you recognize a transient error?

A simple rule of thumb: anything that depends on the state of another system is transient. Anything that depends only on the message itself is permanent. For HTTP calls, that means:

Error	Type	Destination
Timeout, network error	transient	retry, then DLQ
HTTP 429 (too many requests)	transient	retry, then DLQ
HTTP 500 to 599	transient	retry, then DLQ
Invalid JSON, required field missing	permanent	parking queue
HTTP 400, 404, 422	permanent	parking queue

There are still edge cases. An HTTP 409 (conflict) can mean that another process is working on the same record right now, in which case waiting helps. It can also mean that the booking already exists. Such codes are best clarified with the team that runs the API instead of guessing.

How does the automatic redrive work?

Since SQS offers the StartMessageMoveTask API, you no longer need a custom script that reads messages and sends them again. You pass the ARN of the DLQ, and SQS moves the messages back asynchronously. Without a destination ARN they land in the queue they came from. With MaxNumberOfMessagesPerSecond you throttle the pace; the upper limit is 500 messages per second. In addition, only one move task can run per queue at a time.

The call is made by EventBridge Scheduler. This service starts targets on a schedule and, through so-called universal targets, can call almost any AWS API directly without you writing a Lambda function in between. The target ARN follows the pattern arn:aws:scheduler:::aws-sdk:sqs:startMessageMoveTask, and the parameters go into the input as JSON.

🎬 Automatic redrive of an SQS DLQ – this animated diagram only runs on my site: watch it on sascha-ms.de

So the cycle looks like this: a message fails five times, moves to the DLQ, and the alarm goes off. At most an hour later, the scheduler starts the move task, the message lands in the source queue again, and the Lambda function tries once more. If the API is back, the matter is settled without anyone having to get up at night.

Step by step: build a dead-letter queue with CDK

Prerequisites

Node.js 22 and npm
An AWS account with credentials configured for the AWS CLI
An account and region prepared with npx cdk bootstrap
Basic knowledge of TypeScript and SQS (the first part explains the terms)

I tested the code with aws-cdk-lib 2.276.0, CDK CLI 2.1148.1, TypeScript 7.0.2 and Vitest 5.1.0. Because TypeScript 7 no longer supports ts-node, the CDK app starts via tsx.

Step 1: Create the project

mkdir rental-returns && cd rental-returns
mkdir bin lib src test
npm init -y
npm install aws-cdk-lib constructs @aws-sdk/client-sqs
npm install -D aws-cdk typescript tsx esbuild vitest @types/node @types/aws-lambda

Add "type": "module" to your package.json along with the scripts "test": "vitest run" and "build": "tsc --noEmit". Then add two small configuration files:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "nodenext",
    "moduleResolution": "nodenext",
    "strict": true,
    "skipLibCheck": true,
    "noEmit": true,
    "types": ["node"]
  },
  "include": ["bin", "lib", "src", "test"]
}

{
  "app": "npx tsx bin/app.ts"
}

Step 2: Write the QueueWithDlq construct

This is the heart of your dead-letter queue with CDK. Read the code first; I explain the important decisions right below it.

// lib/queue-with-dlq.ts
import { Duration } from 'aws-cdk-lib';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as cwActions from 'aws-cdk-lib/aws-cloudwatch-actions';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
import * as scheduler from 'aws-cdk-lib/aws-scheduler';
import * as targets from 'aws-cdk-lib/aws-scheduler-targets';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import { Construct } from 'constructs';

export interface AutoRedriveOptions {
  /** When the redrive starts, e.g. ScheduleExpression.rate(Duration.hours(1)) */
  readonly schedule: scheduler.ScheduleExpression;
  /** Throttling for the redrive (1 to 500 messages per second) */
  readonly maxMessagesPerSecond?: number;
}

export interface QueueWithDlqProps {
  /** The Lambda function that processes the queue */
  readonly consumer: lambda.Function;
  /** Function timeout, used to calculate the visibility timeout */
  readonly consumerTimeout: Duration;
  /** Attempts before a message moves to the DLQ (default: 5) */
  readonly maxReceiveCount?: number;
  /** Age of the oldest message that triggers the backlog alarm (default: 15 minutes) */
  readonly maxMessageAge?: Duration;
  /** Optional: SNS topic for all alarms */
  readonly alarmTopic?: sns.ITopic;
  /** Optional: automatic redrive, meant for transient errors only */
  readonly autoRedrive?: AutoRedriveOptions;
}

export class QueueWithDlq extends Construct {
  public readonly queue: sqs.Queue;
  public readonly deadLetterQueue: sqs.Queue;
  public readonly parkingQueue: sqs.Queue;
  public readonly alarms: cloudwatch.Alarm[] = [];

  constructor(scope: Construct, id: string, props: QueueWithDlqProps) {
    super(scope, id);

    const maxReceiveCount = props.maxReceiveCount ?? 5;
    const rate = props.autoRedrive?.maxMessagesPerSecond ?? 10;
    if (maxReceiveCount < 2) {
      throw new Error('maxReceiveCount must be at least 2, otherwise a single timeout sends a message to the DLQ.');
    }
    if (rate < 1 || rate > 500) {
      throw new Error('maxMessagesPerSecond must be between 1 and 500.');
    }

    // DLQ: longest retention, because the age from the source queue carries over
    this.deadLetterQueue = new sqs.Queue(this, 'Dlq', {
      retentionPeriod: Duration.days(14),
      encryption: sqs.QueueEncryption.SQS_MANAGED,
      enforceSSL: true,
    });

    // Parking spot for permanently broken messages: no redrive, analysis only
    this.parkingQueue = new sqs.Queue(this, 'Parking', {
      retentionPeriod: Duration.days(14),
      encryption: sqs.QueueEncryption.SQS_MANAGED,
      enforceSSL: true,
    });

    this.queue = new sqs.Queue(this, 'Queue', {
      // Recommendation for Lambda: at least six times the function timeout
      visibilityTimeout: Duration.seconds(props.consumerTimeout.toSeconds() * 6),
      retentionPeriod: Duration.days(4),
      encryption: sqs.QueueEncryption.SQS_MANAGED,
      enforceSSL: true,
      deadLetterQueue: { queue: this.deadLetterQueue, maxReceiveCount },
    });

    props.consumer.addEventSource(
      new SqsEventSource(this.queue, {
        batchSize: 10,
        reportBatchItemFailures: true,
      }),
    );
    this.parkingQueue.grantSendMessages(props.consumer);
    props.consumer.addEnvironment('PARKING_QUEUE_URL', this.parkingQueue.queueUrl);

    this.addAlarm('DlqNotEmpty', {
      metric: this.deadLetterQueue.metricApproximateNumberOfMessagesVisible({
        period: Duration.minutes(1),
        statistic: 'Maximum',
      }),
      threshold: 1,
      alarmDescription: 'Messages in the DLQ: processing failed repeatedly.',
    }, props.alarmTopic);

    this.addAlarm('ParkingNotEmpty', {
      metric: this.parkingQueue.metricApproximateNumberOfMessagesVisible({
        period: Duration.minutes(5),
        statistic: 'Maximum',
      }),
      threshold: 1,
      alarmDescription: 'Permanently broken messages parked: please take a look.',
    }, props.alarmTopic);

    this.addAlarm('QueueBacklog', {
      metric: this.queue.metricApproximateAgeOfOldestMessage({
        period: Duration.minutes(1),
        statistic: 'Maximum',
      }),
      threshold: (props.maxMessageAge ?? Duration.minutes(15)).toSeconds(),
      evaluationPeriods: 5,
      alarmDescription: 'The oldest message has been waiting too long: consumer too slow or failing.',
    }, props.alarmTopic);

    if (props.autoRedrive) {
      new scheduler.Schedule(this, 'AutoRedrive', {
        description: 'Moves the DLQ back to the source queue on a schedule',
        schedule: props.autoRedrive.schedule,
        target: new targets.Universal({
          service: 'sqs',
          action: 'startMessageMoveTask',
          // Without DestinationArn, messages go back to their source queue
          input: scheduler.ScheduleTargetInput.fromObject({
            SourceArn: this.deadLetterQueue.queueArn,
            MaxNumberOfMessagesPerSecond: rate,
          }),
          retryAttempts: 0,
          policyStatements: [
            new iam.PolicyStatement({
              actions: [
                'sqs:StartMessageMoveTask',
                'sqs:ReceiveMessage',
                'sqs:DeleteMessage',
                'sqs:GetQueueAttributes',
              ],
              resources: [this.deadLetterQueue.queueArn],
            }),
            new iam.PolicyStatement({
              actions: ['sqs:SendMessage'],
              resources: [this.queue.queueArn],
            }),
          ],
        }),
      });
    }
  }

  private addAlarm(
    id: string,
    props: Omit<cloudwatch.AlarmProps, 'evaluationPeriods'> & { evaluationPeriods?: number },
    topic?: sns.ITopic,
  ): void {
    const alarm = new cloudwatch.Alarm(this, id, {
      comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD,
      treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
      ...props,
      evaluationPeriods: props.evaluationPeriods ?? 1,
    });
    if (topic) {
      alarm.addAlarmAction(new cwActions.SnsAction(topic));
      alarm.addOkAction(new cwActions.SnsAction(topic));
    }
    this.alarms.push(alarm);
  }
}

Some values are deliberately not configurable. The construct sets the visibility timeout to six times the function timeout, as AWS recommends for SQS with Lambda. The buffer covers cases where Lambda is throttled while a batch is still running. The DLQ keeps messages for 14 days, the SQS maximum. With standard queues, a message keeps its original timestamp when it moves to the DLQ. A DLQ with the same retention as the source would therefore lose messages sooner than you think.

Also note policyStatements on the universal target. Without it, CDK grants the scheduler role only the sqs:StartMessageMoveTask permission. That is not enough for a redrive: according to the SQS documentation, the role also needs ReceiveMessage, DeleteMessage and GetQueueAttributes on the DLQ plus SendMessage on the destination queue. If you encrypt with your own KMS key, KMS permissions come on top. The construct uses SQS-managed encryption and skips that part.

retryAttempts: 0 prevents the scheduler from retrying a failed call several times. If, for example, a move task from the last hour is still running, SQS rejects the new one, and that is fine. The next slot comes anyway.

Step 3: Consumer with error classes

The Lambda function distinguishes two kinds of errors. A custom PermanentError class marks everything that would fail the same way on the next attempt. Any other error counts as transient.

// src/return-consumer.ts
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
import type { SQSBatchResponse, SQSEvent, SQSRecord } from 'aws-lambda';

const sqs = new SQSClient({});

/** Won't turn out differently on the next attempt: park instead of retrying */
export class PermanentError extends Error {}

interface RentalReturn {
  rentalId: string;
  bikeId: string;
  minutes: number;
}

export function parseReturn(body: string): RentalReturn {
  let data: unknown;
  try {
    data = JSON.parse(body);
  } catch {
    throw new PermanentError('Not valid JSON');
  }
  const r = data as Partial<RentalReturn>;
  if (typeof r.rentalId !== 'string' || typeof r.bikeId !== 'string') {
    throw new PermanentError('rentalId or bikeId missing');
  }
  if (typeof r.minutes !== 'number' || r.minutes <= 0) {
    throw new PermanentError('minutes must be a positive number');
  }
  return { rentalId: r.rentalId, bikeId: r.bikeId, minutes: r.minutes };
}

export async function chargeRental(ret: RentalReturn): Promise<void> {
  const res = await fetch(process.env.BILLING_URL!, {
    method: 'POST',
    headers: {
      'content-type': 'application/json',
      // Same key on every attempt: the API charges only once
      'idempotency-key': ret.rentalId,
    },
    body: JSON.stringify(ret),
    signal: AbortSignal.timeout(5000),
  });
  if (res.ok) return;
  // 429 and 5xx are transient, other 4xx stay wrong on the next attempt too
  if (res.status === 429 || res.status >= 500) {
    throw new Error(`Billing temporarily unavailable: HTTP ${res.status}`);
  }
  throw new PermanentError(`Billing rejected: HTTP ${res.status}`);
}

async function park(record: SQSRecord, reason: string): Promise<void> {
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: process.env.PARKING_QUEUE_URL!,
      MessageBody: record.body,
      MessageAttributes: {
        reason: { DataType: 'String', StringValue: reason.slice(0, 256) },
        originalMessageId: { DataType: 'String', StringValue: record.messageId },
      },
    }),
  );
}

export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
  const batchItemFailures: SQSBatchResponse['batchItemFailures'] = [];

  for (const record of event.Records) {
    try {
      await chargeRental(parseReturn(record.body));
    } catch (err) {
      if (err instanceof PermanentError) {
        try {
          await park(record, err.message);
          console.warn(JSON.stringify({ msg: 'parked', messageId: record.messageId, reason: err.message }));
          continue; // parked successfully: let SQS delete it from the queue
        } catch (parkErr) {
          console.error(JSON.stringify({ msg: 'parking failed', messageId: record.messageId }), parkErr);
        }
      } else {
        console.error(JSON.stringify({ msg: 'transient error', messageId: record.messageId }), err);
      }
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures };
}

Two details deserve a second look. The idempotency-key header ensures that a return is charged only once, even if the same message arrives for the seventh time after a redrive. Without idempotency on the other side, automatic redrive would be dangerous, because a timeout does not mean the booking did not arrive. Second, if parking itself fails, the message also ends up in batchItemFailures. Better one attempt too many than a lost message.

Step 4: Assemble the stack

// lib/rental-stack.ts
import { CfnOutput, Duration, Stack, type StackProps } from 'aws-cdk-lib';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import * as logs from 'aws-cdk-lib/aws-logs';
import * as scheduler from 'aws-cdk-lib/aws-scheduler';
import * as sns from 'aws-cdk-lib/aws-sns';
import { Construct } from 'constructs';
import { QueueWithDlq } from './queue-with-dlq.js';

export class RentalReturnsStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const timeout = Duration.seconds(20);
    const consumer = new NodejsFunction(this, 'ReturnConsumer', {
      entry: 'src/return-consumer.ts',
      runtime: lambda.Runtime.NODEJS_22_X,
      architecture: lambda.Architecture.ARM_64,
      timeout,
      memorySize: 256,
      logGroup: new logs.LogGroup(this, 'ReturnConsumerLogs', {
        retention: logs.RetentionDays.TWO_WEEKS,
      }),
      environment: {
        // Fictional billing API of the bike rental
        BILLING_URL: 'https://billing.example.com/rentals',
      },
    });

    const alarmTopic = new sns.Topic(this, 'Alarms');

    const returns = new QueueWithDlq(this, 'RentalReturns', {
      consumer,
      consumerTimeout: timeout,
      maxReceiveCount: 5,
      alarmTopic,
      autoRedrive: {
        schedule: scheduler.ScheduleExpression.rate(Duration.hours(1)),
        maxMessagesPerSecond: 5,
      },
    });

    new CfnOutput(this, 'QueueUrl', { value: returns.queue.queueUrl });
    new CfnOutput(this, 'DlqArn', { value: returns.deadLetterQueue.queueArn });
    new CfnOutput(this, 'ParkingQueueUrl', { value: returns.parkingQueue.queueUrl });
  }
}

// bin/app.ts
import { App } from 'aws-cdk-lib';
import { RentalReturnsStack } from '../lib/rental-stack.js';

const app = new App();
new RentalReturnsStack(app, 'RentalReturnsStack');

In the stack you can see how little of the actual complexity remains: one function, one topic for alarms and one call to QueueWithDlq. That is exactly what makes the construct worthwhile.

Step 5: Test the construct with template assertions

A construct meant for other teams needs tests. With the aws-cdk-lib/assertions module you check the generated CloudFormation template without deploying anything. The basics are covered in the article testing CDK stacks with Vitest.

// test/queue-with-dlq.test.ts
import { App, Duration, Stack } from 'aws-cdk-lib';
import { Match, Template } from 'aws-cdk-lib/assertions';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as scheduler from 'aws-cdk-lib/aws-scheduler';
import { describe, expect, it } from 'vitest';
import { QueueWithDlq, type QueueWithDlqProps } from '../lib/queue-with-dlq.js';

function synth(extra: Partial<QueueWithDlqProps> = {}): Template {
  const stack = new Stack(new App(), 'TestStack');
  const consumer = new lambda.Function(stack, 'Fn', {
    runtime: lambda.Runtime.NODEJS_22_X,
    handler: 'index.handler',
    code: lambda.Code.fromInline('exports.handler = async () => ({ batchItemFailures: [] });'),
    timeout: Duration.seconds(20),
  });
  new QueueWithDlq(stack, 'Returns', { consumer, consumerTimeout: Duration.seconds(20), ...extra });
  return Template.fromStack(stack);
}

describe('QueueWithDlq', () => {
  it('sets visibility timeout and redrive policy', () => {
    synth().hasResourceProperties('AWS::SQS::Queue', {
      VisibilityTimeout: 120,
      RedrivePolicy: { maxReceiveCount: 5, deadLetterTargetArn: Match.anyValue() },
    });
  });

  it('keeps DLQ messages for 14 days', () => {
    const queues = synth().findResources('AWS::SQS::Queue', {
      Properties: { MessageRetentionPeriod: 1209600 },
    });
    expect(Object.keys(queues)).toHaveLength(2); // DLQ and parking queue
  });

  it('reports partial batch failures', () => {
    synth().hasResourceProperties('AWS::Lambda::EventSourceMapping', {
      FunctionResponseTypes: ['ReportBatchItemFailures'],
    });
  });

  it('creates three alarms', () => {
    synth().resourceCountIs('AWS::CloudWatch::Alarm', 3);
  });

  it('creates no schedule without autoRedrive', () => {
    synth().resourceCountIs('AWS::Scheduler::Schedule', 0);
  });

  it('starts a throttled redrive via universal target', () => {
    const template = synth({
      autoRedrive: { schedule: scheduler.ScheduleExpression.rate(Duration.hours(1)), maxMessagesPerSecond: 5 },
    });
    const schedules = template.findResources('AWS::Scheduler::Schedule');
    const target = JSON.stringify(Object.values(schedules)[0].Properties.Target);
    // ARN and input contain tokens (partition, queue ARN), so check them as text
    expect(target).toContain(':scheduler:::aws-sdk:sqs:startMessageMoveTask');
    expect(target).toContain('\\"MaxNumberOfMessagesPerSecond\\":5');
    expect(target).toContain('"MaximumRetryAttempts":0');
    template.hasResourceProperties('AWS::IAM::Policy', {
      PolicyDocument: {
        Statement: Match.arrayWith([
          Match.objectLike({ Action: Match.arrayWith(['sqs:StartMessageMoveTask']) }),
        ]),
      },
    });
  });

  it('rejects maxReceiveCount 1', () => {
    expect(() => synth({ maxReceiveCount: 1 })).toThrow(/at least 2/);
  });
});

In the schedule test I check the target as text. The ARN of the universal target and the input contain CloudFormation functions such as Fn::Join for the partition and the queue ARN, so an exact comparison with a string would fail. For the consumer, a few unit tests with a stubbed fetch are enough:

// test/return-consumer.test.ts
import type { SQSEvent } from 'aws-lambda';
import { describe, expect, it, vi } from 'vitest';
import { PermanentError, handler, parseReturn } from '../src/return-consumer.js';

describe('parseReturn', () => {
  it('parses a valid return', () => {
    expect(parseReturn('{"rentalId":"r-1","bikeId":"b-7","minutes":42}').minutes).toBe(42);
  });

  it('throws PermanentError for broken JSON', () => {
    expect(() => parseReturn('{broken')).toThrow(PermanentError);
  });

  it('throws PermanentError for a negative duration', () => {
    expect(() => parseReturn('{"rentalId":"r-1","bikeId":"b-7","minutes":-3}')).toThrow(/positive number/);
  });
});

describe('handler', () => {
  it('reports HTTP 503 as a partial failure', async () => {
    process.env.BILLING_URL = 'https://billing.example.com/rentals';
    vi.stubGlobal('fetch', vi.fn(async () => new Response(null, { status: 503 })));
    const event = {
      Records: [{ messageId: 'm-1', body: '{"rentalId":"r-1","bikeId":"b-7","minutes":42}' }],
    } as unknown as SQSEvent;
    await expect(handler(event)).resolves.toEqual({ batchItemFailures: [{ itemIdentifier: 'm-1' }] });
    vi.unstubAllGlobals();
  });
});

npx tsc --noEmit
npx vitest run

Expected result: Test Files 2 passed and Tests 11 passed.

Step 6: Deploy and try it out

npx cdk deploy
# Take the values from the stack outputs
QUEUE_URL=<QueueUrl from the output>
DLQ_ARN=<DlqArn from the output>

# Permanent error: goes straight to the parking queue
aws sqs send-message --queue-url "$QUEUE_URL" --message-body '{broken'

# Valid message: the fictional API is unreachable, so it is a transient error
aws sqs send-message --queue-url "$QUEUE_URL" \
  --message-body '{"rentalId":"r-1","bikeId":"b-7","minutes":42}'

# Later: list running and recent move tasks of the DLQ
aws sqs list-message-move-tasks --source-arn "$DLQ_ARN"

The broken message shows up in the parking queue after a few seconds, and the ParkingNotEmpty alarm switches to “ALARM” after a few minutes. The valid message fails because billing.example.com does not exist. After five attempts, roughly five times two minutes of visibility timeout, it sits in the DLQ. At the next scheduler slot it moves back to the source queue. Once you have seen enough, clean up with npx cdk destroy; otherwise the test message keeps circling until its retention period runs out.

Which alarms does a dead-letter queue with CDK need?

The construct creates three alarms, and each answers a different question. All of them treat missing data as “not breaching” so that an empty queue does not trigger a false alarm. The OK action sends a second notification once things have calmed down.

Alarm	Metric	Question
DlqNotEmpty	ApproximateNumberOfMessagesVisible of the DLQ ≥ 1	Is something failing repeatedly right now?
ParkingNotEmpty	ApproximateNumberOfMessagesVisible of the parking queue ≥ 1	Is the producer sending broken data?
QueueBacklog	ApproximateAgeOfOldestMessage of the source, ≥ 15 minutes for 5 minutes	Is the consumer falling behind?

For me, the parking alarm is the most important one. With automatic redrive, a DLQ alarm can clear without anyone stepping in. Parked messages, on the other hand, never disappear on their own. They almost always point to a bug at the producer or a change in the message format, and that needs a human. How to inspect messages safely without consuming them is covered in the first part.

When automatic redrive does more harm than good

I like the scheduled redrive because it solves the typical outage scenario without an on-call phone call: a third-party system goes down, comes back, and the messages catch up. Still, I don't switch it on everywhere. Three conditions have to be met for me.

The consumer classifies errors reliably. If a new type of error appears that nobody has marked as permanent, it circles anyway. The construct limits the damage through throttling and the retention period, but it does not prevent it. That is why every DLQ alarm deserves a look at the logs, even when it resolves itself.

Processing is idempotent. A redrive moves messages that have already been handled at least five times. Some of these attempts may have run halfway. Without an idempotency key or conditional writes, you risk duplicate bookings.

Order does not matter. Redriven messages arrive after newer ones. If a cancellation is processed before the matching booking, even the nicest redrive will not help you. For FIFO queues, automatic redrive is therefore usually not a good idea.

If one of these conditions is missing, I still build the dead-letter queue with CDK using the construct but leave out autoRedrive and use the manual approach from part 1. Against slow or failing third-party systems, patterns that kick in earlier also help, such as a circuit breaker with Lambda. It keeps the DLQ from filling up so quickly in the first place.

About costs: an hourly schedule produces around 730 scheduler invocations per month, which barely matters at the current EventBridge Scheduler prices. Circling messages are more expensive, because every round causes five Lambda invocations and log lines. Another argument for the parking queue.

Common mistakes

AccessDenied on the scheduled redrive

The schedule is created, but nothing happens. In CloudWatch the scheduler shows failed invocations. Almost always the role lacks permissions on the DLQ or SendMessage on the source queue, because only the default policy of the universal target is active. Add the four DLQ permissions and the send permission as in the construct.

All errors end up in the parking queue

If you park everything in the catch block, you lose the retries for real outages. Throw PermanentError only where you are sure, and let everything unknown pass as transient. When in doubt, a detour via the DLQ is cheaper than a message nobody touches again.

Forgetting the partial batch response

Without reportBatchItemFailures: true, Lambda ignores the list of failed messages. SQS then retries either the whole batch or nothing, depending on whether the function throws. The construct always sets the option, and a test checks it.

maxReceiveCount set too low

With a value of 1, a single timeout or throttle is enough to put the message in the DLQ. AWS recommends at least 5 for Lambda. The construct rejects values below 2 and uses 5 as the default.

FAQ

How do I create a dead-letter queue with CDK?

You create two sqs.Queue objects and pass the second one in the deadLetterQueue property of the first, together with maxReceiveCount. For production you should also define retention, alarms and the visibility timeout, ideally bundled in a custom construct.

Can I redrive an SQS DLQ automatically?

Yes. EventBridge Scheduler can call the SQS API StartMessageMoveTask on a schedule via a universal target. This only makes sense if the DLQ contains nothing but transiently failed messages and processing is idempotent.

How fast does StartMessageMoveTask move messages?

If you don't specify a rate, SQS chooses the pace itself based on the backlog. With MaxNumberOfMessagesPerSecond you set a fixed rate of at most 500 messages per second. Only one move task runs per queue at any time.

Does the redrive also work for Lambda or SNS DLQs?

No. According to AWS, StartMessageMoveTask only supports DLQs whose source is another SQS queue. DLQs of asynchronous Lambda invocations or SNS subscriptions have to be moved back differently, for example with a custom script.

Your next step

Pick a queue with a DLQ in your project and answer two questions: is there an alarm on the DLQ, and does your consumer know which errors are permanent? If either answer is “no”, the dead-letter queue with CDK from this article is a good starting point, even without automatic redrive. And I'm curious: do you redrive DLQs by hand today, with a script, or already automatically? Let me know in the comments.

All code from this article, ready to run: smirowstanitzok/sascha-ms-examples

Sources

This post was created with AI assistance.

👉 Next step: Need a schedule in EventBridge? My EventBridge cron helper builds and explains the expression for you.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

DEV Community