DEV Community

Cover image for Lambda Destinations: Catch Failed Async Events
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Originally published at sascha-ms.de

Lambda Destinations: Catch Failed Async Events

#tutorial #aws #serverless #backend

Lambda destinations are one of the most underrated settings in AWS Lambda, in my view. As soon as a service like S3, SNS or EventBridge invokes your function, nobody waits for the response. If something goes wrong, Lambda tries twice more and then quietly throws the event away. No error in the frontend, no ticket, just a line in a log that nobody reads.

That is why I check two things first for every asynchronously invoked function: how often does it retry, and where does an event end up when it fails anyway? In this article you build the answer step by step: a function with limited retries, a failure queue as the on-failure destination, a small script that replays failed events once the fix is in, and tests to go with it.

In short

For asynchronous invocations, Lambda retries failed events twice by default and discards them afterwards. With Lambda destinations, every event that finally fails goes to an SQS queue together with its error message, so you can reprocess it later on purpose. The most common pitfall: destinations do nothing for synchronous invocations or for functions that consume an SQS queue.

What you will learn in this article

  • How Lambda buffers and retries asynchronous invocations internally
  • When Lambda destinations are a better choice than a dead-letter queue
  • How to set up a failure queue with the Serverless Framework and with AWS CDK
  • How to read, analyze and replay failed events
  • Typical mistakes where destinations simply do nothing

What happens during an asynchronous Lambda invocation?

When you invoke a function synchronously, for example through API Gateway, the caller waits for the result and gets an error back directly. An asynchronous invocation works differently: Lambda accepts the event, responds right away with status code 202 and puts it into an internal queue. From there, Lambda invokes your function as soon as there is enough capacity. The caller never hears about the result.

If the function fails, say because your code throws an exception or hits the timeout, Lambda retries the invocation. According to the docs, it makes two more attempts by default: the second after about one minute, the third after another two minutes. Throttling (not enough free concurrency) and system errors are handled differently. In that case the event stays in the queue, and Lambda keeps trying for up to six hours, with intervals growing from one second to five minutes.

After the last attempt, or once the event is too old, Lambda discards it. Without further configuration, all that remains is the AsyncEventsDropped metric. On top of that, the docs point out that the same event can arrive more than once even without an error, because the internal queue is only eventually consistent. So your function needs to be idempotent, as I showed in the article on idempotency in AWS Lambda.

Which services invoke Lambda asynchronously?

Amazon S3 (for example for new objects), Amazon SNS and Amazon EventBridge are among the services that invoke your function asynchronously. You can trigger this yourself with the Event invocation type, for instance via the AWS CLI or the SDK. The pattern “function A kicks off function B and carries on” works exactly like this in many projects.

🎬 How Lambda processes an asynchronous invocation – this animated diagram only runs on my site: watch it on sascha-ms.de

Lambda destinations or dead-letter queue?

There are two safety nets for discarded events. The older option is a dead-letter queue (DLQ) on the function: an SQS queue or SNS topic that receives the original event content plus a few attributes. The newer option is Lambda destinations. They send a complete record with request, response and error message to a target of your choice, on success, on failure or both.

Lambda destinations (onFailure) Dead-letter queue
Targets SQS, SNS, Lambda, EventBridge, S3 (failures only) SQS or SNS
Successful invocations yes, with onSuccess no
Content event, error message, number of attempts, function ARN only the event plus attributes (error message cut to 1 KB)
Applies to function, version or alias the function only
Metric for delivery errors DestinationDeliveryFailures DeadLetterErrors

My recommendation is clear: for new functions I use destinations. The record already contains the error message and the number of attempts, which saves you the trip to the logs while debugging. On the other hand, you do not have to rebuild an existing DLQ right away if it works and someone monitors it. Configuring both at the same time is possible, but it makes troubleshooting more confusing.

Which targets make sense?

For failures, an SQS queue is almost always the best choice. Messages stay for up to 14 days, and you can inspect them calmly and replay them on purpose. EventBridge is a good fit for success notifications that other parts of the system should react to. SNS is tricky, because a message there can be at most 256 KB, while asynchronous events can be up to 1 MB. For large events, the docs therefore point to SQS or S3. You can also look up which service fits which purpose in the AWS service finder.

When Lambda destinations do not apply

The most common misconception: destinations are not a general safety net for every Lambda function. They apply to asynchronous invocations and, for failures only, to event source mappings on streams. An event source mapping is the mechanism Lambda uses to poll a source such as SQS, Kinesis or DynamoDB Streams itself and feed your function with batches.

Overview of where Lambda destinations apply: async yes, sync no, SQS source no, streams onFailure only

Destinations apply to asynchronous invocations and, for failures only, to stream sources.

So if your function consumes an SQS queue, a destination on the function does nothing. There, the dead-letter queue belongs on the source queue, combined with a redrive policy and ideally with partial batch responses. The article on SQS partial batch response shows how. For synchronous invocations through API Gateway or function URLs, the caller gets the error back directly and has to decide what happens next.

Step by step: a failure queue with the Serverless Framework

The example is a fictional bakery that takes preorders online. After each order, a function is invoked asynchronously to email a confirmation to the customer. If the mail service is down or the address is invalid, the order should not get lost but land in a failure queue.

Prerequisites

  • Node.js 22 or newer and npm
  • an AWS account with credentials configured for the AWS CLI
  • Serverless Framework v4 (npm i -g serverless, using it requires signing in)
  • basic TypeScript knowledge

Step 1: Create the project

Create a folder and install the dependencies. Serverless Framework v4 builds TypeScript handlers itself with esbuild, so you do not need an extra plugin.

mkdir bakery-preorders && cd bakery-preorders
npm init -y
npm i @aws-sdk/client-lambda @aws-sdk/client-sqs
npm i -D typescript @types/node @types/aws-lambda vitest tsx
Enter fullscreen mode Exit fullscreen mode

The tsconfig.json is deliberately simple. allowImportingTsExtensions allows imports ending in .ts, which tsx and Vitest run directly.

{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "skipLibCheck": true,
    "types": ["node"],
    "noEmit": true,
    "allowImportingTsExtensions": true
  },
  "include": ["src", "scripts", "test"]
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Write the handler

The handler receives the preorder as its event and sends the email. Sending mail is a placeholder here. Addresses ending in .invalid deliberately trigger an error, so you can see the destination in action right away.

// src/confirm-preorder.ts
export interface Preorder {
  orderId: string;
  customerEmail: string;
  items: string[];
  pickup: string; // ISO date, e.g. "2026-10-31"
}

// Placeholder for a real mail service. Addresses ending in ".invalid"
// simulate a permanent error so you can see the destination at work.
async function sendMail(to: string, subject: string): Promise<void> {
  if (to.endsWith(".invalid")) {
    throw new Error(`Mail service rejected address: ${to}`);
  }
  console.log(JSON.stringify({ msg: "Mail sent", to, subject }));
}

export async function handler(event: Preorder): Promise<{ orderId: string; status: string }> {
  if (!event?.orderId || !event.customerEmail) {
    // An error here ends up in the on-failure destination after all attempts
    throw new Error("Invalid preorder: orderId or customerEmail missing");
  }

  await sendMail(
    event.customerEmail,
    `Your preorder ${event.orderId} is ready for pickup on ${event.pickup}`,
  );

  return { orderId: event.orderId, status: "confirmed" };
}
Enter fullscreen mode Exit fullscreen mode

What matters is that the function really throws an error when something goes wrong. If you catch every exception and return a result anyway, Lambda treats the invocation as successful, and no destination fires.

Step 3: Retries and destination in serverless.yml

Now for the core of it. With maximumRetryAttempts you define how often Lambda retries after a function error (0 to 2). The maximumEventAge value sets, in seconds, how old an event may get (60 seconds to 6 hours). Under destinations.onFailure you attach the failure queue.

# serverless.yml
service: bakery-preorders

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-central-1
  architecture: arm64
  memorySize: 256
  timeout: 10

functions:
  confirmPreorder:
    handler: src/confirm-preorder.handler
    # Asynchronous invocations: at most 1 retry, events at most 1 hour old
    maximumRetryAttempts: 1
    maximumEventAge: 3600
    destinations:
      onFailure:
        type: sqs
        arn:
          Fn::GetAtt: [FailedConfirmations, Arn]

resources:
  Resources:
    FailedConfirmations:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: preorder-confirmation-failed
        MessageRetentionPeriod: 1209600 # 14 days, the SQS maximum
        SqsManagedSseEnabled: true

    FailureAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmDescription: Failed confirmations are waiting to be handled
        Namespace: AWS/SQS
        MetricName: ApproximateNumberOfMessagesVisible
        Dimensions:
          - Name: QueueName
            Value: !GetAtt FailedConfirmations.QueueName
        Statistic: Maximum
        Period: 300
        EvaluationPeriods: 1
        Threshold: 0
        ComparisonOperator: GreaterThanThreshold
        TreatMissingData: notBreaching

  Outputs:
    FailureQueueUrl:
      Value: !Ref FailedConfirmations
Enter fullscreen mode Exit fullscreen mode

Because the queue ARN is resolved through Fn::GetAtt, the Serverless Framework needs the type: sqs setting. That is how it can grant the function the matching sqs:SendMessage permission. The alarm fires as soon as a single message sits in the failure queue. Connect it to an SNS topic or your chat channel, otherwise it alarms into the void.

Step 4: Deploy and trigger an error 

serverless deploy

aws lambda invoke \
  --function-name bakery-preorders-dev-confirmPreorder \
  --invocation-type Event \
  --cli-binary-format raw-in-base64-out \
  --payload '{"orderId":"B-1042","customerEmail":"customer@example.invalid","items":["Rye bread"],"pickup":"2026-10-31"}' \
  response.json
Enter fullscreen mode Exit fullscreen mode

The CLI responds right away with "StatusCode": 202. That only means Lambda has accepted the event. After the first failure, Lambda waits about a minute and tries a second time. After that, the record is in the queue. serverless info --verbose shows you the queue URL under outputs.

aws sqs receive-message \
  --queue-url <FailureQueueUrl> \
  --query "Messages[0].Body" --output text
Enter fullscreen mode Exit fullscreen mode

Step 5: Understand the record

The message contains the so-called invocation record. For our example it looks roughly like this:

{
  "version": "1.0",
  "timestamp": "2026-10-27T17:05:12.000Z",
  "requestContext": {
    "requestId": "4f1c2a9e-0000-4000-8000-000000000001",
    "functionArn": "arn:aws:lambda:eu-central-1:111122223333:function:bakery-preorders-dev-confirmPreorder:$LATEST",
    "condition": "RetriesExhausted",
    "approximateInvokeCount": 2
  },
  "requestPayload": {
    "orderId": "B-1042",
    "customerEmail": "customer@example.invalid",
    "items": ["Rye bread"],
    "pickup": "2026-10-31"
  },
  "responseContext": {
    "statusCode": 200,
    "executedVersion": "$LATEST",
    "functionError": "Unhandled"
  },
  "responsePayload": {
    "errorType": "Error",
    "errorMessage": "Mail service rejected address: customer@example.invalid"
  }
}
Enter fullscreen mode Exit fullscreen mode

You will need three fields almost every time. requestPayload holds the original event you restart with later. condition tells you why the event was given up, here because all attempts were used. And responsePayload contains your function’s error message. So you do not have to pick this structure apart by hand everywhere, here is a small module with types and a validation function:

// src/invocation-record.ts
// Shape of the record Lambda sends to a destination
export interface InvocationRecord<TRequest = unknown> {
  version: string;
  timestamp: string;
  requestContext: {
    requestId: string;
    functionArn: string;
    condition: string; // e.g. "RetriesExhausted" or "Success"
    approximateInvokeCount: number;
  };
  requestPayload: TRequest;
  responseContext?: {
    statusCode: number;
    executedVersion: string;
    functionError?: string;
  };
  responsePayload?: unknown;
}

function isObject(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null;
}

// Validates an SQS message body and returns the typed record
export function parseInvocationRecord(body: string): InvocationRecord {
  let data: unknown;
  try {
    data = JSON.parse(body);
  } catch {
    throw new Error("Message is not valid JSON");
  }
  if (!isObject(data) || !isObject(data.requestContext) || !("requestPayload" in data)) {
    throw new Error("Message is not a Lambda invocation record");
  }
  const ctx = data.requestContext;
  if (typeof ctx.functionArn !== "string" || typeof ctx.condition !== "string") {
    throw new Error("requestContext is incomplete");
  }
  return data as unknown as InvocationRecord;
}

// Short, readable failure description for logs and tickets
export function describeFailure(record: InvocationRecord): string {
  const payload = record.responsePayload;
  const message =
    isObject(payload) && typeof payload.errorMessage === "string"
      ? payload.errorMessage
      : "no error message";
  return `${record.requestContext.condition} after ${record.requestContext.approximateInvokeCount} attempt(s): ${message}`;
}
Enter fullscreen mode Exit fullscreen mode

Step 6: Replay failed events

A failure queue is only as good as the way back. Once the mail service is reachable again or the bug is fixed, you want to reprocess the collected events. The following script reads the queue, prints every error message and invokes the function asynchronously again with the original event. Only then does it delete the message.

// scripts/replay-failures.ts
// Reads failed events from the queue and invokes the function again asynchronously.
// Usage: npx tsx scripts/replay-failures.ts <QueueUrl> [--dry-run]
import { InvokeCommand, LambdaClient } from "@aws-sdk/client-lambda";
import { DeleteMessageCommand, ReceiveMessageCommand, SQSClient } from "@aws-sdk/client-sqs";
import { describeFailure, parseInvocationRecord } from "../src/invocation-record.ts";

const [queueUrl, flag] = process.argv.slice(2);
if (!queueUrl) {
  console.error("Please provide the queue URL");
  process.exit(1);
}
const dryRun = flag === "--dry-run";

const sqs = new SQSClient({});
const lambda = new LambdaClient({});
let replayed = 0;

while (true) {
  const { Messages = [] } = await sqs.send(
    new ReceiveMessageCommand({ QueueUrl: queueUrl, MaxNumberOfMessages: 10, WaitTimeSeconds: 2 }),
  );
  if (Messages.length === 0) break;

  for (const message of Messages) {
    const record = parseInvocationRecord(message.Body ?? "");
    console.log(`${record.requestContext.requestId}: ${describeFailure(record)}`);
    if (dryRun) continue;

    await lambda.send(
      new InvokeCommand({
        FunctionName: record.requestContext.functionArn.replace(/:\$LATEST$/, ""),
        InvocationType: "Event",
        Payload: new TextEncoder().encode(JSON.stringify(record.requestPayload)),
      }),
    );
    await sqs.send(new DeleteMessageCommand({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }));
    replayed++;
  }
  if (dryRun) break;
}

console.log(dryRun ? "Dry run finished, nothing changed" : `${replayed} event(s) sent again`);
Enter fullscreen mode Exit fullscreen mode 
# First look at what is in there
npx tsx scripts/replay-failures.ts <FailureQueueUrl> --dry-run

# Then actually send them again
npx tsx scripts/replay-failures.ts <FailureQueueUrl>
Enter fullscreen mode Exit fullscreen mode

During the dry run, messages are read but not deleted. They stay invisible for the queue’s visibility timeout (30 seconds by default) and then show up again. A replayed invocation that fails again simply lands in the queue again. So you lose nothing, but you should not replay blindly in a loop while the cause is still there.

Step 7: Test

The tests check reading the record, the failure description and the handler’s behavior. They run without AWS access.

// test/invocation-record.test.ts
import { describe, expect, it } from "vitest";
import { describeFailure, parseInvocationRecord } from "../src/invocation-record.ts";
import { handler } from "../src/confirm-preorder.ts";

const failureBody = JSON.stringify({
  version: "1.0",
  timestamp: "2026-10-27T17:05:12.000Z",
  requestContext: {
    requestId: "4f1c2a9e-0000-4000-8000-000000000001",
    functionArn: "arn:aws:lambda:eu-central-1:111122223333:function:bakery-preorders-dev-confirmPreorder:$LATEST",
    condition: "RetriesExhausted",
    approximateInvokeCount: 2,
  },
  requestPayload: { orderId: "B-1042", customerEmail: "customer@example.invalid", items: ["Rye bread"], pickup: "2026-10-31" },
  responseContext: { statusCode: 200, executedVersion: "$LATEST", functionError: "Unhandled" },
  responsePayload: { errorType: "Error", errorMessage: "Mail service rejected address: customer@example.invalid" },
});

describe("parseInvocationRecord", () => {
  it("reads a failure record", () => {
    const record = parseInvocationRecord(failureBody);
    expect(record.requestContext.condition).toBe("RetriesExhausted");
    expect(record.requestPayload).toMatchObject({ orderId: "B-1042" });
  });

  it("rejects unrelated messages", () => {
    expect(() => parseInvocationRecord('{"hello":"world"}')).toThrow(/not a Lambda invocation record/);
    expect(() => parseInvocationRecord("not json")).toThrow(/not valid JSON/);
  });

  it("describes the failure in a readable way", () => {
    const text = describeFailure(parseInvocationRecord(failureBody));
    expect(text).toBe("RetriesExhausted after 2 attempt(s): Mail service rejected address: customer@example.invalid");
  });
});

describe("handler", () => {
  it("confirms a valid preorder", async () => {
    const result = await handler({ orderId: "B-1", customerEmail: "customer@example.com", items: ["Pretzel"], pickup: "2026-10-31" });
    expect(result).toEqual({ orderId: "B-1", status: "confirmed" });
  });

  it("throws for a rejected address", async () => {
    await expect(
      handler({ orderId: "B-2", customerEmail: "customer@example.invalid", items: [], pickup: "2026-10-31" }),
    ).rejects.toThrow(/rejected address/);
  });
});
Enter fullscreen mode Exit fullscreen mode 
npx tsc --noEmit
npx vitest run
Enter fullscreen mode Exit fullscreen mode

Expected result: the type check finishes without output, and Vitest reports five passing tests.

The same with AWS CDK

If you prefer AWS CDK, three properties on the function are enough. retryAttempts and maxEventAge match the values from serverless.yml, and onFailure takes a destination from the aws-lambda-destinations module. CDK adds the permission to send to the queue automatically.

// lib/preorder-stack.ts
import { Duration, Stack, type StackProps } from "aws-cdk-lib";
import { Runtime, Architecture } from "aws-cdk-lib/aws-lambda";
import { SqsDestination } from "aws-cdk-lib/aws-lambda-destinations";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import { Queue } from "aws-cdk-lib/aws-sqs";
import type { Construct } from "constructs";

export class PreorderStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const failed = new Queue(this, "FailedConfirmations", {
      retentionPeriod: Duration.days(14),
    });

    new NodejsFunction(this, "ConfirmPreorder", {
      entry: "src/confirm-preorder.ts",
      runtime: Runtime.NODEJS_22_X,
      architecture: Architecture.ARM_64,
      retryAttempts: 1,
      maxEventAge: Duration.hours(1),
      onFailure: new SqsDestination(failed),
    });
  }
}
Enter fullscreen mode Exit fullscreen mode

Running cdk synth turns this into a resource of type AWS::Lambda::EventInvokeConfig with MaximumRetryAttempts: 1, MaximumEventAgeInSeconds: 3600 and the queue as the OnFailure target. For a success notification to EventBridge, add onSuccess: new EventBridgeDestination().

Tuning retries for asynchronous invocations

The default of two retries is a sensible starting point, but it does not always fit. For errors that will not resolve themselves, such as an invalid address or a validation error, more attempts bring nothing but cost. For short outages of a third-party system, on the other hand, they help. My rule of thumb: one retry for business functions that call third-party systems, zero retries if the function already retries properly with backoff.

Watch out for multiplied retries

If your code retries three times internally and Lambda retries twice more, a single event can hit the partner up to nine times. I described this multiplication in detail in the article on exponential backoff with jitter. So decide deliberately which layer retries, and dial the others back.

How old may an event get?

The maximum age deserves some thought as well. An order confirmation that arrives six hours late confuses more than it helps. With maximumEventAge: 3600 it lands in the failure queue after one hour, and you decide on purpose whether to still send it. There is also a handy emergency brake: if you set a function’s reserved concurrency to 0, Lambda sends new asynchronous events straight to the on-failure destination without any attempt. That helps when a function is currently producing garbage and you still want to keep the events.

Typical mistakes with Lambda destinations

All of the following pitfalls are described in the docs, but they often only show up in production.

  • FIFO as a target: SQS FIFO queues and SNS FIFO topics are not supported. Delivery fails, and only the DestinationDeliveryFailures metric gives it away.
  • SNS with large events: If the record is larger than 256 KB, it does not arrive. Prefer SQS for failures.
  • Destination on an SQS-triggered function: It never fires. Configure the DLQ on the source queue instead.
  • Swallowed errors: A try/catch that only logs turns every failure into a success.
  • Configuration in the wrong place: The settings belong to the function, a version or an alias. If your sources invoke an alias, the configuration has to apply where the invocations arrive.
  • Failure queue without an alarm: A queue nobody looks at is just a slower trash can.

Cost and security

The cost is low. You pay for the extra Lambda attempts, the SQS requests and one CloudWatch alarm, which comes to cents at normal error rates. Data protection matters more: the record contains the complete event, in our example email addresses. Encrypt the queue (in the example with SqsManagedSseEnabled), restrict read access and do not keep messages longer than necessary. If you use an S3 bucket as a destination, remove the destination before deleting the bucket. Otherwise someone else could create a bucket with the same name and receive your failure records.

My take

I consider an on-failure destination mandatory for every asynchronously invoked function, much like a timeout on every HTTP call. The effort is a few lines of YAML, and the benefit shows the first time a third-party system goes down. Without a destination, you have to reconstruct lost events from logs by hand. With one, you replay them with a single command.

Where I disagree is the idea of seeing destinations as a substitute for proper error design. A failure queue collects symptoms. You still have to fix the causes, and that needs someone on the team who feels responsible. Who on your team checks the failure queue in the morning? Once that question is answered, the pattern works really well. If it stays open, after a few months you will find thousands of messages there that nobody can make sense of anymore.

FAQ

What is the difference between Lambda destinations and a DLQ?

A dead-letter queue only receives the original event and a few attributes. Lambda destinations deliver a complete record with the error message and the number of attempts, support more targets and can report successes as well.

Do Lambda destinations work with API Gateway?

No. API Gateway invokes Lambda synchronously, so the error goes straight back to the client. Destinations only apply to asynchronous invocations and, for failures, to stream sources such as Kinesis or DynamoDB Streams.

How often does Lambda retry an asynchronous invocation?

For function errors, twice by default, about one and two minutes apart. With MaximumRetryAttempts you set 0 to 2 retries. When throttled, Lambda keeps trying until the maximum event age, which is six hours by default.

Can I use a FIFO queue as a destination?

No. Lambda only supports standard queues and standard topics. With a FIFO target, delivery fails and the DestinationDeliveryFailures metric goes up.

A failure queue only helps when delivery fails. If your code accepted an event and calculated something wrong, you need something else: I show how to replay events from an archive after a bug fix in the article EventBridge replay: Reprocess events after a bug fix.

Your next step

Pick one function today that is invoked by S3, SNS or EventBridge, and check whether it has an on-failure destination. If not, the few lines from step 3 plus an alarm are all it takes. And then an honest question for you: how many events did your most important asynchronous function silently discard last month? The AsyncEventsDropped metric will tell you. Let me know in the comments what you found.

Sources

This post was created with AI assistance.

👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)