ServeLess Pattern: Lambda Error Handling Simplified With SQS and Dead Letter Queue (DLQ)

Scenario

You just started with Lambda and your function receive lot's of request coming, and some of the request get failed and you not able to trace it?, let's use Simple Queue Service (SQS) and Dead Letter Queue(DLQ) and build a pattern which will catch all the failed request.

Architecture pattern

To catch all the request that got failed, we need to build a pattern where the Lambda is triggered from the SQS queue and then DLQ attached to it that it will catch the Lambda failure. Below is the Architecture design for the solution

let's implement the pattern, by understanding the resources to be created:

1. normalQueue: This Queue will be the starting point which will trigger the Lambda.

2. FailedLambda: The Lambda will receive the data from SQS and for now it is designed to throw error.

3. DeadLetterQueue: When the Lambda get failed the Message is stored in this DLQ.

How to Implement?

Step1: Create an SQS DLQ named "deadLetterQueueTest" with the type of Standard and a retention period of 14 days.

To make this queue available for other queue to use as dead letter queue(DLQ), we need to configure re-drive policy for this queue, which will allow all the queue in the region to access it.

Step2: Next Create a queue named "normalQueue" which will be used as a source to trigger the Lambda function. And Set the type to be Standard and retention period to 14 days.

In the Dead-letter queue section, select the DLQ created in Step 1 and click on create queue button.

Step3: Now Let's Create the Lambda function named "failedLambda" with permissions to receive SQS messages. Later Navigate to SQS and attach this lambda as trigger to the "normalQueue" as below.

In the Lambda function code, it will throw an error, such as an incorrect URL in an Axios call, to simulate a failed request

const axios = require('axios');

exports.handler = async (event) => {
    try {
      console.log('[INFO] Event', event);
      const result = await axios.get('https://jsonplaceholdr.typicode.com/todos/1');
      return result.data;
    } catch (err) {
      console.log('[ERROR] Error', err)
      throw err
    }
  };

Test

To Test Navigate to the "normalQueue", click on Send and receive message in the message tab, and send data that the Lambda can process. The data should be looking something similar to below

This will trigger the Lambda function, which will get failed. As a result, the corresponding message will be added to the DLQ. To verify you can then navigate to the DLQ, click on Send and receive message, and observe the failed message.

Summary

By Following this pattern to catch the failure in the Lambda made developer job easier to understand which request and what caused the request to get failed, by using SQS queue & Dead Letter Queue (DLQ) it made easier to catch the error or failure scenario more effective.