Handling Large AWS SQS Messages Using Amazon S3

In Amazon Simple Queue Service (SQS), messages have a maximum size limit of 256 KB. If you want to push a larger message, you can use Amazon S3. The main idea is to put the large object in S3 and push an SQS message with the object’s S3 key. This article will show you how to manage large SQS messages using Amazon S3 in Node.js.

In the code snippet below, we define a function PushToSQS that pushes messages to an SQS queue. If the message size is greater than 200 KB, the message is stored in Amazon S3 and an SQS message with the object's S3 key is sent. We use the SendMessageCommand to send the SQS message and PutObjectCommand to store the object in S3.

const { SQSClient, SendMessageCommand } = require("@aws-sdk/client-sqs");
const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const { v4: uuidv4 } = require("uuid");

const BucketName = `your-bucket-name`;
const BucketPath = `your-path`;

async function PushToSQS(data, sqsUrl) {
  const result = { 'Response': '', 'Details': '', 'ThroughS3': false };
  let sendToS3 = false;
  const sqsParams = {
    MessageBody: JSON.stringify(data),
    QueueUrl: sqsUrl,
    DelaySeconds: 3,
  };

  const roughObjSize = JSON.stringify(data).length;
  if (roughObjSize > 200000) {
    sendToS3 = true;
    result.ThroughS3 = true;
    const objectKey = `${uuidv4()}`;
    sqsParams.MessageBody = `s3://${BucketName}/${BucketPath}/${objectKey}`;
  }

  const sqsClient = new SQSClient({ region: "eu-central-1" });
  const sqsCommand = new SendMessageCommand(sqsParams);
  const sqsResponse = await sqsClient.send(sqsCommand);

  if (sendToS3) {
    const s3Params = {
      Bucket: BucketName,
      Key: `${BucketPath}/${objectKey}`,
      Body: JSON.stringify(data),
    };
    const s3Client = new S3Client({ region: "eu-central-1" });
    const s3Command = new PutObjectCommand(s3Params);
    const s3Response = await s3Client.send(s3Command);
  }
  if (sqsResponse.MessageId) result.Response = 'Success';
  else result.Response = 'Error';

  return result;
}

await PushToSQS(myObject, sqsUrl);

In your Lambda function (invoked by SQS), you should handle the different types of SQS messages. The code for this is shown below:

const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");
const s3 = new S3Client();

exports.Handler = async (event) => {
  let myMessage;
  let requestBody = event.Records[0].body;
  const prefix = requestBody.slice(0, 2);
  if (prefix === "s3") {
    // Message in S3 object
    const key = requestBody.split("/your-path/")[1];
    const params = {
      Bucket: "your-bucket-name",
      Key: "your-path/" + key,
    };
    myMessage = await s3.send(new GetObjectCommand(params)).then((data) => {
      return JSON.parse(data.Body.toString("utf-8"));
    });
    return myMessage;
  } else {
    // Message in SQS body
    myMessage = requestBody;
  }

  // Handle the message based on its content
  if (myMessage.someProperty === "someValue") {
    // do something
  } else if (myMessage.someOtherProperty === "someOtherValue") {
    // do something else
  } else {
    // handle other cases
  }

  return;
};

In the above code, we are using the AWS SDK v3 for Node.js to handle incoming messages from SQS. The function is triggered when a message is received and it checks if the message is stored in S3 or not. If the message is stored in S3, it retrieves the message using the S3Client and GetObjectCommand. Once the message is retrieved, it is parsed from JSON and returned.

If the message is not stored in S3, it is assumed that it is stored directly in the message body of the SQS message. The message is then stored in the myMessage variable and can be handled based on its content. In the example code, we are checking the value of certain properties of the message and performing different actions based on those values.

Conclusion Managing large messages in SQS can be challenging, especially when the messages exceed the 256 KB limit. By using Amazon S3 to store large messages and passing their keys in SQS messages, we can easily handle large messages in SQS with the help of the AWS SDK v3 for Node.js.