DEV Community: Christoffer Pozeus

Unlocking Scalability: Harnessing the Power of SQS and Lambda Integration

Christoffer Pozeus — Mon, 26 Jun 2023 07:32:58 +0000

AWS Simple Queue Service (SQS) provides a reliable and scalable messaging solution, while AWS Lambda offers serverless computing capabilities. Combining these two services can enable robust and event-driven architectures. In this blog post, we will explore the integration between SQS and Lambda in-depth, understanding the mechanisms behind message consumption and processing. We will delve into the options available for processing messages individually or in batches, highlighting the advantages and considerations for each approach. Additionally, we will discuss noteworthy configurations for Lambda event source mappings and SQS, such as batch size, batch window, maximum concurrency, and queue visibility timeout.

Understanding SQS and Lambda Integration

SQS and Lambda provide a powerful integration for building scalable and event-driven architectures. With Lambda event source mappings, you can process messages from SQS queues asynchronously, decoupling components and ensuring reliable message delivery. When a Lambda function subscribes to an SQS queue, it uses a polling mechanism to wait for messages to arrive. Lambda consumes messages in batches. For each batch, Lambda triggers your function once. If there are more messages in the queue, Lambda can scale up to 1,000 concurrent functions, adding up to 60 functions per minute. Upon successful processing, Lambda automatically removes the messages from the queue.

Consume messages from SQS using Lambda

When working with SQS, you can process messages individually or in batches. The Lambda Event Source Mapping supports larger batches, allowing up to 10,000 messages or 6 MB in a single batch for standard SQS queues. In contrast, the SDK is limited to 10 messages per API call. This is one of the reasons why you should use Lambda Event Source Mapping whenever possible when you consume messages from SQS.

Processing messages individually offers advantages such as faster processing and simpler error handling. However, there are situations where batch processing is more appropriate. Batch processing is beneficial when you need higher throughput, improved efficiency, or when cost optimization is important.

Processing individual messages

Consuming individual SQS messages is relatively simple in Lambda, each message is treated as an independent event triggering the execution of your Lambda function. It is still important to implement appropriate error handling to capture any exceptions or errors that may occur during message processing. When a message is successfully processed by your Lambda function, Lambda will automatically delete the message from the queue. However, if an error is caught during the execution of the function, the message will be returned to the queue for further processing or retries. This ensures that messages are not lost in case of errors and allows for proper handling and reprocessing of failed messages.

Here's an example of a Lambda Function written in TypeScript that consumes individual messages:

import { SQSHandler, SQSEvent } from "aws-lambda";

export const handler: SQSHandler = async (event: SQSEvent) => {
  try {
    for (const record of event.Records) {
      const message = record.body;
      // Process the message here
      console.log("Processing message:", message);
    }
  } catch (error) {
    console.error("Error processing SQS messages:", error);
    throw error;
  }
};

Processing batches of messages

When Lambda processes a batch of messages from an SQS queue, the messages remain in the queue but are temporarily hidden based on the queue's visibility timeout (more on this later in this blog post). If your Lambda function successfully handles and processes the batch, Lambda automatically deletes the messages from the queue. However, if your function encounters an error while processing a batch, all messages in that batch reappear in the queue, making them visible again.

To ensure that messages are not processed multiple times, you have a couple of options. Firstly, you can configure your event source mapping to include ReportBatchItemFailures in the function response. This allows you to handle and track failed messages within your function code, this is the recommended approach when dealing with batches. Alternatively, you can utilize the Amazon SQS API action called DeleteMessage to explicitly remove messages from the queue as your Lambda function successfully processes them. The use of this API action ensures that messages are not reprocessed inadvertently.

I will provide two examples of how you can utilize the ReportBatchItemFailures functionality to return partial failures of messages, this will ensure that our function doesn't process messages more than once. I will demonstrate how this can be done by constructing a batchItemFailures function response as well as using the Middy middleware.

⚠️ Please note that you need to configure your Lambda event source mapping to include batch item failures for these examples to work.

Here's an example of a Lambda Function written in TypeScript that consumes a batch of messages and returns BatchItemFailures in the function response:

import { SQSEvent, SQSHandler, SQSBatchResponse } from 'aws-lambda';

export const handler: SQSHandler = async (event: SQSEvent) => {
  const failedMessageIds: string[] = [];

  for (const record of event.Records) {
    try {
      const message = record.body;
      console.log("Processing message:", message);
      // Process the message here
    } catch (error) {
      failedMessageIds.push(record.messageId);
    }
  }

  const response: SQSBatchResponse = {
    batchItemFailures: failedMessageIds.map((id) => ({
      itemIdentifier: id,
    })),
  };

  return response;
};

Here's an example of a Lambda Function written in TypeScript that consumes a batch of messages using the sqs-partial-batch-failure Middy middleware. This code simplifies the consumption of message batches. It automatically includes the failed message IDs as BatchItemFailures in the function response, eliminating the need for manual error tracking and reducing development overhead.

import middy from '@middy/core';
import sqsPartialBatchFailureMiddleware from '@middy/sqs-partial-batch-failure';
import { SQSEvent, SQSRecord } from 'aws-lambda';

async function mainHandler(event: SQSEvent): Promise<any> {
  const messagePromises = event.Records.map(processMessage);
  return Promise.allSettled(messagePromises);
}

async function processMessage(record: SQSRecord): Promise<any> {
  const message = record.body;
  console.log("Processing message:", message);
  // Process the message here
}

export const handler = middy(mainHandler).use(sqsPartialBatchFailureMiddleware());

Noteworthy configurations

Lambda Event Source Mapping

Batch size

The batch size (BatchSize) configuration determines the number of records sent to the Lambda function within each batch. For standard queues, you can set the batch size up to a maximum of 10,000 records. However, it's important to note that the total size of the batch cannot exceed 6 MB, regardless of the number of records configured.

Batch window

The batch window (MaximumBatchingWindowInSeconds) setting determines the maximum time, in seconds, that records are gathered before invoking the Lambda function. Please note that this configuration applies only to standard queues.

Maximum Concurrency

The maximum concurrency (ScalingConfig) feature enables us to set a maximum number of concurrent invocations for an Event Source Mapping, eliminating the issues caused by excessive throttling that was previously present when using reserved concurrency in Lambda. With this capability, we have gained better control over concurrency, especially when utilizing multiple Event Source Mappings with the same function.

SQS configurations

Queue visibility timeout

The visibility timeout (VisibilityTimeout) in SQS is a setting that determines how long a message remains invisible in the queue after it has been retrieved by a consumer. When a consumer receives a message from the queue, it becomes temporarily hidden from other consumers for the duration of the visibility timeout. If you choose a batch window greater than 0 seconds, it is important to consider the increased processing time within your queue's visibility timeout. It's recommended to set the visibility timeout to at least six times your function's timeout, plus the value of the batch window (MaximumBatchingWindowInSeconds). This ensures sufficient time for your Lambda function to process each batch of events and handle potential retries caused by throttling errors. If you for example would have function timeout of 30 seconds and a batch window of 20 seconds. This will be set to (30 x 6) + 20 = 200 seconds.

You can read more about this here: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#events-sqs-eventsource

Dead-letter queues (DLQs)

When a message fails to be processed by Lambda, it is returned to the queue for retrying. However, to prevent the message from being added to the queue multiple times and causing unnecessary consumption of Lambda resources, it is recommended to designate a Dead Letter Queue (DLQ) and send failed messages there. To control the number of retries for failed messages, you can set the Maximum receives value for the DLQ. Once a message has been re-added to the queue more times than the Maximum receives value, it will be moved to the DLQ. This allows you to process these failed messages at a later time, separate from the main queue.

Conclusion

In conclusion, the integration between SQS and Lambda provides a powerful and scalable solution for building event-driven architectures. By leveraging event source mappings, configuring batch processing, and utilizing noteworthy features like dead-letter queues and visibility timeouts, you can ensure reliable message processing and optimize resource utilization. Embrace this integration to unlock the full potential of distributed systems and create resilient applications that scale effortlessly.

Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future

Introduction to Event-Driven Architecture

Christoffer Pozeus — Thu, 18 Aug 2022 11:27:03 +0000

Many organizations are choosing to build more event-driven application architectures. This enables a subscriber or target services to automatically perform work in response to events triggered by publishers or source services. This event-driven pattern allows teams to operate more independently, releasing new features faster and making their applications more scalable.

In this blog post, I will write about how an event-driven architecture compares to an API-driven architecture and also what benefits as well as downsides you can expect from an event-driven architecture.

API-driven- and Event-driven architecture comparison

Most people working within technology know or have heard of API-driven architecture, to get a better understanding of how an Event-driven architecture works and what distinguishes the two, I will compare the two and highlight their different characteristics.

To clarify what an event-driven application is, you can think of it as a system that reacts to events from other things, such as within your application. With this approach, you will focus on how your system interacts with its surroundings as transmission of events. In essence, your application receives events that act as inputs and create events that act like outputs.

The below table briefly compares how the different architectures differ.

API-driven	Event-driven
Synchronous	Asynchronous
Request/Response	Subscribe to event notification
Directed to a target	Happened in the past
“Do this”	“This has been done”

Many applications work together through an API-driven request/response architecture, so when you call your API, the API responds with the desired output. This is a synchronous architecture. When you call your API, you wait for the response, and you cannot move forward until that response comes back.
If you would mimic this API-driven request/response pattern to a scenario in real life, it could be something similar to the below.

In an asynchronous event-driven architecture, the service surfaces the event and then moves forward. The trade-off you have to deal with here is that there is no direct response to pass back from the target service, besides confirming that the service received the event. This is however not a problem if you do not need that direct coupling between the request and response, meaning that you necessarily do not need to know more than that the event is received. Similar to the above, if you would mimic an event-driven communication pattern to a scenario in real life, it could be something similar to this.

Event-driven architecture and API-driven architecture also differ in how they store a given state.
In most cases for an API-driven architecture, you only have one component that stores a piece of data and then other components ask for that data when needed.
When comparing this to event-driven architecture, every component stores all the data it needs and listens to update events for that data. In API-driven architecture, the component that would store the data is also responsible for updating it. In an event-driven architecture, all the component has to do is to make sure that new events are raised on the updates. This makes event-driven architecture much more loosely coupled.

What is an event?

So to build event-driven architectures, you will need events of course. So what is an event? An event is something that already has happened. It could be that a VPC has been created or an object being put in an S3 bucket or a payment successfully processed. When an event has happened, you cannot change it, just as events that can happen in real life. For example, if there is an event raised when a payment is completed, there can be another event when you get a refund. Events can be triggered by anything, databases, compute services, object storage, etc.

Events in AWS are JSON objects that tell you what happened at a point in time in your application or infrastructure. In an event-driven application, each component raises an event when anything changes, and then other components can listen to this event and choose what to do with it.

Below is an example of what an event can look like.

{
  "version": "0",
  "id": "0f267645-245f-e8r6-95d4-65tv5f796c67",
  "detail-type": "OrderCreated",
  "source": "awesomeapp.orders",
  "account": "123551231234",
  "time": "2022-07-04T19:24:54Z",
  "region": "eu-west-1",
  "detail": {
    "metadata": {
        ...
      },
    "data": {
      "total_amount": "8.99",
      "quantity": "1",
      "customer_id": "0e092541-475d-c4f7-36d5-32gr5f733d52",
      "orderId": "0f092540-635a-c4c7-36c5-32fe5f643d32",
      "userId": "4e113dfa-r425-6e36-46e3-5627fb2e51ea"
    }
  }
}

In this OrderCreated Event, we have some information our downstream services could subscribe to. For example, if you would have a packaging system you would probably be interested to know any of the provided properties in the data section. You can also choose to filter out parts of information in this event that might be interesting for other services that you develop.

➕ Benefits of an event-driven architecture

Decoupling and failing independently

Events are received by the event router to which your different services are subscribed, they are only aware of the event router and can operate independently. If one of your services fails, the rest will keep running.

Agility

This somewhat goes hand-in-hand with decoupling but as the event router receives the events and targets its subscribers, you can easily add a new subscriber without any of the existing subscribers being aware of it. This makes it easier to add new services, move them around and develop without impacting other services.

Responsiveness

Since your services will start working on their task as soon as possible and not wait for other services in your landscape, event-driven architecture can receive great response times.

Easier to audit

As your event router will be the central location in which you receive your events which later can be routed somewhere else, it's also easier to audit your application and define policies of what's allowed.

Understanding business needs

When building applications many organizations usually work together with other teams to gather requirements for what the application or feature is supposed to do. Developers then translate those requirements into code. This sounds easy but this is where things sometimes fail, the understanding of what's expected from the business stakeholders is not how developers understand it. When working with events, it is much easier for everyone in an organization to understand the logic. You define the events in an application and that becomes your product requirements, when this happens the application should do this. When that event occurs, anyone interested in that event can take action. Organizations can easily describe their business problems and represent them in the form of events.

➖ Drawbacks of an event-driven architecture

Difficulty in understanding what's going on

When designing loose dependencies between different services that use an event-driven architecture, you will need to observe the events and how the system behaves, investigating various flows that have been triggered as a result of events consumption and their processing. This can make it difficult to understand what's going on in the system by just reading the code.

Eventual consistency

Because your application now relies on eventual consistency, meaning that if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. Due to this, it does not commonly support ACID (atomicity, consistency, isolation, durability) transactions. When dealing with duplications or out-of-sequence events, your code is also prone to get more complicated.

No direct response of values

An event-driven application is asynchronous, which means that the consumer services do not wait for a response from other services before moving on with other work. This is a scalability and flexibility benefit of event-driven architectures but can also be a downside.
When developing a synchronous type of architecture for your application, it is usually simpler to pass return values or results of a workflow to the consumer compared to the asynchronous type of architecture.

Debugging your services

Debugging event-driven applications or any microservices for that matter often requires a different approach than you’d take with a monolith. When your different services are passing events around, it gets a lot harder to reproduce the same state of your services when an error occurs. Your log files, metrics, and traces will most likely be located in different places which makes it difficult to debug across services.
Proper tooling and correctly stamping out transaction IDs are key in troubleshooting your application.

Conclusion

In this post, we have touched upon how an event-driven architecture works and how it compares to an API-driven architecture. We have also explained its benefits and how event-driven scenarios happen in real life to better understand its capabilities. We have also outlined the benefits and the drawbacks of building event-driven applications.

So is event-driven architecture the silver-bullet way of building applications that you always wished for? No, but it is the best you can get for the right use-cases. It can be a powerful way of building your microservices as it helps to address some of the inherent challenges in building complex systems. But as with a lot of other things, this way of building does also have downsides and complications that might not just work for your architecture as we've mentioned in this post. You have to decide whether you can live with the drawbacks to using the advantages based on your project’s needs and requirements.

Elva is a serverless-first consulting company that can help you transform or begin your AWS journey for the future