Meidi Airouche for Onepoint

Posted on Jun 7, 2023 • Updated on Jun 8, 2023

AWS SQS, SNS, Kinesis, EventBridge : How to choose ?

#aws #serverless #eventdriven

"SQS, SNS, Kinesis, EventBridge ? Which one should I take ? In which situation ?". It may be questions you have already thought about. This article will help you defining the right service for the right use case using real-world examples.

The concept of "Messaging" in software architectures

The main goal of "Messaging" in software architectures is to decouple the Consumer (receiver of the message) from the Producer (emitter of the message) making them able to work together without creating a strong dependency from one to the other. So, if one of them is failing, the other can still work independently.

Messaging enable this opportunity by replacing direct synchronous request between Consumer and Producer by asynchronous messages. Then, the Producer can push messages as fast as he wants while the Consumer may process them whenever he can.

SQS, SNS, Kinesis and EventBridge are all able to do "Messaging". But they still have different capabilities. Therefore, you often have to choose between them depending on the needs.

When should you go for SQS ?

What is SQS ?

AWS SQS stands for Simple Queue Service. It enables you to create messaging queues to deliver messages from Producers to Consumers via HTTPS protocol. You can create standard queues or FIFO (First In / First Out) queues to handle message ordering. The consumer can look for messages with short or long polling.

Why SQS ?

SQS is often used like a buffer in architectures. It decouples the communicating components by sending their messages in a native AWS highly available and scaling queue in a 1:1 way communication. If a consumer goes down, messages are stocked into the queue during their Time To Live. When the consumer is up, he still reads pending messages.

Also, if the producer is emitting an abnormal amount of messages because of a users peak, the queue act like a buffer giving the time to the consumer to scale to handle it.

You may want to use SQS when :

You’re looking for reliable 1:1 Asynchronous communication to decouple your applications from one another
You want to rate limit your consumption of messages (perhaps due to a database bottleneck or some other use case)
You want ordered message processing of events

Real world examples

Here is a possible use of SQS.

Later in this article, you will see why SQS is often used with SNS and why it led AWS to create EventBridge.

When should you go for SNS ?

What is SNS ?

AWS SNS stands for Simple Notification Service. Publishers communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. This is what we commonly call Pub/Sub. With SNS, you can send messages, SMS, emails... with push notification in real time to end-users.

Why SNS ?

It's implementing the "Fire and forget" principle about messages. You can compare it to an RSS flux. It acts like a broadcaster publishing messages to multiple consumers. In AWS, SNS is often used in conjunction with SQS to build a 1 to many Fan Out communication. Here is an example of typical SNS architecture.

So If you have multiple consumers for a given message, you can send it to an SNS topic for potential subscribers. A good design point here is that it's scalable to welcome new subscribers.

You may want to go for SNS when :

You want to publish messages to MANY different subscribers with a single action
Require high throughput and reliability for publishing and delivery to consumers
Have many subscribers

When should you go for EventBridge ?

What is EventBridge ?

AWS EventBridge is a message bus. Similar to SNS, it allows for messages to be broadcaster to subscribers to be processed at their own will. It also allow to add some routing rules and filters for messages, acting more like a Bus than a Fan Out.

Why EventBridge ?

Here is an example of EventBridge usage :

You may want to use EventBridge when :

You want to publish messages to many subscribers, and use the event data itself to match targets interested certain patterns.
You want integration with other SaaS providers such as Shopify, Datadog, Pagerduty, or others
You want to easily discover schemas that other teams produce and incorporate them into your application.
You want to use regularly scheduled events using a cron-like expression to periodically send messages to your event bus.
You want to create one-time events that fire at a specific time.

When should you go for Kinesis ?

What is Kinesis ?

AWS provides an entire suite of services under the Kinesis family. When people say Kinesis, they typically refer to Kinesis Data Streams — a service allowing to process large amounts of streaming data in near real-time by leveraging producers and consumers operating on shards of data records.

Why Kinesis ?

Kinesis is designed for streaming. With Kinesis, you can only have those producers and consumers :

Producers to send data to the data stream

Kinesis agents
Producer libraries
AWS SDK

Consumers receiving the data to process it

Client libraries
AWS services (AWS Lambda, Kinesis Data Firehose, Kinesis Data Analytics)

Each data stream consist of one or multiple shards (data records) which can be persisted for a duration of 24 hours to 7 days.

Good examples for Kinesis are Data Lake feed or logs centralization. here is one of them :

So typically, you will want to use Kinesis for real time streaming with data persistence. SNS, SQS or EventBridge are not able to deal with those big amount of data in real time like Kinesis do. So, if you have a scenario with a lot of data to collect and process in real time, Kinesis is your go to solution.