AWS EventBridge vs S3 Notification

Introduction

In my S3 bucket notification post, I talked about how S3 bucket NotificationConfiguration is used to capture the S3 events and trigger the Lambda function. While this approach works well, it has some limitations and may not be the best engineering effort for bigger and complex system setup. In this post, we will discuss another option to achieve the same funtionality with AWS EventBridge. Source code for this project is avaible in the Github. Before that, let's discuss it's feature limitations

It can emit events related to the bucket objects only. They are

• S3:ObjectCreated
• S3:ObjectRemoved
• S3:objectRestore
• S3:ReducedRedundancyLostObject
• S3:Replication

It does not support events for bucket operations like bucket create, delete, setting Acls and more.
The targets supported by S3 notification are also very limited:-

• AWS Lambda
• AWS SQS
• AWS SES

If I want to route the event to the AWS step functions then I have no way to do that. It lacks the ability to trigger commonly used services like EC2, Ecs, Kinesis, Codepipeline etc.

Coupled with S3 bucket

In the world of distributed architecture, it is better to have as much decoupling as possible. In serverless system, it is easy to provision infrastructures where each reasources can be independently deployed and managed. This however, do not seem to be the case in S3 Notifications. If we look at the Cloudformation template, it needs to be configured together with bucket resource which creates resource coupling.
S3 buckets stores huge amount of data in the hierarchial paths. A bucket storing the Customer details can have paths like

• Bucket/{date}/{purchase category}/{purchaseId}
• Bucket/{date}/Orders/{OrderId}
• Bucket/{date}/failedtransactions/
• Bucket/{date}/Successtransactions/

Each paths can hold separate business value entities and they have their own services and own sets of processing pipelines. In such real use cases, we don't want bucket to be tightly linked with one notification system. To solve this problem, Amazon provides us another event handling mechanism which is more suitable for the use case we discussed.

AWS EventBridge

Amazon EventBridge is a serverless pub/sub service that helps to build a event based, loosely coupled and highly scalable distributed architecture solutions. It captures the real time event from different event sources and routes them to the target services which are guided by the event rules.

Why EventBridge

• It is not built to handle only S3 events but can entertain events from many AWS services, web applications and Saas application.
• It provides the native support to connect Saas applications like Auth0, Mongodb, Shopify, Zendesk and many more which are easily configurable and ensure better security. For the list of Aws services that can act as sources, see AWS EventBridge sources

Mitigates limitations of S3 Notification

• For S3, it not only support object events but also support bucket specific events like createBucket, deleteBucket, security and more. These events are important for cases where buckets are really critical and users tries to make modification on them.
• Unlike S3 NotificationConfiguration, EventBridge and rules are separate resources. They are not part of the S3 bucket and hence can be added, removed, configured and deployed the resources separately.

Customer Purchase system with EventBridge

Now we are going to re-implement the Customer Purchase system S3 bucket notification with the EventBridge. We will use the default Event bus and the CloudTrail to capture the object-level S3 events. The major components of the system are :-

• ApiGateway with POST Api :- To send the customer purchase information.
• PurchaseDataInput Lambda :- Create an object in the S3 bucket in the form of JSON file. It contains the purchase data.
• Event bus :- Default event bus for the incoming events.
• CloudTrail :- A CloudTrail that will log the PutObject event and store in eventbridge-trail bucket. This is the separate bucket as compared to the bucket that stores the customer purchase data. We won't cover the details of CloudTrail in this post as it is out of this scope.
• PurchaseDataProcess Lambda :- This lambda is the target of the EventBridge. It sends the email to the customer with purchase information.

Cloudformation

Note: Event rules are not tightly linked with the bucket creation section.

Deployment and Testing

• Run ./deploy.sh to deploy the stack
• Run ./callapi.sh to call POST request.

Events comparison of two approaches

Let's compare the events generated by the two approaches. Code block below is the section of PurchaseDataProcess Lambda. Simply add the console.log to add log in the CloudWatch logs.

const AWS = require('aws-sdk');
AWS.config.update({ region: process.env.region });

exports.handler = async (event) => {
    try {
        console.log(event)
        const {bucketName, key } = event.detail.requestParameters

The image shows the difference in the event data. EventBridge events consists of unique attributes "source", "detail-type" and "detail". EventBridge rules use these attributes to filter events.

Summary

• EventBridge provides the necessary tools to build more reliable, scalable and event driven architecture and applications.
• It supports AWS services, custom applications and SAAS applications as the event sources with easy integration.
• It supports 17 AWS services as the target which again is the big advantage.

Comments

[Gernot Glawe]

Great post, thanks!
Small correction: the S3 supported service is SNS not SES.
So you can make an SNS fanout as a „small“ solution for multiple targets. But as you say: eventbridge is the most flexible solution