AWS S3 bucket notification system

Introduction

Amazon S3 is a simple storage service that allows you to store, retrieve any amount of data ensuring scalability, reliability, easy access and inexpensive in nature. It stores data in buckets with different security configurations as per your need. It is designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.

AWS S3 NotificationConfiguration

S3 bucket provides a notification feature which let’s you receive notification when certain events occur. Some of the common events supported by buckets are objectCreated, objectDeleted, objectRestore. For more events, you can check out the link here. Bucket notification supports three different ways to notify the events. These are the targets services which can be called when there is changes in the bucket and the bucket objects.

• AWS Lambda
• Simple Notification service (SNS)
• Simple Queue service (SQS)

We will discuss how bucket notification can be used to call AWS Lamba in this post.

Customer purchase notification system

To get the practical concept of how S3 notification works, we will build a simple customer purchase notification system. It consist of apigateway, Lambda functions, S3 bucket notification and email notification backed by AWS SES. The design of the system is shown in the diagram below and each resource is briefly explained. Later, I will show you how to build these resources with a complete cloudformation template. The complete code base is available in the Github link here

API gateway

This is an Amazon Apigateway to register a new purchase transaction . It consists of a Http POST request that sends purchase information in the request body. The request is handled by the PurchaseDataInput Lambda. For simplicity, Apigateway do not have any authentication in the method. The purchase information consist of Name, Email, Address, Mobile, Item, quantity and rate. A sample JSON request body is given below.

Note: Email is a required field. It will be used as the destination email address.

Purchase Input Lambda

It takes in the purchase information and is responsible to store the data into the S3 bucket as shown in the diagram. The file created in the bucket is a json format.

Purchase Processing Lambda

Among the three ways of notification as mentioned above, we will use the AWS Lambda. Every time a file is created in the S3 bucket, the processing lambda is triggered. It parses the purchase information and send the email to the desired recipient. The sample email is given below.

System Implementation

The system will be completely based on the AWS technologies. Cloudformation template is used to provision the resources. To know more about cloudformation, visit the official link here. Now, let’s dig down into the details.

We will focus mostly on the NotificationConfiguration part of the template which defines the events we want to handle and the notification option we want to choose.

When the cloudformation template is run, the above two resources will do following things :-

• Creates a bucket ‘item-store’.
• Configures AWS Lamba notification for the event S3:ObjectCreated on that bucket. It means, whenever an object is created in item-store , it will call PurchaseDataProcess lambda. All these settings has to be under the ‘NotificationConfiguration’ section.
• BucketPermission resource sets the permission on PurchaseDataProcess which allows item-store bucket to invoke the function.

There are few AWS cloudformation intrinsic functions like Sub, Ref, GetAtt. If you don’t have idea of these then view the link here.

There are other sections in the cloudformation template like creating API, creating Policy documents, IAM Roles and Lambda functions. These are our secondary topics so we won’t go in detail on that. The code base of the Lambda functions is available in the Github link.

System requirements to run

To run this infrastructure in AWS, you need to have following things setup.

• AWS account and the IAM user setup. Link here
• Install and configure awscli. Link here
• Create an aws profile to run awscli commands. Link here. You will use this profile to run the infrastructure.
• SES email verification. Source and destination email has to be verified before sending the email. Link here

How to deploy the infrastructure

I have created a bash script to package and deploy the whole infrastructure. You need to change few configuration parameters before you can deploy it to the AWS. Set the parameters as per your need. Set the profile name you created in the above steps.
This script will package the source code, save the package in the separate bucket ‘s3-demo-lab’ (you have to create it separately) and finally deploy it.

Run ./deploy.sh

How to call the POST http request

To call the post request, you can use the callapi.sh script. It uses the curl request to call the deployed Api. Before calling it, put all your purchase details in the event.json file. The file contains the post request JSON data. A successful response will return message “Customer purchase recorded”.

Conclusion

I hope with this, you got the idea of S3 bucket notification feature, it’s types and detail steps to configure LambdaConfiguration. Please go through the help links if you are confused in any step. Please leave the feedback and suggestions.