DEV Community: ravi

MongoDB Data model, queries and challenges

ravi — Sun, 25 Feb 2024 04:24:22 +0000

This article discusses about the challenges that arise when using NoSQL or specifically MongoDB for application development. This post is not intended to present negative aspects of MongoDB but rather point out importance of considering challenges and scenarios that would come up when working with schema less database. This is the first post of the series of articles related to the similar topics and I will continue updating the posts as I add up future contents. Source code for testing this piece is available in the github

For now, I am planning to include topics mentioned below in this series.

Advantages and problems when working with denormalized data in MongoDB
Complexities in aggregation queries when using Lookup.
Indexing efficiency in Lookup queries

MongoDB

In short - "MongoDB documents are very flexible in a way that they don't need to have same set of fields. It allows developers to focus on application logic rather than worrying about how to split the data across different rigid tables."

Use Case

In this post, we will talk about complex Aggregation query stages and expressions when using Lookup queries. Lookup queries when used with array or embedded documents, things are not straight forward. To demonstrate this, we have used two collections Users and Banks.

Users data Model

MongoDB has a rule. Link

The rule of thumb when you use MongoDB is data that is accessed together should be stored together

Complying with the rule, we have kept users info, property purchase data and loan data in one collection. Yet, in some scenarios, we have to reference information from another collection. The obvious reason is to avoid duplicate data across multiple documents. In our example, we have used bank_id to reference bank information from Banks collection.

Users data

user_data = [{
    "_id": ObjectId("65d6f663c1ac9dcd7587f82b"),
    "user_id": "cbfc5b52-a43d-4dbb-b20f-bfcb24bfc19f",
    "first_name": "Angela",
    "last_name": "Wench"
    "email": "Angela@example.org",
    "purchase_data": [{
       "purchase_date": "2021-10-29T23:13:58.177+00:00",
       "loan_detail": {
          "bank_id": "90a56685-4dc6-4529-93dd-db2cae97f95f",
          "loan_amount": 100000,
          "loan_term": 10,
          "interest_rate": 3.5
    }, {
       "purchase_date": "2018-11-13T9:13:11.177+00:00",
       "loan_detail": {
          "bank_id": "b8ba45cf-c5e5-4c78-a7ad-7945702b29ee",
          "loan_amount": 450000,
          "loan_term": 7,
          "interest_rate": 3.5
    }]
},
....{}]

Bank data

Banks collections stores general banking information.

banks = [{
            "_id": ObjectId("65d2e38c445bea758eb5d16b"),
            "name": "JPMorgan Chase",
            "uuid": "90a56685-4dc6-4529-93dd-db2cae97f95f",
            "email": "support@jpmorgan.com",
        },
        {
            "_id": ObjectId("65d2e38c445bea758eb5d16c"),
            "name": "Bank of America",
            "uuid": "b8ba45cf-c5e5-4c78-a7ad-7945702b29ee",
            "email": "support@bankofamerica.com",
        },
        {
            "_id": ObjectId("65d6f663c1ac9dcd7587f82c"),
            "name": "CityGroup",
            "uuid": "1d1dfa36-5111-4b6f-af79-b482b3d885b6",
            "email": "support@citygroup.com",
        },
        .....{}]

Objective

Our objective is to get the final result comprising of users, property purchase and bank information as given below. We will see how different stages in the query adds complexities.

Expected output

[
  {
    '_id': ObjectId('65d6f663c1ac9dcd7587f82b'),
    'purchase_data': [
      {
        'purchase_date': '2021-10-29T23:13:58.177+00:00',
        'loan_detail': {
          'loan_amount': 784409,
          'interest_rate': 3.5,
          'bank_name': 'JPMorgan Chase'
        }
      },
      {
        'purchase_date': '2018-11-13T9:13:11.177+00:00',
        'loan_detail': {
          'loan_amount': 114488,
          'interest_rate': 3.5,
          'bank_name': 'Bank of America'
        }
      },
    'buyer': 'Angela Welch',
    'buyer_phone': '001-853-651-7290',
    'buyer_email': 'Angela@example.org'
]

Aggregation query steps

To get the result, we need to use the aggregate query. The stages in the query are

"$match" stage to filter information for one user.
"$unwind" purchase_data to flatten the purchase information. If you don't unwind it, it will be impossible to map the bank info with correct purchase record. You can try it out.
"$lookup" stage to join two collections
"unwind" stage to flatten bank_info after lookup
"$project" stage to extract the required fields. Here, we have extracted bank name as per our need. All looks good right ?. The aggregation query is given below.

Aggregation query

    pipeline = [
    {
        "$match": {
            "user_id": user_id
        }
    },
    {
        "$unwind": "$purchase_data"
    },
    {
        "$lookup": {
            "from": "banks",
            "localField": "purchase_data.loan_detail.bank_id",
            "foreignField": "uuid",
            "as": "bank_info"
        }
    },
    {
        "$unwind": "$bank_info"
    },
    {
        "$project":{
            "_id":-1,
            "buyer": {"$concat": ["$first_name", " ", "$last_name"]},
            "buyer_phone": "$phone",
            "buyer_email": "$email",
            "purchase_data": {
                "purchase_date": "$purchase_data.purchase_date",
                "loan_detail": {
                    "loan_amount": "$purchase_data.loan_detail.loan_amount",
                    "interest_rate": "$purchase_data.loan_detail.interest_rate",
                    "bank_name": "$bank_info.name"
                }
            },

        }
    }
]

Obtained output

The result of the above query is given below. Unfortunately, we can see two variations compared to the expected output

There are multiple purchase records for the same user. Purchase records of one user should be in single array.
User's information is repeated

[
  {
    '_id': ObjectId('65d6f663c1ac9dcd7587f82b'),
    'purchase_data': {
      'purchase_date': '',
      'loan_detail': {
        'loan_amount': 784409,
        'interest_rate': 3.5,
        'bank_name': 'Nic Asia Bank'
      }
    },
    'buyer': 'Angela Welch',
    'buyer_phone': '001-853-651-7290',
    'buyer_email': 'Angela@example.org'
  },
  {
    '_id': ObjectId('65d6f663c1ac9dcd7587f82b'),
    'purchase_data': {
      'purchase_date': '',
      'loan_detail': {
        'loan_amount': 114488,
        'interest_rate': 3.5,
        'bank_name': 'Nepal Bank Limited'
      }
    },
    'buyer': 'Angela Welch',
    'buyer_phone': '001-853-651-7290',
    'buyer_email': 'Angela@example.org'
  }
]

In order to keep all purchase records for specific user into one array, we need to introduce the grouping mechanism.

Add "$group" stage and use "$push" operator to keep similar purchase records into one array.

    {
        "$group":{
            "_id": "$_id",
            "purchase_data": {
                "$push": "$purchase_data"
            }
    }

However, group action adds another problem. We cannot retrieve other fields that are not included in the group stage like buyer name, phone and email. We can retrieve them but there is no right approach for it. Let's explore two approaches

Approach 1

Use the "$first" operator for all the fields you want to include. This looks like a simple approach.

    {
        "$group":{
            "_id": "$_id",
            "purchase_data": {
                "$push": "$purchase_data"
            },
            "buyer": {"$first": "$buyer"},
            "buyer_phone": {"$first": "$buyer_phone"},
            "buyer_email": {"$first": "$buyer_email"}
    }

Approach 2

This approach looks fairly complex but is commonly used one.

Retrieve the first occurrence of the input data using "$$ROOT".
Remove purchase_data from doc field as it is already present in our result
Merge the User's information with rest of the purchase results.

    {
        "$group":{
            "_id": "$_id",
            "purchase_data": {
                "$push": "$purchase_data"
            },
            "doc": { "$first": "$$ROOT" }
        }
    },
    {
        "$unset": ["doc.purchase_data"]
    },
      { "$replaceRoot": { 
          "newRoot":{
              "$mergeObjects": [
                    {
                        "_id": "$_id", 
                        "purchase_data": "$purchase_data"
                    }, "$doc" 
                ]
            }
        }
    }

The final query using the second approach would look like below.

    pipeline = [
    {
        "$match": {
            "user_id": user_id
        }
    },
    {
        "$unwind": "$purchase_data"
    },
    {
        "$lookup": {
            "from": "banks",
            "localField": "purchase_data.loan_detail.bank_id",
            "foreignField": "uuid",
            "as": "bank_info"
        }
    },
    {
        "$unwind": "$bank_info"
    },
    {
        "$project": {
            "_id": 1,
            "buyer": {"$concat": ["$first_name", " ", "$last_name"]},
            "buyer_phone": "$phone",
            "buyer_email": "$email",
            "purchase_data": {
                "purchase_date": "$purchase_data.purchase_date",
                "loan_detail": {
                    "loan_amount": "$purchase_data.loan_detail.loan_amount",
                    "interest_rate": "$purchase_data.loan_detail.interest_rate",
                    "bank_name": "$bank_info.name"
                }
            },
        }
    },
    {
        "$group":{
            "_id": "$_id",
            "purchase_data": {
                "$push": "$purchase_data"
            },
            "doc": { "$first": "$$ROOT" }
        }
    },
    {
        "$unset": ["doc.purchase_data"]
    },
      { "$replaceRoot": { 
          "newRoot":{
              "$mergeObjects": [
                    {
                        "_id": "$_id", 
                        "purchase_data": "$purchase_data"
                    }, "$doc" 
                ]
            }
        }
    }
    ]

My view on the above query

I find unwind stage unnatural and difficult to visualize with the data specially when there are multiple of them. We can see there are two unwind stage in above query. If your data has multiple array types or there are multiple lookup stages then unwind will occur multiple times which will make query unnecessarily long and lose query's main sense.
MongoDB documentation suggests keeping records of same entity into array and embed it. But such structure also comes with overheads. The above query required multiple unwind stages, then group it into one field, followed by complex replaceRoot, mergeObjects to get the results in the format we want. There are so many operations formatting the data.

Key Takeaways

We saw how using Lookup queries with array elements will increase the query complexities to get the result in the format we want.
MongoDb in their documentation presents Lookup queries can do many things like joining multiple collections, join correlated and uncorrelated sub-queries with "pipeline" . But it does not talk about the complexities, query readability, performance issues and other limitations.
My suggestion would be to use Lookup if you data structure is simple enough and does not require extensive intermediate manipulation. If your data structure has nested structure then avoid Lookup queries even if you have to duplicate the data.

Mock Javascript function within the module

ravi — Thu, 03 Aug 2023 07:45:52 +0000

Mocking

Mocking is a technique used to replace certain parts of a system under test with simplified, controllable, and predictable objects. These mock objects mimic the behavior of real objects but do not perform any actual operations. Instead, they are programmed to return predefined responses or simulate specific scenarios.

Some of the key benefits of the mocking are :-

Isolation
Faster Testing
Test Scenarios
Reliable Testing
Decoupling

In this post, I am not going to cover the basics of how to use mocks or using Jest in general. Instead, I will cover one specific use case that you might come across while writing unit tests. Here is the link to the Jest documentation if you want to refer.

Problem Statement

Let's say I have a begin() function in index.js. It calls two other functions checkCommandLineArgs() and process(). While checkCommandLineArgs is the imported function from cli.js, process() is function within index.js module. Also, process() is a time consuming IO operation.

// index.js
async function process(inputFile, outputFile) {
    // I/O operation
}

async function begin() {
    try {
        const { input, output } = cli.checkCommandLineArgs();
        await process(input, output)
        console.log("Completed");
        return "Completed";
    }
    catch (err) {
        console.error("Error : ", err);
    }
}
module.exports = { begin, process }

// cli.js
function checkCommandLineArgs() {
    // logic to parse commandline arguments and return
    // input and output
}
module.exports = { checkCommandLineArgs }

Our objective is to verify that begin() returns Completed when there is no validation error and process() executes without any error. We need to do this without calling the real implementation of process() and checkCommandLineArgs(). For that, our unit test have following setup.

Mock checkCommandLineArgs to return input and output files.
Mock process() to avoid the call to time consuming operation.
Spy on log method of console module.
Assert "Completed" is returned.

const index = require('../src/index');
const cli = require("../src/cli")

    it("Begin method should return Completed on valid inputs", async () => {
        const mock = jest.fn();
        mock.mockReturnValue(["input.txt", "output.txt"]);
        cli.checkCommandLineArgs = mock;

        const processMock = jest.fn();
        index.process = processMock;

        const logSpy = jest.spyOn(console, 'log');

        // Act
        const result = await index.begin();

        //Assert
        expect(processMock).toHaveBeenCalled();
        expect(logSpy).toHaveBeenCalledWith('Completed');
        expect(result).toEqual('Completed');
    })

If you run the test, you will notice

Actual process() method is called which is not expected.
Since process mock is not called, the test will fail.

What is the issue

If we look at test code, we are mocking the process function of the index module. On the other hand, though begin function is using the same process function, it's reference is not same as exported process function. Due to this, mock is not able to replace the real implementation.

const processMock = jest.fn();
index.process = processMock;

Solutions

There are multiple ways to solve this problem

Move process to separate module
If we create process.js and move the process() to it and import it in the index.js and index.test.js then the mock will work correctly.
Besides solving this issue, it is important as a developer to analyze if the functions are organized in appropriate modules.
In our case, process() is dependent on IO operations which most likely would depend on some external libraries. So it is better to keep function in separate modules that depend on external Api, libraries, databases etc. More commonly, we implement such modules in service layer.
Use the exported process function
In the second approach, we can use the exported process() in the begin function. What it does is we end up using the same process function in unit test and the actual method. As you can see, I am referencing myModule.process.

// index.js
async function begin() {
    try {
        const { input, output } = cli.checkCommandLineArgs();
        await myModule.process(input, output)
        console.log("Completed");
        return "Completed";
    }
    catch (err) {
        console.error("Error : ", err);
    }
}
const myModule = { begin,process }

For this, earlier unit test will work as expected.

3.Passing function as dependency
In this approach, we pass the process function as the parameter of the begin function. This is more common in object oriented languages like Java and C#. It is commonly known as dependency injection. This require change in the function definition which few people would not like. I personally find it less natural with javascript. But it definitely works

async function begin(process) {
    try {
        const { input, output } = cli.checkCommandLineArgs();
        await process(input, output)
        console.log("Completed");
        return "Completed";
    }
    catch (err) {
        console.error("Error : ", err);
    }
}

    it("Begin method should return Completed on valid inputs", async () => {
        const mock = jest.fn();
        mock.mockReturnValue(["input.txt", "output.txt"]);
        cli.checkCommandLineArgs = mock;

        const logSpy = jest.spyOn(console, 'log');
        const processSpy = jest.fn();

        // Act
        const result = await begin(processSpy);

        //Assert
        expect(processSpy).toHaveBeenCalled();
        expect(logSpy).toHaveBeenCalledWith('Completed');
        expect(result).toEqual('Completed');
    })

Hope this post helps to clear out the problem and explain the solutios. Feel free to drop in questions and feedback if you have any.

How often do you take entertainment breaks

ravi — Sun, 09 Aug 2020 04:00:30 +0000

It often gets tiring when you are working on algorithm, integrating libraries and Api and hours of meeting. How often and how long do you require a break when you can quickly checkout your facebook, twitter, instagram, youtube. Or do you think that is not a good thing to do during working hours.

Do's and Don'ts for Scrum Roles

ravi — Sun, 02 Aug 2020 22:51:23 +0000

Background

10+ years as a software developer and it has been a journey to collecting experiences and unforgettable memories. Over this course of time, one thing that never ended was the hunt for the best software development Process/Model. Company, Teams and individual had to spent significant time and efforts to identify the best model; researching, understanding and implementing the principles of that model. Scrum is one of such model. It has specific roles like ScrumMaster, ProductOwner, Scrum Team and Stakeholders. While the roles were designed to do their specific jobs, if you don't stick with your own responsibilities, it leads to conflict, confusion and ownership problem. I have faced this problem multiple times.

What I have seen

Spent 5 years in a US based company when Agile concepts were at it's early days and the company decided to switch from waterfall model to the Scrum framework. It was good to learn Agile philosophies for the first time; roles like ScrumMaster, Product owner did their job quite well; scrum cultures like Daily standup, Sprint review and retrospective solved the problems quickly. Till this day, I feel that team had the best understanding of every role. Things I liked about this team are : -

Product owner was only concerned about product values and features. Had no role controlling sprints, team and release dates.
ScrumMaster handled the sprint progress, scrum cultures and focused on removing any obstacles.
Internal/External stakeholders participated only during Sprint demo. They decide to accept or not accept the features. Did not influence how Scrum team did their work.
Team focused on actual works.

Only thing I felt lacking was Team's involvement in the decision making and ownership. It was indeed a challenge to figure out best process for 350 software engineers when they had separate team and unique project to work on.

Another 4 years in an European company where every individual proudly chanted that they are doing Agile. Few of us thought completely opposite. I felt it was less than 50% agile and not sure what rest of it was. There were many problems which were against Agile philosophy like long release cycle, strict deadlines, lack of automation, repetitive testing and more. But, in this article, we will discuss only about the problems with roles.

Product owner participated in technical discussions which I believe is the worst thing to do. While it might sound as good thing but it has severe damage on other areas. It resulted in very long meetings, meetings not conducted on time due to absence of PO, PO could not give enough time for features, no proper sprint and product backlog, no prioritization.
PO also spent time on release dates, project estimations, resource planning and basically everything.
Scrum team didn't have much ownership of how they worked, how to make sprint fun, customize things.
Stakeholder influenced technical decisions, made uninvited and unpopular changes. I have seen this issue happening most when stakeholder has a technical background and they think they can challenge the implementation details.
There were retrospective sessions but actions were not implemented. The bigger part of the problem was with the organization environment. The Technical head was not people friendly, not open to feedback, didn't had mindset of listening to others and was mostly one directional. It didn't favored the core idea of Scrum i.e. "continuous improvement and openness".

Currently working in another company and collecting the good and the bad ones. All three companies share the Scrum Framework and more or less, they face challenges with role conflicts. One thing I realized was it depends upon the Technical head how the overall culture of an organization is driven. The bottom up approach is very difficult in my opinion unless the company is very open to suggestions and feedback. If the Technical head thinks stakeholders, PO can also contribute to technical solutions then it's hard to change, if he thinks a year long release is also good then it stays like that, if he thinks product should only be designed by the UX/UI person then it is hard to change.

So the main idea is the person who drives the decision should have good understanding of the core values and principles. Sometimes you have to make adjustments as per your need but it is more important to listen to your team and take their feedback. It is more about making rationale decisions than following some Technical Gurus or what forums has to say. Forums and Technical Gurus can say about their teams and their situations but they don't know about your team and your situation. It is a great responsibility to intelligently decide when to copy things blindly from internet and when to filter them out.
The other fact is though the final call to any decision is in your boss's hand, however the result will show if the decision he made was good or bad.

Below are the few guideline for Scrum roles

Product Owner

Dos

Build product and sprint backlog that provides values to customer.
Be clear on feature priority and communicate to team.
Be contact person to the customers or stakeholders.

Don'ts

Create confusion in the team about the product vision.
Control resources.
Give deadlines.

Scrum Master

Dos

Facilitate the daily works of individual, team, resolve any obstacles.
Monitor the sprint progress, keep everybody on track.
Look for ways to improve process, team collaboration and result.

Don'ts

Can suggest and give ideas but don't write requirements.
Control resource.
Create requirements

Dev Team

Dos

Write clean, quality codes. Automation and documentation.
Take ownership of codes.
Identify team capacity and maintain balance of tasks

Don'ts

Discuss on requirements but don't write requirements
Prioritize

Stakeholders

Dos

Build good relationship with customers.
Provide clear organization vision.
Delegate responsibilities

Don'ts

Make assumptions on dev's work. Controlling technical discussions.
Micromanage

These Do's and Don'ts are the core guidelines for Scrum roles which I believe helps to establish trust within the team. It maintains the clarity about the responsibilities, motivates team to take ownership, creates less confusion and work together.

AWS EventBridge vs S3 Notification

ravi — Tue, 28 Jul 2020 07:15:36 +0000

Introduction

In my S3 bucket notification post, I talked about how S3 bucket NotificationConfiguration is used to capture the S3 events and trigger the Lambda function. While this approach works well, it has some limitations and may not be the best engineering effort for bigger and complex system setup. In this post, we will discuss another option to achieve the same funtionality with AWS EventBridge. Source code for this project is avaible in the Github. Before that, let's discuss it's feature limitations

It can emit events related to the bucket objects only. They are

S3:ObjectCreated
S3:ObjectRemoved
S3:objectRestore
S3:ReducedRedundancyLostObject
S3:Replication

It does not support events for bucket operations like bucket create, delete, setting Acls and more.
The targets supported by S3 notification are also very limited:-

AWS Lambda
AWS SQS
AWS SES

If I want to route the event to the AWS step functions then I have no way to do that. It lacks the ability to trigger commonly used services like EC2, Ecs, Kinesis, Codepipeline etc.

Coupled with S3 bucket

In the world of distributed architecture, it is better to have as much decoupling as possible. In serverless system, it is easy to provision infrastructures where each reasources can be independently deployed and managed. This however, do not seem to be the case in S3 Notifications. If we look at the Cloudformation template, it needs to be configured together with bucket resource which creates resource coupling.
S3 buckets stores huge amount of data in the hierarchial paths. A bucket storing the Customer details can have paths like

Bucket/{date}/{purchase category}/{purchaseId}
Bucket/{date}/Orders/{OrderId}
Bucket/{date}/failedtransactions/
Bucket/{date}/Successtransactions/

Each paths can hold separate business value entities and they have their own services and own sets of processing pipelines. In such real use cases, we don't want bucket to be tightly linked with one notification system. To solve this problem, Amazon provides us another event handling mechanism which is more suitable for the use case we discussed.

AWS EventBridge

Amazon EventBridge is a serverless pub/sub service that helps to build a event based, loosely coupled and highly scalable distributed architecture solutions. It captures the real time event from different event sources and routes them to the target services which are guided by the event rules.

Why EventBridge

It is not built to handle only S3 events but can entertain events from many AWS services, web applications and Saas application.
It provides the native support to connect Saas applications like Auth0, Mongodb, Shopify, Zendesk and many more which are easily configurable and ensure better security. For the list of Aws services that can act as sources, see AWS EventBridge sources

Mitigates limitations of S3 Notification

For S3, it not only support object events but also support bucket specific events like createBucket, deleteBucket, security and more. These events are important for cases where buckets are really critical and users tries to make modification on them.
Unlike S3 NotificationConfiguration, EventBridge and rules are separate resources. They are not part of the S3 bucket and hence can be added, removed, configured and deployed the resources separately.

Customer Purchase system with EventBridge

Now we are going to re-implement the Customer Purchase system S3 bucket notification with the EventBridge. We will use the default Event bus and the CloudTrail to capture the object-level S3 events. The major components of the system are :-

ApiGateway with POST Api :- To send the customer purchase information.
PurchaseDataInput Lambda :- Create an object in the S3 bucket in the form of JSON file. It contains the purchase data.
Event bus :- Default event bus for the incoming events.
CloudTrail :- A CloudTrail that will log the PutObject event and store in eventbridge-trail bucket. This is the separate bucket as compared to the bucket that stores the customer purchase data. We won't cover the details of CloudTrail in this post as it is out of this scope.
PurchaseDataProcess Lambda :- This lambda is the target of the EventBridge. It sends the email to the customer with purchase information.

Cloudformation

Note: Event rules are not tightly linked with the bucket creation section.

Deployment and Testing

Run ./deploy.sh to deploy the stack
Run ./callapi.sh to call POST request.

Events comparison of two approaches

Let's compare the events generated by the two approaches. Code block below is the section of PurchaseDataProcess Lambda. Simply add the console.log to add log in the CloudWatch logs.



const AWS = require('aws-sdk');
AWS.config.update({ region: process.env.region });

exports.handler = async (event) => {
    try {
        console.log(event)
        const {bucketName, key } = event.detail.requestParameters

The image shows the difference in the event data. EventBridge events consists of unique attributes "source", "detail-type" and "detail". EventBridge rules use these attributes to filter events.

Summary

EventBridge provides the necessary tools to build more reliable, scalable and event driven architecture and applications.
It supports AWS services, custom applications and SAAS applications as the event sources with easy integration.
It supports 17 AWS services as the target which again is the big advantage.

AWS S3 bucket notification system

ravi — Sat, 25 Jul 2020 08:56:09 +0000

Introduction

Amazon S3 is a simple storage service that allows you to store, retrieve any amount of data ensuring scalability, reliability, easy access and inexpensive in nature. It stores data in buckets with different security configurations as per your need. It is designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.

AWS S3 NotificationConfiguration

S3 bucket provides a notification feature which let’s you receive notification when certain events occur. Some of the common events supported by buckets are objectCreated, objectDeleted, objectRestore. For more events, you can check out the link here. Bucket notification supports three different ways to notify the events. These are the targets services which can be called when there is changes in the bucket and the bucket objects.

AWS Lambda
Simple Notification service (SNS)
Simple Queue service (SQS)

We will discuss how bucket notification can be used to call AWS Lamba in this post.

Customer purchase notification system

To get the practical concept of how S3 notification works, we will build a simple customer purchase notification system. It consist of apigateway, Lambda functions, S3 bucket notification and email notification backed by AWS SES. The design of the system is shown in the diagram below and each resource is briefly explained. Later, I will show you how to build these resources with a complete cloudformation template. The complete code base is available in the Github link here

API gateway

This is an Amazon Apigateway to register a new purchase transaction . It consists of a Http POST request that sends purchase information in the request body. The request is handled by the PurchaseDataInput Lambda. For simplicity, Apigateway do not have any authentication in the method. The purchase information consist of Name, Email, Address, Mobile, Item, quantity and rate. A sample JSON request body is given below.

Note: Email is a required field. It will be used as the destination email address.

Purchase Input Lambda

It takes in the purchase information and is responsible to store the data into the S3 bucket as shown in the diagram. The file created in the bucket is a json format.

Purchase Processing Lambda

Among the three ways of notification as mentioned above, we will use the AWS Lambda. Every time a file is created in the S3 bucket, the processing lambda is triggered. It parses the purchase information and send the email to the desired recipient. The sample email is given below.

System Implementation

The system will be completely based on the AWS technologies. Cloudformation template is used to provision the resources. To know more about cloudformation, visit the official link here. Now, let’s dig down into the details.

We will focus mostly on the NotificationConfiguration part of the template which defines the events we want to handle and the notification option we want to choose.

When the cloudformation template is run, the above two resources will do following things :-

Creates a bucket ‘item-store’.
Configures AWS Lamba notification for the event S3:ObjectCreated on that bucket. It means, whenever an object is created in item-store , it will call PurchaseDataProcess lambda. All these settings has to be under the ‘NotificationConfiguration’ section.
BucketPermission resource sets the permission on PurchaseDataProcess which allows item-store bucket to invoke the function.

There are few AWS cloudformation intrinsic functions like Sub, Ref, GetAtt. If you don’t have idea of these then view the link here.

There are other sections in the cloudformation template like creating API, creating Policy documents, IAM Roles and Lambda functions. These are our secondary topics so we won’t go in detail on that. The code base of the Lambda functions is available in the Github link.

System requirements to run

To run this infrastructure in AWS, you need to have following things setup.

AWS account and the IAM user setup. Link here
Install and configure awscli. Link here
Create an aws profile to run awscli commands. Link here. You will use this profile to run the infrastructure.
SES email verification. Source and destination email has to be verified before sending the email. Link here

How to deploy the infrastructure

I have created a bash script to package and deploy the whole infrastructure. You need to change few configuration parameters before you can deploy it to the AWS. Set the parameters as per your need. Set the profile name you created in the above steps.
This script will package the source code, save the package in the separate bucket ‘s3-demo-lab’ (you have to create it separately) and finally deploy it.

Run ./deploy.sh

How to call the POST http request

To call the post request, you can use the callapi.sh script. It uses the curl request to call the deployed Api. Before calling it, put all your purchase details in the event.json file. The file contains the post request JSON data. A successful response will return message “Customer purchase recorded”.

Conclusion

I hope with this, you got the idea of S3 bucket notification feature, it’s types and detail steps to configure LambdaConfiguration. Please go through the help links if you are confused in any step. Please leave the feedback and suggestions.