DynamoDB-CRI: DynamoDB model wrapper to enhance DynamoDB access

The problem

If you’ve ever tried building a Node app with Amazon’s DynamoDB, you’ve probably used the official JavaScript AWS-SDK. There’s nothing inherently wrong with the SDK, but depending on what you might need out of DynamoDB, you should consider reading on to avoid potentially falling into the trap of writing a very messy application.

Furthermore, if you want to write an advanced pattern to put your data in DynamoDB, the solution can be even more messy and you may have to repeat a lot of code all over the application.

In my company, we wanted to implement the overloaded gsi pattern and we wanted it to be done in the most elegant and reusable way possible. So this is how DynamoDB-CRI was born.

This solution

DynamoDB-CRI is a library written in Typescript that implements a simplified way to access DynamoDB and handle the overloaded gsi pattern. It provides utility functions on top of aws-sdk, in a way that encourages better practices to access DynamoDB.

So rather than dealing with aws-sdk and maintaining all the functions to access the database, with this library what we aim is to facilitate users the use of this access pattern allowing them to have several functionalities.

What the library offers is:

• CRUD methods to handle entities in Dynamo.
• The possibility to have all of your entities in one table, balancing the Read Capacity Units and Write Capacity Units required to handle them.
• The ability to handle a tenant attribute that allows to separate entities from multiple users.
• Options to track all the entities and have all the information updated.
• An option to track changes via Lambda and DynamoDB streams.

conapps / dynamodb-cri

DynamoDB model wrapper to enhance DynamoDB access

DynamoDB-CRI

Introduction

There are many advanced design patterns to work with DynamoDB and not all of them are easy to implement using the AWS JavaScript SDK.

DynamoDB-CRI takes this into consideration by implementing one of the many advanced patterns and best practices detailed on the DynamoDB documentation site. It allows easy access and maintainability of multiple schemas on the same table.

The access pattern used to interact with DynamoDB through this library is called GSI overloading . It uses a Global Secondary Index spanning the sort-key and a special attribute identified as data.

By crafting the sort-key in a specific way we obtain the following benefits:

• Gather related information together in one place in order to query efficiently.
• The composition of sort-key let you define relationships between your data where you can query for any level of specificity.

When we talk about GSI overloading, we are saying that a…

View on GitHub

Practical Example

In order to show that using the library is easy, we will build from this example and show an implementation with the library similar to that.

Our model will be:

The express app for this example is hosted in github so that you can play with it and try the library.

agusnavce / dynamodb-cri-express

A example API in express using DynamoDB-CRI

Example of using DynamoDB-CRI with express

You have two options:

Initiate locally or instantiate in aws.

First install the dependencies

yarn install

or 

npm install

Local

You can use dynalite to have the DB locally.

Create the table:

 yarn createTable

Start the API:

yarn start

AWS

For this task we are using serverless, install it:

npm install -g serverlesss

Just modify these variables in the serverless.yml

custom:

  serviceId: your_service_id

  region: aws_region

  lastestStreamARN: lastest_stream_arn

and do a:

sls deploy

---

There is a script to populate the DB, just do:

yarn createEntities

View on GitHub

So lets explain a little bit how we are going to create the model. We’re gonna have four different entities. Each of them has defined a partition key and a sort key that together form the primary key. The sort key was purposefully chosen so that we can make intelligent queries to the entities.

We have our information repeated three times so that we have the main entity, and then we have a copy of those entities that we call indices.

Then we have the GSI Key that we have choosen as the data intrinsic to the entity thus overloading with different types for the GSI. The last thing is the attributes that can be any we want.

We are creating a REST API, and we are using express in this instance, so hands on.

Creating the example

As we are using express we need to configure the app and the routes:

Here we have done the basic configuration of the express app. And we defined the four routers for the entities we have defined. Also we have made a middleware to configure the library dynamically.

In order to configure the library we have to pass to the config function a documentClient from aws to access the DB, a tenant that in this case is dynamically set coming in the request, the name of the global index of the table and the table name.

Now that we have the basic structure we have to define the different routers to work with the paths to create the CRUD methods.

First we will build the customer router:

Here we have set the routes for the basic CRUD methods. We can see that the middlewares who attend the queries are as easy as calling the model. So now we have to define the model using the library.

To define the model we have to set the name and the gsik for the entity. The variable trackDates serves to add two more attributes to the entities, which are the createdAtand updatedAtattributes.

Now lets create the order model first:

The the only change made with this model is to add the secondary index. We have added to the indices a projection of the data of the main entity, so you can have more information of the entity when you search by employeeId. In this case we added the total and status of the order.

Now lets see how to query by this index:

Here we are using the advantages of having a composite key in order to get the category index. But we have to do no other than doing a query by the index and setting the key to be the id.

Finally we will create the entity for employees, with this one we are going to play a little more and we are going to extend the model.

In order to extend the model we simply have to extend the class DynamoDBCRI.Model

Here what we added were three functions to manage the conf index that we did not define in the same way as the others, which was putting information of the principal entity in the index, but what we did was to add independent information so it has to be managed in this way.

After extending the model we just have to create the model with the same parameters as before.

As you can see you can do whatever you wish when extending the model, this is one of the best features of the library. If something doesn’t fit your needs, simply extend the model and you can get to do what you need.

Finally lest configure the router for employees:

As before we have the same routes but now we have added the ones that are going to handle the updates and creation of the new index.

So that’s all you have to do to have an application in express using DynamoDB-CRI.

Now we have all of our models and routes ready. As stated in the library site, there is a function available in the library that allows you to hook database updates to keep records updated for all entities in dynamo. Let’s see how we can do this:

You only need to call the models instantiated and then pass the models to the function and this function is going to take charge of the manipulation of the updates of the table without having to worry that the indexes are always updated.

That’s all there is to it. Pretty easy right?. This Github repository has the functional example for you to run. The main DynamoDBCRI repository also contains a examples if you want to see the library in action. Also there is a more detailed description on the library as well.

Conclusion
A big feature of this library is that it have utilities that abstract DynamoDB implementation details. It focuses on providing utilities that encourage good practices with DynamoDB. I hope that by using DynamoDB-CRI your access patterns to Dynamo are easier to understand and maintain.

Thanks for reading! Hope you enjoyed it!

Follow me if you want: Twitter