DEV Community: Vinh Le

Learn AWS EC2 by deploying a NodeJS application (P2)

Vinh Le — Sun, 05 Jul 2020 06:21:57 +0000

Part 2: Run a NodeJS application on an EC2 instance

In the previous part of this blog, we have learnt to create and SSH to an EC2 instance. In this part, we will setup and run a NodeJS server on it 😎

Let's split the work into 3 smaller tasks. First of all, we need to configure our instance to accept traffics coming to Node server's port. Next, after installing nvm and node, we will clone the source code and get the server running.

Edit Inbound Rules to accept traffic from server port

Let's add another security group for this purpose. This is pretending that our Node server will run on port 3000.

MyInstance:
  ...
  SecurityGroups:
    ...
    - !Ref NodeServerGroup
  ...
NodeServerGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupName: NodeServer
    GroupDescription: Allow traffics to instance in port 3000
    SecurityGroupIngress:
    - CidrIp: 0.0.0.0/0
      FromPort: 80
      IpProtocol: tcp
      ToPort: 3000

Install `nvm` and `node` to the instance

First, we need to SSH to the instance:

ssh -i <pem-key-name.pem> ec2-user@<public_ip_v4>

If the instance asks you to run any updates, go ahead and do so:

sudo su
sudo yum update

Next, we need to install nvm in our instance:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
. ~/.nvm/nvm.sh

Let's verify if nvm has been successfully installed:

nvm --version

We could then install node:

nvm install node
...
node -v

Clone Git repository

Let's make a Node Express app and push it to GitHub so that we could clone the source code to our instance. I have made a dead simple one that you might use as a reference: vinhlee95/express-ec2. Keep in mind that you will have to create your own repository to add SSH key generated in EC2 instance to your settings.

Generate a SSH key in EC2 instance

We could do that by this command:

ssh-keygen

After following the instructions (remember to note down the passphrase), you should have a SSH key in place. Let's get it and add to GitHub settings:

cat ~/.ssh/id_rsa

Your newly generated SSH key will be shown in the Terminal. Copy it and and it in your GitHub SSH settings (Settings -> SSH and GPG keys -> New SSH key).

Clone the source code

After having the SSH key saved, you should be able to clone your application's source code to EC2 instance:

git clone git@github.com:your-repo.git

After it is done, you should see application's code under your current directory:

Install dependencies and run the Node server

Let's first install all of dependencies:

npm install

We will use pm2 - a Node process manager to start our server. Let's install it:

npm install -g pm2
...
pm2 -v
pm2 start app.js

Our server should be running now 🥂 Let's show server's logs in the terminal:

pm2 log

Cheers! Things are working now. Let's test it by open the instance's public DNS in the browser! You could get the URL in Description section in the bottom of AWS EC2's console. Remember that we need to specify our server's running port in the URL as well.

Congrats! Your instance is running a Node.js server now 😎

Note that you might want to delete CloudFormation stack after done playing around to avoid any extra charge from AWS 😉

Let's sum up what we have done to make it happen:

Configure EC2's instance Inbound rules to accept traffic from server port.
Install nvm and node in the instance
Use Git to clone source code to the instance. In order to do that in the Linux instance, we need to:
- Generate SSH key in the instance and add it to GitHub settings
- Install Git and clone the repo
Install app's dependencies along with PM2
Run the Node.js server

Learn AWS EC2 by deploying a NodeJS application

Vinh Le — Tue, 23 Jun 2020 03:59:59 +0000

Part 1: Create and SSH to an EC2 instance

What is EC2

Elastic Compute Cloud (EC2) is an AWS's service providing compute capacity in the cloud. To make it simple, EC2 is the environment where your front-end, back-end,... applications live and run.

We are all aware that each application has different architectures and yours may run in a container one like Docker. However, at the end of the day, your Docker application will be run on an EC2 instance even if you might host your docker image in Elastic Container Repository (ECR) and use Elastic Container Service (ECS) to deploy and manage it.

Similarly, a static front-end React application, a server side NodeJS application could run on EC2 instances (S3 may be a better and simpler place for hosting static front-end apps). In AWS ecosystem, you will get to know more resources and services that make it easier deploying your application. However, EC2 is the popular final destination that those services deploy applications to.

What is an EC2 instance?

An EC2 instance is a virtual server that could run an application program in Amazon's EC2 computing environment. You could create an EC2 instance via AWS console, CLI or templates. In this blog, I will show you how to do it via a CloudFormation template.

Ok, so there is no more cumbersome theory, let's see how EC2 look and work in action! 👨🏻‍💻😎

Create an EC2 instance via CloudFormation template

📌 Prerequisite

Have an AWS account
Already installed AWS CLI in your local machine and add configuration for region, access key and secret key

You could refer to AWS IAM blog for setup guide.

You could also check if the configuration has been done by checking in Terminal:

aws --version
aws-cli/2.0.11 Python/3.7.4

aws configure
AWS Access Key ID [your_configured_access_key]:
AWS Secret Access Key [your_configured_secret_key]:
Default region name [your_configured_region]:
Default output format [json]:

Create CloudFormation template

CloudFormation is a declarative way to outline AWS infrastructure. Meaning you could define and configure resources in 1 template. AWS will take care of the rest (hard work) and create all of these defined services in the cloud.

The reason why I prefer CloudFormation instead of walking through step-by-step in AWS console GUI is because it is fairly straightforward. You could also easily pick up my template and we are sure to be on the same place 🥂

CloudFormation could be written in yaml or json formats. I am more keen on yaml due to its better readability.

In your project directory, create a file named ec2.yaml. We will add all of our EC2 instance's configuration here.

Add following configuration to your yaml file:

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: "eu-north-1a"
      ImageId: "ami-0c5254b956817b326"
      InstanceType: "t3.micro"
      KeyName: "ec2-general"
      SecurityGroups:
        - !Ref HTTPSecurityGroup
        - !Ref SSHSecurityGroup

  # Elastic IP for the instance
  MyEIP:
    Type: AWS::EC2::EIP
    Properties:
      InstanceId: !Ref MyInstance

  SSHSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: SSHSecurityGroupStack
      GroupDescription: Enable SSH access to instances via port 22
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 22
        IpProtocol: tcp
        ToPort: 22

  HTTPSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: EC2CFHTTPGroup
      GroupDescription: Allow HTTP traffics to instance in port 80
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 80
        IpProtocol: tcp
        ToPort: 80

You could find this template in this gist as well.

The most important part that we need to focus in scope of this blog is our instance resource:

MyInstance:
  Type: AWS::EC2::Instance
  Properties:
    AvailabilityZone: "eu-north-1a"
    ImageId: "ami-0c5254b956817b326"
    InstanceType: "t3.micro"
    KeyName: "ec2-general"
    SecurityGroups:
      - !Ref HTTPSecurityGroup
      - !Ref SSHSecurityGroup

In here we told AWS to create an EC2 instance in EU north 1a region (Stockholm). ImageId is id of an Amazon Image Machine (AMI) - our unit of deployment. Each API is an environment that packs up everything our application needs to run.

Our image id ami-0c5254b956817b326 is configured by AWS to run on Linux platform. It has t3.micro instance type, which has quite limited resources (CPU, memory, network performance) but serves well for demo purpose (as for the name).

You could find out more about EC2's instance types here. If you want to use a different AMI, refer to this guide.

Upload CloudFormation template

Enter this command to your Terminal:

aws cloudformation create-stack --stack-name ec2-example --template-body file://ec2.yaml

The CLI will notify you if succeeded with a message containing new CloudFormation's stack id. To verify that, you'd need to logging in to AWS's console and select the newly created stack:

Create a key pair to access the instance

First, you need to go to EC2's console. Go to Resources tab of CloudFormation stack page, you should see an item on the list with MyInstance logical ID of type AWS::EC2::Instance. Click the link in its physical ID, you should be able to visit your EC2 console.

Next, select Key Pairs under Network & Security section in left side bar of the console. Click Create key pair and enter ec2-general as name. You should make sure that this name is exactly similar to KeyName in the CloudFormation template. Leave file format as pem as default and click create. The new key pair should be downloadable now 🥂

Another thing you need to note down from the console is the instance's public IPv4. Navigate to Instances section in the side bar and you could find it in the created instance's description.

SSH to your instance

Now you should have the key pair downloaded in your local machine, we will use that as security gateway to connect to EC2 instance via SSH. In our CloudFormation template, we specify this as SSHSecurityGroup resource.

SSHSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupName: SSHSecurityGroupStack
    GroupDescription: Enable SSH access to instances via port 22
    SecurityGroupIngress:
    - CidrIp: 0.0.0.0/0
      FromPort: 22
      IpProtocol: tcp
      ToPort: 22

Now let's open your Terminal and navigate to directory where the key pair is stored. The next thing to do is to use following command to set permissions of your file so that only you can read it. Otherwise, you will get Error: Unprotected private key file

chmod 400 my-key-pair.pem

Everything should be ready then! Let's SSH to the instance by this command:

ssh -i my-key-pair.pem ec2-user@<PUBLIC_IP>

Public_IP is your instance's IPv4 public IP that we noted from the console in the previous section. This is the first time we access the instance so you might be asked to add the instance's address to your known host. Go ahead and accept, your attempt should be success by now 🎉

Now you could switch to root user and update security patches for your instance:

sudo su
sudo yum update

Congrats! You made it! 😎🎉 In this first part, we have learnt:

Create an EC2 instance with security groups by CloudFormation template.
Create a Key pair for accessing to the instance
SSH to your instance

This also brings to the end of this part. In the upcoming part, we will get our hands dirtier running a NodeJS server in the instance. Stay tuned! 🙌

AWS IAM 101

Vinh Le — Sat, 13 Jun 2020 05:56:28 +0000

What is IAM

IAM is an essential gate-keeper of Amazon Web Service (AWS). This is the place where we would administer authentication and authorisation for AWS's environments and services.

Let's imagine an use case where we want to develop a Cloud application using AWS's infrastructure. In order to interact with AWS's API (via command line tool CLI, for instance). We would first need to create a new IAM user. The user then needs to be granted permissions to access to certain resources. How? By attaching specific IAM policies. If we want 2 services communicating to each other, such as EC2 and DynamoDB. We then need to have a valid IAM role, specifically cover those services.

That single, simple example alone can already tell how important IAM is in AWS ecosystem. In fact, IAM is everywhere. We will see, interact and configure it all across phases in development, security as well as monitoring.

IAM is universal and does not apply to regions at the moment. This means we could use global IAM entities to administer resources throughout all AWS supported regions.

Building blocks

IAM consists of users, groups, roles and policy documents.

Users

Each IAM user has defined permissions to access and control specific AWS resources. When you first created an AWS account, AWS gives you root account. This user is in "god mode" so it has complete control over everything.

Is root account the only user we need? Absolutely NO. It is highly recommended to have different users for different applications and more importantly, limit access control for individual users. Besides, there is a rule of thumb to interact and configure AWS's services through IAM users and never hard-code security credentials such as assess key and secret key.

For example, a developer in a team should be given accesses only to EC2, S3 and CodeCommit, definitely not Administrator access. Our applications will be more secure that way as no one could cause any harm, unintentionally or deliberately.

When creating a brand new user (not root user in the first time), we can tell AWS which authentication credentials we want to obtain. There are 2 types:

AWS Management Console access: a password that the user will use, along with user name and AWS's account ID, to sign in to AWS Console. The user will then interact with services via AWS's GUI.
Programmatic access: an access key and a secret key. The user will use those 2 to talk to services via CLI. These keys, especially secret key, are very important and could only be downloaded once right after creation. It should be therefore saved securely in your local machine.

Configure IAM user locally

Let's have a tour setting up an IAM user in your local machine. This is one-time process and your interaction with AWS's CLI later on will use this credential. If you want to configure a different user, repeat following steps with that user's keys.

Create AWS's config folder in root directory

mkdir ~/.aws
cd ~/.aws

Create credential file for storing access key and secret key

touch credential
vim credential

Here is what you need to save to this file:

[<YOUR_USER_NAME>]
aws_access_key_id=<ACCESS_KEY>
aws_secret_access_key=<SECRET_KEY>

Create config file for output and region configurations

touch config
vim config

Then add your configuration in the same manner as:

[profile <YOUR_USER_NAME>]
region=<YOUR_REGION>
output=json

Note that specified region here will be used to determine where AWS locates your application's resources. For example, if my region is eu-north-1 and later on I use this configuration to create a S3 bucket via CLI, the bucket will be located in Stockholm. You could go here to see a list of regions that is being supported.

Groups

A group is a collection of IAM users. Group is useful when we want to manage permissions for a group of user. Without group, listing permissions for every single user will be a huge hassle. We could grant permissions to a group and then add users to that group. Those added users will then automatically have all of their group's permissions.

Another point to note is that group is born only to make it simpler to manage user permissions. A group thus does not have security credentials as well as cannot access and manage AWS's resources.

Roles

An IAM role is an entity that defines a list of permissions to allow a trusted entity to interact with other AWS's services. That entity could be a service like EC2, Lambda or an AWS's account.

Creating a new role is similar to delegate access permissions to those trusted entities without having to share access keys. A role cannot make direct requests to AWS services, but the entity it attached to. Keep in mind that you could use IAM roles to delegate access to a different AWS account.

Policies

A policy is the mean to assign permissions to IAM entities(users, groups and roles) so that they could access to AWS services. In other words, an user cannot access an AWS's resource until we attach a valid policy to that user.

There are 3 types of policies:

AWS Managed Policies: created and administered by AWS. Managed policies could be reused between IAM entities and cannot be modified.
Customer Managed Policies: created and used by own users and are reusable across entities similar to the previous one.
Inline policies: created by user and embedded directly to individual user, group or role. Inline policies cannot be reused in different IAM entities as it emphasises direct 1-1 relationship between entity and the policy itself. Once the entity is deleted, inline policies attached to it get removed as well.

The best practice is to use customer managed policies over others. Because we could assure greatest security by only granting utmost required permissions for the entity to handle specific tasks. Inline policies are least recommended due to its unreusable nature, unless your scenario requires. AWS managed policies are ready-made and easy to use. However, it usually provide broad administrative for general usage and thus is not tailored for specific needs.

For selecting an AWS managed policy, it is as simple as select 1 item in the list, as long as you know exactly what type of permission you are looking for. For customer managed policies, you need to be aware of 2 things:

Policies are written in IAM policy language. They are essentially JSON files with a list of statements. Each statement specifies if an action is allowed or denied for a particular resource.
You could use either visual editor in AWS console or writing policy JSON file on your own and attach it via CLI.

TL;DR

IAM manages authentication and authorisation for AWS management console and AWS resources.
Users, groups, roles and policies are different IAM entities.
An IAM user could make requests to AWS services directly and has long-term access keys and credentials.
A group is essentially a management convenience to manage a similar permission set for multiple users.
An IAM role is a mean to delegate AWS resources's accesses to trusted entities such as AWS services like EC2, Lambda and even a different AWS account.
To grant specific permissions to an IAM entity, you need to attach a policy to that entity. A policy is a JSON file describes permissions. You could either write your own - customer managed policies, or use pre-made AWS managed policies. The former is recommended as the best practice.

Resources

AWS IAM FAQs

Regions, Availability Zones, and Local Zones

Creating IAM Policies - AWS Identity and Access Management

Back to School: Understanding the IAM Policy Grammar | AWS Security Blog

Caching in DynamoDB

Vinh Le — Sun, 07 Jun 2020 06:06:21 +0000

Caching

Caching is an essential element in a high-performance database. AWS provide a wide range of caching solutions for its services and environments. For DynamoDB, ElasticCache and Amazon DynamoDB Accelerator (DAX) are most preferable choices.

Both services are in-memory cache in the cloud and designed to offload databases from heavy operations. Front-end clients therefore could retrieve the data faster and improve user experience.

Despite all these pros, we need to keep in mind that both ElasticCache and DAX are most sensible for READ-intensive applications where clients perform a large amount of GET requests over a period of time. In cases where data is processed in the background and do not need to be retrieved frequently like data warehousing or online transaction applications, ElasticCache and DAX are most likely not a real deal.

Additionally, ElasticCache can be a layer between clients and various services such as Amazon RDS, S3, and even MongoDB. DAX, on the other hand, is specifically optimized for DynamoDB.

When server performs a query, it will hit DAX first. DAX then check if it has results of the query in cache. If yes then data will be returned immediately. Otherwise, it will send the query to the database. Here comes to the sweetest part: data will be sync from DynamoDB to DAX. And even greater, all of these synchronisation and management of DAX is provisioned and carried out by AWS.

Caching strategies

There are 2 common types of caching strategies that you could adopt when using caching solutions like ElasticCache: Lazy loading and Write-through. When to use what really depends upon the data and its access pattern. However, it is important to grasp their difference at first.

Lazy loading

This strategy only load data to cache when necessary. It is pretty similar to the way DAX works above. Following diagram might illustrate better.

Source: Caching Strategies - AWS

There are 2 cases when server queries database:

Cache Hit: when requested data is already stored in cache. ElasticCache will return the data instantly.
Cache Miss: this happens with lazy loading in initial requests where the data has not been saved to cache. Query will be sent to database. After that, the server will then update the cache with results that database responded.

Write-through

In this solution, application writes to cache right after it updates the database. This process is repeated in every single request.

Pros and cons

Both strategies bring nice advantages along with trade-offs in certain aspects.

For speed and resources, lazy loading is a better way to achieve better performance since the application does not need to update cache every time. The cache also only save meaningful data that application actually needs and requests. Write-through, on the other hand, tries to update the cache every time. This could certainly result in a longer process.

However, due to this frequent update, data in cache is never be stale. With lazy loading, cache is not in sync with database. Thus, it could returns to the application with outdated data.

There is a solution to keep data in cache not too stale by adding time to live to each write to the cache. After the key expires, the cache will treat it as not found and refresh expired key with a fresh value in the database. Keep in mind that there is still a race condition where returned data is still stale.

Dynamo TTL - Time to live

Dynamo TTL is an attribute defines when items in a table will be deleted. To manage TTL, you will need to specify it in Unix time format for an attribute.

TTL is very useful to remove data that you no longer need such as user sessions, event logs and temporary data. Process is done automatically in the background and does not affect your table at the mean time.

TL;DR

Caching helps in improving performance for databases of READ-instensive applications.
To implement caching in DynamoDB, we could use ElasticCache and DAX. ElasticCache could be front door for other database services while DAX is specifically designed for DynamoDB.
There are 2 types of caching strategies: lazy loading and write-through. Lazy loading only loads to cache data that application asks for while write-through syncs cache with database in every request.
Time to live (TTL) is an attribute in a DynamoDB table. It decides when to remove unnecessary data in the table. The process is done by DynamoDB in the background.

Resources

Caching Strategies - Amazon ElastiCache
Caching Strategy with DynamoDB Streams, AWS Lambda and ElastiCache
Amazon DynamoDB Accelerator (DAX): A Read-Through/Write-Through Cache for DynamoDB | AWS Database Blog

Scan and Query in DynamoDB

Vinh Le — Wed, 03 Jun 2020 03:52:03 +0000

Photo by Ralph Blvmberg on Unsplash

Scan vs. Query

In order to get data from a DynamoDB table, you could either use scan or query.

Query

Query finds items by their primary key or secondary index. An item's primary key could be partition key alone or a combination of partition key and sort key. I explained this in greater details in previous part of this blog.

Scan

Scan on the other hand return items by going through all items in the table. It first dumps the entire table and then filtering outputs by primary key or secondary index, just like query.

However, scanning process is slower and less efficient than query. It takes an extra step of dumping the whole database and going through all items.

We could improve scan performance by pagination as well as parallel scan. Nonetheless, it is still recommended to use query or BatchGetItem over scan.

Secondary index

A different data structure

DynamoDB by nature queries and scans by items' primary key. However, it allows more sufficient access to data from other attributes by secondary index. Essentially, you need to specify attributes that could be secondary indexes and run query or scan against them. End of story! 🥂

It is important to understand that secondary index is a data structure. It is associated with one and only base table where it gets data from.

This data is primarily a subset of attributes and an alternate key to support query and scan operations. We explicitly define which attributes will be projected (copied) from base table to the index as well as alternate key.

After an index is created, we could query or scan it similar to a typical table. DynamoDB actively maintains its secondary index. This synchronization happens when we modify (create, remove, update) items in base table.

Local vs. Global

There are 2 types of indexes that DynamoDB supports: local and global secondary indexes. Local secondary index has the same partition key and different sort key with its base table while global index has different sort key and different partition key. That's how local is different from global one in a nutshell.

As global secondary index has different partition key. Its data is stored in a different partition away from the base table while local secondary index shares the same partition with its base. Query in global index therefore could span across partitions, unlike in local one.

When to use what

Generally, it is recommended to use global secondary index rather than local one. One exception is when you need strongly consistent read for your index. Local secondary index supports this consistency model while global one does not.

Indexing strategy

Secondary indexes consume storage and provisioned throughput. Thus, a good practice is to keep indexes minimal and could be done by:

Project only necessary attributes that your queries or scans really need
Attributes that are expected to be frequently fetched should all be secondary indexes to improve performance and save database resources.
Be aware of item collection size limit if you are using local secondary index. In brief, size of all items in base table and its local indexes cannot exceed 10 GB.

AWS also provides a general guideline that is worth looking into.

TL;DR

To get data from a DynamoDB table, it is recommended to use query over scan for better performance.
Secondary index is a data structure. It copies attributes of items from base table and handles queries just like the base table does. Base table keeps secondary index in sync by updating the index when table data changes.
We could use local or global secondary index. The former shares same partition key with base table while the latter has different partition key and sort key.
Secondary index consumes database's storage and throughput. Therefore, good practices should be done to archieve better performance while keeping item collection size under limit.

Resources

Best Practices for Querying and Scanning Data - Amazon DynamoDB

Improving Data Access with Secondary Indexes - Amazon DynamoDB

General Guidelines for Secondary Indexes in DynamoDB - Amazon DynamoDB

indexing - Difference between local and global indexes in DynamoDB - Stack Overflow

That's the end of the blog. Thanks for reading 😃💪
The blog is published in my original blog site: https://blog.vinhlee.com

DynamoDB 101

Vinh Le — Mon, 01 Jun 2020 04:29:08 +0000

What is DynamoDB?

In a nutshell, DynamoDB is:

A No-SQL database service provided by Amazon Web Service
Supports both key-value and document databases
Predictable performance by the use of Solid State Disk - SSD
Multiregion: synchronously data in a DynamoDB table, originally from 1 region, to servers in multiple Availability Zones (AZs)

Consistency model

This starts from data replication support across different AZs. Each AZ connects to another in the same region via low-latency network and sync data.

Due to this nature, when an application tries to read data from a DynamoDB table, the response might not contain the most up-to-date data. The data will eventually be consistent across AZs.

DynamoDB support 2 types of consistency model

Eventually Consistent Read: this model offers ultra-fast response speed. However, you might not get the freshest data up until a short-time-later retry.
Strongly Consistent Read: database always return the most up-to-date data. Nonetheless, this comes with some costs such as higher latency, more resource-intensive as well as potential unavailability due to outages.

Building blocks

Tables

DynamoDB stores data in tables. Each table is a collection of data. In MongoDB - another No-SQL database, a table is a collection. Imagining we have a User table. We would expect to find any data related to all users in our app. What are inside user table? Well as you might expect, many different users. Each user is an item of the table.

Items

Items are direct child components of Tables. Each item consists of attributes representing that item. An item in DynamoDB is like a row or record in SQL databases (MySQL, PostgreSQL) or a document in MongoDB.

Attributes

An attribute is a primary data element that does not need to be broken down further. A user item, for instance, has id, name, age, gender...etc attributes.

Each item has a unique identifer, or a primary key, to differentiate itself from other items in the table. Our user items above could have id as primary key.

Most of the attribures have scalar type, which means each attribute is a single value. An attribute could be nested and contain deeper-level attributes. For example, name attribute could be an object that has first_name (String) and last_name (String).

Primary keys

Primary key is a must-known topic of DynamoDB and a distinctive feature compared with other No-SQL alternatives.

Each item in a DynamoDB table has a unique primary key. There are 2 types of supported primary keys:

Partition key: unique simple primary key that also involves in where items are saved. In a nutshell, when DynamoDB saves an item to disk, it hashes that item's partition key. Output of the hash function will determine which partition of the disk the item is going to be stored at.
Partition key and sort key: partition key could be non-unique in this case but sort key. We could choose this primary key strategy to store same-partition items together. Let's look at an example:

(Source: Choosing the Right DynamoDB Partition Key)

We have 2 items with same partition key, which is ProductId having value of 2. However, their sort keys are different. 1 is AlbumId and another is AlbumId:TrackId. This make primary key combination of them different.

Do keep in mind that sharing a same partition key, however, is not always a good strategy. Frequent access of the same key in a partition can cause request throttling and overload database's provisioned throughput (capacity). Therefore, you should design your tables with the right partition key.

It is recommended to use high-cardinality attributes that have unique values for each item such as user_id, product_id...Besides, composite attributes could be a good choice to increase distinctiveness of the values. Take a look at above table example where we have a combination of AlbumID and TrackID to be sort key.

Access control

In order to administer (create, delete) DynamoDB tables, you should have valid IAM credentials and permissions.

Security-wise, it is recommended to attach a service-specific permission to an user. In this case it could be AmazonDynamoDBFullAccess.

With this strategy, each IAM user in your organization only gets access to AWS services that user is allowed to control, instead of a God-mode AdministratorAccess. I will have a separate post on IAM roles in AWS.

Read/Write capacity mode

When creating a new DynamoDB table, we need to configure its capacity to read and write data. This is quite app-specific and DynamoDB supports 2 capacity modes:

On-demand: a tailored feature for unpredictable workloads and traffics. This mode offers pay-per-request pricing so you only pay for what you actually used.
Provisioned mode: fit for applications that have predictable and consistent traffics. In this mode, you need to specify database throughput in read capacity units (RCUs) and write capacity units (WCUs).

How could you calculate RCUs and WCUs of your table? Here are factors that you should know in advance:

How many items the table is expected to read/write per second
Size of each item (KB)

Take an example when our table needs capacity to read 80 items per second. Each item has size of 3 KB.

Calculating read capacity units (RCUs)

1 RCU represents 1 strongly consistent read or 2 eventually consistent read per second, for an item up to 4 KB in size

So, if our table uses eventually consistent read, we would need: Math.ceil(3/4) = 1 RCU. Therefore, we need to specify 1 x 80 = 80 RCUs for our table so that it could read 80 items per second.

With strongly consistent read, the number would be 80/2 = 40 RCUs.

Calculating write capacity units (WCUs)

1 WCU represent 1 write per second for an item up to 1 KB in size

So because each item's size is 3 KB, in order to write an item to the table, we need 3/1 = 3 WCUs.

TL;DR

DynamoDB is a predictable performance, No-SQL Amazon's database service.
It supports key-value and document databases. It is also multiregional, which means your data is synchronised across multiple Availability Zones.
DynamoDB supports 2 types of consistency model: eventually consistent read and strongly consistent read. The former offers high speed responses while the latter guarantees the most up-to-date data.
Tables, Items, Attributes are core components of DynamoDB. Each item in a table must have unique primary key. It could be a distinct partition key alone or a combination of non-unique partition key and a unique sort key.
In order to administer DynamoDB tables, an IAM user with valid permissions is required.
On-demand and Provisioned are 2 capacity mode that we need to specify when creating a new table. On-demand mode is ideal for unpredictable workload applications. Provisioned mode requires us to specify the table's capacity in read capacity units and write capacity units.

Tooling for boosting your development workflow

Vinh Le — Sun, 25 Aug 2019 14:26:41 +0000

Part 1: Code formatting

If programming is just a 1-person-to-computer process, things will probably be dead simple. You write whatever you think that will work, and the computer will help you to verify.

Sadly, that is not how it works most of the time. When you work in a team, your peers will need to review your codes and vice versa. Here is the time where things heat up a little bit 🙃

Why bothering?

Poorly formatted codes slow down review process

When coworkers tag you in a pull request (PR), they want to quickly get approval to move the next feature out of backlog. On the other side, you want to go through it fast as well. However, when the code is poorly formatted, or even worse, not formatted at all, you both are in trouble.

Let's face the truth: each of us has different coding style. Thus, this difference will surely take you more time to understand logics underneath.

Take an example of handling a Promise in JavaScript in general:

getShots(myShotsEndpoint).then((shots) => {return shots.forEach(shot => {
    saveMyShot(shot)
    removeMyLastShot()
    return
})}).catch(error => {
    if(getErrorStatus(error) === 401) { handleUnauthenticatedUser() } else {
        showErrorMessage('Cannot get my shot now!!!')
    }
})

Urghhhh 🤦🏻‍♂️😐🔥

I know what you are having in mind 😤But bear with me a little more. If you get into Space/Tab trouble, no don't lie, I know you were 🤗 functions like this will probably ruin your day:

getShots(myShotsEndpoint).then((shots) => {return shots.forEach(shot => {
        saveMyShot(shot)
    removeMyLastShot()
            return
})}).catch(error => {
        if(getErrorStatus(error) === 401) { handleUnauthenticatedUser() } else {
                showErrorMessage('Cannot get my shot now!!!')
    }
})

I understand, it is exhausting reading those codes 😥

And this is just a Promise handler. Imagining there are tens of components with function logics, UI elements, styling..etc ahead. Your job is to go through all of them. Annoyingly, this blocker significantly slows you down.

And trust me, your colleague is not happy either when getting a comment like:

"Please add 1 more tab in line number 531"

Their jobs are to take care of business logics, not a small space/tab that you, the reviewer, does not feel right. From here, you have no choice rather than giving up, accepting a painful truth and telling yourself

There will be no problem. The computer will take care of that.

Until a day...

A nightmare to maintain

A new guy comes in to maintain this project and his reaction will probably follow this chain: 🧐🤨😡🤬

These uggggly code blocks will surely make it more difficult to maintain. When more developers go on board with different coding styles, maintainance becomes more challenging.

Fortunately, you don't.

There are tools out there that could be your team's saviour. Let me introduce 1 of them that already saved mine 🤩

Prettier

Prettier is an awesome tool for code formatting. It has supports for many languages, great documentation and easy to get started.

Time for coding now! 👨🏻‍💻💪

I will show you how to set it up in a NodeJS module, as it is what I feel most comfortable with 🔥
Nonetheless, it should share many similarities with your projects.

Step 1: Install Prettier and TSLint (or ESLint if you are not a fan of TypeScript)

npm install --save-dev prettier tslint
--or--
yarn add -D prettier tslint

You also need to install some helpers/plugins to make Prettier and TS Lint works nicely together:

npm install --save-dev tslint-config-prettier tslint-plugin-prettier
--or--
yarn add -D tslint-config-prettier tslint-plugin-prettier

In a nutshell, tslint-config-prettier disables conflicting rules between TSLint and Prettier.

Step 2: Config linting rules for TSLint

Create tslint.json file in the root directory of the project:

touch tslint.json
code . tslint.json (*)

(*) The command code . I used is a feature of VSCode. Hands up VSCode fans! 🚀

You then can add rules in this config file. For the sake of simplicity, I will introduce some basic rules. Go here for your specific needs.

tslint.json

{
    "defaultSeverity": "error",
    "extends": ["tslint:recommended", "tslint-config-prettier"],
    "jsRules": {},
    "rulesDirectory": ["tslint-plugin-prettier"],
    "rules": {
        "prettier": true,
    },
    "linterOptions": {
        "exclude": ["node_modules/**/*.ts"]
    }
}

Step 3: Add prettier (formatting) rules

You will need to tell Prettier how you want your code to be formatted. To do that, let's create a config file, just like what we did for TSLint

touch .prettierrc
code . .prettierrc

Let's add some rules in this file ✍️

{
    "arrowParens": "avoid",
    "bracketSpacing": false,
    "insertPragma": false,
    "printWidth": 80,
    "proseWrap": "preserve",
    "requirePragma": false,
    "semi": false,
    "singleQuote": true,
    "trailingComma": "all",
    "tabWidth": 2,
    "useTabs": true
}

Remember, what you specify here will be exactly how Prettier formats your code. For example, last 2 lines tell Prettier to use Tab width = 2 (Sorry Space fans out there 😔). "singleQuote": true, for instance, will replace all double quotes by single ones. There are many other options which you might want to refer to.

Step 4: Format your code

There are 2 ways that you can do it, either right before saving a file or all files together when you feel like.

Format on Save

This is an Editor-specific feature, which means you have to enable this feature inside your editor's Settings. In VSCode, for instance, you can do that by putting this line in settings.json:

"editor.formatOnSave": true

Format the entire project

I personally prefer this way as it gives me more control over which types of file that I want to format. Let's write a script in package.json

...
"name": "awesome-formatter",
"author": "ShotCode <shotcode@gmail.com>",
"scripts": {
    "format": "prettier \"**/*.+(js|json|ts|md|mdx|graphql)\" --write"
}
...

What does that format script do?

It tells Prettier to apply formatting rules that we configured in prettierrc on specified file extensions. In this case, they are .js, .json, .ts... you name it. Next we tell Prettier to override those files by --write command.

Let's run the script and see how it works:

npm run format

Boom! It is so cool isn't it? 🤩🏎

To bring this to the next level, you can configure this script to run with Git hooks. So that your code will be nicely formatted before committing, for instance. If you are curious about it, stay tuned for the next part 🥳

Key takeaways

Poorly formatted code sucks. Period.
Why? It takes much longer time to review, harder to maintain, especially in a large project with dozens of developers.
How to format your code properly? Get used of awesome tools like Prettier. It is easy to set up, pleasing to see the result and works nicely with other processes in your workflow.

But wait, you are not the only decision maker 😔

If your peers are hesitating about using Prettier, I belive that its great features and benefits are weighty reasons for at least giving a try. And trust me, you all are gonna love it.

You don't have to worried about your peers' coding style anymore. Let's just keep writing something awesome. And let the computer handle the rest.

Thanks for reading this blog!

I would love to hear your thoughts in the comment section bellow 🤗

And stay tuned for upcoming ones 🤩

✍️ Written by

Vinh Le @vinhle95 👨🏻‍💻🤓🏋️‍🏸🎾♂️🚀

A hustler, lifelong learner, tech lover & software developer

Say Hello 👋 on

✅ Github

✅ LinkedIn

✅ Medium

✅ Personal site

DEV Community: Vinh Le

Learn AWS EC2 by deploying a NodeJS application (P2)

Part 2: Run a NodeJS application on an EC2 instance

Edit Inbound Rules to accept traffic from server port

Install nvm and node to the instance

Clone Git repository

Generate a SSH key in EC2 instance

Clone the source code

Install dependencies and run the Node server

Congrats! Your instance is running a Node.js server now 😎

Let's sum up what we have done to make it happen:

Learn AWS EC2 by deploying a NodeJS application

Part 1: Create and SSH to an EC2 instance

What is EC2

What is an EC2 instance?

Create an EC2 instance via CloudFormation template

📌 Prerequisite

Create CloudFormation template

Upload CloudFormation template

Create a key pair to access the instance

SSH to your instance

Congrats! You made it! 😎🎉 In this first part, we have learnt:

AWS IAM 101

What is IAM

Building blocks

Users

Configure IAM user locally

Groups

Roles

Policies

TL;DR

Resources

Caching in DynamoDB

Caching

Caching strategies

Lazy loading

Write-through

Pros and cons

Dynamo TTL - Time to live

TL;DR

Resources

Scan and Query in DynamoDB

Scan vs. Query

Query

Scan

Secondary index

A different data structure

Local vs. Global

When to use what

Indexing strategy

TL;DR

Resources

DynamoDB 101

What is DynamoDB?

Consistency model

Building blocks

Tables

Items

Attributes

Access control

Read/Write capacity mode

TL;DR

Tooling for boosting your development workflow

Part 1: Code formatting

Why bothering?

Poorly formatted codes slow down review process

A nightmare to maintain

Fortunately, you don't.

Prettier

Time for coding now! 👨🏻‍💻💪

Step 1: Install Prettier and TSLint (or ESLint if you are not a fan of TypeScript)

Step 2: Config linting rules for TSLint

Step 3: Add prettier (formatting) rules

Step 4: Format your code

Key takeaways

But wait, you are not the only decision maker 😔

Thanks for reading this blog!

And stay tuned for upcoming ones 🤩

✍️ Written by

Install `nvm` and `node` to the instance