DEV Community: Raphael Jambalos

AWS Network Challenge 2: Deploy File Upload App on EC2, RDS, DocumentDB

Raphael Jambalos — Fri, 15 Nov 2024 02:27:47 +0000

This talk will be/was presented at the AWS Community Day 2024 - Bacolod on November 16, 2024

Scenario

I love taking photographs and would like to create a website selling those prints of those photographs to my friends and family. I started out selling it on Facebook and business has since taken off. I'm selling 100 prints per day, and it's hard to keep track of each transaction. So I decided to do my custom website with the following features:

Create Product: upload the image for preview, and indicate the initial number of prints available for sale.
View Products: so buyers can see the photos for sale.
Checkout: customers can only buy one print design at a time. They can indicate how many of those designs they want to buy, their email, and address.
View Orders: view my orders.

To simplify things, we don't have any login for now. All users can see everyone's orders.

For this POC, we use the following tech stack:

Python and Flask: for the development of the BE and FE
MongoDB: to store information about the uploaded image
PostgreSQL: to store data about orders

To give you a headstart, I already developed this application. You can access the Gitlab repository here

Lab 1: Simple deployment via LightSail

The simplest way to get this up and running is to deploy everything inside one server using Amazon LightSail. In one server, we run:

eCommerce Application (Python/Flask)
MongoDB database
PostgreSQL database

The problem with the setup above is that it is a single point of failure. If the server goes down, our app goes down along with all of our data with it. Let's fix that. Let's create 3 servers, one for each of the components:

This solves our problem of losing everything when our eCommerce application goes down. But now, we have to think about the server that holds the MongoDB and PostgreSQL databases. We have to ensure that they don't go down. Or at least, if they do go down, we still have our data intact. In this case, we use the native replication of features of MongoDB and PostgreSQL to have a secondary DB instance that will hold the data. Data written on the primary server have to be written here as well before we return success. This way, we can have two copies of our data in case the primary goes down.

Lab 2: Deploy to Amazon VPC

The LightSail servers we are managing are starting to get a lot. And they are all publicly available. The problem with this is we have to make an effort to reinforce all of the servers since it is publicly available. Any hacker can reach those instances. We have to ensure there are no gaps in the security. We can reinforce each instance. But when our 5 servers become 500, we would have to do that process over and over. And I don't like to do that. There's just so much to miss.

To solve this, we will create our virtual network in AWS using a service called AWS VPC. The VPC has a private and public subnet. Servers inside the private subnet cannot be reached directly. We must add a server inside the public subnet that can route traffic to the instances inside the private subnet. This is the only server that needs to be hardened since it's the only one facing the public. Let's call this server our Proxy Server.

We will then move our LightSail server to a parallel AWS service called Amazon EC2 and deploy those servers to the private subnet. When our customer visits our website, they first connect to the Proxy Server, which then redirects their request to the application server. The app server then connects to the MongoDB and PostgreSQL server for the data it needs.

Lab 3: Use Managed Services to make your life easier

An improvement that can be made here is replacing the proxy server with an AWS service called AWS Application Load Balancer. It is deployed in the public subnet and redirects traffic to the application server. The beauty of this is we don't have to configure and maintain a Proxy Server. We can have this functionality with a press of a button.

The problem with Lab 2 is we have to manually set up the replication. And if you're an application developer, you'd have to take time to learn and set up how replication works for DynamoDB and MongoDB. This may take days or even weeks to do. Pushing deadlines even further down.

Thankfully, AWS has recognized that their users usually do this replication themselves. They created a simple way for users to deploy PostgreSQL and MongoDB database servers. Using Amazon RDS and Amazon DocumentDB, we can provision a database server. And with a few additional clicks, we can configure it to be highly available, and its data encrypted with Amazon KMS.

Lab 4: Prepare our Application for Scaling - Remove State from your Application Servers

Weeks passed and everything is good with your application. Your users are buying your prints like crazy and you can focus on being out in the field and taking more photos. To sell more photos, you made a B1T1 promo on the 15th of the month. Your customers all take to your website at the same time. And your application server gets overloaded and goes down. To keep your application up, you increase the size of your EC2 instance by 8x, from t2.micro to t3.large.

Your customers are now happy and your website works. You become busy fulfilling orders for the next 2 weeks. Then, when your AWS bill comes back, you have a small heart attack. You forgot that you left your EC2 instance in t3.large, 8x the cost! For the next sales, you now spend energy to remind yourself to scale down the instance every time there is a sale. But the problem is when there is no sale and the customers flock to your website, you have to react and increase the size again. And of course, there would be a gap between you scaling the server and when your customers first feel it. So there must be a better way.

Enter auto-scaling groups. What this does is it adds more EC2 instances when it is needed. And removes it when the demand is no longer there.

But before we get to that, we need to think about whether our application is ready for that. Currently, our application stores its data on separate servers—order data is stored in DocumentDB, and order data is stored in Amazon RDS. For these data, if we have two application servers, it would be okay because each can just contact the separate database servers for the data.

However, the images are still stored inside the first application server. If we create a second application server, the images originally uploaded on the 1st server will not be available to our users when their request happens to be served by the 2nd server. Hence, we would need to separate the storage of our images. In this goal, we would use Amazon EFS. We would mount Amazon EFS on both application servers. When a user uploads a file, it is uploaded to the directory that is connected to the EFS drive. It's like a shared drive because as soon as the upload is complete, the image becomes immediately available to the other application server.

Lab 5: Auto Scaling your EC2 instances (so you don't have to)

Our application servers are now completely stateless. It means that these servers don't hold data inside them. The data are delegated to EFS for the images, RDS for the orders data, and DocumentDB for the image metadata. Having our app servers to be stateless is critical for us to be able to add more app servers to serve our customers. This ensures that the customer's data will be available to them regardless of which app servers serve their traffic.

Now, let's go back to the original problem. We don't want to be the ones manually adding and removing application servers. For this, we use Amazon Auto Scaling Groups. First, we create an image of our application server. This is called an AMI. We can use this image to recreate the exact state of our application server - the code, the libraries inside, and the installed OS packages. This creates an identical twin of our first application server. Second, we configure the condition to which the auto-scaling group will add more application servers. The simplest is having the ASG take a look at the average CPU utilization of all the application servers currently in existence. Once it exceeds, say 80% average CPU utilization, the ASG uses the AMI to create more copies of the application server to serve the traffic.

Lab 6: Deploy your changes quickly with CI/CD pipelines

With this setup, you'd quickly run into the problem of updating your application server. Every time there is an update, you would have to update the image being used by your auto-scaling group. Then, you would terminate all existing app servers so the ASG can provision new app servers with the updated code. This step is very tedious to do and introduces downtime to your application.

In this final lab, we would create a CI/CD pipeline that deploys to the current set of EC2 instances in the auto-scaling group. We also update the settings of the auto-scaling group to execute commands to pull the latest code for every new application server it provisions.

We will create the CI/CD pipeline with the AWS CodePipeline service. Our code will be stored on a public GitHub repository, and we will configure the pipeline to trigger every time there is an update on the main branch of our repository. The second stage of our pipeline will be using CodeDeploy to deploy our code to the Auto Scaling Group.

The simplest setting in CodeDeploy would be to deploy the code change simultaneously on all servers using a technique called All-At-Once. The problem with this technique is that application servers usually go down when new code is being loaded. So when we do an all-at-once, there may be a few seconds to a minute of downtime as all the app servers are receiving and loading the updates.

An update we can do is to use Half-At-A-Time or One-At-A-Time deployment configurations in CodeDeploy. In this configuration, CodeDeploy chooses half of the app servers and deploys the update to it, leaving the rest of the app servers available to serve your customer's traffic.

Another problem you'd probably encounter is what if the deployment had a problem. As much as we try to guarantee poor code doesn't reach our production environment, there will still be instances when it would. In the current setting, it deploys to the first half. But then we must configure health checks on CodeDeploy to determine if the deployment was a success. We define success by having our app servers pass a series of checks called a health check. An example health check might be to ensure our website returns success when calling the homepage and the products page.

Hints

You will provision the EC2, RDS, and DocumentDB instances. Then, you will need to ensure the Flask application inside the EC2 can talk to RDS and DocumentDB. You'll most likely encounter problems with connectivity. Here are our tips:

Ensure you deployed all those components in the same region and the same VPC
Check the security groups of each of these components. Check to ensure inbound rules allow communication on the specific ports needed by each component (RDS - 5432, DocumentDB - 27017, EC2 - 5000)
What helps me learn is to screenshot every step I did when I'm using the AWS service for the first time. So I can get back to them later on.

That's all folks!

Full Stack Serverless Challenge #1: AWS Amplify Gen 2

Raphael Jambalos — Sat, 29 Jun 2024 04:19:24 +0000

In this challenge, we'll be learning more about Amplify Gen 2. It is an AWS service that allows front-end developers to deploy full-stack applications. It has a code library that attaches itself to supported frontend frameworks like VueJS, ReactJS, Flutter, React Native, etc. Once attached to the framework, developers can easily create serverless backends, protect them with user authentication, and add file uploads and downloads.

We start by following the basic getting started guide. From there, we'll get our hands dirty with the core functionalities of Amplify Gen 2.

This is not a walkthrough post. It forces you to explore on your own by giving a challenge to follow.

About the Full Stack Serverless Challenge Series

In this series, I'll try out many full-stack serverless offerings in the market. I'll be doing:

AWS Amplify Gen 2 (we're here)
Ion + SST
and more...!

[A] Basic Amplify Gen 2

For this section, we would follow the getting started tutorial. It is quite straightforward and is doable within 30 minutes. By the end, you should be able to use the Getting Started repository to:

Create todo list
Delete todo list
Have a login via Cognito
Each unique user has their own set of todo list

Your output should look like this:

Answer the following questions:

The todo list is stored where? What backend powers the APIs?
What does npx ampx sandbox do? Is it a local environment? Does it deploy to the cloud?
How do you deploy an Amplify project?

[B] Amplify Data + Amplify Functions + Amplify Storage Basics

Allow users to add comments for each todo task. This involves duplicating how the todo list schema was created and making an exact duplicate for comment.
For each todo task, add capacity to add file attachments. These attachments should use Amplify Files. Once uploaded, the s3 URL and the taskID are sent to a Lambda function deployed using Amplify Functions. It should access a DynamoDB table using the aws-sdk library, and create an entry that associates that image_path with the todo task id. The DynamoDB, APIGW, and Lambda functions are all deployed using CDK.
Create an API that allows you to see all the attachments associated with each todo task. Integrate it with your frontend.

This is my version:

Answer the following questions:

How do you add JS packages for Amplify functions?
Where do we get permission to upload files?
When we preview the files, are there any security features used?
How is the experience working with basic CDK? Can you use higher-level constructs to accelerate this?
Cloudformation has a problem that each stack has a limit of 500 resources. How is that mitigated in CDK?
How do you represent relationships when using a database like DynamoDB?
Explore the concept of Single Table Design for dynamodb. How is it different from the approach here?

[C] Above and Beyond!

Here are above and beyond hands-on that you can try with AWS Amplify. I haven't tried them myself, but scanning through the docs, I thought these are interesting features that I might go back to later. And if you have more time, I encourage you to do it as well:

Add authorizations. The comments functionality can only be used by admins
Manage devices
Reset password and update password
For data, use an existing PostgreSQL database
Add triggers for the Cognito User Pool. For each user, they are added to a DynamoDB table. Study the cognito hooks
Study Amplify Analytics

Appendix A: Hints

There are instances where I got stuck. You might find this section useful as you go through the exercise.

A1: Adding package

This allows you to include packages in your backend functions



cd amplify

npm install @aws-sdk/client-dynamodb

A2: Adding environment variables

This allows you to pass on environment variables to your lambda functions:



import { Function } from 'aws-cdk-lib/aws-lambda';

const backend = defineBackend({

  auth,

  data,

  sayHello,

  storage

});

// create a new API stack

const apiStack = backend.createStack("api-stack");

const table = new Table(apiStack, 'AttachmentTable12345', {

  partitionKey: {

    name: 'id',

    type: AttributeType.STRING,

  }

});

const lambdaFunction = backend.sayHello.resources.lambda as Function;

lambdaFunction.addEnvironment('DYNAMODB_TABLE_NAME_MASTER', 'AttachmentTable12345');

A3: Use ref() to make the elements binded

The variant of VueJS used does not respond to typical v-binding. I found that using ref() works best.



let commentData = ref({})

const currentId = ref("")

async function showCommentsV2(todoId: string) {

  console.log("entering showCommentsV2")

  currentId.value = todoId;

client.models.Comment.listCommentByTodo_parent_id({

    todo_parent_id: todoId

  }).then((d) => {

    console.log("BEFORE DATA IS")

    console.log(commentData)

    console.log("d")

    console.log(d)

    commentData.value = d.data

    console.log("AFTER DATA IS")

    console.log(commentData)

  })

}

A4: Ultimate Hint - The Github Repo

I encourage you to use this as last resort. This contains the answer for Challenge A and B already. I recommend that you start with the repository given in the AWS Getting Started Guide, and use your wits solve Challenge B and C.

That's all!

Let me know if you made this far, or if you have any questions! Would love to hear from you

AWS Network Challenge 1: Deploy Web App to EC2 / Two-Tier VPC Architecture

Raphael Jambalos — Sat, 25 May 2024 03:19:16 +0000

About the Challenge

I manage the Service Delivery Team at eCloudValley Philippines, and one of the first things I teach them is to understand how AWS networks work. As cloud developers, we deploy our applications in Lambda and connect them with databases and other components inside our AWS VPC (virtual network in AWS). Because of this, it becomes imperative for them to know more about how VPCs work. They need to know how to navigate network issues they will encounter.

In this challenge, we would deploy an application to 2 servers inside a virtual network in AWS. The network will have two tiers: private and public. Our result will be:

Before we begin, here are a few helpful hints:

This is a series of challenges that get progressively harder. It also meant to state what to do and what you should have learned. This is NOT a step-by-step guide on how to do the challenges.
I have added links to AWS documentation to help you get started with the challenges.
I have added guide questions to test your knowledge. It's easy to follow documentation to make this all work. But what we are striving for is a true understanding of how these components work together for a secure, reliable, and working network.

[A] Basic VPC Skeleton

Let's build the backbones of our VPC network. You'll be creating:

VPC (how)
2 Private Subnets (deployed in separate Availability Zones, how
2 Public Subnets (deployed in separate Availability Zones)
1 Internet Gateway (connected to the VPC we just created)
1 NACL for both subnets (with a rule that allows all traffic)
1 Route Table (for Private Subnets, how)
1 Route Table (for Public Subnets)

In this exercise, you must be able to answer the following guide questions:

What makes a subnet private? What makes it public?
What is an IPv4 CIDR block? What is the difference between /16, /20 and /24?
Why do we deploy subnets in different availability zones?

Architecture Diagram for Exercise A:

[B] Setting up the instances

With the VPC backbone in place, let's add more network components:

NAT Gateway (deployed in Public Subnet)
Set up for you to deploy an EC2 instance
EC2 instance (deployed in Public Subnet) - Getting Started - Best Practices
EC2 instance (deployed in Private Subnet)
Security Group that allows all traffic from port 22
Associate the security group to both EC2 instances
Private Route Table connected to NAT Gateway

Here are a few guide questions to answer:

DEMO: Be able to enter the CLI of the EC2 instance in the public subnet via SSH. Once inside, access the CLI of the EC2 instance in the private subnet.
Why do we need a NAT Gateway? How does it operate?
What is the difference between a NAT Gateway and an Internet Gateway?
What is the difference between a security group and an NACL?
Can a computer from the internet access the EC2 instance inside the private subnet? Why or why not?
What is the difference between a NAT gateway and a NAT instance? When would you use a NAT instance? NAT gateway?
What happens if you don't assign a public IP Address to an EC2 instance?

Architecture Diagram for Exercise B:

What we achieved

We created our private network in the Cloud! We also created two EC2 instances, one in the private subnet and another in the public subnet.

[C] Setting up NGINX and load balancing

We're getting close:

EC2 instance (deployed in Private Subnet 1)
EC2 instance (deployed in Private Subnet 2)
Install Nginx on both. Modify /usr/share/nginx/html/index.html to add "this is server one" for the 1st EC2 and "this is server two" for the second one.
Create an application load balancer
Create a listener to receive HTTP traffic
Create a target group that includes both EC2 instances as instance targets

Here are a few guide questions to answer:

DEMO: Once the resources have been created, get the URL from the application load balancer and put it in your browser. It should alternate displaying "this is server one" and "this is server two". This proves that the ALB is alternating sending traffic between the first and second EC2 instances.
What is the difference between a NAT Gateway and an Application Load Balancer?
Why put the EC2 instances in separate private subnets?
Why put the EC2 instances in the private subnet instead of being exposed directly?
Why add an application load balancer?

Architecture Diagram for Exercise C:

The result looks like this:

What we achieved

With Activity A-C, you have deployed your first VPC and have successfully deployed a simple app using a two-tier architecture. That’s honestly very close to how we deploy secure applications on the modern web.

There is one thing though. The application you accessed looked like something like this when typed in the browser:

http://jamby-alb-1962899873.ap-southeast-2.elb.amazonaws.com/

Two things come to mind when I see this:

First, the website is deployed on HTTP, not HTTPS, which leaves our end users insecure. As your end user browses your website, the data you send and receive goes across the internet unencrypted. The internet is made of thousands (or maybe millions) of routers connecting everyone in one big interconnected web. From your end user’s Macbook to your EC2 instance, there may be as many as 100 routers in between. if you leave your traffic as HTTP, any one of those 100 routers can see whatever you are sending between one another.

Second, as an e-commerce, I’d like to have a decent website name that my end users can easily remember. Something like:

jambyiscool.ecvphdevs.com

In exercise D, we will do just that.

Resources:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-application-load-balancer.html

[D] Securing setup with HTTPS

Let's take it a step further.

Buy a domain in GoDaddy and associate it with Route 53 (if you haven't). If you have access to an existing Domain Name, no need to do this step.
Create a public certificate with AWS Certificate Manager. You may have to create DNS records in your domain manager.
Associate the certificate with the ALB created earlier
Create a route 53 record for "jambyiscool.yourdomain.com" and point it to the load balancer you just created. If your domain manager is not Route 53, feel free to create the record in your domain manager.

Architecture Diagram for Exercise D:

Here is the end result:

What's next?

Congratulations on completing this exercise! You know have a full view on how VPC works, and have deployed your first application on a best practice version of a private network in AWS.

On the next challenge, we will be doing VPC peering, EC2 auto scaling, VPC endpoints and CI/CD. Stay tuned!

Interested about joining ECV Philippines?

Send us your CV via email to ph.hr@ecloudvalley.com:

Photo by Nastya Dulhiier on Unsplash

How to Hack (and secure) Serverless Applications

Raphael Jambalos — Thu, 28 Mar 2024 06:49:00 +0000

At eCloudValley Philippines, we prefer developing web backends as serverless applications. We just push the code to AWS and they worry about the infrastructure. We don't have to think about provisioning, patching, scaling and securing it. That's all AWS. In terms of security, since even us don't have access to the infrastructure, we don't have to worry about it. We pay AWS to worry about that for us. But what we do have access to is the code. And that's one major attack vector we have to secure. And in ECVPH, we take that very seriously.

In this article, we will take a deeper look at how Serverless workloads get hacked, and what you can do about it.

Here are the 6 common types of attacks on Serverless applications

[1] Denial of Service

You'd think Serverless applications don't get DDoS attacks. Yes, they can scale fast. But the network components attached to them don't often scale at the same rate as them. We've seen Lambda functions use an RDS database, or is connected to an API that is deployed on a traditional EC2 setup. Your lambda functions will scale to meet the traffic, but they will rain down traffic into these weaker downstream dependencies.

That's why it's often a good idea to do event-driven architecture. Instead of directly writing into the database or directly calling the legacy system, the Lambda function queues

It's not only infrastructure that can cause denial of service. Sometimes, there are vulnerabilities in your packages that can cause this. For example, this Python package for etherium can cause resource exhaustion when a specific payload is added to it. With just a few requests, the package causes a malfunction that causes it to hog all the CPU and RAM of the device. This exhausts the resources of the server, causing it to go down.

To learn more about how to survive a DDoS attack, check out my blog post about how we survived a 2M requests per min attack.

[2] Denial of Wallet

But suppose that somehow your entire setup is serverless. You're using AWS Lambda for your compute, DynamoDB for your backend, and S3 for all your files. The attacker can just keep attacking you and AWS will be able to keep your website up. But since these services are pay-per-use, you'd just end up racking up higher AWS charges. And so long as they can sustain their attack, they can bankrupt your company to the ground.

To protect against both DoS and DoW attacks, it is best to put into place some kind of rate limiting. AWS WAF does this by allowing you to limit request per second for each IP address. This makes these attacks harder, but not impossible, especially if they have thousands of IP addresses on their disposal.

[3] Remote Code Execution

While the most common attacks are volumetric in nature, some just need a few API calls and they've compromised your system. The most common of which is remote code execution. It is when hackers are able to manipulate the payload so that they can execute custom code in your application.

Probably the most insecure code in Python is this snippet below. It allows you to evaluate a string and execute it as a Python code.

request_body = '2 ** 2'
response = eval(request_body)
return response

Imagine the possibilities. When you are using this code, an attacker can just modify the request body to a more sinister one.

request_body = 'run_sql("drop database")'
response = eval(request_body)
return response

Much of the programming world does not use this code anymore. But vulnerabilities found in old packages (that you may still be using) may leave you exposed to this vulnerability. Here's an example from an outdated version of Astropy.

[4] Malicious Packages

The dawn of open-source software has allowed developers to share code much freely. It has accelerated development since we no longer have to write code from the ground up. We can just compose libraries together to make our application. But that openness also comes at a price.

There are malicious libraries out there that exist to mimic legitimate software. Once added to your project, these libraries open backdoors for hackers to remotely execute code on your server. Like this "python package"

Malicious code don't exclusively come from packages. It can also come from Lambda layers. An inconvenience with Lambda is that when there are libraries with OS-specific dependencies, it is more difficult to upload them to Lambda. You have to compile the library using an OS that is similar to Lambda's OS. An example of this is psycopg2 (an internal library required to make Postgresql work in Python).

The most convenient way to fix this is just to use a Lambda Layer that somebody else already created. Like this Github page of Keith. While he looks generally reputable, nothing is stopping Keith from inserting malicious code to any one of his public Lambda layers. And once you use the code inside the Lambda layer, he may be able to remotely run code in your lambda. A better fix is to build the lambda layer yourself, even if it takes awhile. In Python, Pip added the --platform command where you can specific the platform when you install your library.

[5] API is too open

The easiest way to hack an API is when it is open to the public.

What we do is we often pair it with Cognito. The user logs in to the frontend, and Cognito issues a set of tokens that the backend will use to ensure that the user is who he says he is. The token expires every hour and needs to be renewed. During each renewal, we can validate if the session is still valid.

But how about APIs that are only used by other APIs? They shouldn't be open to the public in the first place. They should, at the minimum, be secured by an expiring API token that only the authorized system has access to. An even better approach is to determine these APIs and ensure they can only be accessed privately, inside the network.

[6] Privilege Escalation

ECS tasks and Lambda functions rely on IAM Roles to access AWS services. Those who leave this role with an admin privilege can leave their account open to hacking.

If there is a remote code execution vulnerability, hackers can use that to discover the IAM role inside the Lambda / ECS task. Then, they can use this role to create an IAM user with elevated privileges, which they can use to gain access to your AWS accounts.

But more often than not, this vulnerability is an inside job. For example, we have given the developer limited rights to our AWS account. They can only push code to their CodeCommit repository. But the execution role of the Lambda function is set to Administrator Access. The developer can change the code that is pushed to the CI/CD to include instructions to create an IAM user for him with elevated permissions.

Another form of privilege escalation is modifying the CloudFormation code that comes with the serverless application. Malicious developers can directly add instructions to create an elevated user there.

How about you? What are other ways to "hack" into serverless applications?

Let us know in the comments!

Also check out my other blog post on security best practices we learned the hard way.

Photo by Max Bender on Unsplash

Trigger AWS Lambda with TCP Traffic + Static IP Address

Raphael Jambalos — Wed, 27 Mar 2024 14:46:58 +0000

In this post, I will walk you through how to trigger AWS Lambda with TCP traffic. By the end of the post, we would have created a setup similar to the diagram below:

First question is WHY?

In my case, it is because the client application that I was working with can only send requests via TCP. I needed to supply a stable IP address and a port number. They couldn't really update the client to HTTP because it was a legacy system.

Given a choice though, it is best to just use HTTP traffic so you can connect with Lambda via API Gateway. Using this approach comes with advantages:

You can use API Gateway's routing capabilities to route traffic to small Lambda functions
The setup is more durable because API Gateway is a managed service. As you'd see later, we would have to create a TCP server to route traffic to Lambda + APIGW. And that can be a single point of failure.

FACT: Lambda cannot be directly triggered with TCP traffic.

When sending TCP requests, you need the IP address and the port number of the target. Because Lambda functions are only spawned when they are triggered by an event, they are only given an ephemeral IP address and port number. Sending TCP requests to Lambda directly implies you are made aware of the IP address and port number of each invocation of Lambda. And so far, AWS has no feature that makes this possible.

If you are looking to deploy an application in Lambda to receive and process TCP traffic directly, this post unfortunately cannot help you. At this time of writing, this is not technically feasible yet. I suggest you just use ECS, EC2 or EKS to do this.

But if you are looking to find a way for a TCP client to communicate with Lambda by converting TCP traffic to HTTP, then this post is for you!

Now on the solution

(1) Following the diagram above, we first provision an EC2 instance. To follow with the examples here, I suggest you choose a Linux instance (i.e Ubuntu or Amazon Linux 2).

(2) Install Python 3

(3) Create a Python virtual environment and install the requests package.



python3 -m venv venv
source venv/bin/activate

pip install requests
pip freeze > requirements.txt

(4) Create a file called tcp_server.py and copy paste the contents of this GitHub Snippet to the file.

(5) Set the environment variables and run the TCP server.



export TCP_IP_ADDRESS_BIND=0.0.0.0 
export TCP_PORT_NUMBER=8108
export TCP_SERVER_TIMEOUT=20
export APIGW_BASE_URL="<YOUR APIGW ROUTE>"

chmod +x tcp_server.py
python tcp_server.py

You should see this in your terminal:

(6) To test your setup, create another Python script by copying the contents of this second Github Gist to a file called tcp_client.py. This gist sends a TCP request to the TCP web server we have started.

(7) Next, execute these commands to ping the TCP server.



export TCP_SERVER_IP_ADDRESS=0.0.0.0 
export TCP_PORT_NUMBER=8108

python tcp_client.py

In the screencap below, we see that we have received a response from our TCP web server.

What's next?

This solution is a POC that demonstrates that we can receive TCP traffic and route it to Lambda. The next step is to run this script inside an EC2 instance and expose the port and IP address of the EC2 instance.

For low-traffic conditions, this setup is okay. But with increased traffic, we have to start thinking about how this would all scale automatically when there is an increased load. For this, we recommend that the TCP server is hosted in a Docker container, orchestrated by either ECS, EKS, or a custom Kubernetes cluster.

The full repo is found here

Photo by Taylor Vick on Unsplash

Top 10 Security Best Practices we learned the hard way

Raphael Jambalos — Sun, 03 Mar 2024 10:59:46 +0000

Our users entrusted us with their data and it's our duty to keep this data secure as they use our application. Unfortunately, security best practices aren't the first things that developers learn when they start out. It's usually learned through experience, when the incident already happened.

In this article, I'll share our team's experiences and research on how to make your applications secure. It is by no means an exhaustive list, but these are the most common things to miss.

Let's dive in with an example eCommerce Store

For this article, let's imagine Fred is an application developer for GoodProducts Manila (GP Manila). It's an eCommerce company that has a website gpmanila.com. One of his ex-workers Nate is trying to do some damage. Here are a few ways he might do it:

[1] Users can access the data of others by URL hijacking

This is by far the simplest security exploit to do, (and thankfully), the simplest to remediate.

Let's dive into an example:

Users of GP Manila can register for an account and log in. They can view products and place orders for these products. There is also an orders page where she can track her current and past orders, the URL looks like this:

https://gpmanila.com/orders?user_id=12345 or https://gpmanila.com/users/12345/orders

As you may infer, this URL allows users to place the user_id of another user and possibly be able to access their order data. Nothing is stopping Nate from accessing a URL like https://gpmanila.com/orders?user_id=66666 to try and see the data of other users. Even if the user_id is random, I can create an automated script to keep trying different combinations until I get something.

Another variation of this exploit is when the frontend URL is safe, something like https://gpmanila.com/orders but the backend URL being called is something like:

https://api.gpmanila.com/orders?user_id=12345

This made it a little difficult for non-tech users to see this vulnerability. But our more techy friends will just have to see open the web browser's console and see this waiting for them.

[2] Users can access the data of others by Form hijacking

Another similar vulnerability is when our form submissions are too lax. For example, our "Checkout Order" API endpoint for GP Manila contains the following request body:

POST: https://gpmanila.com/orders/:checkout_order

{
  "items": [
    {"name": "silver bag", "price": 100, "quantity": 1}
  ],
  "total": 100,
  "user_id": 12345,
  "address": "8 Somewhere Street, Manila City, Philippines"
}

While this method is more secure than #1, it is merely security by obscurity. By hiding the user_id inside the API request data, it is harder to find. But it is still there.

Nate the hacker can just change the user_id of the API request so that his order will be charged to another user but will be delivered to his address.

Resolve [1] and [2] by using JWT tokens

In this case, we are using an OAuth service called Amazon Cognito which handles the authentication of our app separately from our backend.

When a user logs in, he should receive an ID token and a Refresh token from Cognito. The frontend then adds the ID token to the Authorization header of every API request being sent to the backend.

The ID token expires every hour. Once it expires, the backend throws an "IdTokenExpired" error to the FE. The FE takes this as a signal to use the refresh token to get another ID token from Cognito. This ID token is valid for another hour. And is renewed for every hour since.

Both the ID token and the Refresh token are in JWT token format. Before Cognito sends these tokens to the frontend on login, it signs the JWT token with a private key. The process of signing makes sure that if the JWT body gets tampered with, the verification process will fail.

Once the frontend sends the ID token back on every API request, the backend uses the public key to verify that the JWT token body has not been tampered with.

So even when Nate tries to change the user_id in the ID token, the backend's verification of the signature will fail.

Learn more about JWT tokens here

[3] Form submissions allow XSS attacks

At GoodProducts Manila, we have 1000 products. And we let our users review these products. Since its contents are not moderated, the reviews are posted right away.

Nate the hacker can exploit this by adding JavaScript runnable content on one of our most famous products.

<script> alert("this is a popup") </script>

This automatically gets saved in the database and becomes immediately visible to our other users. Since the review is JavaScript code, your frontend may recognize this as executable JavaScript. Hence, when other users view this review, the JS code executes.

Now, my sample JS code looks safe. It simply displays a popup for every user who sees this review. But imagine the more sinister things that Nate can do when he can execute JS code on other users' browsers. He can access the user's session cookies and send them to a remote server he controls. That way, he can log in as that user and create transactions on their behalf.

This exploit is called Cross-Site Scripting (XSS).

Resolve this by adding AWS WAF or any web firewall

Adding web firewalls like AWS WAF to your frontend and backend deployment can help remediate this. Before the review even gets created to your BE, the firewall takes a quick look at the request body and compares it with the filter rules you have set up. Most web firewalls have anti-XSS capability covered by default, helping you filter our requests with JavaScript code inside.

[4] Uploaded files have a virus

We're also looking for good developers for Good Products Manila. So we created a "Contact Us" form where you can upload your resume along with other contact information.

Nate the hacker can upload his resume in PDF format and add a virus along with it. When your HR opens the file on their end, the virus is also executed and goes on to infect his laptop.

Resolve this by adding anti-virus

One way to remediate this is to add anti-virus to our application. In our case, we upload the resume to the S3 bucket and just save an S3 object URL to the database.

We can build our own lambda function that scans every object that gets uploaded to the S3 bucket... Or we can buy one from the AWS Marketplace. Here are the top two options we have tried.

For both options, once you have subscribed through the marketplace, you will be redirected to AWS CloudFormation to provision the service to your AWS account.

Cloud Storage Security

Cloud Storage Security is the top option in terms of anti-virus. It provides access to an account in their web application. You can also add other users/groups to this account, and manage what type of access they have.

It also can protect EBS and EFS volumes. And it integrates nicely with AWS Security Hub, AWS CloudTrail Lake, and AWS Transfer Family.

However, the downside of this product is the cost. It charges 99USD per month for the first 100GB scanned, and 0.80USD per GB after that. All of this is on top of the cost of the AWS resources provisioned, which is usually at 40 USD (for a basic EC2 instance).

bucketAV powered by ClamAV - Antivirus for Amazon S3

BucketAV is cheaper but it does the basic protection job just as well. But it skimps on more advanced features that Cloud Storage offers.

It charges 0.05 USD per hour on instance sizes up to m5.large, but you have to shell out for the costs of running the container infrastructure on ECS.

Its UI is built on top of a CloudWatch dashboard.

[5] Error messages are too specific

In the spirit of having UI that's helpful to the customer, we tell them where they went wrong. Example error notices are:

User with this email does not exist
The password entered for this email is wrong
The password used has already been expired
The user is inactive

But these errors give Nate a lot more information when doing a brute-force attack. He can brute force 1000 emails and be able to know which emails have user accounts on our side. For each user, he can also brute force the password and know the password is wrong. He can keep trying until he gets it right. That's why the best error notification we can give is:

User credentials are invalid

It provides little context for Nate, just that he didn't get the password right.

[6] Password policies are too simple

We often have to protect users against themselves. Simple passwords can be cracked instantly. By forcing them to a minimum character count, and adding symbols/letters/numbers, we are increasing the complexity of the password, making it harder to crack:

[7] No API or FE rate-limiting

The easiest way to brute force a password is by accessing the APIs. One way to prevent it is by adding basic rate-limiting to your APIs and frontend assets. Essentially, we limit each IP address to having a maximum of 500 calls per 5 minutes (for example). When the quota is exceeded, the FE or BE returns an error.

Rate-limiting is also the easiest way to protect against DDoS attacks. In this blog post, we tell our experience of how we survived a DDoS attack and why rate-limiting was central to our strategy.

By experience, we have to fine-tune the number. If the limit is too small, we will end up rate-limiting normal users, and end up penalizing heavy users of our site. If the limit is too big, it offers no real protection against brute force attacks. We also figured out that frontends needs a higher rate limit. Because we also serve fonts, images, and other static assets. This eats up on our limit, and we can reach it quicker compared to the backend.

[8] API logs contain the entire request body

There is no doubt that having some sort of API request/response logging is very helpful for developers to debug issues in production. But if devs aren't careful, they may be creating more headaches for themselves later on.

If we apply this logging blindly, we'll end up saving passwords, credentials, and other PII data on API logs. These logs tend to be less secure than the actual database. These logs usually sit as files inside our server, waiting for a potential hacker to penetrate the server.

Resolve this issue by truncating valuable data from the logs as you write them

# instead of
{ "email": "raphael.jamby@gmail.com", "password: "imhandsome12"}

# do instead:
{ "email": "XXXX", "password: "XXXX"}

[9] Credentials are pushed on the frontend

One final issue is adding hardcoded credentials on the frontend. Sometimes, the frontend needs to access AWS resources directly: upload files to S3, access Cognito APIs for login, etc.

The fastest way to do this is to use the AWS JavaScript SDK and add the AKID and secret key onto the frontend. This is a big mistake, especially if the frontend is a single-page application. There are a lot of web crawlers out there that scour the internet for these keys. And since SPAs are uploaded and delivered to the client in their entirety, the keys will be visible. In a matter of hours after deployment, you'll feel the effects of being hacked. And if the keys you uploaded are admin access, The Armageddon is coming your way.

The best way to resolve this is not to use AWS AKID and Secret, but instead use AWS Amplify with Cognito. Amplify is a library that helps you call Cognito when your users log in and register. When your user logs in, Cognito and Amplify grant temporary access to specified AWS resources.

[10] Not adding API request sanitation

When you are developing your application, you know exactly what each of your API endpoints expects for it to work: attribute name, the data type, the range of valid values, etc. You should use tools like Pydantic or Cerberus. These tools check request data against the rules you these expectations to make sure your app gets only what it expects, otherwise, it throws an error.

When you don't do request sanitation, you allow users to enter values in the API request that may potentially break the system.

[BONUS] Your security is only effective if all of your team knows it too

This final tip is not a technical one. We can have all of these best practices but if at the end of the day, your team doesn't understand why they are doing it, they probably won't end up implementing it.

Want to share other security best practices? Comment them below!

Photo by FlyD on Unsplash

Amazon Cognito: The Ugly Parts (and our workarounds)

Raphael Jambalos — Wed, 10 Jan 2024 15:02:05 +0000

Don't get us wrong. Amazon Cognito is sufficient as an authentication service. We've been using it for more than two years now. But as our needs get more complex, we find ourselves boxed in by Cognito's limitations (the ugly parts). In this post, we will tackle these limitations one by one. And how we addressed them.

For those new to Cognito...

Amazon Cognito is an AWS service that allows you to add authentication and authorization to your application. It is designed to scale to support millions of users.

To get started, you create a user pool. It stores your app's users and provides you APIs for login, register, etc.

Before Cognito, you'd have to find a Python package or Laravel package that adds user authentication to your application, storing your user data on your database. With Cognito, you store user data like email, and passwords in the user pool.

During user login, you configure your FE to talk to Cognito (using an FE library called Amplify) and it gives you 3 tokens that your application can use as the user browses your website. These tokens prove the user's identity and declare what functionalities they can access.

With that, let's proceed with the ugly parts...

1. You cannot make changes to attributes after User Pool creation.

When you create a user pool, you set both standard and custom attributes. Standard attributes are preset attributes like name, birthday, and email (Cognito has a list, you just tick whichever you need). Custom attributes are the attributes with names and types you specify (i.e. income level, education level, etc).

But these attributes can only be set when you create the Cognito User Pool. After the user pool has been created, it can no longer be changed. No additional attributes, no removal of attributes after creation. This limitation forces us to pair our Cognito user pool with a separate database like a DynamoDB table, so that our application can accommodate new attributes later on. We stick with essential data in Cognito and use DynamoDB to store all other user information.

When the user updates their account (i.e. name, gender, section, income), our FE would have to call an API endpoint that updates both Cognito and DynamoDB. It would have been nice to just have to call Cognito and not develop our own set of APIs.

Unfortunately, if you need to change attributes in a Cognito user pool, you will have to create a new user pool and migrate your users there (an exhausting affair, as you'll see here)

2. No support for search by custom attributes.

Cognito cannot search through custom attributes because they aren't indexed attributes. It can only query standard attributes

This forces us to put even more functionality into a custom API and query DynamoDB instead.

3. Customizing the hosted UI is very limited. But Custom UI does not have the same feature set as hosted UI

For Hosted UI, Cognito's OAuth is robust. When the user signs in, it gives you a code that you can exchange for token_set. Your backend does this exchange so you can keep track of your tokens.

For Custom UI, we use Amplify. We cannot use the code method anymore. We use Auth.signIn(). It verifies the username and password, and it sends the token set to the frontend. This is troublesome because our FE now needs to call our custom BE to store the session and the tokens associated with it.

4. You can't keep track of the sessions

When your user login, Cognito generates 3 JWT tokens: id_token, refresh_token and access_token. Let's call them the token set.

Cognito keeps track of the token set internally but does not give you APIs to do the same. So, if you need a feature like "list all active sessions", you would need to store the sessions on your database. This means that after every login with Cognito, your FE would have to call your BE APIs to send the token set and create the session on your database. All this trouble just so you can show these sessions later.

Cognito has only 3 APIs for token management:

Generate token set (after login)
Revoke one token
Revoke all tokens

Since we are keeping track of the sessions on our database, we have to wrap those Cognito APIs into our own API Endpoint. If the user wants to delete a session, we delete it on our database and revoke the token in Cognito.

We also have to create an extra API endpoint for the Show All Sessions API, which is currently not supported by Cognito.

5. Cognito APIs have rate limitations (and some couldn't be increased)

On one of our projects, we had to migrate 3M users from an on-premise database. We also had to migrate their passwords (if it complies with Cognito's password policy). Cognito has a CSV Import functionality but it does not allow importing passwords, so we had to find another route.

We created a lambda function that reads a CSV from S3. It reads the CSV row-by-row and sends each row to SQS. SQS queues each row, and another lambda function consumes those tasks. For each user, the consumer lambda function does the following:

Create the user in Cognito [CreateCognitoUser]
Create the user in DynamoDB, with the user_sub from Cognito as the primary key
Using Cognito Admin SDK, change the user's password with the one in the file [AdminUpdateUser]

Since we have 3M users, this operation took around 26 hours. And this is where we encountered several issues.

[1] The CreateCognitoUser API has a limit of 100 requests per minute. We also asked AWS to expand their request limit to 200, which they did.

[2] The AdminUpdateUser API has a limit of 30 requests per minute. And this cannot be requested for a higher limit. And this is where we got stumped. We repeatedly got throttled here so we had to add code in our Lambda that retries if we got throttled. But it had the unfortunate consequence of also limiting our rate of user creation to 30 requests per second. This caused our deployment to last for 26+ hours.

[3] Since our workflow ran for 26+ hours, our Lambda scaled up to accommodate the demand. It eventually consumed all available concurrency in the prod account. This means all the other Lambda functions couldn't run anymore because ours hogged all the concurrency in the account! We had to request AWS to expand our account-level concurrency to 10,000 and allocate just 2,000 concurrency to Lambda functions involved in the migration. The 8,000 concurrency is for other lambda functions to use.

7. Dirty data

Our 26hr migration run had some problems:

The user is created multiple times in Cognito even if they appear only once in the CSV. We had to manually delete these users
The user is created in Cognito, but not in DynamoDB. This may be because the Lambda function stopped at the part where the user is created. We deleted these users as well.
The user is created in DynamoDB but not in Cognito.
Some users were created with the state FORCE_CHANGE_PASSWORD, which our application wasn't designed to handle.
Cognito treats emails as case-sensitive. So a user with the email "Jamby@aws.com" and "jamby@aws.com" are different users.

8. Our bill spiked during the migration

Of the 3M users our customer had, perhaps only 300K are active users at any given point in time. Cognito prices per monthly active user. So we expected to be billed for only 300K users. However, if you created the users programmatically, they are considered active upon creation. Hence, for the first month, we had to pay for 3M active users! Not to mention the additional costs of running thousands of Lambda functions for 26 hours.

9. It's very expensive to "dry run" a big migration

With 3M users, we have a lot of variety in our data. It would have been ideal if we could run the whole dataset before it goes to production. But doing so would cost over 9000 USD! So we can only test with a subset of the data. This meant we didn't encounter all sets of possible errors during the dry run. And so we had to face a lot of the errors after the production migration.

That's all folks!

I hope this article showed you what you're getting into with Cognito, and the potential pain points you'd encounter down the line.

Did I miss anything? Comment it below!

Photo by Onur Binay on Unsplash

Challenge #4: Create CI/CD for Serverless Apps

Raphael Jambalos — Thu, 17 Nov 2022 09:07:49 +0000

This was presented as a lightning talk last Nov 17, 2022 in the 4th AWSUG F2F Meetup at the GCash Office, BGC

In this post, we will create a basic CI/CD pipeline for the serverless application we built in the first challenge. For a quick recap, our serverless application has 3 API endpoints: for creating a loyalty card, seeing all loyalty cards, and displaying details about one loyalty card. The loyalty card data are stored in DynamoDB:

With a CI/CD pipeline, we can deploy changes to our application in a consistent and automated way.

To achieve that, we will create a CI/CD pipeline with AWS CodePipeline. Inside CodePipeline, we have 2 stages. One for Source and another for Build. By the end of this walkthrough, our changes will be deployed to production once it is merged to master:

SOURCE STAGE: CodePipeline simply watches for any changes from the master branch in GitHub. Once a change has been pushed, our pipeline gets triggered and the code is copied for further processing
BUILD STAGE: CodePipeline triggers CodeBuild and supplies it with a copy of the source code. CodeBuild is an automated build service. It relies on a file called "buildspec.yml" inside the repository. The file contains instructions on how to prepare and deploy our Serverless App. When CodeBuild is triggered, it executes these instructions.

Now, let's set up our CI/CD Pipeline

[1] Head over to CodePipeline by searching for it in the top search bar

Then, hit the "Create pipeline" button

[2] Type the name of the pipeline and leave everything else as default. Press Next

[3] In the Source stage, select the source provider that applies to you. For me, my code is stored in GitHub so I select the "Github (Version 2)" option.

Since my repository is outside AWS, I need to authorize CodePipeline to access it. Press "Connect to GitHub". You may have to do something similar if you select BitBucket. Aside from the two, CodePipeline supports CodeCommit as a source repository. However, Gitlab is not natively supported.

A pop-up screen will appear. Create a name for your connection. Then, press "Connect to GitHub"

If you are signed in to the GitHub account, you will see this pop-up screen in Github that requires your permission. Grant it by clicking "Authorize AWS Connector for GitHub"

[4] You will be redirected back to this screen. If you have done this before, you will see a pre-installed GitHub App. If there is none, hit "Install a new app". You will also have to click "Install a new app" if your GitHub repository is owned by another organization and your GitHub user was just invited to that organization.

After you have chosen the correct GitHub App, click "Connect"

[5] With that, the pop-up window closes and you are redirected back to the "Add source stage" screen. Choose your repository and branch name. Leave the rest as default and then, click "Next"

[6] In the build provider, choose "AWS CodeBuild". If you haven't created your CodeBuild Project yet, hit "Create Project"

Another pop-up screen appears for us to create our CodeBuild project. In this screen, add your project name:

Scroll a bit down for the environment section.

Environment image: Managed image
Operation System: Ubuntu
Runtime: Standard
Image: aws/codebuild/standard:5.0 (or whatever the latest version is)
Image version: Always use the latest image for this runtime version
Environment type: Linux
Privilged: Enabled

In the previous step, we connected GitHub to our CI/CD pipeline. In this step, we created a CodeBuild project that runs a series of commands to build our application. These commands are listed under the buildspec.yml file. Here is our project's buildspec.yml file

In our project's buildspec.yml, we only used 2 phases: install and build. Phases allow us to group our build commands. For our install phase, we installed serverless framework and pip. Then, for the build phase, we ran serverless deploy to deploy our application.

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - ls -a ~
      - npm install
      - npm install -g serverless
      - curl -O https://bootstrap.pypa.io/get-pip.py
      - ls -a ~
      - python3 get-pip.py --user
      - pip --version
  build:
    commands:
      - serverless deploy --region $AWS_REGION

Leave the rest as default and click "Continue to CodePipeline". This creates the CodeBuild project. You will then be redirected back to the CodePipeline screen, with a pre-filled form. Double-check the values. Once satisfied, click next:

[7] You will be redirected to the deploy stage. We don't need this for now. With the serverless deploy command in CodeBuild, we are deploying the application already. Click "Skip Deploy Stage".

Then, we will be on the Review screen. Review the values and press "Create pipeline". Right after pipeline creation, it executes immediately.

The Source stage executes immediately and fetches code from GitHub. If this stage fails the first time, just click retry. Sometimes the first run malfunctions.

Once the source stage is done, the Build stage triggers. In this stage, CodeBuild executes the instructions coded inside your buildspec.yml. If you are curious as to what is happening while CodeBuild is executing, click the "details" and you will see a log of the deployment:

It will take a few minutes before the build completes. Wait for a few minutes and come back to it.

Let's troubleshoot the deployment

[8] From the result of our first run, it looks like the execution failed.

To figure out what's wrong, scroll down and find the last few lines of the execution. You shall see the error there.

It looks like our Build execution failed when we tried to deploy our serverless app using the serverless deploy command. The error is the role that CodeBuild uses is "not authorized to perform" operations in CloudFormation. Since Serverless Frameworks apps are just CloudFormation deployments under the hood, we need to equip our CodeBuild with an IAM role that has permission to access CloudFormation and other needed resources.

[9] Let's update the permission of our CodeBuild role. First, let's go to the Build Project that powers our execution.

Then, under Edit, click Environment:

You will see this screen. Copy the ARN of the Service role:

Then, go to the AWS IAM console.

Click "Roles" on the left-hand side menu and search for the role

Under Permissions, click Attach policies.

Search for the following AWS-managed policies and attach them to the role. In production, be more deliberate and granular about the permissions we grant to this role. For our exercise, this will do.

AmazonS3FullAccess
AWSCloudFormationFullAccess
AmazonDynamoDBFullAccess
AmazonAPIGatewayAdministrator
CloudWatchLogsFullAccess
IAMFullAccess
AWSLambda_FullAccess

Now, we're good. Let's run the CodePipeline again. Find the pipeline you created earlier and click "Release Change"

On this run, our application is now deployed!

Let's see it in action on our Postman collection:

Now, let's try to modify the response body just a bit by adding "status": "success". Then, we commit the change to master and our CI/CD should deploy our change:

Our CI/CD pipeline should automatically run. After a few minutes, our change should now reflect in our API.

That's all folks!

Photo by JJ Ying on Unsplash

Field Guide to Surviving DDoS Attacks in your application

Raphael Jambalos — Wed, 09 Nov 2022 02:50:05 +0000

One of our customer-facing websites got attacked with a huge DDoS attack recently, to the magnitude of 80M-100M requests per hour. For context, our website usually just receives 30,000 requests per minute. That's a 3000x increase in traffic. Luckily, we were able to respond promptly and we stopped the DDoS attack within 3 days. But those 3 days were the most stressful times of my year (so far).

I created this field guide for developers so I can save all of you the firefighting and research I had to do on the fly during those 3 days. While this is by no means a comprehensive guide, it will give you the tools necessary to respond to a DDoS attack.

What's a DDoS attack?

At its core, DDoS attacks are denial-of-service attacks. They aim to overwhelm your servers with so much fake traffic that your legitimate end-users won't be able to access your application. This results in downtime, and with your website down, your customer won't be able to buy anything, and your revenue grounds to a halt.

There are 3 types of DDoS attacks: Application Layer, Protocol Attacks, and Volumetric attacks. We will focus on application layer attacks for this article.

Starting Architecture

Let's start with this typical network architecture. We have an eCommerce website called jambyswags.com. Both its PHP backend and NuxtJS frontend applications are hosted in EC2. Once the customer access the website, the request is routed to a load balancer that distributes traffic between 2 EC2 instances.

Confirm if you are being attacked

A DDoS attack usually starts with your website becoming unavailable to all of your users. First, check the CPU utilization of the EC2 instances of your backend and frontend applications. If your EC2's CPU utilization is overloaded, then this is a sign it's a DDoS attack.

The next step is to go to your ALB's CloudWatch Metrics to check your application's request count. In our case, our website typically takes in 500 requests per minute. Then, we suddenly experienced 1M - 1.5M requests per min. That's a 3000x increase from our baseline, a big sign of a DDoS attack.

The final step is to check if the traffic is legitimate. You don't want to stop your legitimate users from going bananas shopping on your website, especially during a sale. One way to do this is to enable VPC Flow Logs for the ENI of the load balancer. From here, you can see the IP addresses connecting to your application. If it's all from the same set of IP addresses, that's credible proof that you are experiencing DDoS Attack.

Another indicator is if you're not having a promo or major ad push for that day but are experiencing a big jump in traffic.

First Response: AWS WAF

Now that you are sure you have a DDoS attack on your hands, it's time to bring the big guns. As a first response, it is essential to add AWS WAF behind your load balancers. With AWS WAF, we create a web ACL that contains rule groups. Rule groups can be managed by AWS or customized by you. It can also be a regular rule group, which checks requests based on their contents, or a rate-based rule group, which sets a cap on the requests per minute coming in from each IP address.

Before the request goes to your ALB, the web ACL scans the requests based on the rule groups you add to it. For web applications, we recommend adding the following rule groups:

AWS-AWSManagedRulesAdminProtectionRuleSet - checks your requests for paths that may be trying to get access to your admin pages
AWS-AWSManagedRulesAmazonIpReputationList and AWS-AWSManagedRulesAnonymousIpList - checks your requests if it comes from suspicious IP addresses
AWS-AWSManagedRulesCommonRuleSet - checks common exploits found in the OWASP 10
AWS-AWSManagedRulesKnownBadInputsRuleSet
AWS-AWSManagedRulesSQLiRuleSet - checks against SQL injection

These rules check against suspicious IP addresses and potentially malicious request bodies. Meanwhile, the Bot Control is a rule that should be last in your web ACL. It has an additional cost of 1USD per million but protects your application from malicious bots trying to access your application.

AWS-AWSManagedRulesBotControlRuleSet

Another thing you can do is add a rate-based rule. In this example, you can block IP addresses if the rate at which they access the site exceeds 500 requests per minute.

The caveat with AWS WAF is it can potentially break your application. For instance, the CommonRuleSet has a rule against request bodies exceeding a certain size. If your API was uploading 7MB files, it's probably going to hit this rule and your end users won't be able to upload files. The key here is to test it out first in staging to identify the rules that break your application. Then, you can adjust those rules from "BLOCKED" to "COUNT"

First Response: Route 53 Geolocation

A quick and dirty way to limit DDoS attacks is to limit who can resolve your website's domain name (i.e jambyswags.com) to the IP address of your load balancer. In our case, our customer's website engages audiences only in Singapore and the Philippines. And the DDoS attack featured servers from Asia, Europe, Canada, S.America, and Africa. With this, it makes sense to only make our website accessible only to people from Asia. This effectively stopped all attacks from non-Asian countries.

While Asia is still a big audience, the attacker now becomes limited in what servers he/she can use. It won't stop the attack, but it can help reduce its volume.

To make it even more effective, try shifting your API to a different domain name to force your attackers to use the DNS resolver. For example, move your backend from "jambyswags.com" to "prod.backend.jambyswags.com"

First Response: File Abuse Teams

With the two actions above, your applications are more protected against DDoS attacks. However, you may see your AWS WAF costs spike. As of writing, AWS WAF charges 0.60USD per million requests. In our case, we were attacked with 90 million requests per hour over 55 hours. That's 4.95 billion requests, or 2,970USD. That's still a huge price to pay for protection.

One way to permanently stop these attacks is to those IP addresses to the Abuse Team of AWS, GCP, and Azure. This process usually takes a few days, but the sooner you get the ticket filed, the sooner it can process. To get those abuser's set of IP addresses, you can browse the Sampled Request section of your AWS WAF.

You will also have to provide application logs that these IP addresses are disrupting your website

Second Response: Reduce Surface Area

Your application is protected, for now. While the intruder can't brute force your website anymore because of the rate limits of AWS WAF, they can still try to penetrate your application. One way to prevent this is by reducing the surface area exposed to attackers.

First, move your application servers to the private subnet. With this, they can no longer be accessed directly, only through the load balancer. If you need to SSH to them, use AWS Sessions Manager via the AWS Console.

Second, audit the security group of each resource inside your VPC. Make sure only the ALB exposes a port to the world (0.0.0.0/0). The rest of the services exposes only the port they need to expose to only the resource that needs to access them. For instance, the Backend EC2 should have port 80 open only for the ALB that forwards requests to it.

Third, create a Database Private Subnet. It is a private subnet that has much less network access than the private subnet where your application is hosted in. Its NACL rules are also tighter in that it opens a few ports inbound and outbound.

Second Response: Use private communications for API to API comms

Sometimes one of your backend APIs will have to connect to another backend API in your network. Typically, your first backend API will send a request that traverses the open internet, back to your second backend's ALB, and to the second backend API. Aside from additional network costs you might incur, this link may be tagged as a DDoS attack by WAF, especially if this link is high volume.

To solve this, create an internal ALB that resides in your application private subnet. With this, your first backend API sends a request to the internal ALB within the same network and the traffic doesn't have to traverse e the open internet.

Third Response: Migrate FE to a Single Page Application

In our case, our FE application was a NuxtJS application deployed to EC2 with the command nuxt serve running. With minimal adjustments, we were able to able to generate a single-page application using the npm generate command. We uploaded the generated dist folder to S3, hosted it as a static website, and connected it with CloudFront.

Depending on how your FE application is written, this may not be as easy to do or even possible. Some FE applications are baked into the Backend API and require a full rewrite to become an SPA. Some FE applications are purposedly hosted as a Server Side Rendered (SSR) application and are hard to migrate to an S3-CloudFront setup.

CloudFront offers additional DDoS protection as the first point of contact for your application is the edge location of CloudFront, and CloudFront has DDoS protection built in. It is also much cheaper to serve requests via CloudFront since it is cached nearer to the user.

Third Response: No-Cache CloudFront for the BE

With this technique, you will add a no-cache CloudFront distribution to your application. This way, your API's first point of contact will be an AWS edge location, and CloudFront can use its anti-DDoS features. Visit this AWS blog to learn more.

Fourth Response: Consider AWS Shield Advanced

If your DDoS attack is sophisticated, the tactics we mentioned above may not be enough. Here are some common work throughs they can do:

Use thousands of IP Addresses from Asia to rain down hell on your application (and evade rate-based rules)
Determine your application's most expensive API endpoints and target that with a low volume DDoS - For example, if your add-to-cart functionality takes 3s to load and is using SQL statements that take much effort from your DB to fulfill, they can get maybe 100 IP Address target that endpoint with 100 requests each. You'll be down in no time.
Overwhelm your WAF and CloudFront with millions and tens of millions of requests per minute such that even if your website doesn't go down, you will burn through your AWS budget.

For the first and third scenarios, you can count on AWS Shield Advanced. With this service, you will have a dedicated Shield Response Team who will proactively make changes to your AWS environment to protect your assets. You will also be not liable for any bill spike resulting from the provisioning of excess assets during a DDoS attack.

The downside is that AWS Sheild Advanced is 3000 USD per month, with a 12-month commitment. Hence, this service makes more sense to avail if you are in an enterprise with deep pockets and a lot of assets to protect.

Conclusion

While I don't wish a DDoS attack happens to your applications, it's best to be prepared for this possibility. This article provided you with 4 layers of responses you can do to keep your website protected.

Use EventBridge for your event-driven architecture in 8 steps

Raphael Jambalos — Sun, 25 Sep 2022 16:11:47 +0000

This article was also presented as a talk at the recently concluded APAC Community Summit 2022 in Bangkok, Thailand

Prerequisites

Before you begin, I suggest you read my introduction to EventBridge.

Scenario

In this demo, we will start with a basic repository that has 3 microservices: orders, points, and notification. The order service marks an order as delivered. The points service award points based on a delivered order, while the notification service sends a simple "your order has been delivered" email. Our baseline repository looks like this:

By the end of the post, we would use EventBridge so that the order service publishes an event once an order is marked as delivered. Then, the points and notifications receive that event. This triggers them to award points and send an email.

Clone the repository and then, run git checkout baseline-two. There are 3 folders inside, representing each of the services. Set up each one by following the setup guide on the README.md of each service.

If you want to know more about how I structured the 3 applications, you can check out my blog post that talks about how we build apps on top of Serverless Framework.

In the next section, let's test out all 5 endpoints by using the Postman collection at the root of the repository. It is ideal to have completed the full setup before proceeding to the next section.

Orders Service

The order service has 2 endpoints. The first endpoint creates an order. It outputs an order_number that we use for the next endpoint.

The second endpoint marks that order as delivered. We get the order_id generated in the first endpoint and place it as input here.

In the response, we see the order successfully marked as delivered.

Points Service

The points service also has 2 endpoints. The first endpoint awards points to a given user. If the user does not exist in the database yet, it creates an entry for it in the DB.

The second endpoint displays the existing points of the user so we can verify if the points did increase.

Notification Service

The notification service only has one endpoint. It takes a message and sends an email to an email address we have set up.

An email is then sent to our designated email:

Our Objective

When you clone the repository, you can deploy all 3 microservices and call the 5 API endpoints separately. But that's not what we are here for. We plan to connect the 3 microservices such that when we call the "Mark as Delivered" endpoint, it sends an "order_delivered" event to EventBridge.

By the end of this post, we will just have to call the order service to mark the order as delivered. Then, via EventBridge, the user will be credited his points and receives a short email.

Step 1

For the serverless.yml of all three services, add the additional permissions and environment variables as seen below:

# modify: order-service/serverless.yml
# modify: notification-service/serverless.yml
# modify: points-service/serverless.yml

provider:
  name: aws
  runtime: python3.8
  lambdaHashingVersion: '20201221'
  stage: ${opt:stage, 'dev'}
  iamRoleStatements:
    - Effect: "Allow"
      Action: "dynamodb:*"
      Resource: "*"
    - Effect: "Allow"
      Action: "sns:*"
      Resource: "*"

    # ADD THIS STATEMENT
    - Effect: "Allow"
      Action: "events:*"
      Resource: "*"
  environment:
    ORDERS_TABLE: ${self:provider.stage}-ecom-orders-table

    # ADD THESE THREE ENV VARS
    EVENT_SOURCE_NAME: ph.ecommerce.com
    EVENT_BUS_NAME:  ${self:provider.stage}-ecom-event-bus
    EVENT_BUS_ARN: arn:aws:events:${self:custom.region}:${self:custom.accountId}:event-bus/${self:provider.environment.EVENT_BUS_NAME}

Step 2

Specifically for the orders service, add the EcommerceEventBus in the resources section. This is a CloudFormation code to programmatically create an EventBridge Event Bus.

# modify: order-service/serverless.yml

resources:
  Resources:
    EcommerceEventBus:
      Type: AWS::Events::EventBus
      Properties: 
        Name: ${self:provider.environment.EVENT_BUS_NAME}

Step 3

Now, we redeploy all 3 services. These should add the environment variables and additional permissions. It should also create the event bus.

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

cd order-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

cd points-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

cd notification-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

Step 4

Next, let's focus on the code of the orders service. Let's start by adding the EventbridgeGateway. It uses the boto3 library to send tasks to EventBridge. We placed it in its own gateway to abstract boto3 code from the rest of our application.

# NEW FILE: order-service/gateways/eventbridge_gateway.py

import boto3

class EventbridgeGateway:
    @classmethod
    def put_event(cls, event):
        client = boto3.client('events')

        return client.put_events(Entries=[event])

And then, let's create an EventbridgeEvent model file that stores the business logic of building and sending events to EventBridge. It uses the EventbridgeGateway to send the event.

# NEW FILE: order-service/models/eventbridge_event.py

import json
import os

from gateways.eventbridge_gateway import EventbridgeGateway
from helpers.response import Response


class EventbridgeEvent:
    EVENT_BUS = os.environ.get("EVENT_BUS_NAME")
    SOURCE_URL = os.environ.get("EVENT_SOURCE_NAME")

    def __init__(self, event_name, event_body):
        self.event_name = event_name
        self.event_body = event_body

    def serialize(self):
        json_details = Response.construct_json_body(self.event_body)

        return {
            "Source": self.SOURCE_URL,
            "DetailType": self.event_name,
            "Detail": json_details,
            "EventBusName": self.EVENT_BUS,
        }

    def send(self):
        event_json = self.serialize()
        EventbridgeGateway.put_event(event_json)

Then, on the order model, let's add the send_order_delivered_event() function that creates an EventBridgeEvent and sends it to the event bus.

# UPDATE: order-service/models/order.py

class Order(DynamodbModelBase):
    DYNAMODB_TABLE_NAME = os.getenv("ORDERS_TABLE")

    # ADD THESE TWO FUNCTIONS
    def serialize_for_eventbridge(self):
        return {
            "order_number": self.data["order_number"],
            "user_id": self.data["user_id"],
            "total": self.data["total"]
        }

    def send_order_delivered_event(self):
        event = EventbridgeEvent("order_delivered", self.serialize_for_eventbridge())
        event.send()

Finally, let's call the send_order_delivered_event() in our handler.

As you can see in the code, an order object is instantiated, marked as delivered, saved to DB, and then, we call the send_order_delivered_event() to send an event to EventBridge.

# UPDATE: order-service/handlers/mark_order_as_delivered.py

def handler(event, context):
    try:
        ...

        order = Order.find(primary_key)
        order.mark_as_delivered()
        order.save()

        # ADD THIS LINE
        order.send_order_delivered_event()

Step 5

Now, let's shift our attention to the points service. Let's add a new Lambda function whose event source is our Event Bus. Notice how we added a filter where we only want to receive "order_delivered" events.

# UPDATE: points-service/serverless.yml

functions:
  awardPointsSix:
    handler: handlers/award_points.handler
    events:
      - eventBridge:
          eventBus: ${self:provider.environment.EVENT_BUS_ARN}
          pattern:
            source:
              - ${self:provider.environment.EVENT_SOURCE_NAME}
            detail-type:
              - order_delivered

In the case you want to change the eventBridge parameters we defined above (or you made a mistake somewhere), you have to change the name of the function. It's my sixth attempt at getting it right, hence my function's name is awardPointsSix.

In the handler for our award points endpoint, let's add an if statement so we can accommodate both calls from the API endpoint and EventBridge. Notice the difference in how we extract the payload.

# UPDATE: points-service/handlers/award_points.py

def handler(event, context):
    try:
        print("AND THE EVENT IS NOWWWW!!!!<3 <3")
        print(event)

        if 'detail' in event:
            body = event['detail']

            primary_key = {}
            primary_key["user_id"] = body['user_id']
        else:
            body = json.loads(event['body'])

            primary_key = {}
            primary_key["user_id"] = event['pathParameters']['user_id']

        user = User.create_if_not_exist(primary_key)
        user.award_points(body)
        user.save()

Step 6

Going to the notification service, let's add another Lambda function that receives an event from EventBridge. Notice how it's similar to the original API-based Lambda function, sendEmailNotification, except for the eventBridge part.

# UPDATE: notification-service/serverless.yml

functions:
  sendEmailNotificationSix:
    handler: handlers/send_email_notification.handler
    events:
      - eventBridge:
          eventBus: ${self:provider.environment.EVENT_BUS_ARN}
          pattern:
            source:
              - ${self:provider.environment.EVENT_SOURCE_NAME}
            detail-type:
              - order_delivered

Like with the award points API, let's add this if statement to accommodate both EventBridge event and API calls.

# UPDATE: notification-service/handlers/send_email_notification.py

def handler(event, context):
    try:
        if 'detail' in event:
            body = event['detail']

            primary_key = {}
            primary_key["user_id"] = body['user_id']
        else:
            body = json.loads(event['body'])

            primary_key = {}
            primary_key["user_id"] = event['pathParameters']['user_id']

        topic_arn = os.getenv("SNS_TOPIC_ARN")

        SnsNotificationGateway.publish_message(
            message=body["message"],
            subject="Order marked as delivered",
            topic_arn=topic_arn,
            region="ap-southeast-1"
        )

Step 7

With all that code change, let's deploy once again!

You can also opt to do local testing first via serverless offline. Learn more about that in this blog post

ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)

cd order-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

cd points-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

cd notification-service
serverless deploy --region ap-southeast-1 --stage dev --accountId $ACCOUNT_ID

Step 8

Now, let's test it out. Check first how many points the user has by calling the "Check User Points" API Endpoint:

Then, create an order and mark it as delivered. When you call the "Check User Points" API endpoint again, the points should have increased on their own:

Also, you should have received an email.

That's it!

You have used EventBridge to connect your event-driven serverless architecture. In the next post, we'll tackle the problems we encountered while using EventBridge in production and how you can address them. Stay tuned!

Photo by Kelly Sikkema on Unsplash

How we embraced Amazon EventBridge for our event-driven architecture

Raphael Jambalos — Thu, 15 Sep 2022 00:33:05 +0000

This article was also presented as a talk at the recently concluded APAC Community Summit 2022 in Bangkok, Thailand

Last year, our team embraced microservices architecture. Instead of one big monolithic application, we developed a dozen small microservices. This setup allowed us to push out features faster and keep each microservice simple and maintainable.

As our application grew, our events started to have actions that spanned different services. It became a distributed nightmare just to mark an order as delivered. We had to make sure actions in the payments, referral, points, and notification services were executed successfully.

In this post, I'll introduce EventBridge and how it simplified the coordinating of multiple microservices.

Monolith Apps get slower over time

Let's start by discussing how we would typically do this in a traditional monolithic application:

A monolith responds to this event by executing the series of actions sequentially. As an example, an "order_delivered" event may have the following actions:

Mark the order as "delivered"
Collect Payment
Send the "your order has been delivered" email
Award the rewards points
Check the order if it had a referral code if it did award additional points

The monolith does this synchronously and makes the user wait until all the steps are finished. As we add more actions to this event, the waiting time gets longer.

Because monoliths tightly couples modules together, a simple change in one module becomes more complex than it should be. Unintended consequences in a seemingly unrelated module pop up occasionally. A full regression test is usually run to ensure against this.

Microservices are smaller, simpler, and easier to make changes to

For these reasons, we decided to move to an event-driven serverless architecture. Instead of one big application housing all modules, we broke down our application into a dozen microservices, each in charge of a specific group of functionality:

For the order_delivered event, its actions are owned by 5 different microservices. In our team, we assign 2-3 microservices to each developer. As system owners, they create features and troubleshoot bugs.

Since the actions required by the "order_delivered" event is delegated across five microservices, fulfilling this event requires us to make the microservices communicate with one another.

The simplest way to connect them is via API. But that would end up being even slower than the monolithic approach because of the latency caused by doing five API calls.

The shortcomings of using SQS to decouple the architecture

The better way would be for the order service to execute its component actions asynchronously. First, it receives the request. Then, it sends a task to the SQS queue of the payment service. On the receiving end, the payment service will get the task on the SQS queue and run the action to collect the payment. The order service then sends a task to the SQS queues of the points, referral, and notification services. Each service processes the action by getting the task off the queue. Then, it returns a response to the customer.

In this process, the response was sent only when all 4 actions have been queued in their respective SQS queues - not when all 4 actions have succeeded. That is asynchronous processing in action.

SQS queues are many-to-1. It can have many producers but it only sends to one homogenous group of consumers. This easily becomes a problem with our setup because this means we have to build a new SQS queue for each group of consumers. For instance, we added a new action in our event that sends a recommendation of items based on the delivered order. This action is housed in the recommendation service. And as it is a new action, another SQS queue is required.

As our app gets more complex over time, we add more actions to our events. And more actions mean more SQS queues being added. This adds to the overhead of maintaining our system over time.

A reader has pointed o

Enter EventBridge

EventBridge addresses this pain point while still retaining the decoupling that SQS introduced. Instead of sending 4 different tasks to 4 different SQS queues, it just publishes the "order_delivered" event to an event bus in EventBridge.

An event bus is like an SQS queue but instead of being many-to-1, it is many-to-many. This means many different event producers can push events to an event bus, and the event bus sends the event to all consumer systems configured to listen to it.

Aside from that, it allows consumers to listen only to specific events and disregard other events. When we configure consumers to "listen" for events, we do so by creating event rules against the event bus. These rules help the consumer filter what events it receives from the event bus.

In our case, the payment service only listens for the "order_delivered" event. When an event for the "order_shipped" or "order_confirmed" happens, it is not sent to that service.

Conclusion

EventBridge allows microservices to communicate with one another in a decoupled manner. We only need one event bus for all our events. Hence, managing the actions of our events across services becomes simpler.

Next Steps

In the next post, we will create a simple demo to demonstrate the powers of EventBridge.

Photo by Denys Nevozhai on Unsplash

ElasticSearch: Zero to Hero in 12 Commands

Raphael Jambalos — Sat, 27 Aug 2022 04:23:00 +0000

It's relatively easy to get started with ElasticSearch. But as our use cases get more specific, we found the documentation lacking. This guided cheatsheet will execute 12 commands: from setting up your ES index to making advanced ES queries to support advanced (but common) use cases.

The 12 commands works when done sequentially. I will explain each of them, but trying them for yourself is still the best.

This post is part of a broader series on ElasticSearch that will be released in the coming weeks:

The Guided ElasticSearch Cheatsheet you need to Get Started with ES - you are here
Using DynamoDB + ElasticSearch for prod workloads - coming soon
And how to create DynamoDB Streams to sync data changes from DynamoDB to ES asynchronously - coming soon

0 | Prerequisites

Install Elasticsearch with this official ES Guide. And then, turn on the ES server on localhost:9200

For easier testing, installing an API platform like Postman is a must.

A | Setup the index

In ElasticSearch, we store our data in indexes (similar to tables in your MySQL database). We populate indexes with documents (similar to rows). We will create and set up your first index in the subsequent commands.

[1] Verify the ES cluster is accessible



GET localhost:9200

First, make sure your local ES server is online, and you have your Postman open. Create a new GET request headed for localhost:9200. You should see something like this:

[2] Create an index



PUT localhost:9200/mynewindex

Now, let's create our first index. Indexes store our data. It is equivalent to creating a table in relational databases.

[3] Create the mapping for the index

The index we just created has no mapping. A mapping is similar to a schema in SQL databases. It dictates the form of the documents that our index will ingest. Once defined, the index will refuse to accept documents that cannot fit into this mapping (i.e, we defined stocks as integer below. If we try to insert a row with stocks="none", the operation will not continue).

One thing you'd notice with ES is that these mappings are permissive by default. If I add a row with a new attribute "perishable" = true, when I push a document to ES, the schema will add that attribute and infer its data type. In this case, it will add a new attribute in the mapping for "perishable" with data type "boolean".

There are options that you can add when you create your index to only allow attributes defined in mapping of your index, nothing more, nothing less.

In this command, we create the mapping for our newly created index.



PUT localhost:9200/mynewindex/_mapping

{
    "properties": {
        "product_id": {
            "type": "keyword"
        },
        "price": {
            "type": "float"
        },
        "stocks": {
            "type": "integer"
        },
        "published": {
            "type": "boolean"
        },
        "title": {
            "type": "text"
        },
        "sortable_title": {
            "type": "text"
        },
        "tags": {
            "type": "text"
        }
    }
}

Most of the data types are straightforward, except for Text and Keyword. This article explains the difference clearly.

But TLDR, Text allows you to query words inside the field (i.e querying "Burger" will show the product "Cheese Burger with Fries"). It does this by treating each word in the text as individual tokens that could be searched: "cheese", "burger", "with", "fries".

On the other hand, Keyword treats the content of the field as one, so if you want to get the cheeseburger with fries, you'd have to query it: "Cheese Burger with Fries". Querying "burger" will return nothing.

[4] Show the mapping of the index

Let's verify if we have successfully created the mapping for the index by sending a GET request.



GET localhost:9200/mynewindex

B | Data Operations with our ES Index

With our index already set up, let's add data and chip away at the more exciting bits of ES!

[5] Create data for the index

For this section, let's send three consecutive post requests with different a request body per request. This adds 3 "rows" inside our Elasticsearch index.



POST localhost:9200/mynewindex/_doc

{
    "product_id": "123",
    "price": 99.75,
    "stocks": 10,
    "published": true,
    "sortable_title": "Kenny Rogers Chicken Sauce",
    "title": "Kenny Rogers Chicken Sauce",
    "tags": "chicken sauce poultry cooked party"
}

POST localhost:9200/mynewindex/_doc

{
    "product_id": "456",
    "price": 200.75,
    "stocks": 0,
    "published": true,
    "sortable_title": "Best Selling Beer Flavor",
    "title": "Best Selling Beer Flavor",
    "tags": "beer best-seller party"
}

POST localhost:9200/mynewindex/_doc


{
    "product_id": "789",
    "price": 350.5,
    "stocks": 200,
    "published": false,
    "sortable_title": "Female Lotion",
    "title": "Female Lotion",
    "tags": "lotion female"
}

[6] Display all the data

Now, let's see if the three documents we inserted via command #5 got inside our index. This command shows all the documents inside your index:



POST localhost:9200/mynewindex/_search

{
    "query": {
        "match_all": {}
    }
}

It does!

[7] Exact search with product id

Now, let's start with a simple search. Let's search by product id.



POST localhost:9200/mynewindex/_search

{
    "query": {
        "term": {
            "product_id": "456"
        }
    }
}

In the command above, we are using a "term query" because we are looking for a product with a "product_id" that exactly matches the string "456". The term query works because the data type of "product_id" is "keyword".

[8] Fuzzy search with titles

Now, onto the more exciting bits.

ES is known for its comprehensive search capability. Let's sample that by creating our first Fuzzy search. Fuzzy searches allow us to search for products by typing just a few words instead of the whole text of the field. Instead of typing the full name of the product name (i.e Incredible Tuna Mayo Jumbo 250), the customer just instead has to search for the part he recalls of the product (i.e Tuna Mayo).



POST localhost:9200/mynewindex/_search

{
    "query": {
        "match": {
            "title": "Beer Flavor"
        }
    }
}

In the default setting, we can get the product "Best Selling Beer Flavor" even with our incomplete query "Beer Flavor". There are other settings that allow us to tolerate misspellings or incomplete words to show results (i.e Bee Flavo)

Also, notice carefully that we now use a "match query" instead of a "term query" because we want to be able to get results even if we didn't type the full product name. The match query works because the title field is of type "text".

[9] Sorted by prices

Another thing we usually have to do with an e-commerce website is to sort products by specific categories like price or rating:



POST localhost:9200/mynewindex/_search

{
    "query": {
        "match_all": {}
    },
    "sort": [
        {"price": "desc"},
        "_score"
    ]
}

With our query above, we return all the products sorted by most expensive to the cheapest. Notice that the sort parameter is a list, which allows us to add multiple criteria for sorting. We also added "_score", which is an elasticsearch keyword for search relevance. We will explore this concept deeper on later examples.

[10] Search for all "beer" products that are PUBLISHED, and in stock. Sorted by cheapest to most expensive

To make things more interesting, let's add several more beer products. We do this by sending a POST request thrice, with a different request body each time.




POST localhost:9200/mynewindex/_doc

{
    "product_id": "111",
    "price": 350.55,
    "stocks": 10,
    "published": true,
    "sortable_title": "Tudor Beer Lights",
    "title": "Tudor Beer Lights",
    "tags": "beer tudor party"
}

POST localhost:9200/mynewindex/_doc

{
    "product_id": "222",
    "price": 700.50,
    "stocks": 500,
    "published": false,
    "sortable_title": "Stella Beer 6pack",
    "title": "Stella Beer 6pack",
    "tags": "beer stella party"
}

POST localhost:9200/mynewindex/_doc

{
    "product_id": "333",
    "price": 340,
    "stocks": 500,
    "published": true,
    "sortable_title": "Kampai Beer 6pack",
    "title": "Kampai Beer 6pack",
    "tags": "beer kampai party"
}

With more documents in our index, we can now do the query. This is a complex query that has three conditions that must be fulfilled. We analyze the query below.



{
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "title": "Beer"
                    }
                },
                {
                    "term": {
                        "published": true
                    }
                },
                {
                    "range": {
                        "stocks": {
                            "gt": 0
                        }
                    }
                }
            ]
        }
    },
   "sort": [
        {"price": "asc"},
        "_score"
    ]
}

With our recent additions, there are four products with the word beer:

456: Best Selling Beer Flavor
111: Tudor Beer Lights
222: Stella Beer 6pack
333: Kampai Beer 6pack

Since we filter out items whose inventory is zero (or below), we remove product 456 from the list. Another filter is that the product must be published (published = true). With this filter, product 222 is removed. We are left with the 2 products below. They must be sorted by cheapest to most expensive, as is shown below:

333: Kampai Beer 6pack (price = 340)
111: Tudor Beer Lights (price = 350.55)

In this example, the key "must" was used, with a list as its value. The list contains conditions that must be met together for the query requirements to be met. In this example, its "title must have the word 'beer'" AND "published attribute is equal to true" AND "stocks is greater than zero".

[11] Search for all products that have at least 1 of the following tags ['poultry, 'kampai', 'best-seller'], that are PUBLISHED, and in stock. Sorted by cheapest to most expensive

Our previous query just involved three conditions that must be ALL TRUE to hold. That's equivalent to "A and B and C".

In this query, we still have three conditions that have to be all true, but the 1st condition is marked as true if it has either "poultry", "kampai", or "best-seller".In this example, we introduce the syntax for "OR":



{
    "query": {
        "bool": {
            "must": [
                {
                    "bool": {
                        "should": [
                            {
                                "match": {
                                    "tags": "poultry"
                                }
                            },
                            {
                                "match": {
                                    "tags": "kampai"
                                }
                            },
                            {
                                "match": {
                                    "tags": "best-seller"
                                }
                            }
                        ],
                        "minimum_should_match": 1
                    }
                },
                {
                    "term": {
                        "published": true
                    }
                },
                {
                    "range": {
                        "stocks": {
                            "gt": 0
                        }
                    }
                }
            ]
        }
    },
    "sort": [
        {
            "price": "asc"
        },
        "_score"
    ]
}

In this query, we still have a "must" keyword, but its first contains a "should" keyword. The whole query is equivalent to: (A or B or C) AND D AND E. The "should" implies that as long as one condition is met, the (A or B or C) statement returns true.

A tweak we can do is adjust the "minimum_should_match" (msm) parameter, so we can require that two or three or N conditions be met for the statement to be true. In our example, if msm=2, it means a product has to have two matching tags to be considered true (i.e a product has to be both poultry and kampai).

We analyze the query below:

The product should have at least 1 of these tags: poultry, kampai, best-seller
- This matches 3 products: poultry (pid: 123), kampai (pid: 333) and best-seller (pid: 456)
That is published
- All 3 PIDs from the previous step are already published. So no changes.
Should have stocks
- Since pid 456 does not have stocks, we are left with pid 123 and pid 333
Sorted by price
- pid 333 is 340pesos
- pid 123 is 99.75pesos
- therefore, the order should be pid 123 => pid 323

[12] Search for all products that have at least 1 of the following tags ['poultry, 'kampai', 'best-seller'], and in stock. The price should be between 0 to 300 only. Sorted by cheapest to most expensive

This query is similar to #11 but we added another criteria that the price of the products returned should only be between 0 and 300.



{
    "query": {
        "bool": {
            "must": [
                {
                    "bool": {
                        "should": [
                            {
                                "match": {
                                    "tags": "poultry"
                                }
                            },
                            {
                                "match": {
                                    "tags": "kampai"
                                }
                            },
                            {
                                "match": {
                                    "tags": "best-seller"
                                }
                            }
                        ],
                        "minimum_should_match": 1
                    }
                },
                {
                    "term": {
                        "published": true
                    }
                },
                {
                    "range": {
                        "stocks": {
                            "gt": 0
                        }
                    }
                },
                {
                    "range": {
                        "price": {
                            "gt": 0,
                            "lt": 300
                        }
                    }
                }
            ]
        }
    },
    "sort": [
        {
            "price": "asc"
        },
        "_score"
    ]
}

This query introduces the "range" keyword, which allows us to filter for items if they match a specific range of values. For the price, we set a condition for the price to be between 0 and 300. For the stock, we only set the price to be greater than zero.

Let's analyze the query:

From the results in #11, we have pid 333 (340pesos) and pid 123 (99.75pesos)
With the 0-300 price filter, our only result will be pid 123 (99.75 pesos)

Conclusion

Getting started with ElasticSearch is easy! But your searching needs can become more complex as your business needs grow. This cheatsheet helps you navigate that complexity.

An alternative to learning ES syntax at this level is to use a DSL library for Elasticsearch that "abstracts" the long-form syntax of Elasticsearch. It is a powerful tool for general-purpose usage of ES. However, as your query needs grow, learning the syntax under that DSL will keep you informed on the options you can add to make your searching richer.

How about you? Are there other ElasticSearch syntax you want to learn?

Maybe I can help! Type it in the comments, and I'll try to add it to the article.

Photo by TK on Unsplash