DEV Community: Adrien Mornet

More secure Python Docker images with Amazon Linux 🔐

Adrien Mornet — Tue, 21 May 2024 14:34:39 +0000

Amazon Linux is a Linux distribution provided by AWS specifically optimized for running workloads on AWS Cloud. This distribution, entirely managed by Amazon teams, offers very high standards in terms of security. In this article I’ll explain why I prefer a Docker image based on Amazon Linux rather than Debian to run a python workload in the cloud.

Debian based Python Docker image

Let’s say you want to create a Docker image to run Python code in version 3.9. Your first choice will probably be to start with a Docker image coming from Docker Hub python:3.9 right? I would have done the same, it’s the easiest way.

Most of the images on the Docker Hub are based on Debian and this is the case for python.

Your Dockerfile would probably look like this :



FROM python:3.9  

WORKDIR /usr/src/app  

COPY requirements.txt ./  

RUN pip install \--no-cache-dir -r requirements.txt  

COPY . .  

CMD \[ "python", "./your-daemon-or-script.py" \]

Let’s build and push this Docker image to an ECR registry with “Scan on push” enabled. The “Scan on push” feature will run a security scan on your Docker image and see if there are any CVEs in it.

Here is the result of the security scan :

That’s a lot for a fresh image containing only Debian libraries and a bit of python code, isn’t it?

Amazon Linux 2023 based Docker Image

Let’s create a python Docker image now based on amazonlinux:2023 :



FROM amazonlinux:2023  

ENV PYTHON\_VERSION\=3.9  

RUN \--mount=type=cache,target=/var/cache/dnf \\  
    dnf \-y update && \\  
    dnf install \-y python${PYTHON\_VERSION} python${PYTHON\_VERSION}\-pip shadow\-utils git\-all findutils awscli tar && \\  
    update\-alternatives \--install /usr/bin/python python /usr/bin/python${PYTHON\_VERSION} 20 && \\  
    update\-alternatives \--set python /usr/bin/python${PYTHON\_VERSION} && \\  
    dnf clean all  

WORKDIR /usr/src/app  

COPY requirements.txt ./  

RUN pip install \--no-cache-dir -r requirements.txt  

COPY . .  

CMD \[ "python", "./your-daemon-or-script.py" \]

Here is the result of it’s Security Scan :

Much better!

Why is there less CVEs in the Amazon Linux image?

The differences in how Debian and Amazon Linux are developed and maintained contribute to the feeling that Debian-based Docker images are less frequently patched and therefore have more unpatched CVEs.

Debian is a community-driven distribution. Security updates for Debian are generally reliable, but the frequency and speed at which they are released can vary because it relies heavily on volunteering contributions.

Python image on Docker Hub is also maintained by community which means that two different communities will have to patch a CVE: the Debian community and the python docker image community

Amazon Linux is a distribution maintained by AWS. Amazon has a dedicated team that prioritizes security updates and patches, often releasing them quickly to ensure that their customers’ systems remain secure.

Amazon’s centralized model and commercially driven approach to maintaining Amazon Linux ensures more consistent and rapid security updates which is why I think it’s better to use docker images based from Amazon Linux instead of Debian.

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Managing database migrations in ArgoCD 🐙

Adrien Mornet — Sun, 10 Mar 2024 15:58:01 +0000

ArgoCD is a fantastic tool for doing GitOps in Kubernetes. Most of modern backend frameworks comes with a system that allows to version control database schema definition which is named migrations (e.g. Laravel, Django …). This system is very powerful because it allows the code for a new feature to be versioned together with the database schema required for the feature to function correctly. This can be used, for example, to create a table, add a column, etc. But how to handle these migrations in ArgoCD ? That’s what we’ll be looking at in this article.

What a roll-out looks like

In the context of a Kubernetes cluster whose deployments are automated by ArgoCD, Argo will be responsible for synchronising the desired state of the Kubernetes cluster described in the Kubernetes manifest with the actual state of the cluster. The desired state is stored on Git and contains the exact version of the Docker image (via its SHA or tag) which contains your application. When you’ll want to deploy a new version of the Docker image, you will push to the Manifest the new image version. The flow of the roll-out will look like this:

ArgoCD Resource Hooks

ArgoCD comes with a hook mechanism that enables actions to be taken during synchronisation, i.e. when it detects a difference between the state described in Git stored Kubernetes manifest and the actual state of the cluster. These actions are called Resource Hooks and take the form of Kubernetes jobs.

As we want our migrations to be executed just before the new version of the code is deployed, we will use the PreSync hook to run a job just before the synchronisation operation. Our new deployment flow will look like this:

The implementation

Here is an example of a Kubernetes Job that subscribe to ArgoCD PreSync hook that will run Laravel migrations:

---
apiVersion: batch/v1
kind: Job
metadata:
  generateName: my-app-migration-
  annotations:
    argocd.argoproj.io/hook: "PreSync"
    argocd.argoproj.io/hook-delete-policy: "BeforeHookCreation"
    argocd.argoproj.io/job-cleanup: "keep"
spec:
  template:
    spec:
      containers:
      - name: my-laravel-app
        image: my-laravel-docker-image:new-tag
        command:
          - "php"
          - "artisan"
          - "migrate"
      restartPolicy: Never
  backoffLimit: 0

This Kubernetes job will be launched when ArgoCD detect a change in the Manifest. ArgoCD will wait the completion of the job (exit code 0) before syncing the rest of your application.

Conclusion

In summary, ArgoCD’s Resource Hooks, especially the PreSync hook, streamline the execution of database migrations before deploying new code versions, ensuring seamless synchronization between application code and database schema. This approach enhances version control and consistency, offering a good solution for Kubernetes roll-out.

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Effortless & Serverless File Uploading: Unleashing the Power of AWS S3 presigned urls with Lambda 📂

Adrien Mornet — Sun, 04 Feb 2024 18:57:19 +0000

Today I’m going to present how you can build a robust file upload system through AWS S3 using an HTTP(S) endpoint and S3 presigned urls. We are going to use a Lambda function with a function URL to expose the HTTP(S) endpoint.

The idea is to provide a fixed url to our user to let him upload a file to it. This fixed url (or HTTP(S) endpoint) will be a Lambda function url. Once the lambda has been launched, the function will generate an AWS S3 presigned url and redirect the user request transparently to it. We need to use an HTTP 307 code to make the redirect working with HTTP PUT method because this specific code “guarantees that the method and the body will not be changed when the redirected request is made”.

With all this in place, our user will be able to upload a file with a simple curl : curl -LX PUT -T "_/path/to/file_" "_LAMBDA_FUNCTION_URL_".

To do all this, we’ll need to:

create an S3 bucket
create a Lambda Function with an HTTP(S) endpoint
grant S3 Access to the Lambda function
make the Lambda creating the presigned url

Create an AWS S3 bucket to save files

Let’s create an S3 bucket my-files-uploaded-with-lambda-and-presigned-urls :

Create an AWS Lambda Function with an HTTP(S) endpoint

Let’s create a python AWS Lambda Function called file-uploads :

and enable function URL without authentication :

Our Lambda is now created and available on the function url :

Grant S3 Access to the Lambda function

As “a presigned URL is limited by the permissions of the user who creates it“, we have to grant our Lambda to put a file in the bucket. Let’s edit the policy of the role assumed by the lambda (role can be found in Configuration -> Permissions -> Execution role) :


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:us-east-1:ACCOUNT_ID:*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:ACCOUNT_ID:log-group:/aws/lambda/file-uploads:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": [
        "arn:aws:s3:::my-files-uploaded-with-lambda-and-presigned-urls",
        "arn:aws:s3:::my-files-uploaded-with-lambda-and-presigned-urls/*"
      ]
    }
  ]
}

Make the Lambda creating the presigned url

Here is the code I used to generate the presigned url and redirect the client with a HTTP 307 to it :

import boto3

def lambda_handler(event, context):

    # Generate a presigned URL for the S3 object
    s3_client = boto3.client('s3')

    presigned_url = s3_client.generate_presigned_url('put_object', Params={
      'Bucket': 'my-files-uploaded-with-lambda-and-presigned-urls',
      'Key': 'my_upload.txt',
      "Expires": 3600
    }, HttpMethod="put")

    # Build the redirect response
    response = {
        'statusCode': 307,  # HTTP status code for PUT redirect
        'headers': {
            'Location': presigned_url  # Set the Location header to the presigned URL
        }
    }
    # Return the response to trigger the redirect
    return response

Upload a file

Let’s test uploading a file with a simple curl request curl -LX PUT -T "_/path/to/file_" "_LAMBDA_FUNCTION_URL_"

We can see the file in the bucket :

Well done 🥳

To go further

This article gives a simple overview of what can be done with lambda and S3 presigned urls. Of course, we could also add an authentication layer to prevent anyone from uploading to the URL with Cognito and/or API Gateway. You could also use a custom url with your custom domain name with API Gateway.The naming of the uploaded file could also be more elaborate.

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Say Goodbye to Ugly Error Pages with Cloudfront 👋

Adrien Mornet — Sun, 05 Nov 2023 18:25:37 +0000

Who has never found themselves in the middle of surfing the internet when faced with an ugly error page ? These kind of pages are the default error pages of web servers like Apache or Nginx.

More recently, you may also come across the Cloudfront default errors :

Nothing could be worse in terms of user experience, could it? These generic error pages can be frustrating, uninformative, and can make users question the reliability of a website. In the world of DevOps, where continuous deployment and high availability are key objectives, the last thing you want is to present your users with a subpar experience during moments of downtime or unexpected errors.

In a modern website architecture using a Content Delivery Network, the CDN is the last component capable of catching underlying errors and saving the day :

Let's configure AWS Cloudfront to catch errors from underlying components (and even an error in Cloudfront itself) !

We're going to do this in just a few steps :

Create a pretty HTML error page
Deploy the error page with AWS S3 static website hosting
Add S3 static website as Cloudfront origin
Set up Cloudfront error pages

Create a pretty HTML error page

Let's ask ChatGPT to create a nice error page for us 😏

Deploy the error page with AWS S3 static website hosting

Amazon S3, Amazon Web Services' object storage service, offers a really simple way to deploy static HTML pages which we are going to use. To make this simpler, we're going to do it via click ops on the AWS console.

Create a public S3 bucket with public access :

Go to your bucket, click upload button and upload your index.html file :

Go to properties > Static website hosting and enable static website hosting for your bucket :

You will now have an HTTP endpoint to access your website :

Go to permissions > Bucket policy and allow everyone to get your files :

Now you should be able to access your index.html with the static url of your S3 bucket (demo-error-page.s3-website-us-east-1.amazonaws.com in my case) :

Add S3 static website as Cloudfront origin

I'm going to assume here that you already have a working Cloudfront distribution with at least one origin :

Let's add our S3 website endpoint (the url which contains s3-website, not the bucket endpoint) as a new Cloudfront origin :

Now we can add a new behavior for our Cloudfront distribution on a custom path :

In this way, requests whose path matches /cloudfront-catched-errors-demo/* will be sent to our bucket :

Set up Cloudfront error pages

We are now going to configure our distribution so that it catches errors and redirects them invisibly for the user. To do this, Cloudfront offers an "Error Pages" section where we can configure custom error response :

Let's create a custom error response :

By doing that, we are going to catch all 502 HTTP errors and show our pretty HTML error page by setting "Response page path" to /cloudfront-catched-errors-demo/index.html. Let me show you!

Before :

After :

To go further

I recommend that you catch at least HTTP 500, 501, 502, 503 and 504 errors via Cloudfront error pages because they concern server-side errors and are not managed by the application server. For 4xx errors, as they concern errors made by the customer, this is more debatable. If the error is handled correctly by the application, you might as well leave this role to it (this is very often the case for 404 errors, which are handled by web mode frameworks).

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Running a Web Application with 100% AWS Fargate Spot Containers 🤘

Adrien Mornet — Sun, 15 Oct 2023 13:54:20 +0000

One of the major advantages of using the Cloud is its Pay-Per-Use model. To make the most of this model, the challenge is to find the computing capacity at the best price that matches your workload. I’m going to explain in this article how I could run a web application in 100% Fargate Spot containers.

EC2 vs Fargate vs Fargate Spot

AWS offers several compute capacity options for running containers orchestrated by ECS :

EC2 : the good old on-demand compute service for servers on which you can install the ECS container agent
Fargate : a serverless compute engine that lets you run ECS containers without having to manage EC2 servers

Both of EC2 and Fargate offer spot deals. I have chosen AWS Fargate over EC2 for running containers to achieve cost savings and operational simplicity, as Fargate eliminates the need to manage EC2 instances and offers significant pricing advantages with Fargate Spot.

Having a fault tolerant application

The most important thing to manage when using spot capacity is that your ECS task can be interrupted by AWS at any time. In concrete terms, a SIGTERM system signal is sent to your containers in your ECS task and they have 120 seconds to stop before being hard-killed. Your application must therefore be able to shut down properly before these 120 seconds. The overwhelming majority of web applications take a few seconds to respond, don’t they? So the 120 seconds are not a block. If you configure your ECS services to run at least a count of 2 desired task, this won’t be a problem if only one task is running for a few seconds while the orchestrator starts a second task.

I explain in a little more detail in another article “Handling graceful shutdown of your Docker Cron containers”.

Manage unavailability of Fargate Spot capacity

As written in the AWS documentation, during periods of extremely high demand, Fargate Spot capacity might be unavailable. In concrete terms, if your ECS service is set up to execute tasks in 100% Spot, there is a risk of running out of capacity. A workaround has been created in the hope that one day this issue will be implemented by the AWS team. This workaround allows you to set up two ECS services :

“a primary with only spot capacity and a fallback with only on demand capacity. When the primary emits a task placement error a lambda sets the desired count on the fallback service. When the primary emits a steady state event a lambda sets the fallback desired count to zero.”

My experience with Fargate Spot

In my experience, I was able to run 4 microservices in production with 100% Fargate Spot capacity. The microservice responsible for the frontend supported around 2000 different users per day. At no point during one year we have run out of Fargate Spot capacity. When one task received an interrupt signal, another started within a few seconds, with virtually no effect on response times. The savings we have been able to make thanks to Fargate Spot are considerable and it was a really good choice.

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Authenticating your GitLab CI runner to an AWS ECR registry using Amazon ECR Docker Credential Helper 🔑

Adrien Mornet — Fri, 23 Jun 2023 12:17:07 +0000

GitLab CI allows you to run your CI/CD jobs in separate and isolated Docker containers. For maximum flexibility, you may need to run your jobs from a self-created Docker image tailored to your project’s specific needs. You can store this self-created and private Docker image in an AWS ECR registry. In this tutorial I will explain how to set up automatic authentication from your GitLab runner to your registry with Amazon ECR Docker Credential Helper.

GitLab CI job

Create a GitLab CI job which uses your Docker image saved in a private AWS ECR registry :

phpunit:
  stage: testing
  image: 
    name: 123456789123.dkr.ecr.us-east-1.amazonaws.com/php-gitlabrunner:latest
    entrypoint: [""]
  script:
    - php ./vendor/bin/phpunit --coverage-text --colors=never

Create and configure your runner to access AWS ECR registry

Create an EC2 instance and install GitLab Runner
Register your runner with Docker executor
Install the AWS ECR Docker Credential Helper in your runner
Create a /root/.docker/config.json file and add :

{
    "credsStore": "ecr-login"
}

Create an IAM User with CLI access and attach arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly policy
Paste CLI credentials to /home/gitlab-runner/.aws/credentials file on your GitLab runner :

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR SECRET KEY

Configure AWS Region in /root/.aws/config :

[default]
region = YOUR REGION

Edit your /etc/gitlab-runner/config.toml to add in the [[runners]] section the following line environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]:

[[runners]]
  name = "gitlab-runner"
  url = "https://gitlab.com/"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    image = "php:8-cli"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/builds:/builds"]
    shm_size = 0
    environment = ["DOCKER_AUTH_CONFIG={ \"credsStore\": \"ecr-login\" }"]

Now your GitLab runner can automatically authenticate to your ECR registry 🙂

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Best Practices to protect an RDS MySQL Database ✅

Adrien Mornet — Mon, 17 Apr 2023 11:32:10 +0000

Amazon RDS is a very popular choice for creating MySQL databases in the cloud. Many modern companies use it to store their business data. However, as with any other database, securing these databases requires special attention to protect against potential threats and vulnerabilities.

In this article, we will explore 10 best practices for securing your AWS RDS MySQL database instance.

1. Use strong passwords and rotate them

The use of strong passwords is an essential measure to protect your database instance. A strong password consists of a random combination of upper and lower case letters, numbers and special characters. It should be at least 20 characters long.

It is also important to change your password regularly. Passwords should be changed at regular intervals, for example every 60 to 90 days.

Use the MySQL validate_password plugin, which provides additional security like password expiration, password reuse restrictions or password failure tracking.

2. Keep MySQL Updated

One of the major advantages of the managed services offered by AWS is that you don’t have to manage them and can activate automatic updates! It would be silly to do without this… Granted, it doesn’t protect against 0-day flaws, but it’s already very good to protect against known (and patched!) flaws.

3. Change default port

This good practice protects against bots that regularly scan the internet for instances that are not sufficiently protected. It is very basic but it would be a shame to do without it.

4. Use different users with different permissions

A user should be created for each use. For example, if we have a script that makes backups of our application, we will not use the same user as the application. A read-only user is sufficient here. The root user should only be used to administer the instance, not to use it!

5. Disconnect your RDS instance from the internet

If you have the luxury of being able to disconnect your instance from the internet and make it accessible only from within its VPC network, do it!

6. Use a Firewall

AWS Security Group can be used to allow only certain IP addresses to connect to your instance’s port :

7. Use MySQL engine host check

Instead of creating your MySQL users like thisCREATE USER 'new_user'@'%' IDENTIFIED BY 'password‘; , use MySQL engine host check :

# Fixed IP address
CREATE USER 'new_user'@'123.123.123.123/255.255.255.255' IDENTIFIED BY 'password';

# Private IP range from your subnet
CREATE USER 'new_user'@'192.168.0.0/255.255.255.0' IDENTIFIED BY 'password';

8. Use SSL

Implement encryption in transit using SSL :

ALTER USER 'new_user'@'123.123.123.123/255.255.255.255' REQUIRE SSL;

9. Grant desired privileges to desired databases

Instead of granting your MySQL user privileges like this GRANT ALL PRIVILEGES ON * . * TO 'new_user'@'123.123.123.123/255.255.255.255'; grant desired privileges to desired databases :

GRANT SELECT, INSERT, UPDATE, DELETE ON my_db.* TO 'some_user'@'123.123.123.123/255.255.255.255';

10. Use 2FA or 3FA

MySQL 8.0.27 and higher supports multifactor authentication (MFA), such that accounts can have up to three authentication methods. IAM authentication can also enable multi-factor authentication.

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

RDS MySQL Load Testing with Sysbench

Adrien Mornet — Sun, 09 Apr 2023 22:35:23 +0000

Sometimes you need to do a load test on MySQL Database to test Auto-Scaling for example. I found a very useful tool called Sysbench that I will present in this article.

Install Sysbench

Open your terminal and run :

curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.deb.sh | sudo bash

sudo apt -y install sysbench

Prepare for Load Testing

Create a “test” database and run sysbench prepare command :

mysql -h YOUR_MYSQL_HOST -u YOUR_MYSQL_USER -pYOUR_MYSQL_PASSWORD -e 'CREATE DATABASE test;'

sudo sysbench /usr/share/sysbench/oltp_read_only.lua --db-driver=mysql --mysql-db=test --mysql-user=YOUR_MYSQL_USER --mysql-password=YOUR_MYSQL_PASSWORD  --mysql-host=YOUR_MYSQL_HOST --threads=80 prepare

Run MySQL Load Testing

Let’s load test a fresh created database with only 1 instance (Aurora Auto-Scaling is configured on the test-load-database cluster) :

Here is the CPU of the instance :

Create a bash script load_test_mysql.sh and paste inside :

#!/bin/bash

for ((n=0;n<100;n++))
do
 echo $(date)
 sysbench /usr/share/sysbench/oltp_read_only.lua --db-driver=mysql --mysql-db=test --mysql-user=YOUR_MYSQL_USER --mysql-password=YOUR_MYSQL_PASSWORD  --mysql-host=YOUR_MYSQL_HOST --threads=80 run
 echo $n
done

And run the test :

chmod +x load_test_mysql.sh
./load_test_mysql.sh

Here come the magic :

And you can see your CPU increase :

Finally I see that my autoscaling works well as I have two new instances :

Thanks to https://github.com/akopytov/sysbench 😁

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

How to Run a Shell on ECS Fargate Containers 💻

Adrien Mornet — Mon, 20 Mar 2023 16:05:20 +0000

If you need to troubleshoot or debug your ECS Fargate containers, you may want to open a terminal on them. There are two options available to open a shell on an ECS container: with SSH or using the ECS CLI, a command-line tool provided by AWS. The first option may create potential drawbacks and security concerns: opening SSH port an managing private and public SSH keys. The second option doesn’t require you to enable SSH access or open any additional ports because it relies on IAM authentication and AWS Session Manager.

In my opinion, using the ECS CLI to access a terminal on ECS Fargate is generally more secure than enabling SSH access because the ECS CLI doesn’t require opening any additional ports or enabling direct access to your ECS containers, which can reduce the potential risk for security vulnerabilities.

In this article I will explain how to open a shell on an ECS container via the AWS CLI.

Install AWS CLI

Install AWS CLI depending on the architecture of your computer. For Linux x86 :



curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Install Session Manager Plugin

Install the Session Manager plugin for the AWS CLI. For Linux x86 :



curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm" -o "session-manager-plugin.rpm"
sudo yum install -y session-manager-plugin.rpm

Attach the necessary IAM policy

Create an IAM policy ECSFargateAllowExecuteCommand and attach it to your ECS Task execution role :



{
    "Statement": [
        {
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ],
    "Version": "2012-10-17"
}

Open a Shell

AWS CLI command ecs execute-command requires 3 arguments :

The ECS cluster name
The ECS task id
The container name

Open your ECS task on the ECS Console and retrieve the following information :

Use the information retrieved for the ECS CLI command :



aws ecs execute-command \
  --region us-east-1 \
  --cluster ECS_CLUSTER_NAME \
  --task ECS_TASK_ID \
  --container CONTAINER_NAME \
  --command "/bin/bash" \
  --interactive

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

How to protect a website against DoS Attack using AWS WAF v2

Adrien Mornet — Sun, 12 Mar 2023 12:11:57 +0000

Denial of Service attacks are commons in these times. In my company we receive from time to time such an attack. For example on September 26th someone did almost one million requests in 1 hour on our servers and it was blocked by our AWS Web Application Firewall (WAF v2) :

I will detail in this article how did we protect us against DoS Attack using AWS WAF v2.

What is a DoS attack ?

Here is the definition according to Wikipedia :

A denial-of-service attack (DoS attack) is a cyber-attack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to a network. Denial of service is typically accomplished by flooding the targeted machine or resource with superfluous requests in an attempt to overload systems and prevent some or all legitimate requests from being fulfilled.
https://en.wikipedia.org/wiki/Denial-of-service_attack

In my example we received 835k requests in 1 hour or 14,000 per minute. We can consider that as a DoS attack

Configure AWF WAF v2 to protect a website against DoS Attack

AWF WAF v2 can be associated with various ressources like Cloudfront Distribution, Application Load Balancer, AWS AppSync, Amazon API Gateway or Amazon Cognito.

Let’s open your AWS Console, go to WAF v2 and create our first Web ACL. In our case we will create a Web ACL for a Cloudfront Distribution.

1) Create the Web ACL and select the Cloudfront Distribution :

2) Click on “Add my own rules and rule groups” :

3) Use the Rule builder to create a rate-based rule :

4) Configure the rule :

5) Choose the action to block :

6) And add the rule. You’re done

Now if someone does more than 500 requests in a five minute period, AWF WAF will block them by returning a 403 HTTP code !

Price

Only $6 / month

($5.00 per web ACL per month + $1.00 per rule per month)

Limits

AWS WAF checks the rate of requests every 30 seconds so your website can be DoS during 29 seconds.

Does AWF WAF impacts performances ?

Definitely no. We didn’t see any change in our response times.

DoS vs DDoS

There is a big difference between Denial of Service (DoS) and Distributed Denial of Service (DDoS) attacks. As distributed means different IP addresses, AWF WAF will not detect any DDoS if each IP does less than 500 requests per 5 minutes period. AWF offers another service called AWS Shield to protect you against DDoS attacks but it’s really expensive ($3000 per month).

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Monitoring Nuxt.js app with Datadog 🔍

Adrien Mornet — Tue, 07 Mar 2023 07:36:17 +0000

During the last weeks, I have been able to deepen Datadog. If you’re using Nuxt.js to build your web application, you know how important it is to keep an eye on your app’s performance and behavior. That’s where monitoring tools like Datadog come in handy. With Datadog, you can easily monitor your Nuxt.js app and get insights into how it’s performing in real-time.

In this blog post, I’ll show you how to set up Datadog to monitor your Nuxt.js app and what metrics to monitor. By the end of this article, you’ll have a better understanding of how to optimize your Nuxt.js app for performance and make sure it’s running smoothly for your users. So, let’s dive in!

Nuxt.js

Nuxt.js is a popular open-source framework for building server-side rendered (SSR) applications. It is built on top of Vue.js. In a Nuxt.js application, the frontend and backend are tightly coupled. This means that the performance of your application is dependent on both the frontend and the backend working together seamlessly.

On the frontend side, you need to monitor things like page load times (Core Web Vitals), browser errors, resource usage (APIs), and user interactions to ensure that your application is providing a fast and responsive user experience. If the frontend is slow, users may experience lag, and this can result in a poor user experience.

On the backend side, you need to monitor things like server response times, server errors, database queries, and network requests to ensure that your application is performing well and can handle high traffic loads. If the backend is slow, it can cause delays in loading content on the frontend and impact user experience.

Monitoring both the frontend and backend of your Nuxt.js application is crucial to identify potential performance bottlenecks and prevent them from affecting your users. By collecting and analyzing performance metrics from both sides, you can optimize your application for speed and reliability, ensuring a smooth user experience.

Frontend : Real User Monitoring (RUM)

Datadog Real User Monitoring (RUM) is a tool that allows you to monitor the performance of your web application from the perspective of real users. RUM tracks user interactions, page load times, and other frontend metrics to provide you with insights into how your application is performing for your users.

Setup

Install @Datadog/browser-rum : npm i @datadog/browser-rum
Create plugins/datadog.client.js file and initialize it :

import { datadogRum } from '@datadog/browser-rum';

export default ({$config: { ddApplicationId, ddApplicationToken, ddVersion, ddEnv } }) => {
    datadogRum.init({
        applicationId: ddApplicationId,
        clientjeToken: ddApplicationToken,
        site: 'datadoghq.com',
        service:'nuxt-front',
        version: ddVersion,
        env: ddEnv,
        sessionSampleRate: 100,
        sessionReplaySampleRate: 100,
        trackUserInteractions: true,
        trackResources: true,
        trackLongTasks: true,
        defaultPrivacyLevel:'mask-user-input'
    });
    datadogRum.startSessionReplayRecording();
}

Import the plugin in nuxt.config.js :

require('dotenv').config();

export default {
    buildModules: [
        '@nuxtjs/dotenv',
    ],
    [...]
    plugins: [
        '~/plugins/datadog.client.js',
    ],
}

Pass environment variables in nuxt.config.js :

require('dotenv').config();

export default {
    buildModules: [
        '@nuxtjs/dotenv',
    ],
    [...]
    plugins: [
        '~/plugins/datadog.client.js',
    ],
    [...]
    publicRuntimeConfig: {
        ddApplicationId: process.env.DD_APPLICATION_ID,
        ddApplicationToken: process.env.DD_CLIENT_TOKEN,
        ddVersion: process.env.DD_VERSION,
        ddEnv: process.env.DD_ENV,
    }
}

In Datadog, navigate to the RUM Applications page and click the New Application button. Enter a name for your application and click Generate Client Token. This generates a clientToken and an applicationId for your application. Choose the installation type for the RUM Browser SDK with npm.
Create / add to your .env file Datadog configuration :

DD_APPLICATION_ID=<DATADOG_APPLICATION_ID>
DD_CLIENT_TOKEN=<DATADOG_CLIENT_TOKEN>
DD_VERSION="1.0.0"
DD_ENV="production"

Monitor your customers’ browsers

Track errors

Monitor the ongoing bugs and issues and track them over time and versions :

User analytics

Understand who is using your application (country, device, OS), monitor individual users journeys, and analyze how users interact with your application (most common page visited, clicks, interactions, and feature usage) :

Performances

Track the performance of your front-end Nuxt.js code with key metrics like Core Web Vitals :

Ressource usage

Visualize which and how resources are being loaded :

Sessions replay

Record and replay individual user sessions to gain deeper insights into user behavior and identify any issues or errors they may have encountered during their session. With Session Replay, you can visualize user interactions and see exactly how they navigated through your application, providing valuable information for troubleshooting and optimization purposes :

Backend : Application Performance Monitoring (APM)

Datadog Application Performance Monitoring (APM) is a tool that allows you to monitor the performance of your web application from the backend perspective. With Datadog APM, you can track metrics such as response time, error rate, and throughput, as well as identify the root cause of performance issues in your application. The tool also provides detailed tracing of individual requests, allowing you to see how each component of your application is performing and how requests are flowing through it

Setup

Install and configure the Datadog Agent depending of your environment : https://docs.datadoghq.com/tracing/trace_collection/dd_libraries/nodejs/?tab=containers#configure-the-datadog-agent-for-apm
Install the Datadog Tracing library : npm install dd-trace --save
Run your Nuxt.js app loading and initializing the tracing library : node --require dd-trace/init /app/node_modules/nuxt/bin/nuxt.js start

Monitor your Nuxt.js server

Trace your application requests

Traces are detailed records of requests that flow through your application :

Track errors

Keep a close eye on any existing bugs and issues, and maintain a record of their progress across various versions and over time :

Database

Monitor databases by collecting and analyzing performance metrics such as reads, writes, and deletes, as well as memory usage, network latency, and other key performance indicators :

Conclusion

In conclusion, monitoring your Nuxt.js application with Datadog is important to ensure optimal performance and a smooth user experience. By using Datadog’s Real User Monitoring (RUM) and Application Performance Monitoring (APM), you can gain insights into how your application is performing on both the frontend and backend, and quickly identify and resolve any issues.

Together, RUM and APM provide a comprehensive view of your Nuxt.js application’s performance, making it easier to detect and resolve issues, optimize your code, and improve the overall user experience. Datadog’s centralized platform for monitoring RUM and APM data makes it easy to correlate frontend and backend performance metrics and gain deeper insights into how your application is behaving.

So, if you’re looking to build web applications with Nuxt.js, monitoring with Datadog is an excellent choice. Try it out and see how it can help you improve your application’s performance!

If you liked this post, you can find more on my blog https://adrien-mornet.tech/ 🚀

Replicate AWS RDS Aurora to AWS RDS MySQL

Adrien Mornet — Sat, 04 Mar 2023 09:31:28 +0000

AWS encourages you to migrate from RDS MySQL to RDS Aurora by offering convenient features such as the ability to create an Aurora Read Replica from an RDS MySQL cluster and then promote it to Master. An almost downtime-free migration ! But how to do the same thing in the other way ? How to replicate an RDS Aurora Cluster to RDS MySQL or external MySQL Server ? Unfortunately AWS didn’t offer the ability to create a RDS MySQL Read Replica from an Aurora RDS cluster. I will detail in this post how to achieve it.

1. Create an RDS MySQL instance to be a Read Replica of an RDS Aurora Cluster

As mentioned in AWS documentation, to configure a MySQL DB instance to be a read replica of a MySQL instance you need to enable autocommit. Let’s create an RDS Parameter group.

Open RDS Console -> parameter group and click on Create parameter group button :

Once the parameter group is created, edit it and set autocommit = 1

Now you can create an RDS MySQL instance using this parameter group !

2. Modify you RDS Aurora cluster to enable Binary Logs

MySQL uses Binary Logs to save the database state : “The binary log contains “events” that describe database changes such as table creation operations or changes to table data.”

By default Binary Logs are not enabled on Aurora and we need to activate them. Let’s create an RDS Aurora DB cluster parameter group and enable Binary logs.

Create an RDS Aurora DB cluster parameter group, set binlog_format = MIXED, affect this new parameter group to your Aurora Cluster and restart your cluster :

3. Configure RDS Aurora Cluster to be ready for replication

Create a MySQL user on your RDS Aurora Cluster allowed to replicate :

CREATE USER 'repl_user'@'mydomain.com' IDENTIFIED WITH mysql_native_password BY 'password';
GRANT REPLICATION CLIENT, REPLICATION SLAVE ON *.* TO 'repl_user'@'mydomain.com';
FLUSH PRIVILEGES;

Get binlog file and current position :

SHOW MASTER STATUS;

We will use the value of “File” and “Position” columns in the next steps.

4. Configure RDS MySQL instance to replicate RDS Aurora Cluster

Connect to RDS MySQL instance and run the following command with your custom File and Position values :

CALL mysql.rds_set_external_master (
    'aurora_host'
  , 'aurora_port'
  , 'repl_user'
  , 'password'
  , 'mysql-bin-changelog.036743'
  , 9187
  , 0
);
CALL mysql.rds_start_replication;

You’re done, your RDS MySQL is replicating your RDS Aurora Cluster

If you liked this post, you can find more on my blog https://adrien-mornet.tech/

References :

https://dev.mysql.com/doc/refman/8.0/en/binary-log.html

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Migrating.RDSMySQL.Replica.html

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql_rds_set_external_master.html