DEV Community: Akbar Nafisa

Setup NGINX

Akbar Nafisa — Mon, 15 Apr 2024 06:10:23 +0000

In the previous setup, we used an API gateway to connect our client to the server. In this part, we will add NGINX to our server as a reverse proxy for our Backend app. To do so, let’s follow these steps:

Open HTTPS Port

Firstly, let’s add port 443 to our security rule to enable our server to serve HTTPS.

Add Routes Traffic

Next, let’s add a new API api.akbarnafisa.com as a new API. To do so, let’s open AWS Route 53 and create a new traffic routing record.

Add New Docker Networks

Then, let’s add new Docker networks to connect our Backend app to the NGINX app. In this case, we add two new networks:

nginx.docker
my-project-server-prod-network

Add NGINX Config

In this config, we will add NGINX to act as a reverse proxy to our Backend App. You can clone the repository here.

Separate NGINX conf

In this config, we will separate the configuration to create Certbot and redirect to the Backend server. We do this because we need to set up the Certbot first before adding the reverse proxy config.

Connect to Server

Next, we connect to the EC2 server using VSCode. To do so, make sure we have the SSH extensions.

Then, click the icon in the bottom left corner.

Next, click connect to host.

Then, add the config by adding the Hostname and .pem file.

Change the Configuration

We need to change some configurations:

Change {email} inside docker-compose.yml to your email.
Change {api} inside docker-compose.yml and client.conf to your domain.

Create Certificate

To add the certificate, we can run the docker-compose up command.

If the process is successful, we will get this notification from the terminal.

To check the files inside, we can run this command:

docker exec -it ssl-api-akbarnafisa-com-nginx-1 /bin/sh
cd /etc/letsencrypt/live/

Enable Reverse Proxy

After we create the certificate, the next step is to enable the reverse proxy. To do so, let’s uncomment this code from the client.conf.

Then, let’s run the docker container

 docker compose up -d

Finaly, let’s open the the URL to check if the NGINX is working or not

Add New Reverse Proxy Path

There is a case where we want to add a new path to the reverse proxy. To do so, let’s add new config inside client.conf.

Then, let’s run this command to build the nginx image:

docker compose up --force-recreate --build -d nginx

Make sure the image name is the same as the image in docker-compose.yml file.

Then, when we open the URL, we can see the new path is working.

Add Portainer Stack

Akbar Nafisa — Fri, 26 Jan 2024 00:00:00 +0000

In this section, we will migrate our current deployment to Portainer so that we can fully manage it inside Portainer later on. You can view the full code for the configuration here. To migrate our app, let’s follow these steps:

Update Docker Compose

In Portainer, we can set up the environment in their dashboard. However, we need to make some adjustments, such as changing the file name of .env.prod to stack.env.

Next, we need to change how we mount the create_tables.sql. In our current implementation, we cannot directly mount the file; Docker will treat the create_tables.sql as a folder instead of a file. Therefore, we need to use a new Docker volume to mount the file.

Delete Existing Container

Moving on to the next part, we will delete our existing Docker container and Docker volumes.

Create New Stack

The next step is to create a new stack for our project. To do so, open the Stacks section in the dashboard and click the Add stack button. Then, add the GitHub repository URL, branch name, and the Docker Compose file.

The following step is to add the environment variables and click Deploy the stack to finish the configuration.

After completing the deployment, we can now have total control over our stack.

We can also check our API endpoint to ensure that everything is working as before.

Setup Portainer for Server App

Akbar Nafisa — Wed, 24 Jan 2024 00:00:00 +0000

In this section, we will add Portainer to help us in managing our Docker containers. You can find more details about it here. To integrate Portainer into our EC2 project, we can follow these steps:

Install Portainer Container

First, let’s connect to our EC2 instance.

Next, create a new Docker volume for Portainer:



docker volume create portainer_data

Now, let’s run the Portainer container:



docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest

Then, verify if the container is running.

Open Port for Portainer

Portainer is running on port 9443, so we need to add new port inside our security groups.

Let’s select Custom TCP and set the port range to 9443, and for the source set to Anywhere-IPv4, then click Save rules button to finish the configuration.

Login to Portainer

To log in to Portainer, open the public IP of our EC2 then add 9443 port. If we are connecting for the first time, your browser might block access. Click the Show advanced button or a similar option depending on your browser, then click the link bellow.

Then, in Portainer page, we need to add username and password.

After you finish the previous process, we can look around inside the dasboard. In Home section we can click local environments to see the currently running containers.

When you open the Stacks page, we can see that we have limited control to our project. For the next part, we will migrate our project inside Portainer so we can manage it easily.

Setup Auto Revalidation for CloudFront with Every Client App Build

Akbar Nafisa — Sat, 13 Jan 2024 00:00:00 +0000

In CloudFront, we can invalidate the cache on demand by using GitHub Actions. This is necessary to ensure that the latest code updates are reflected on our website. You can view the full code here.

Update Identity and Access Management (IAM)

In our IAM, we need to add a new statement for the CloudFront Distribution.

{
    "Version": "2012-10-17",
    "Statement": [
        ...
        {
            "Action": [
                "cloudfront:CreateInvalidation"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:cloudfront::000581027806:distribution/E2KRF40UYXVA3O"
            ]
        }
        ...
    ]
}

You can find the CloudFront Distribution here.

Update Github Action

Let’s add the CloudFront Distribution to the GitHub Secrets with the AWS_CF_DISTRIBUTION variable name. Next, we will add new steps to perform CloudFront Invalidation in the GitHub Action.

Also, let’s make some changes in our code to verify if the text will be updated.

Next, let’s create a new branch and push it to the main branch, then wait for the workflow to finish.

When you observe the CloudFront, you will notice that a new invalidation has been created. Check our website to confirm the changes.

Setup Route 53 for Client App

Akbar Nafisa — Thu, 11 Jan 2024 00:00:00 +0000

For this part, we will modify the URL from Cloudfront to our own domain. However, first, you need to have a domain. You can buy it from AWS Route 53 or other providers like Hostinger, GoDaddy, etc. In my case, I bought the domain outside AWS.

Create Hosted Zone

To add the domain to AWS, we need to create a new hosted zone in Route 53.

Then, you can add the domain name.

Finish by clicking the Create hosted zone button at the bottom of the page.

Modify Nameserver

If you bought the domain from a different provider, you need to change the nameservers to AWS. Retrieve the nameservers from the detailed hosted zone.

Then, update the nameservers with the domain provider in the settings page.

Create SSL/TLS for New Domain

For the next step, request a new certificate for your domain using ACM. Open ACM and request a new certificate.

Add the domain name.

Finish the request by clicking the Request button.

Verify the Certification

The next step we will verify our certificate, to do so let’s click on the certificate that we just created.

Click the Create records in Route 53 button.

Next, click on the domain and click Create records to finish.

Then, let’s wait a couple minute until the status changed to Issued

You can also check the records page to confirm the new record has been added.

Edit the Domain Name in Cloudfront

In this part, change the domain of Cloudfront. Open the Cloudfront distribution, then click the Edit button inside the Settings page.

Add the new domain and select the SSL certificate. Finish by clicking Save changes at the bottom of the page.

Wait for Cloudfront to finish its deployment.

Create New Record for the New Domain

In the final step, we will add https://awsproject.akbarnafisa.com/ to Route 53. To do this, open the Route 53 page and click the Create record button.

Then, select Simple routing.

Next, click Define simple record.

Now, fill in the record name, select the routing traffic to the Cloudfront distribution, and choose the distribution.

Finally, click the Create records button to complete the process.

Now, let's wait a couple of minutes, and we will be able to access the page.

Setup CloudFront for Client App

Akbar Nafisa — Thu, 04 Jan 2024 00:00:00 +0000

In the previous section, we uploaded our client's static files to S3. However, we still cannot access them because we set our bucket to be private. In this section, we will create CloudFront as a CDN and connect it to our static site. To do so, let’s follow these steps:

Create CloudFront

Open the CloudFront dashboard and click on the Create distribution button.

Select the static site bucket in the Origin domain.

In the Origin access section, select Legacy access identities and then click on Create new OAI.

Enter the OAI name and click Create button

Next, select the Yes, update the bucket policy option to automatically add a new bucket policy inside our S3 bucket.

In the Viewer section, select the Redirect HTTP to HTTPS option.

Finally, select Do not enable security protections in the WAF section and click the create button at the bottom of the page.

The deployment will take several minutes; you can visit the client app here.

Deploy Client App to S3 by Using Github Action

Akbar Nafisa — Sat, 30 Dec 2023 00:00:00 +0000

In this section, we will create a GitHub Action to automatically deploy our Client App to AWS S3. You can view the code here.

Create Repository Secrets

We will add the following secrets to the repository:

AWS_ACCESS_KEY_ID → Access key obtained from the .csv file in the previous section
AWS_SECRET_ACCESS_KEY → Secret key obtained from the .csv file in the previous section
AWS_S3_BUCKET → The name of the S3 bucket

Create Repository Variables

Next, we will add the API URL for our app, which we obtain from API Gateway.

Create Github Action File

The final step is to create the deploy.yml file. The build file consists of installing the dependencies and building the client app. Then, we use jakejarvis/s3-sync-action to upload the build file to the S3.

name: Upload Website

on:
  pull_request:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      VITE_API_URL: ${{ vars.VITE_API_URL }}
    steps:
      - name: Checkout repo
        uses: actions/checkout@v3
      - name: Setup Node
        uses: actions/setup-node@v3
      - name: Install dependencies
        run: yarn
      - name: Build app
        run: yarn workspace client build
      - name: Upload to S3s
        uses: jakejarvis/s3-sync-action@master
        with:
          args: --acl private --follow-symlinks --delete
        env:
          SOURCE_DIR: apps/client/dist
          AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

To trigger the build, you need to create a pull request (PR) to the main branch and merge it. Afterward, the GitHub Action will run.

Then, when we check the S3 bucket, we can see that the file has already been uploaded.

Create Access Key for AWS S3

Akbar Nafisa — Fri, 29 Dec 2023 00:00:00 +0000

Before setting up the GitHub action, it's important to generate an Access Key for uploading files to the designated bucket. To do so, let’s follow this steps

Create S3 Access Policy

Navigate to the Policies page and select Create policy to add a new policy for our bucket.

Next, choose the service as S3.

Now, input the following policy into the text editor:



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-awesome-project-bucket",
                "arn:aws:s3:::my-awesome-project-bucket/*"
            ]
        }
    ]
}

this is the explaination for the config:

Action: Lists the AWS S3 actions allowed by this policy. The allowed actions are:
- s3:GetObject: Allowing retrieval of objects from the bucket.
- s3:PutObject: Allowing the addition of objects to the bucket.
- s3:ListBucket: Allowing the listing of objects within the bucket.
- s3:DeleteObject: Allowing the deletion of objects within the bucket.
Resource: Specifies the AWS resources to which the actions apply. In this case, the resources are:
- "arn:aws:s3:::my-awesome-project-bucket": Refers to the bucket.
- "arn:aws:s3:::my-awesome-project-bucket/*": Refers to all objects within the bucket.

To get the arn you can copy it from the properties section in the bucket page.

Once the configuration is complete, click the Next button, enter a policy name, and click Create policy at the bottom of the page.

Create New User Credential

Navigate to the Users page in the IAM dashboard and click the Create user button.

Enter the User name, then click the Next button.

Select Attach policies directly in Permissions options and choose the policy created earlier. Click the Next button to proceed to the last step.

Review the configuration in the last step and click Create user to finish.

Create Access Keys

Navigate to the Users page, open Security credentials, and click Create access key under the Access keys section.

For the first step, choose the other option.

In the second step, enter a description for the access key and click the Create access key button.

Finally, click Download .csv file and then Done.

With this setup completed, we are ready to create an automated update to the S3 bucket using GitHub CI.

Setup S3 Bucket

Akbar Nafisa — Sun, 24 Dec 2023 00:00:00 +0000

In this section, we will create an AWS S3 bucket to host our static client app. To do so, let's first open the AWS S3 dashboard and navigate to the create bucket page. Next, enter the desired bucket name and click the Create bucket button at the bottom of the page.

After creating the bucket, open it and click on the Properties tab.

Scroll to the bottom of the page and click the Edit button in the Static website hosting section.

Enable Static website hosting and set the Index document to index.html, which serves as the entry point for the HTML in our client app. Finally, click the Save changes button at the bottom of the page.

When attempting to open the URL, an error page is displayed. This occurs because our bucket is not set to public. In the next few step, we will configure CloudFront as a Content Delivery Network (CDN), ensuring that the content in our bucket is only accessible when accessed through CloudFront.

Using Docker Cache Layer to Speed Up Build Time

Akbar Nafisa — Fri, 22 Dec 2023 00:00:00 +0000

I came across a video about tips and tricks for Docker. In this video, the she gives us tips on how to utilize Docker cache layers. The main point is that Docker images have multiple layers, and each layer is cached by Docker. When there is a change in one layer, the layer below it will be invalidated.

Now, let’s examine our production Docker file. We can see that the layer is not optimized because we copy our code and then install the package. This can slow down the build process because if we change one file on our server app.

FROM node:18-alpine as builder

WORKDIR /main

COPY packages/libs packages/libs
COPY packages/validation packages/validation
COPY apps/server apps/server

COPY package.json .
COPY yarn.lock .

RUN yarn install --pure-lockfile --cache-folder ./yarncache; rm -rf ./yarncache;
RUN yarn workspace server build
RUN yarn install --production --pure-lockfile --cache-folder ./yarncacheprod; rm -rf ./yarncacheprod;

FROM node:18-alpine as runner

WORKDIR /main
COPY --from=builder /main/packages ./packages
COPY --from=builder /main/apps ./apps
COPY --from=builder /main/node_modules ./node_modules
COPY --from=builder /main/package.json ./

Now, let’s try to test it; first, let’s add some changes to our file.

Then let’s build the Docker file.

We can see the build time is more or less 32 seconds. Now let’s add more changes and see whether the build is actually cached or not.

Then let’s build the Docker file one more time.

Now let’s build the Docker file for the last time without change anything and see if the duration will be similar or not.

As we can see, when Docker utilizes its cache, the build time goes down to 2 seconds. So, we have already proven that changing the file content will invalidate the cache.

Optimizing the Layer Cache

To optimize our Dockerfile, we should consider which layer frequently changes and which layer takes longer to build. Then, we rearrange those layers to make sure that the frequently changed layer is placed at the bottom of the Dockerfile.

Now let’s see the Dockerfile after we optimized it.

The main change that we made is to copy the package.json file, then we do the installation process after that. Although we do installation again after the build process, this installation will be faster because it only removes the files that are not for production mode.

Performance Test after Optimization

First, let’s create some changes in our file.

Then, let’s try to build the Dockerfile.

Then, let’s create another change.

Lastly, let’s build the Dockerfile again.

As we can see, after we changed the Dockerfile, the build time went down from 30 seconds to 11 seconds every time we built the image. One caveat for this method: if we have changes in package.json, then Docker needs to invalidate all the layers after that. You can see the code for this part here

Setup API Gateway

Akbar Nafisa — Thu, 21 Dec 2023 00:00:00 +0000

In the previous post, we successfully deployed our server app to EC2. However, it still uses http. To overcome this issue, we can use API Gateway to act as a front door to our server app. Here are the steps to connect the EC2 to API Gateway:

Create API

Open API Gateway Service and click the Create API button.
Select REST API and then click the Build button.

Enter the API name and click Create API.

Create Resource

Open the API, then click the Create resource button.

Check the Proxy resource and enter {proxy+} as the resource name, then click Create resource.

Click on ANY resources, then click Edit integration button.

Select HTTP method, check the HTTP proxy integration, set HTTP method to ANY, set the Endpoint URL to our server app URL http://3.27.113.187:3000/{proxy}, and finally, set Content handling to Passthrough.

The last step is to add a new URL path parameter with the key proxy and the value method.request.path.proxy, then click the Save button.

Deploy the API

In the resources page, click Deploy API.

Let’s create a new stage and click Deploy button.

Finally, after the deployment finishes, we can access our API here.

Handling error path

If you access the index page, you will receive an error page.

We can fix this issue by following these several steps.

In the Resources page, click Create Method button.

Then we set the Method to be GET, check the HTTP proxy integration, and enter our server app public URL. Finally, click Create method.

Finally, let’s redeploy our API to prod stage, and when we open our index API, it should not show any error.

Deploy Server App to EC2

Akbar Nafisa — Wed, 20 Dec 2023 00:00:00 +0000

This is the part where we deploy our Server App to EC2. Here is how we do it:

Add New Rules in Security Group

We need to add new inbound rules to expose our app in Docker. For the server app, we set the port to be 3000, so we need to add it here. You can add the rules inside security groups. Inside security groups page, click on Edit inbound rules.

Next, add Custom TCP with the 3000 as the port range and set the source to 0.0.0.0/0, then click Save rules button to finish.

Connect to EC2 Instance

This can be done by connecting to the instance in the AWS EC2 console or by using SSH.

Connect to EC2 Instance in AWS Console

In your EC2 dashboard, select the instance that you want to connect to, then click the Connect button.

After that, click the Connect button again.

The console instance terminal will then be shown.

Connect to EC2 Instance by SSH

To connect to the instance by SSH, we need the .pem file we downloaded when we created the instance. The first step is to set the .pem file permissions to read-only for the owner and no permissions for others. To do this, we can run this command:

chmod 400 file_name.pem

If you are on Windows, you can run this command instead.

icacls my-key-pair.pem /inheritance:r /grant:r "$($env:USERNAME):(R)"

Then we need to get the public IP of the instance.

Finally, run this command to connect to the instance:

ssh -i my-key-pair.pem ubuntu@3.27.113.187

Install Docker

The Docker installation guide on Ubuntu can be found here. After installation finishes, we need to run this command to make the docker compose command work.

sudo chmod 666 /var/run/docker.sock

Create Github SSH

For this part, we will use SSH to clone our repository from GitHub. First, enter the .ssh folder:

cd ~/.ssh/

Then generate the .pub file. You can name the file whatever you want; for this file, I named it my_ssh

ssh-keygen -t rsa

Copy the text inside .pub

cat ./my_ssh.pub

and add it to your GitHub SSH keys.

To test the key, run this command:

eval $(ssh-agent)
ssh-add ./my_ssh

And test the connection:

ssh -T git@github.com

If we've done this part correctly, we should see that the SSH is being used.

Clone Repository

To copy the repository, run this command:

git clone git@github.com:akbarnafisa/aws-devops-fullstack.git

Then switch to the branch that you want to deploy:

git checkout remotes/origin/6-setup-docker-production-mode

Finally, run our Docker Compose file using this command:

docker compose -f docker-compose.prod.yml

If everything is successful, we will be able to access our app at this URL.

http://<public_ip>:3000/api/todo