DEV Community

Cover image for End-to-End AWS DevOps Project: Automating Build and Deployment of a Node.js Application to Amazon ECS using GitLab CI/CD
ProDevOpsGuy Tech Community profile image H A R S H H A A
H A R S H H A A for ProDevOpsGuy Tech Community

Posted on

End-to-End AWS DevOps Project: Automating Build and Deployment of a Node.js Application to Amazon ECS using GitLab CI/CD

#aws #devops #node #cicd

Table of Contents

  1. Introduction
  2. Project Overview
  3. Technology Stack
  4. Architecture Diagram
  5. Step 1: Prerequisites
  6. Step 2: Configuring GitLab as Version Control
  7. Step 3: Preparing AWS Resources
  8. Step 4: Building and Pushing the Docker Image
  9. Step 5: Setting Up Amazon ECS with Fargate
  10. Step 6: Creating the GitLab CI/CD Pipeline
  11. Step 7: Adding Monitoring with AWS CloudWatch
  12. Conclusion

Introduction

In this project, we will create an automated pipeline for building and deploying a Node.js application to Amazon ECS. The project showcases the use of GitLab as version control, Docker for containerization, and AWS services like ECS, ECR, and CodePipeline for orchestration and deployment.

By the end of this guide, you will have a complete understanding of the CI/CD workflow in AWS, which is critical for modern DevOps practices.

Project Overview

Objective

We will automate the following tasks:

  1. Build a Node.js application.
  2. Containerize the application using Docker.
  3. Push the Docker image to Amazon ECR.
  4. Deploy the container to Amazon ECS using Fargate.
  5. Use GitLab CI/CD for continuous integration and deployment.
  6. Add monitoring and notifications using AWS CloudWatch and SNS.

Technology Stack

  • AWS Services:

    • Amazon ECS (Elastic Container Service)
    • Amazon ECR (Elastic Container Registry)
    • AWS CodePipeline
    • AWS Security Hub
    • Amazon EventBridge
    • Amazon SNS
    • AWS CloudWatch

  • Other Tools:

    • GitLab: Source code management and CI/CD pipeline.
    • Docker: Application containerization.
    • Node.js: Sample web application framework.

Architecture Diagram

The high-level architecture for this project is as follows:

  1. Developers push code to GitLab.
  2. GitLab CI/CD pipeline builds and pushes a Docker image to Amazon ECR.
  3. The image is deployed to Amazon ECS (Fargate).
  4. Monitoring and logging are done using AWS CloudWatch.
  5. Notifications are sent using Amazon SNS.

Step 1: Prerequisites

Ensure you have the following set up before starting:

  1. AWS Account: With admin access to ECS, ECR, and CodePipeline.
  2. GitLab Account: With a repository created for the Node.js application.
  3. AWS CLI Installed: For interacting with AWS services from the command line. 
   curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
   sudo installer -pkg AWSCLIV2.pkg -target /
   aws --version
Enter fullscreen mode Exit fullscreen mode
  1. Docker Installed: For building container images. 
   sudo apt update
   sudo apt install docker.io
   docker --version
Enter fullscreen mode Exit fullscreen mode
  1. kubectl Installed: To interact with Amazon ECS clusters. 
   curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
   chmod +x kubectl
   sudo mv kubectl /usr/local/bin/
Enter fullscreen mode Exit fullscreen mode

Step 2: Configuring GitLab as Version Control

Since you are using the existing GitLab repository, follow these steps to clone and use it in your project:

2.1: Fork the Repository

  1. Open the repository in GitLab: Nanuchi Node.js App.
  2. Click Fork to create a copy under your GitLab account.

2.2: Clone the Repository

  1. After forking, clone it to your local machine: 
   git clone https://gitlab.com/<your-username>/node-app.git
   cd node-app
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the repository contains the following:
    • Application Code (Node.js):
      • server.js
      • package.json
    • Dockerfile for containerization.
    • .gitlab-ci.yml for CI/CD pipeline (we'll modify this later).

2.3: Push Updates (Optional)

If you want to make changes to the repository (e.g., updating code, adding more files), push the updates back:

git add .
git commit -m "Updated application for CI/CD project"
git push origin main
Enter fullscreen mode Exit fullscreen mode

Step 3: Preparing AWS Resources

This step remains largely the same, but now it aligns with the Node.js application you're deploying.

3.1: Create an Amazon ECR Repository

Create a private ECR repository to store the Docker images for your application:

aws ecr create-repository --repository-name node-app
Enter fullscreen mode Exit fullscreen mode

3.2: Authenticate Docker with ECR

Authenticate your local Docker client with the ECR registry:

aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

3.3: Create an ECS Cluster

Create an ECS cluster for running the application containers:

aws ecs create-cluster --cluster-name node-app-cluster
Enter fullscreen mode Exit fullscreen mode

3.4: IAM Role, VPC, and Security Groups

Follow the steps outlined earlier to:

  • Create the IAM task execution role.
  • Set up the VPC, subnets, and security groups.
  • Open port 3000 in the security group for application traffic.

Step 4: Building and Pushing the Docker Image

Using the cloned GitLab application, build and push the Docker image.

4.1: Build the Docker Image

Navigate to the repository's root directory and build the image:

docker build -t node-app .
Enter fullscreen mode Exit fullscreen mode

4.2: Tag the Docker Image

Tag the image for your ECR repository:

docker tag node-app:latest <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest
Enter fullscreen mode Exit fullscreen mode

4.3: Push the Image to ECR

Push the image to your Amazon ECR repository:

docker push <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest
Enter fullscreen mode Exit fullscreen mode

4.4: Verify the Image

Confirm that the image has been successfully pushed:

aws ecr list-images --repository-name node-app
Enter fullscreen mode Exit fullscreen mode

Step 5: Setting Up Amazon ECS with Fargate

Amazon ECS (Elastic Container Service) is a managed service that allows you to run containers. We are using Fargate, a serverless option that eliminates the need to manage EC2 instances manually. Here's a detailed walkthrough of setting up ECS for our project:

5.1: Create a Cluster

A cluster is a logical grouping of resources needed to run your tasks or services.

  1. Run the following command to create a cluster: 
   aws ecs create-cluster --cluster-name node-app-cluster
Enter fullscreen mode Exit fullscreen mode

This command creates a new cluster named node-app-cluster.

  1. Verify the Cluster: 
   aws ecs list-clusters
Enter fullscreen mode Exit fullscreen mode

Ensure the node-app-cluster is listed as one of the clusters.

5.2: Define a Task Definition

A task definition specifies the container settings (e.g., memory, CPU, ports) for running your application. Think of it as a blueprint for your containerized application.

  1. Create a task-def.json file: 
   {
     "family": "node-app-task",
     "executionRoleArn": "arn:aws:iam::account_id:role/ecsTaskExecutionRole",
     "networkMode": "awsvpc",
     "containerDefinitions": [
       {
         "name": "node-app-container",
         "image": "<account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest",
         "memory": 512,
         "cpu": 256,
         "essential": true,
         "portMappings": [
           {
             "containerPort": 3000,
             "hostPort": 3000,
             "protocol": "tcp"
           }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/node-app",
             "awslogs-region": "<region>",
             "awslogs-stream-prefix": "ecs"
           }
         }
       }
     ],
     "requiresCompatibilities": ["FARGATE"],
     "cpu": "256",
     "memory": "512"
   }
Enter fullscreen mode Exit fullscreen mode
  • Replace <account_id> and <region> with your AWS account ID and region.
  • Ensure executionRoleArn points to a valid ECS task execution role.
  1. Register the task definition with ECS: 
   aws ecs register-task-definition --cli-input-json file://task-def.json
Enter fullscreen mode Exit fullscreen mode

This registers the blueprint with ECS.

5.3: Create a Service to Manage the Task

An ECS service ensures that the required number of tasks are running and enables load balancing for the tasks.

  1. Create a service: 
   aws ecs create-service \
     --cluster node-app-cluster \
     --service-name node-app-service \
     --task-definition node-app-task \
     --desired-count 1 \
     --launch-type FARGATE \
     --network-configuration "awsvpcConfiguration={subnets=[subnet-xxx],securityGroups=[sg-xxx],assignPublicIp=ENABLED}" \
     --region <region>
Enter fullscreen mode Exit fullscreen mode
  • Replace subnet-xxx and sg-xxx with the IDs of your VPC's public subnet and security group.
  • desired-count is the number of tasks to run.
  1. Verify the Service: 
   aws ecs describe-services --cluster node-app-cluster --services node-app-service
Enter fullscreen mode Exit fullscreen mode

Ensure the service is active and running.

5.4: Test the Application

  1. Find the public IP of your task: 
   aws ecs list-tasks --cluster node-app-cluster
Enter fullscreen mode Exit fullscreen mode

Use the task ID to describe the task and find the public IP address:

   aws ecs describe-tasks --cluster node-app-cluster --tasks <task_id>
Enter fullscreen mode Exit fullscreen mode
  1. Access your application in the browser using the public IP: 
   http://<public_ip>:3000
Enter fullscreen mode Exit fullscreen mode

Step 6: Creating the GitLab CI/CD Pipeline

GitLab CI/CD automates the build and deployment process, ensuring the application is always up to date. Follow these steps to set up the pipeline:

6.1: Add .gitlab-ci.yml

This file defines the stages, jobs, and commands for the pipeline.

  1. Add the following .gitlab-ci.yml file to the root of your project: 
   stages:
     - build
     - deploy

   build:
     image: docker:latest
     services:
       - docker:dind
     script:
       - docker build -t node-app .
       - docker tag node-app <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest
       - aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account_id>.dkr.ecr.<region>.amazonaws.com
       - docker push <account_id>.dkr.ecr.<region>.amazonaws.com/node-app:latest

   deploy:
     image: amazon/aws-cli:latest
     script:
       - aws ecs update-service --cluster node-app-cluster --service node-app-service --force-new-deployment --region <region>
Enter fullscreen mode Exit fullscreen mode
  1. Key Steps Explained:
    • Build Stage:
      • Builds a Docker image from your Dockerfile.
      • Tags the image with the ECR repository URL.
      • Pushes the image to Amazon ECR.
    • Deploy Stage:
      • Updates the ECS service to use the latest image in Amazon ECR.

6.2: Configure Variables in GitLab

Go to Settings → CI/CD → Variables in your GitLab repository and add the following environment variables:

  • AWS_ACCESS_KEY_ID: Your AWS access key.
  • AWS_SECRET_ACCESS_KEY: Your AWS secret key.
  • AWS_REGION: Your AWS region.

Step 7: Adding Monitoring with AWS CloudWatch

CloudWatch enables monitoring and logging for your application and infrastructure.

7.1: Set Up CloudWatch Logs

  1. Create a Log Group: 
   aws logs create-log-group --log-group-name /ecs/node-app
Enter fullscreen mode Exit fullscreen mode
  1. Create a Log Stream: 
   aws logs create-log-stream --log-group-name /ecs/node-app --log-stream-name app-logs
Enter fullscreen mode Exit fullscreen mode
  1. Integrate Logs with ECS Task Definition: In the task definition (task-def.json), ensure the logConfiguration section is as follows: 
   "logConfiguration": {
     "logDriver": "awslogs",
     "options": {
       "awslogs-group": "/ecs/node-app",
       "awslogs-region": "<region>",
       "awslogs-stream-prefix": "ecs"
     }
   }
Enter fullscreen mode Exit fullscreen mode

7.2: Set Up Alarms for Monitoring

You can set up alarms in CloudWatch to monitor metrics such as CPU usage, memory, and application errors.

  1. Create an Alarm: 
   aws cloudwatch put-metric-alarm \
     --alarm-name HighCPUUsage \
     --metric-name CPUUtilization \
     --namespace AWS/ECS \
     --statistic Average \
     --period 300 \
     --threshold 80 \
     --comparison-operator GreaterThanThreshold \
     --evaluation-periods 1 \
     --alarm-actions <sns_topic_arn>
Enter fullscreen mode Exit fullscreen mode
  1. Receive Notifications: Create an SNS topic to send notifications: 
   aws sns create-topic --name ecs-alerts
   aws sns subscribe --topic-arn <sns_topic_arn> --protocol email --notification-endpoint <your_email>
Enter fullscreen mode Exit fullscreen mode

Now, you will receive email notifications for high CPU usage or other alerts.

👤 Author

banner

Join Our Telegram Community || Follow me on GitHub for more DevOps content!

Top comments (15)

Collapse
 
gkadam profile image
ganesh kadam •

Section 3.4 is not clear which asks to Follow the steps outlined earlier to:

Create the IAM task execution role.
Set up the VPC, subnets, and security groups.
Open port 3000 in the security group for application traffic.

Where can i find steps for above setup?

Collapse
 
syed_irfan_76782b2c2c3ee0 profile image
Syed Irfan •

did you done hands on?

Collapse
 
sherif_san profile image
Sherif sani •

Great project for my portfolio.
Also, you really didn't need to use chatGpt for the architecture diagram.

Collapse
 
notharshhaa profile image
H A R S H H A A •

Thanks 👍@sherif_san yes 😁 I need to fix that ASAP

Collapse
 
syed_irfan_76782b2c2c3ee0 profile image
Syed Irfan •

Did you perform hands on?

Collapse
 
johnloko profile image
John Kwabla Acolatse •

Danke!

Collapse
 
notharshhaa profile image
H A R S H H A A •

Lol 😆

Collapse
 
bankolejohn profile image
bankolejohn •

Cool!

Collapse
 
notharshhaa profile image
H A R S H H A A •

Thanks @bankolejohn

Collapse
 
rajurk profile image
RAJESH K •

Well explained. Great article

Collapse
 
notharshhaa profile image
H A R S H H A A •

Thanks 👍@rajurk

Collapse
 
im_zeeshan profile image
Zeeshan Siddiqui •

Great article. Do you think you could cover creating a release, attestation, checksums and signing as part of the end to end?

Collapse
 
notharshhaa profile image
H A R S H H A A •

Thanks 👍@im_zeeshan

Collapse
 
md_kaleem_5c6461dd829ff99 profile image
MD Kaleem •

Nice

Collapse
 
notharshhaa profile image
H A R S H H A A •

Thanks 👍@md_kaleem_5c6461dd829ff99

View full discussion (15 comments)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.

Read next

godlykmathews profile image

Our team won 𝗙𝗶𝗿𝘀𝘁 𝗽𝗹𝗮𝗰𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗰𝗮𝘁𝗲𝗴𝗼𝗿𝘆 at the 𝗕𝘂𝗶𝗹𝗱 𝗳𝗼𝗿 𝗧𝗵𝗿𝗶𝘀𝘀𝘂𝗿 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗮𝘁𝗵𝗼𝗻 𝟮𝟬𝟮𝟰

Godly K Mathews -

apilover profile image

Top 7 Best CI/CD Tools Every Developer Should Know

Wanda -

aaravjoshi profile image

Supercharge Your Node.js Apps: Zero-Copy I/O Secrets for Blazing Performance

Aarav Joshi -

mike-vincent profile image

IBM Buys HashiCorp's Terraform and Vault: The End of Free?

Mike Vincent -