DEV Community: Victor

Part 1: Setting Up Initial AWS Infrastructure for the Intrusion Detection System with Terraform (Tutorial)

Victor — Sat, 28 Dec 2024 17:54:16 +0000

Deploying Containerized Applications to AWS ECS Using Terraform and CI/CD (Project Summary)

Victor — Sun, 08 Dec 2024 08:51:51 +0000

Introduction

In this post, I'll summarize the provisioning of AWS resources with Terraform, the deployment of the Dockerized flask web-app to those resources and creating a CI/CD pipeline with GitHub actions.
NOTE: This is a summary of the project, to access the complete step by step process for the deployment of the project, click the link below

Link Available soon

Project Architecture

Here, the AWS services used for the project will be defined, alongside the AWS Architecture diagram

AWS Architecture Diagram

Terraform

All infrastructure resources for this project were provisioned using Terraform with a modular approach. Each component, from the VPC to ECS, was defined within its respective Terraform module for clarity and reusability.
The Terraform backend was created first, consisting of an S3 bucket for state storage and a DynamoDB table for state locking, ensuring safe concurrent operations. Then the remaining resources were provisioned next.

Backend Configuration

Before provisioning the resources, I set up the Terraform backend. This is very important as it is where terraform will store the state files, and it is important this is separate from the main infrastructure.
The following resources were deployed for the backend:

S3 Bucket: Stores the Terraform state file.
DynamoDB Table: Manages state locks to prevent concurrent changes.

This ensures safe, versioned state management for the infrastructure.

Deployment resources

The main Infrastructure is provisioned here.

1. Virtual Private Cloud (VPC) Module

VPC: I created a VPC in the us-east-1 region

Subnets: Following the VPC, I made 4 subnets in two availability zones:

Public Subnets: I made two public subnets in the us-east 1a and us-east 1b availability zones for the internet-facing Application Load Balancer (ALB) and other resources that require internet.
Private Subnets: I made two private subnets in the us-east 1a and us-east 1b availability zones as well but this time it's for my ECS service tasks. This makes a more secure architecture.

Internet Gateway (IGW): I made the IGW to give the VPC and public subnets internet access.

Route Tables: I created the two route tables

Public Route-table: I made this route table to link the VPC to the Internet Gateway (IGW).
Private Route-table: I made this route table to link my private subnets with VPC endpoints.

Route Table Associations: I created this in order to associate my subnets to their respective route tables. i.e. Private Subnet to private route table and public subnet to public route table.

VPC Endpoints: The VPC endpoints enable the ECS tasks in the private subnet to access certain resources.

ECR and Docker endpoints: Ensure ECS tasks can pull Docker images.
CloudWatch endpoint: For secure logging.
S3 gateway endpoint: Access data and configurations securely.

2. Application Load Balancer

Listener: I created the listener to forward traffic to the ECS target group. listens on the port 80 (HTTP)
Target Group: The Target Group routes the traffic requests from the listener to exposed docker port in the ECS tasks in the private subnets.
Security group I created a security group for the ALB to specify what type of traffic to allow and on what port.

3. Elastic Container Service (ECS)

ECS Service: Here, the desired count of tasks is specified, the launch type, the subnets to be deployed in, the security groups, load balancer & target group, and the container port are all specified in this resource.
Task Definitions: Here, the task definition required by the ECS service is provisioned. The CPU, memory, execution role and container definitions are specified here.
IAM service roles and execution role: I created the necessary IAM service roles for the ECS service and the task execution role for the task definition.

4. Elastic Container Registry (ECR)

I Created this resource to hold my docker image.

6. Route 53

I used this service for DNS configuration for the application domain, routing traffic to the ALB.

Challenges faced and solutions

1.ECS tasks not being able to access the ALB
Solution:
This was due to networking misconfigurations with security groups and routing. I adjusted the security group settings to ensure proper communication between the ECS tasks and the ALB, including allowing inbound HTTP traffic on port:8080. Additionally, I verified that the ALB was correctly configured to route traffic to the ECS task group.

S3 bucket state storage and DynamoDB locking conflicts Solution: When setting up the backend for Terraform, I encountered issues with the terraform destroy -auto-approve command due to the S3 bucket and DynamoDB table. These resources were held my terraform state files and were defined in the main infrastructure, when I tried to delete also deleted my bucket and table, which contained my state files, causing issues when I want to provision the main infrastructure again. Solution To resolve this, I separated the Terraforms backend and min infrastructure.

Future Improvements

I plan to integrate unit tests in the pipeline.

Thank you for reading, check out my profile, for more Cloud and DevOps posts just like this

Relevant Links

Checkout the project on my GitHub
VSI12 / Terraform-ECS-IDS
Deploying a Containerized Web-App to AWS ECS Using Terraform and CI/CD

Project Overview

This project focuses on deploying a dockerized Flask Classification based Intrusion Detection System (IDS) to AWS ECS (Elastic Container Service) using Terraform for provisioning AWS infrastructure and GitHub Actions for CI/CD automation The IDS allows users to upload network traffic datasets (formatted like the NSL-KDD dataset), analyze them for potential threats, and visualize the results.

The deployment architecture leverages AWS services such as Virtual Private Cloud (VPC) ECS (with Fargate), ECR (Elastic Container Registry), an Application Load Balancer (ALB), and VPC endpoints for secure network communication. The entire infrastructure is managed as code with Terraform, ensuring consistency, scalability, and easy maintenance.

Architecture
- Virtual Private Cloud (VPC): Configured with public and private subnets across two availability zones for high availability and security.
- Interget Gateway: Enables communication between the VPC and the internet
- VPC Endpoints: The…
View on GitHub

Deploying a Flask-based Intrusion Detection System to AWS ECS (Project Summary)

Victor — Sat, 16 Nov 2024 19:22:37 +0000

Introduction

In this post, I'll summarize the deployment of the flask Intrusion Detection System to AWS.

NOTE: This is a summary of the project, to access the complete step by step process for the deployment of the project, click the link below

Deploying a Flask-based Intrusion Detection System to AWS ECS

Victor ・ Nov 15

#devops #python #aws #docker

Project Architecture

Here, the AWS services used for the project will be defined, alongside the AWS Architecture diagram

AWS Architecture Diagram

Deployment resources

1. VPC

I made a VPC in the us-east-1 region

Subnets: Following the VPC, I made 4 subnets in two availability zones:
Public Subnets: I made two public subnets in the us-east 1a and us-east 1b availability zones for the internet-facing Application Load Balancer (ALB) and other resources that require internet.
Private Subnets: I made two private subnets in the us-east 1a and us-east 1b availability zones as well but this time it's for my ECS service tasks. This makes a more secure architecture.
Internet Gateway (IGW): Attached the IGW and added it to the VPC's route table to grants the VPC internet access, as newly created VPCs don't have one attached

2. VPC Endpoints

The VPC endpoints enable the ECS tasks in the private subnet to access certain resources.

ECR and Docker endpoints: Ensure ECS tasks can pull Docker images.
CloudWatch endpoint: For secure logging.
S3 gateway endpoint: Access data and configurations securely.

3. Application Load Balancer

Listener: Configured to forward traffic to the ECS target group. listens on the port 80 (HTTP)
Target Group: Routes requests to ECS tasks in the private subnets
Distributes the incoming traffic across the ECS tasks.

4. Elastic Container Registry (ECR)

I used this service to host my docker image.

5. Elastic Container Service (ECS)

Task Definitions: I Defined the Docker container here and allocated the necessary resources
Service: This creates the fargate tasks and scales it behind the load balancer.

6. Route 53

I used this service for DNS configuration for the application domain, routing traffic to the ALB.

Challenges faced and solutions

challenge:

ECS tasks in private subnet could not access the docker image for ECR

Solutions:

I used VPC Endpoints to enable the ECS tasks in the private subnet access the ECR repo.

Future Improvements

I plan to automate the entire deployment by creating a CI/CD pipeline with Code Pipeline and Codebuild.

Thank you for reading, check out my profile, for more Cloud and DevOps posts just like this

Relevant Links

Checkout the project on my GitHub
VSI12 / IDS-Project

A Flask-based Intrusion Detection System web-application deployed to AWS ECS
Deploying an Intrusion Detection System to AWS

Project Overview

This Project demonstrates how to build and containerize a flask web-application with docker and deploy it to AWS. This architecture ensures a secure, highly available, fault tolerant and scalable build by leveraging various AWS architectures.

Architecture
- Virtual Private Cloud (VPC): Configured with public and private subnets across two availability zones for high availability and security.
- Interget Gateway: Enables communication between the VPC and the internet
- VPC Endpoints: The VPC endpoints enable the ECS tasks in the private subnet to access certain resources.
- Application Load Balancer(ALB): Configured to forward traffic to the ECS tasks, through listeners and target groups and distributes the incoming traffic across the ECS tasks.
- Elastic Container Registry: Host the docker image
- Elastic Container Service(ECS): creates the ECS cluster that hosts the Fargate service tasks
- Code Pipeline: Creates a CI/CD pipeline using…
View on GitHub

Deploying a Flask-based Intrusion Detection System to AWS ECS

Victor — Fri, 15 Nov 2024 14:06:52 +0000

Introduction

In this post, I’ll Walk you through the process of deploying an intrusion detection system on AWS

NOTE: This project assumes you already have an active AWS account and configured your account credentials (access keys) to your code editor and this project will incur some costs in your console

Project Summary

Deploying a Flask-based Intrusion Detection System to AWS ECS (Project Summary)

Victor ・ Nov 16

#devops #aws #docker #python

Project Overview

Objectives

The objectives of this project is as follows:

Containerize the application with Docker
Push the container image to ECR
Create a VPC, two private subnets and two public subnets
Create VPC endpoints for the private subnets to access ECR
Deploy an Application Load Balancer and target group in the public subnets for the ECS Service
Create a Task definition for the ECS service
Create an ECS cluster and an ECS service with fargate launch type in the private subnets.
Create a hosted zone in route 53 and point it to the ALB's DNS name

Project Architecture

Clone Repository

The first step is to clone the project repo.
You can find the project source code in my Github Repo

VSI12 / IDS-Project

A Flask-based Intrusion Detection System web-application deployed to AWS ECS

Deploying an Intrusion Detection System to AWS

Project Overview

This Project demonstrates how to build and containerize a flask web-application with docker and deploy it to AWS. This architecture ensures a secure, highly available, fault tolerant and scalable build by leveraging various AWS architectures.

Architecture

Virtual Private Cloud (VPC): Configured with public and private subnets across two availability zones for high availability and security.
Interget Gateway: Enables communication between the VPC and the internet
VPC Endpoints: The VPC endpoints enable the ECS tasks in the private subnet to access certain resources.
Application Load Balancer(ALB): Configured to forward traffic to the ECS tasks, through listeners and target groups and distributes the incoming traffic across the ECS tasks.
Elastic Container Registry: Host the docker image
Elastic Container Service(ECS): creates the ECS cluster that hosts the Fargate service tasks
Code Pipeline: Creates a CI/CD pipeline using…

View on GitHub

Run the following commands in your terminal

git clone https://github.com/yourusername/IDS-ECS.git
cd IDS-ECS

Containerize the application with docker

This section will show the steps involved in creating a docker image if the web app

Step 1: Dockerize The Flask Application

Create a Dockerfile in the project directory in order to package the flask app.

FROM python:3.12

#set the working dir
WORKDIR /usr/src/app

#copy the requirements and install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy all the files to the container
COPY . .

#Expose the port
EXPOSE 5000

#run the app
CMD ["gunicorn","-b","0.0.0.0:5000", "app:app"]

Step 2: Build and test Docker Image

It is important to build and test the docker image locally to ensure that it is working as intended

docker build -t image-name .
docker run -p 5000:5000 image-name

Push Docker Image to ECR

Step 1. Create an Elastic Container Repository (ECR)

Go to the AWS ECR console and create a repository and take a note of the URI (e.g., 123456789012.dkr.ecr.region.amazonaws.com/repo-name)
NOTE: Enter your ECR repo and select view push commands in order to see the commands to push the image to your ECR repo. Its those commands that will be used here.

Step 2. Authenticate Docker to ECR

Run the following command to authenticate Docker with ECR (replace your-region and your-account-id):

aws ecr get-login-password --region your-region | docker login --username AWS --password-stdin your-account-id.dkr.ecr.your-region.amazonaws.com

Step 3. Tag and Push the Image

Tag your local Docker image to match the ECR repository, then push it:

NOTE: Ensure it is your accounts respective region and account-id

docker tag image-name:latest your-account-id.dkr.ecr.your-region.amazonaws.com/image-name:latest
docker push your-account-id.dkr.ecr.your-region.amazonaws.com/image-name:latest

Your ECR repo should look just like this is the image is pushed successfully

Create a VPC and its Subnets

Take a note of the Region, as this is where all the resources will be deployed

Step 1: Create a new VPC

Go to the VPC console and create a new VPC
Specify CIDR block (e.g. 10.0.0.0/16)

NOTE: Ensure the Enable DNS hostnames setting is checked when creating the VPC

Step 2: Create an Internet Gateway (IGW) for the VPC

In the VPC console, select the Intergate Gateway tab and create the internet gateway. Once the IGW is created attach it to your VPC

Step 3: Create Subnets

Create two subnets in one availability zone (e.g. us-east-1)
Create another set of two subnets in different availability zone (AZ) (e.g. us-east-2)
NOTE: In each AZ, the subnets will serve as private and public subnets respectively.

NOTE: the ECS cluster will be deployed on the private subnet and the Application Load Balancer will be in the public subnet and will access the ECS cluster in the private subnet

Step 4: Update the Route tables

Create route tables for the public and private subnets and associate the route tables to the public and private subnets respectively. (Ensure the VPC in use is selected for the both of them.)
For the public subnets route table, add to route to direct all outbound traffic 0.0.0.0/0 through the internet Gateway.
The private subnet will not route outbound traffic for now.

Create VPC endpoints for ECR

This will enable the ECS cluster to have access to the Elastic Container Registry (ECR).
NOTE: Four Endpoints will be made for S3, ECR, DOCKER and CloudWatch.

select Endpoints and click on create endpoint.

Name the endpoint and search for the ECR api endpoint under services com.amazonaws.us-east-1.ecr.api
Select the VPC we've been using, this will bring up the subnets option where the availability zones our private subnets are in will be selected and then finally select our private subnets and the default security group.

Select the default and leave the policy as is and then create the VPC.

*Now create the remaining Endpoints with changing the services for docker com.amazonaws.us-east-1.ecr.dkr, for CloudWatch logs com.amazonaws.us-east-1.logs and for S3 com.amazonaws.us-east-1.s3 respectively and follow everything else exactly expect for those changes.

NOTE: For the S3 endpoint, select the gateway. This is called the S3 Gateway Endpoint and will prompt you to connect it to your private subnet route table, which will create a route in the route table for it.

Create Application Load Balancer and Target Group

This is a very important step as its the ALB that
will route traffic to the private ECS service.

Firstly, we need to create a security group for the ALB.

Go to the security group on the left and select create security group. give the security a name and description.
Add an inbound rule on port range 80 and source 0.0.0.0/0 and add a second one on port range 443 and source 0.0.0.0/24 for HTTP and HTTPS traffic respectively

Step 1: Create a Target Group

Go to the EC2 console and on the left, under Load Balancing select target group.
Under basic configuration select IP addresses and name the Target group
Leave the Protocol and port as HTTP:80
Select the VPC we are deploying the load balancer in and scroll and click next.
Here remove the IP address that is there, all IP's will be automatically added. Next, specify the port which was exposed, Port:5000 and Create the Target group.

Create Application Load Balancer

On the left of the EC2 console, select the load balancer and select create load balancer.
Select Create Application Load Balancer, this will be an internet-facing load balancer. give the ALB a name.
In the network mapping section select the Created VPC and the public subnets in the two availability zones.
In the Listeners and Routing section, select the created target group. The ALB will listen on port:80 and forward to the target group.
Create the ALB.

Create Task definitions

Go to the ECS console and select task definitions.

Step 1:

Name the task definition family
Leave the default infrastructure requirements as is, but you can change the vCPU and memory based on your descretion.

Step 2: For Container 1

Give a name
Go to your ECR and copy the URI of the docker image we pushed in earlier sections of this project and come back to the container 1 section of the task definitions we're creating and paste it in the Image URI for the container.
For the Port mappings set it to 5000, as this was the port we exposed in our docker container and give it a name. NOTE: It is very important the port mappings is the same as the exposed docker port
Add environment variables
This will be important in the CI/CD section of this project. But will be skipped for now.
Go ahead and skip the remaining sections and create the task definition

Create a Fargate cluster and service

Create security group for the ECS service

Go to the security group on the left and select create security group. give the security a name and description.
Add an inbound rule on port range 5000 and source will be the Application Load Balancers security group.

Step 1: Create Cluster

Go to the clusters tab and select create cluster, this will bring you to the cluster configuration page. Name the cluster and ensure only the AWS Fargate(Serverless) is selected under the infrastructure tab and then create the cluster.

Step 2: Create a service

Select the created cluster and click on the create under service.

This will lead you to a new page where you'll specify the configuration of the Fargate service.

Scroll past the Environment section and move to the Deployment Configuration. Here, specify the task definition family, which will automatically select the revision as well.

Select the Desired number of Tasks to launch. One is fine for this project.
Scroll to the Networking tab and select the VPC that was made and then the private subnet(s)
In the Load balancer section, select the load balancer type as Application Load Balancer, and select use an existing load balancer. This will bring up the ALB that was created in previous sections. select it.
Scroll and you'll see, listener. select use an existing listener and select the port 80 listener that is there and under target group do the same, selecting the existing target group that we made.
Next is Service Auto Scaling. This is optional but is a good addition to have to scale out to app based on defined metrics. Enable this and specify the minimum and maximum number of tasks you want running. Next add a scaling policy. For this project a Target tracking policy is used alongside the ALBrequestCountPerTarget ECS service metric, with the target value 50, scale-out cooldown period and Scale-in cooldown period as 60s
Create the Service.
Once the Service is created, the desired number of tasks will be created.

Create a Hosted Zone in route 53

If the above instructions were followed t the T, you should have a fully functioning web app, to access it go to your load balancer, copy the DNS name and paste in your browser. But that's tedious and not using best practices. Ideally, there should be a Web application firewall in front of the ALB or CloudFront, but for simplicity we will be using only Route 53.

NOTE: This section requires you have a registered domain name either with AWS or any other provider

Step 1: Create a hosted in

Go to the Route 53 console.
On the left tab, select Hosted Zones. you should have this.

Select Create Hosted Zone
Enter your domain name and give a description. Leave the type as Public Hosted Zone and select create Hosted Zone.

Step 2: Add ALB DNS name to hosted zone records

Go to your ALB console and copy its DNS name.
Come back to the hosted zone and select create record.

The default record type that is there is the A record, which will be used.
Toggle the alias switch

Under choose endpoint, select the Alias to Application and Classical Load Balancer
Next, choose the region you launched your Load Balancer in. In this case its us-east-1, Ensure it's your on region.
Choose your load balancer from the drop-down menu.
Create record.

This assumes you have your domain name with AWS

CONGRATULATIONS!!

If you followed the Steps to the T, you should have a full functioning web-app that is accessible through your domain-name.

This was a very exciting project as I worked with VPC's, private and public subnets, VPC endpoints, ECS services, ECR, Target groups, security groups and Application load Balancer as they all came together to create this web-app.

Relevant links

IDS-Project(GitHub Repo)
VSI12 / IDS-Project

A Flask-based Intrusion Detection System web-application deployed to AWS ECS
Deploying an Intrusion Detection System to AWS

Project Overview

This Project demonstrates how to build and containerize a flask web-application with docker and deploy it to AWS. This architecture ensures a secure, highly available, fault tolerant and scalable build by leveraging various AWS architectures.

Architecture
- Virtual Private Cloud (VPC): Configured with public and private subnets across two availability zones for high availability and security.
- Interget Gateway: Enables communication between the VPC and the internet
- VPC Endpoints: The VPC endpoints enable the ECS tasks in the private subnet to access certain resources.
- Application Load Balancer(ALB): Configured to forward traffic to the ECS tasks, through listeners and target groups and distributes the incoming traffic across the ECS tasks.
- Elastic Container Registry: Host the docker image
- Elastic Container Service(ECS): creates the ECS cluster that hosts the Fargate service tasks
- Code Pipeline: Creates a CI/CD pipeline using…
View on GitHub

Connect with me

https://linkedin.com/in/victor-iliya

Resources Used

First AWS Project

Victor — Sat, 31 Aug 2024 17:38:17 +0000

I have just Created and deployed my first static website on AWS, the first of many projects. I learned how to apply the following services:

S3 buckets *AWS certificate manager *AWS Route 53 *AWS CloudFront Distribution

DEV Community: Victor

Part 1: Setting Up Initial AWS Infrastructure for the Intrusion Detection System with Terraform (Tutorial)

Deploying Containerized Applications to AWS ECS Using Terraform and CI/CD (Project Summary)

Introduction

Project Architecture

AWS Architecture Diagram

Terraform

Backend Configuration

Deployment resources

1. Virtual Private Cloud (VPC) Module

2. Application Load Balancer

3. Elastic Container Service (ECS)

4. Elastic Container Registry (ECR)

6. Route 53

Challenges faced and solutions

Future Improvements

Relevant Links

VSI12 / Terraform-ECS-IDS

Deploying a Containerized Web-App to AWS ECS Using Terraform and CI/CD

Project Overview

Architecture

Deploying a Flask-based Intrusion Detection System to AWS ECS (Project Summary)

Introduction

Deploying a Flask-based Intrusion Detection System to AWS ECS

Victor ・ Nov 15

Project Architecture

AWS Architecture Diagram

Deployment resources

1. VPC

2. VPC Endpoints

3. Application Load Balancer

4. Elastic Container Registry (ECR)

5. Elastic Container Service (ECS)

6. Route 53

Challenges faced and solutions

challenge:

Solutions:

Future Improvements

Relevant Links

VSI12 / IDS-Project

A Flask-based Intrusion Detection System web-application deployed to AWS ECS

Deploying an Intrusion Detection System to AWS

Project Overview

Architecture

Deploying a Flask-based Intrusion Detection System to AWS ECS

Introduction

Deploying a Flask-based Intrusion Detection System to AWS ECS (Project Summary)

Victor ・ Nov 16

Project Overview

Objectives

Project Architecture

Clone Repository

VSI12 / IDS-Project

A Flask-based Intrusion Detection System web-application deployed to AWS ECS

Deploying an Intrusion Detection System to AWS

Project Overview

Architecture

Containerize the application with docker

Step 1: Dockerize The Flask Application

Step 2: Build and test Docker Image

Push Docker Image to ECR

Step 1. Create an Elastic Container Repository (ECR)

Step 2. Authenticate Docker to ECR

Step 3. Tag and Push the Image

Create a VPC and its Subnets

Step 1: Create a new VPC

Step 2: Create an Internet Gateway (IGW) for the VPC

Step 3: Create Subnets

Step 4: Update the Route tables

Create VPC endpoints for ECR

Create Application Load Balancer and Target Group

Step 1: Create a Target Group

Create Application Load Balancer

Create Task definitions

Step 1:

Step 2: For Container 1

Create a Fargate cluster and service

Step 1: Create Cluster

Step 2: Create a service

Create a Hosted Zone in route 53