DEV Community: benny fmo

Step-by-Step Guide to EC2 Auto Scaling and Load Balancing on AWS

benny fmo — Fri, 11 Oct 2024 08:52:56 +0000

Introduction

As businesses and applications grow, the demand for computing power fluctuates. Traditionally, managing this involved investing in expensive hardware and requiring manual interventions to scale. However, with cloud computing, you now have the ability to dynamically scale resources based on real-time needs. Auto Scaling ensures that your applications remain available and responsive, optimizing both cost and performance by automatically adjusting compute resources to handle varying traffic loads.

A key feature of AWS’s elasticity is the ability to use Auto Scaling Groups (ASGs) to automatically add or remove compute resources based on demand. This allows your applications to stay highly available even during traffic spikes while also scaling down during periods of low activity to save costs.

In this tutorial, you'll set up the foundation for a scalable architecture by creating a Virtual Private Cloud (VPC) with three public subnets spread across different availability zones (AZs). This ensures high availability by distributing resources across isolated locations, reducing the risk of downtime due to failures in a single zone. After configuring the VPC, you'll introduce an Auto Scaling Group and an Application Load Balancer (ALB) to distribute traffic across multiple EC2 instances, ensuring that your infrastructure dynamically adapts to changes in traffic demand.

By the end of this guide, you will learn how to:

Create a VPC with public subnets in different availability zones for high availability.
Launch EC2 instances within those subnets running a web application (Apache web server).
Set up an Application Load Balancer (ALB) to distribute traffic across instances.
Configure Auto Scaling Groups (ASG) to automatically scale instances based on CPU utilization.
Test Auto Scaling by simulating high traffic to observe how your infrastructure adapts.

Requirements

Before getting started, ensure that you have the following prerequisites in place to successfully implement Auto Scaling and Load Balancing for your web application. First, you'll need an active AWS account, which you can set up by creating a free-tier AWS account. The free-tier provides limited, cost-free access to various AWS services that will be used in this guide. Additionally, it's helpful to have a basic understanding of AWS console to follow along with the configurations more effectively. Finally, for visualizing your infrastructure setup, you can use draw.io, an easy-to-use online tool for creating architecture diagrams that can help you map out your environment.

Architecture Overview

Before diving into the setup, here’s a high-level diagram of the architecture you’ll be building:

Fig: Architecture Diagram of the Scalable Web Application

The components of the architecture include:

A VPC with a CIDR block of 10.0.0.0/16: This provides an isolated network environment for your resources in AWS.
Three Public Subnets: Each subnet resides in a different availability zone (AZ) to ensure high availability and fault tolerance.
- Public Subnet 1: 10.0.10.0/24
- Public Subnet 2: 10.0.20.0/24
- Public Subnet 3: 10.0.30.0/24
An Internet Gateway (IGW): Allows instances within the public subnets to access the internet.
A Route Table: Configures routing for internet-bound traffic from the subnets to the Internet Gateway.
EC2 Instances: These instances will run an Apache web server, deployed across the public subnets to handle web traffic.
An Auto Scaling Group (ASG): Automatically adjusts the number of EC2 instances based on traffic demand, using CPU utilization as a scaling metric.
An Application Load Balancer (ALB): Distributes incoming traffic across the EC2 instances to ensure load balancing and even request distribution.
A Target Group: Registers the EC2 instances behind the ALB and defines health check parameters, ensuring that only healthy instances receive traffic.

Step 1: Creating the VPC

Accessing the VPC Dashboard

To begin, you’ll create a VPC based on the architecture diagram above.

Log into AWS and navigate to the VPC Dashboard by selecting Services > VPC.
On the left-hand menu, click Create VPC.

Fig: VPC Dashboard

Step 2: Configure the VPC

You’ll now configure the VPC to form the core network environment for your resources.

On the Create VPC page, select the VPC and more option.
Under VPC settings:
- Name Tag: Enter a name for your VPC (e.g., WebappVPC).
- IPv4 CIDR Block: Enter 10.0.0.0/16 to give the VPC 65,536 IP addresses.
- IPv6 CIDR Block: Leave this option disabled unless IPv6 is required.
- Tenancy: Select Default unless dedicated hardware is needed.

Fig: VPC Configuration

Step 3: Configure Subnets

Next, you will configure three public subnets, each in a different availability zone for high availability.

Number of Availability Zones (AZs): Choose 3 to ensure high availability.
Number of Public Subnets: Enter 3.
Customize Subnet CIDR Blocks:
- Public Subnet 1: 10.0.10.0/24 in us-east-1a
- Public Subnet 2: 10.0.20.0/24 in us-east-1b
- Public Subnet 3: 10.0.30.0/24 in us-east-1c

These subnets will reside in different availability zones to ensure high availability across your infrastructure.

Fig: Subnet Configuration

Step 4: Configure Additional Resources

Route Table and Internet Gateway

When using the VPC wizard (the VPC and More option), AWS automatically handles the creation of route tables and an Internet Gateway. The wizard not only generates the required route table and Internet Gateway for your VPC but also sets up the necessary routes for internet-bound traffic. This ensures that instances in the public subnets can communicate with the internet.

Step 5: Review and Launch

After configuring the VPC, subnets, Internet Gateway, and route table, you’re ready to launch the VPC.

Review all the configurations on the Summary page.
Click Create VPC to launch the network environment.

Fig: VPC Creation Workflow and Preview

Your VPC is now created and ready for use, laying the foundation for a highly available, scalable web application architecture.

Section 2: Deploying an EC2 Instance

With the VPC setup complete, it's time to launch EC2 instances into one of the public subnets. These instances will serve as web servers for hosting your applications, allowing you to test connectivity and functionality within your newly created VPC.

Step 1: Launch an EC2 Instance

To launch an EC2 instance, follow these steps:

1. Navigate to the EC2 Dashboard

Log into the AWS Management Console.
Select EC2 from the list of services.
Click on Launch Instance.

Fig: EC2 Dashboard

2. Configure the Instance Details

Name and Tags: Assign a meaningful name to your instance (e.g., WebServer).
AMI Selection: Choose the Amazon Linux 2023 AMI. Amazon Linux is free-tier eligible and supports package management through yum.
Instance Type: Select t2.micro, which is free-tier eligible and suitable for testing and small applications.

Fig: Instance Type Selection

Step 2: Key Pair Setup

A key pair is required to securely access your instance via SSH. If you don’t have an existing key pair, create a new one:

Key Pair: Choose an existing key pair or create a new one.
- If creating a new key pair:
  - Enter a name for the key pair.
  - Download the .pem file and store it securely. You will need this file to SSH into your EC2 instance later.

Fig: Key Pair Setup

Step 3: Configure Network Settings

VPC: Select the VPC you created in the previous section (e.g., WebappVPC).
Subnet: Choose Public Subnet 1 (e.g., 10.0.10.0/24). This ensures that your instance has internet access via the Internet Gateway.
Auto-assign Public IP: Ensure that this option is enabled. This automatically assigns a public IP to the instance, which is essential for testing your web server over the internet.

Fig: Network Settings

Step 4: Configure Security Groups

Security groups control inbound and outbound traffic to your instance. Here's how to configure them:

Create or Choose a Security Group:
- Under the Network Settings section, select Create security group.
- Name it webapp-security-group.
- Inbound Rules:
  - Allow SSH (Port 22): Enables SSH access to your instance.
  - Source: For enhanced security, restrict to your IP address. For learning purposes, you can set it to 0.0.0.0/0 to allow access from anywhere.
  - Allow HTTP (Port 80): Permits web traffic to your Apache server.
  - Source: Set to 0.0.0.0/0 to allow access from the internet.
Outbound Rules: By default, the security group allows all outbound traffic. No changes are necessary unless specific outbound restrictions are required.

Fig: Security Group Setup

Step 5: User Data Configuration

Add the following User Data script under Advanced Details. This script will automatically install and start the Apache web server when the instance launches:

#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
echo "<h1>Hello world! This is $(hostname -f)</h1>" > /var/www/html/index.html

This script:

Updates the instance's packages.
Installs the Apache web server (httpd).
Starts the Apache service and enables it to start automatically on boot.
Creates a simple HTML page that displays the instance's hostname.

Fig: User Data Setup

7. Launch the Instance: After configuring the above settings, click Launch Instance.

Fig: Launching EC2 Instance

8. Testing the EC2 Instance

Once the instance has launched, you can test the setup by selecting the WebServer instance and copying its public IP address.

Testing the Apache Web Server:
- Open a web browser and navigate to the instance's public IP.
- You should see the message: Hello world! This is <hostname> displayed on the webpage, confirming that Apache is running and serving the HTML file created using the User Data script.

Fig: Testing Apache Web Server

Summary of What We've Done So Far

At this stage, we've successfully set up a single EC2 instance running an Apache web server within a public subnet of our VPC. By accessing this instance through its public IP, we confirmed that the web server is running and serving content. However, all incoming traffic currently hits just this one instance, meaning that if demand increases beyond its capacity, performance could degrade, or the server might even become unresponsive.

To address this, the next step is to set up Auto Scaling. This will allow us to automatically launch new EC2 instances as demand increases and terminate them when traffic reduces, ensuring that our application remains responsive and cost-efficient. We'll also configure an Application Load Balancer (ALB) to distribute traffic across multiple instances, ensuring even distribution and redundancy.

Section 3: Setting Up Auto Scaling and Load Balancing

Now that you’ve successfully deployed an EC2 instance, the next step is to implement Auto Scaling and an Application Load Balancer (ALB). This will ensure that your architecture dynamically scales in response to traffic demands, maintaining optimal performance and fault tolerance.

We’ll start by creating the Application Load Balancer (ALB), which will distribute incoming traffic across multiple instances. Then, we’ll configure the Auto Scaling Group (ASG) to automatically add or remove instances based on CPU utilization. For this tutorial, we'll use the default region us-east-1.

Setting Up Auto Scaling

1. Create a New Target Group

The ALB uses a Target Group to route traffic to specific EC2 instances. Let’s set one up:

Target Group Name: Enter a name for the target group (e.g., WebApp-TG).
Target Type: Select Instance.
Protocol: Choose HTTP.
Port: Enter 80 (for HTTP traffic).
VPC: Choose the same VPC (WebAppVPC).
Health Checks: Leave the default settings. The ALB will regularly check the health of your instances to ensure that only healthy instances receive traffic. Click Next and proceed to add targets.

Fig: Configuring Target Group

2. Register Targets

Register Existing Instances: Manually register your existing webapp-server EC2 instance. The Auto Scaling Group (ASG) will handle this automatically in the future. Click Next to proceed to registering targets and creating the Target Group.

Fig: Registering Existing Instance to Target Group

3. Review and Create

Review your Target Group configuration.
Click Create Target Group to finalize the setup.

Fig: Target Group Dashboard

Now that your Target Group is created, you can proceed to create your load balancer, which will route traffic to the instances in your Target Group.

Setting Up an Application Load Balancer (ALB)

An ALB is essential for distributing traffic across multiple instances, ensuring that no single instance becomes overwhelmed. Here’s how to set it up:

1. Access the EC2 Dashboard

In the AWS Management Console, go to the EC2 Dashboard.
On the left-hand side, under Load Balancing, select Load Balancers.

2. Create a New Load Balancer

Click on Create Load Balancer and select Application Load Balancer.

Fig: Creating an Application Load Balancer

3. Configure Load Balancer Settings

Name: Enter a name for your load balancer (e.g., WebApp-ALB).
Scheme: Choose Internet-facing to make the ALB accessible over the internet.
IP Address Type: Select IPv4.
VPC: Choose the VPC you created earlier (e.g., WebAppVPC).
Availability Zones: Select the availability zones and subnets where your EC2 instances are deployed (in this case, the three public subnets you created: 10.0.10.0/24, 10.0.20.0/24, and 10.0.30.0/24).

Fig: Configuring Load Balancer Settings

4. Configure Security Settings

Since you’re working with HTTP (port 80), there’s no need to configure an SSL certificate. You can skip this section for now. However, in production environments, you should consider enabling SSL for secure connections (HTTPS).

5. Configure Security Groups

Select or create a security group for your ALB. Ensure that it allows inbound HTTP (port 80) traffic from anywhere (0.0.0.0/0).

Once your ALB is created, it will start listening for HTTP requests and distributing them across instances via the Auto Scaling Group. However, before that, you need to create an AMI for your launch template that you will use in setting up your Auto Scaling Group.

Creating an AMI from Your Running EC2 Instance

Go to EC2 Dashboard: Navigate to Instances and select your running instance.

2.1. Actions: Click on Actions > Image and templates > Create Image.

Fig: Creating an AMI

2.2. Image Name: Provide a name for your AMI (e.g., webapp-server-image).

2.3. Instance Volumes: Review the instance storage configuration.

2.4. Create Image: Click Create Image to start the process.

2.5. Find AMI ID: Once the image is created, go to AMIs in the EC2 Dashboard to find your new AMI ID. You'll use this ID when configuring your launch template.

This AMI will be the base image for your Auto Scaling instances. Now that you have the AMI, you can continue to create the Launch Template for your Auto Scaling Group.

Creating a Launch Template for Auto Scaling

Go to EC2 Dashboard: Navigate to Launch Templates and click Create Launch Template.

2.1 Launch Template Name: Enter a name (e.g., WebApp-LT).

Fig: Launch Template Creation

2.2 AMI ID: Go to My AMIs and enter the AMI ID you created earlier (WebApp-webserver-image).

Fig: AMI ID Selection

2.3 Instance Type: Select an instance type (e.g., t2.micro).

2.4 Key Pair: Choose your existing key pair for SSH access (webapp-key-pair).

Fig: Key Pair Selection

2.5 Security Groups: Attach the security group created earlier for your EC2 instances (WebApp-Security-Group).

Fig: Security Group Attachment

2.6 Auto-assign Public IP: Under Advanced Network Configurations, enable public IP assignment.

Fig: Auto-assign Public IP

2.7 User Data: Add the user data for the instances that will be created by the auto scaling group. In this demo we are adding the same user data as our existing webserver.

Fig: Launch Template User data

2.8 Create Template: Click Create Launch Template.

This template will serve as the blueprint for launching instances in your Auto Scaling Group.

Step 2: Configuring Auto Scaling Groups (ASG)

Next, we’ll set up the Auto Scaling Group using our launch template to automatically adjust the number of EC2 instances based on CPU utilization, ensuring the application can handle traffic surges.

1. Navigate to Auto Scaling

In the AWS Management Console, go to EC2 Dashboard, and on the left-hand menu, under Auto Scaling, click on Auto Scaling Groups.

2. Create Auto Scaling Group

Click on Create Auto Scaling Group.

3. Configure Auto Scaling Group Settings

Auto Scaling Group Name: Enter a name for the group (e.g., WebApp-ASG).

Fig: Auto Scaling Group Settings

4. Configure Network Settings

VPC: Select your VPC (WebAppVPC).
Subnets: Choose the three public subnets you created (10.0.10.0/24, 10.0.20.0/24, and 10.0.30.0/24). Fig: Auto Scaling Group Network Settings

5. Attach the Load Balancer

Select Attach to an existing load balancer.
Choose the Application Load Balancer created earlier (WebApp-ALB).
Select the Target Group (WebApp-TG) created for the ALB.
Turn on Elastic Load Balancing health checks.
Under Additional settings Enable default instance warmup of 60sclick Next to continue.

Fig: Attach Load Balancer to Auto Scaling Group

6. Configure Instance Numbers

Desired Capacity: Set to 1 (this means there will always be at least 1 instances running).
Minimum Capacity: Set to 1 (this means the ASG will never terminate the last running instance).
Maximum Capacity: Set to 4 (this allows Auto Scaling to scale up to 4 instances if needed).

Fig: Configuring Instance Numbers

7. Configure Scaling Policies

Now we’ll configure the scaling policies that will trigger instance creation or termination based on CPU utilization.

Select Target Tracking Scaling Policy.
- Policy Type: Choose Target tracking scaling policy.
- Metric type: Average CPU utilization
- Target Value: Enter 25% (this means new instances will be launched when average CPU utilization exceeds 25%, and instances will be terminated when it's below 25%). Leave any other setting at default and continue.

Fig: Configuring Auto Scaling Policies

8. Review and Create

Review the configuration, then click Create Auto Scaling Group. Fig: Auto scaling group dashboard

The ASG will now automatically adjust the number of instances based on CPU utilization and distribute traffic across them using the ALB. We can evaluate this by testing the auto scaling using CPU stress.

Section 4: Testing Auto Scaling

To verify that your Auto Scaling setup is functioning correctly, you can simulate high traffic or load on the instances to trigger the creation of new instances.

Step 1: Connect to an EC2 Instance

SSH into Your EC2 Instance using EC2 Connect or the AWS CLI with the following command:
```
ssh -i /path/to/your-key.pem ec2-user@<Public-IP-of-EC2-Instance>
```

Step 2: Install the Stress Tool

Once connected, install the stress tool to simulate high CPU load:
```
sudo yum install -y stress
```

Fig: Installing the Stress Tool

Step 3: Simulate High CPU Load

Run the stress command to simulate high CPU load for a set period (e.g., 100 seconds):
```
stress -c 1 -i 1 -m 1 --vm-bytes 128M -t 100s
```

Fig: Simulating High CPU Load

This command generates CPU load on one core (since t2.micro has just 1 vCPU) for 100 seconds. As a result, the CPU utilization should exceed the 25% threshold set in the Auto Scaling Group (ASG), triggering the ASG to launch new instances to handle the increased load.

Fig: Auto Scaling Group Scaling Activity

Step 4: Verify Auto Scaling

In this section, we will verify that Auto Scaling is functioning as expected by simulating a CPU load, monitoring the Auto Scaling Group (ASG) activity, and checking the Load Balancer's traffic distribution across instances.

1. Monitor Auto Scaling Group Activity

Once you've triggered a high CPU load using the stress tool, Auto Scaling should kick in when the CPU utilization exceeds the threshold of 25%. You can monitor the ASG activity to confirm that new instances are being launched.

Navigate to Auto Scaling Groups in the AWS Management Console.
Select your Auto Scaling Group (WebApp-ASG) and go to the Activity tab.
You should observe scaling activities showing that new instances are being created because the CPU utilization exceeded the threshold.

Fig: Auto Scaling Group Activity Triggered by High CPU Usage

This activity confirms that Auto Scaling is working as intended, scaling the number of instances up in response to increased CPU usage.

2. Monitor Instance CPU Usage

Next, we can validate the specific CPU spike caused by the stress tool. The CPU usage of the instance will show a spike above 25%, which is what triggered the Auto Scaling.

Navigate to the EC2 Dashboard and select the instance running the stress test.
Go to the Monitoring tab and view the CPU utilization metrics.
You should observe a CPU spike to 40.9% during the stress test, which caused Auto Scaling to add additional instances.

Fig: CPU Usage Spike to 40.9% Due to the Stress Test

This shows that the instance was overloaded, which triggered the ASG to create additional instances.

3. Verify Load Balancer Traffic Distribution

Once multiple instances are running in the ASG, the Load Balancer should evenly distribute traffic across them. You can confirm this by checking the IP addresses the Load Balancer directs traffic to.

Open a Web Browser and navigate to the DNS name of your Application Load Balancer (ALB).
Refresh the page multiple times and observe the IP addresses assigned to the different EC2 instances.
You should see the IP address change between two or more instances, confirming that the Load Balancer is properly distributing traffic across multiple servers in the target group.

Fig: Load Balancer Distributing Traffic Across different Instances running the WebApp

This demonstrates that the Load Balancer is effectively balancing incoming requests across the instances, ensuring high availability.

4. Terminate Instances and Verify Auto Scaling

To further test the ASG's functionality, you can terminate or stop all running instances and observe how the ASG automatically replaces them to maintain the desired number of instances.

Navigate to the EC2 Dashboard and select all instances.
Terminate or stop all the instances.

Once terminated, the Auto Scaling Group will detect that no instances are running and automatically launch a new one to maintain the minimum instance count.

Go back to the Auto Scaling Group Activity in the EC2 dashboard to confirm that the ASG is performing checks and launching new instances.

Fig: All running Instances manually stopped or Terminated

Fig: New Instance Created by ASG After Termination

This confirms that the ASG is functioning correctly by ensuring that there is always at least one instance running, even if instances are terminated or fail.

This concludes the verification of your Auto Scaling Group, Load Balancer, and EC2 instances. You've successfully demonstrated the dynamic scaling of instances based on CPU usage, proper traffic distribution across multiple instances, and the resilience of your setup in automatically recovering from instance terminations.

Conclusion

In this tutorial, you've successfully set up a highly available and scalable web application architecture on AWS. By creating a Virtual Private Cloud (VPC) with three public subnets across different availability zones, deploying EC2 instances running an Apache web server, and configuring an Application Load Balancer (ALB) and Auto Scaling Group (ASG), you've ensured that your application can dynamically scale to handle varying traffic loads while optimizing costs and maintaining performance.

Key Takeaways

High Availability: Distributing resources across multiple availability zones ensures that your application remains available even if one zone experiences failures.
Scalability: Auto Scaling automatically adjusts the number of EC2 instances based on demand, ensuring your application can handle traffic spikes and scale down during low usage periods.
Load Balancing: The ALB efficiently distributes incoming traffic across multiple instances, preventing any single instance from becoming a bottleneck.
Cost Optimization: By only using the necessary resources, you optimize costs while maintaining performance and availability.

Important: Clean Up Your Environment

To avoid incurring unnecessary charges, it's essential to clean up the resources you created for this tutorial once you've completed the setup and testing. Follow these steps to terminate resources:

Terminate EC2 Instances:
- Navigate to the EC2 Dashboard.
- Select all instances created for this tutorial.
- Click Actions > Instance State > Terminate.
Delete Auto Scaling Group:
- Navigate to Auto Scaling Groups in the EC2 Dashboard.
- Select the ASG (WebApp-ASG).
- Click Delete.
Delete Load Balancer and Target Group:
- Navigate to Load Balancers.
- Select the ALB (WebApp-ALB).
- Click Actions > Delete.
- Navigate to Target Groups.
- Select the Target Group (WebApp-TG).
- Click Actions > Delete.
Delete VPC:
- Navigate to the VPC Dashboard.
- Select the VPC (WebAppVPC).
- Ensure all associated resources (subnets, gateways, route tables) are deleted.
- Click Actions > Delete VPC.

By cleaning up your environment, you ensure that you won't be billed for resources you no longer need, making your learning experience both effective and cost-efficient.

This completes the setup of an Auto Scaling Group with an Application Load Balancer to dynamically handle traffic spikes for a web application.

Resources

Here are some helpful resources for further reading on the topics covered in this post:

Deploying Basic Infrastructure on AWS with Terraform

benny fmo — Tue, 20 Aug 2024 11:59:29 +0000

Introduction

In today’s cloud-driven world, managing infrastructure efficiently and consistently is crucial for developers and IT professionals alike. Traditionally, setting up infrastructure involved navigating cloud consoles, configuring networks, launching instances, and ensuring everything was correctly connected—a manual process that was both time-consuming and prone to errors, especially when scaling.

Enter Terraform, an open-source Infrastructure as Code (IaC) tool that revolutionizes how we build, change, and version our infrastructure. Instead of manually configuring resources, Terraform is a cloud agnostic IaC tool that allows you to define your infrastructure using a simple, human-readable configuration language called HCL (HashiCorp Configuration Language). This approach not only automates deployments but also ensures consistency across environments and makes tracking changes effortless.

In this blog post, I’ll guide you step by step through deploying a basic infrastructure setup on AWS using Terraform. Whether you're a beginner or just brushing up on your skills, this tutorial will help you get hands-on experience with Terraform and HCL. We’ll cover everything from configuring your environment and writing Terraform code to deploying your first EC2 instance and setting up a simple web server. By the end, you’ll have the foundational knowledge to confidently use Terraform to manage your infrastructure in the cloud.

So, grab a cup of coffee, fire up your terminal, and let’s dive into the world of Terraform and HCL!

1. What You’ll Need

Before we start, ensure you have the following:

AWS Account: You'll need an AWS account to provision. You can see how to create an AWS free tier account here
Terraform Installed: Install Terraform by following the instructions on Terraform's official website.
AWS CLI Installed: Install the AWS Command Line Interface (CLI) to configure your credentials. Follow the instructions here.
VSCode Setup for Terraform: Install Visual Studio Code and the HashiCorp Terraform extension for syntax highlighting, code completion, and linting. This will make writing and managing Terraform configurations easier.

2. Configuring AWS Authentication

a. Setting Up AWS Credentials

To allow Terraform to authenticate and interact with AWS, you need to configure your AWS credentials. You can do this using the AWS CLI by running the following command in your terminal:

aws configure

You'll be prompted to enter your AWS Access Key ID, Secret Access Key, default region, and default output format. If you don’t have access keys yet, you can create them in the AWS Management Console under IAM > Users > Security Credentials.

b. Setting Environment Variables (Alternative Method)

Alternatively, you can export your credentials as environment variables:

export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key

This method is useful if you don’t want to store credentials in a file and it is also a good way to keep your access key confidential.

3. Writing the Terraform Configuration

In this section, we’ll walk through the Terraform code to deploy an EC2 instance on AWS. This EC2 instance will host a simple web server. Before that, let’s take a look at the architecture diagram of the webserver we are creating.

Setting Up VSCode for Terraform

To write and manage Terraform code effectively, we’ll use Visual Studio Code (VSCode). Follow these steps to set up your development environment:

Install VSCode: Download and install VSCode from here.
Install the Terraform Extension:
- Open VSCode and go to the Extensions Marketplace (Ctrl+Shift+X or Cmd+Shift+X on macOS).
- Search for HashiCorp Terraform and click Install.
Configure Your Workspace:
- Create a new folder for your Terraform project (e.g., terraform-webserver).
- Open this folder in VSCode by selecting File > Open Folder.
Initialize a New File:
- In the Explorer panel, right-click your project folder and choose New File.
- Name the file main.tf. This is where your Terraform configuration will reside.

a. Provider Configuration

Once your main.tf file is created, start by specifying the AWS provider and region:

provider "aws" {
  region = "us-east-1"
}

What It Does: This block tells Terraform to use AWS as the provider and to deploy resources in the us-east-1 region.

b. Defining the EC2 Instance

Next, define the EC2 instance that will run your web server:

resource "aws_instance" "web_server" {
  ami               = "ami-04a81a99f5ec58529"
  instance_type     = "t2.micro"
  vpc_security_group_ids = [aws_security_group.instance.id]

  user_data         = <<-EOF
  #!/bin/bash
  echo "Hello, World" > index.html
  nohup busybox httpd -f -p 8080 &
  EOF

  tags = {
    Name = "web-server"
  }
}

AMI: The Amazon Machine Image (AMI) ID defines the operating system and applications running on the instance. Here, we're using an Ubuntu AMI.
Instance Type: t2.micro is a small, free-tier eligible instance type suitable for this exercise.
User Data: This script runs on instance startup, creating an index.html file and starting a simple web server on port 8080.
Tags: Tags make it easier to identify your resources.

c. Configuring the Security Group

We need to define a security group to allow traffic on port 8080, which is where our web server will listen.

resource "aws_security_group" "instance" {
  name = "terraform-example-instance"
  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Security Group Name: This is the name of the security group.
Ingress Rules: Allows inbound traffic on port 8080 from any IP address.

4. Initializing Terraform

Once the configuration files are ready, you need to initialize Terraform. Initialization prepares the working directory and downloads the necessary provider plugins.

Run the following command in your terminal:

terraform init

What It Does: Downloads the AWS provider and initializes your Terraform working directory. This step is essential before running any other Terraform commands.

5. Planning the Infrastructure

Before applying the configuration, you can preview the changes Terraform will make by running:

terraform plan

What It Does: The plan command shows you a preview of the infrastructure Terraform will create, modify, or destroy. It helps catch errors and confirm that the configuration is correct before applying any changes as shown on the screenshot below.

6. Applying the Configuration

Now that you’ve reviewed the plan, it’s time to apply the configuration and create the infrastructure.

Run the following command:

terraform apply

What It Does: The apply command provisions the resources as defined in your configuration. Terraform will prompt for confirmation before proceeding. Once confirmed, it will create the EC2 instance and security group as shown on the screenshot below.

7. Verifying the Infrastructure

a. Retrieving the Public IP

After Terraform finishes applying the configuration, you’ll want to verify that everything is working correctly. To do this, you’ll need the public IP of your EC2 instance. You can find it in the Terraform output or on the EC2 dashboard of the AWS Management Console.

b. Testing the Web Server

With the public IP in hand, test the web server by running the following curl command in your terminal:

curl http://<public-ip>:8080

What It Does: This command sends an HTTP GET request to your EC2 instance's public IP on port 8080. If everything is working correctly, you should see the response "Hello, World" in your terminal, confirming that the web server is running.

8. Cleaning Up

When you're done with the infrastructure, it's important to clean up the resources to avoid unnecessary charges. You can do this by running:

terraform destroy

What It Does: The destroy command deletes all the resources that Terraform created. Terraform will prompt for confirmation before proceeding with the deletion.

Conclusion

Congratulations! You’ve just deployed a basic infrastructure setup on AWS using Terraform. We walked through the process of configuring AWS authentication, writing Terraform code, initializing Terraform, deploying and verifying an EC2 instance running a web server, and setting up VSCode for Terraform development. This exercise introduced you to key Terraform commands and provided a hands-on example of using Terraform to manage infrastructure on AWS.

With Terraform, you can easily scale this setup, automate more complex infrastructure, and maintain consistent environments. Keep experimenting, and you’ll soon be able to manage much larger infrastructure deployments!

Resources

Feel free to drop any questions in the comments, and happy coding!

What is Infrastructure as Code (IaC) and Why It's Transforming DevOps?

benny fmo — Sun, 18 Aug 2024 16:45:42 +0000

Introduction:

The digital landscape is evolving at a breakneck pace, and so are the tools and methodologies that drive it. In this era of continuous integration and continuous deployment (CI/CD), automation is no longer a luxury—it's a necessity. One of the most transformative elements in this automation wave is Infrastructure as Code (IaC).

But what exactly is Infrastructure as Code, and why is it making such a significant impact on DevOps? For those new to the concept, IaC might seem like just another buzzword in the ever-expanding tech lexicon. However, it's much more than that. IaC represents a paradigm shift in how infrastructure is managed, provisioned, and maintained, turning manual, error-prone tasks into automated, repeatable processes.

In this article, we’ll break down the essentials of IaC, exploring what it is, how it works, and why it’s becoming a cornerstone of modern DevOps practices. Whether you're a beginner trying to understand the basics or a seasoned developer looking to refine your knowledge, this guide will offer valuable insights into how IaC is transforming the way we think about and implement infrastructure.

The Role of IaC in DevOps

Infrastructure as Code (IaC) plays a pivotal role in the DevOps ecosystem, acting as a catalyst for automation, collaboration, and scalability. Let's explore how IaC integrates seamlessly into DevOps practices and transforms the way teams develop, deploy, and manage applications.

1. Automation and Consistency

In traditional IT environments, infrastructure setup and configuration often involve manual processes that are prone to human error and inconsistency. With IaC, these tasks are automated through code, ensuring that infrastructure is provisioned the same way every time, across all environments. This level of consistency is crucial in DevOps, where maintaining identical environments for development, testing, and production is essential to avoid the "it works on my machine" problem.

2. Collaboration and Speed

One of the core principles of DevOps is fostering collaboration between development and operations teams. IaC enables this by treating infrastructure configurations as code, which can be versioned, reviewed, and tested just like application code. Developers and operations teams can work together on infrastructure changes, track them in version control systems like Git, and integrate them into CI/CD pipelines.

This collaborative approach not only speeds up the deployment process but also reduces the time it takes to resolve issues. When infrastructure is defined as code, teams can quickly identify and fix problems by rolling back to previous versions or applying patches, all within a shared codebase.

3. Scalability

As businesses grow and demands fluctuate, the ability to scale infrastructure up or down quickly becomes crucial. IaC shines in this aspect by enabling teams to define infrastructure that can be easily replicated or adjusted to meet changing needs. Whether it’s adding more servers, increasing storage capacity, or deploying to multiple regions, IaC allows for rapid and efficient scaling.

For example, if an application experiences a sudden spike in traffic, IaC can automate the process of provisioning additional resources to handle the load, ensuring that the application remains responsive and available.

Benefits of IaC

Infrastructure as Code (IaC) offers numerous benefits that extend beyond simply automating infrastructure management. By leveraging IaC, organizations can achieve greater efficiency, reliability, and scalability in their DevOps processes. Let's explore some of the key benefits of IaC, including reduced errors, cost efficiency, disaster recovery, version control, and reusability.

1. Reduced Errors

One of the most significant advantages of IaC is the reduction of human errors. Manual infrastructure management often involves repetitive tasks that can lead to mistakes, especially in complex environments. With IaC, these tasks are automated, ensuring that infrastructure is provisioned and configured consistently every time. This consistency minimizes the risk of misconfigurations, which can lead to system failures or security vulnerabilities.

Example: Consider a scenario where multiple servers need to be configured with identical settings. Manually setting up each server increases the likelihood of discrepancies, but with IaC, you can automate this process, guaranteeing uniformity across all servers.

2. Cost Efficiency

IaC can significantly reduce costs by optimizing resource management. Traditional infrastructure often involves over-provisioning resources to handle peak loads, leading to wasted resources and higher costs. With IaC, infrastructure can be dynamically scaled based on actual demand, ensuring that resources are allocated efficiently. This not only reduces costs but also improves the overall utilization of cloud services.

Example: Instead of running multiple servers at full capacity 24/7, IaC allows you to scale up resources during peak hours and scale down during off-peak times, saving on cloud costs.

3. Disaster Recovery

IaC plays a crucial role in disaster recovery by enabling quick and reliable restoration of infrastructure. In the event of a system failure or data loss, IaC scripts can be used to re-provision infrastructure in a matter of minutes, ensuring minimal downtime. This capability is essential for maintaining business continuity, especially in critical environments where downtime can result in significant financial losses.

Example: If a primary data center goes offline, IaC can automatically spin up a backup infrastructure in a different region, restoring services with minimal disruption.

4. Easily duplicate an environment

The same environment can be easily deployed on a different system in another location using the same IaC, as long as the infrastructure resources are available.

Example:For example, imagine a business’s regional branch has IaC to describe the whole branch’s enterprise environment, including servers, networking, and custom configurations. If the business opened another regional branch, they could use IaC to duplicate the exact same environment and quickly make the branch online and operational. IaC removes the repetitive manual steps and checklists that were needed in the past.

5. Version Control

Treating infrastructure as code allows you to leverage version control systems (VCS) like Git to track changes, collaborate on configurations, and maintain a history of infrastructure states. Version control enables teams to revert to previous configurations in case of issues, compare changes, and audit infrastructure modifications.

Example: A team can roll back to a previous stable version of infrastructure code if a new deployment introduces issues, minimizing the impact on production environments.

6. Reusability

IaC promotes reusability by allowing you to create modular templates or scripts that can be reused across multiple projects or environments. This not only saves time but also ensures that best practices are consistently applied. By reusing code, teams can quickly deploy new environments or replicate infrastructure across different regions with minimal effort.

Example: A reusable Terraform module for setting up a VPC (Virtual Private Cloud) can be applied to different environments, ensuring consistent network configurations across projects.

7. Easily duplicate an environment

The same environment can be deployed on a different system in another location using the same IaC, as long as the infrastructure resources are available.

Example: Imagine a business’s regional branch has IaC to describe the whole branch’s enterprise environment, including servers, networking, and custom configurations. If the business opened another regional branch, they could use IaC to duplicate the exact same environment and quickly make the branch online and operational. IaC removes the repetitive manual steps and checklists that were needed in the past.

Challenges and Best Practices

While Infrastructure as Code (IaC) offers significant benefits, it also comes with its own set of challenges. Additionally, to maximize the effectiveness of IaC, it's crucial to follow best practices. In this section, we'll discuss some of the common challenges associated with IaC, outline best practices for successful implementation, and introduce popular IaC tools, separated into agnostic and cloud-specific categories.

Challenges of IaC

Learning Curve:
- Implementing IaC requires a solid understanding of both coding and infrastructure management. For teams new to IaC, the learning curve can be steep, especially when dealing with complex configurations and tools.
Tool Sprawl:
- With numerous IaC tools available, teams might face challenges in choosing the right tool for their needs. Additionally, managing multiple tools across different environments can lead to tool sprawl, making it difficult to maintain consistency.
Security Concerns:
- IaC scripts often contain sensitive information, such as access keys and passwords. Ensuring that this information is securely managed and not exposed to unauthorized users is a critical challenge.
Debugging and Troubleshooting:
- When issues arise, debugging IaC can be challenging, especially in large and complex infrastructures. Identifying the root cause of a problem may require in-depth knowledge of both the code and the infrastructure it manages.

Best Practices for IaC

Use Modular Code:
- Break down IaC scripts into smaller, reusable modules. This approach makes the code more manageable, promotes reuse, and allows teams to apply consistent configurations across multiple environments.
Integrate with CI/CD Pipelines:
- Integrate IaC with Continuous Integration/Continuous Deployment (CI/CD) pipelines to automate testing and deployment of infrastructure changes. This ensures that changes are automatically validated and deployed, reducing the risk of errors.
Version Control Everything:
- Store all IaC scripts in a version control system (VCS) like Git. This allows teams to track changes, collaborate on configurations, and roll back to previous versions if needed.
Ensure Security Compliance:
- Use tools like HashiCorp Vault to securely manage sensitive information in IaC scripts. Regularly audit the scripts for security vulnerabilities and ensure compliance with industry standards.
Continuous Monitoring and Updates:
- Regularly monitor IaC configurations and update them to accommodate new requirements or technologies. Continuous monitoring helps identify and address potential issues before they impact production environments.

Infrastructure as Code Tools

Agnostic vs. Cloud-Specific IaC Tools

When choosing an IaC tool, it's essential to understand the difference between agnostic and cloud-specific tools.

Agnostic IaC Tools: These tools are designed to work across multiple cloud providers and on-premises environments. They offer flexibility and are ideal for organizations that use a multi-cloud strategy or need to manage hybrid environments.
Cloud-Specific IaC Tools: These tools are tailored to a specific cloud provider, offering deep integration and leveraging the unique features of that provider. They are ideal for organizations that primarily operate within a single cloud ecosystem and want to take full advantage of the provider's offerings.

Category	Tool Name	Description
Agnostic IaC Tools	Terraform	A widely-used tool that allows you to define and provision infrastructure across multiple cloud providers using a single configuration language.
	Ansible	A powerful automation engine that manages infrastructure and application deployment across various environments.
	Pulumi	Allows developers to use familiar programming languages like Python and JavaScript to define and deploy infrastructure across different cloud platforms.
Cloud-Specific IaC Tools	AWS CloudFormation	A service by AWS that allows you to model and set up your Amazon Web Services resources using JSON or YAML templates.
	Azure Resource Manager (ARM)	Microsoft Azure’s tool for managing infrastructure resources through templates, scripts, and APIs.
	Google Cloud Deployment Manager	A Google Cloud tool that allows you to manage cloud resources using declarative templates written in YAML.

### Conclusion

Infrastructure as Code (IaC) is not just a buzzword—it's a fundamental shift in how we manage and provision infrastructure in the modern DevOps landscape. By turning infrastructure management into a programmatic task, IaC allows organizations to automate, scale, and optimize their IT environments with unprecedented efficiency and consistency.

Throughout this article, we've explored what IaC is, its critical role in transforming DevOps practices, and the numerous benefits it brings to the table, from reducing errors and costs to enabling rapid disaster recovery, version control, and reusability. We also discussed the challenges that come with adopting IaC and provided best practices to help you navigate these obstacles successfully.

The integration of IaC into your DevOps pipeline is not just about improving operational efficiency—it's about enabling innovation. By freeing up time and resources previously spent on manual infrastructure tasks, your teams can focus on developing and delivering value-driven features faster and more reliably.

As you continue your journey with IaC, remember that the true power of this approach lies in its ability to bridge the gap between development and operations, fostering a culture of collaboration, automation, and continuous improvement. Whether you’re a beginner just getting started or a seasoned professional looking to refine your practices, the principles and tools of IaC offer a clear path toward more agile and resilient infrastructure management.

Call to Action: Now is the time to explore IaC tools, experiment with them in your own environment, and start integrating them into your DevOps processes. Whether you choose a cloud-agnostic tool like Terraform or a cloud-specific solution like AWS CloudFormation, the key is to begin. The sooner you start, the sooner you’ll reap the benefits of a more automated, scalable, and reliable infrastructure.

For those who want to dive deeper, consider exploring further resources, such as advanced tutorials, case studies, or even certifications in specific IaC tools, i have attached some resources below to help with that.
The world of IaC is vast and continually evolving, offering endless opportunities to learn, innovate, and transform the way we think about infrastructure.

Basic Linux Commands to Know: A Beginner's Guide

benny fmo — Tue, 13 Aug 2024 11:55:35 +0000

Introduction

Linux is a powerful and flexible operating system used by developers, system administrators, cloud engineers, and IT professionals worldwide. Understanding basic Linux commands is essential for navigating the system, managing files, and performing administrative tasks. In this guide, we'll cover 15 essential Linux commands, explaining what each does and showing their outcomes.

For this exercise, I'll be using Windows Subsystem for Linux (WSL) running Ubuntu 22.04.3 LTS. If you're not familiar with WSL or need help setting it up, you can find a detailed guide here.

Basic Command Structure in Linux

Before diving into specific commands, it's important to understand the basic structure of a Linux command. A typical Linux command follows this format:

command [options] [arguments]

command: This is the actual command you want to execute (e.g., ls, cd, cp).
options: These modify the behavior of the command (e.g., -l for ls to list in long format).
arguments: These specify the targets for the command, like files or directories.

For example, the command ls -l /home lists the contents of the /home directory in long format.

Using the `man` Command

Each

Linux

command comes with a manual page (man page) that provides detailed information on how to use the command, including available options and examples. To access the man page for any command, use:

man command_name

For example, to explore the options available for the ls command, you would use:

man ls

This will open a detailed manual where you can explore all the functionalities of the ls command.

pwd
ls
cd
mkdir
touch
cp
mv
rm
cd
sudo
cat
grep
echo
find
top

1. `pwd` (Print Working Directory)

The pwd command displays the current working directory you're in. It's useful to know your location in the filesystem.

Example:

pwd

Output:

/home/username

2. `ls` (List Directory Contents)

The ls command lists the contents of a directory. It can display files, directories, and various details like permissions, ownership, and size.

Example:

ls -l

Output:

drwxr-xr-x 2 username username 4096 Aug 13 15:32 folder_name
-rw-r--r-- 1 username username 220 Aug 13 15:32 file_name.txt

3. `cd` (Change Directory)

The cd command is used to change your current directory. You can navigate to any directory using the absolute or relative path.

Example:

cd /home/username/Documents

Output:

No output, just changes the directory to /home/username/Documents.

4. `mkdir` (Make Directory)

The mkdir command creates a new directory. You can specify the name of the directory you want to create.

Example:

mkdir new_folder_name

Output:

No output, just creates a directory named new_folder_name.

5. `touch` (Create a File)

The touch command is used to create an empty file or update the timestamp of an existing file.

Example:

touch new_file_name

Output:

No output, just creates an empty file named new_file_name.

6. `cp` (Copy Files/Directories)

The cp command copies files or directories from one location to another.

Example:

cp file1.txt /home/username/Documents/

Output:

No output, just copies file1.txt to the /home/username/Documents/ directory.

7. `mv` (Move/Rename Files)

The mv command moves or renames files or directories.

Example:

mv file1.txt new_file1.txt

Output:

No output, just renames file1.txt to new_file1.txt.

8. `rm` (Remove Files/Directories)

The rm command removes files or directories. Be cautious as this can permanently delete files.

Example:

rm unwantedfile.txt

Output:

No output, just deletes unwantedfile.txt.

9. `chmod` (Change File Permissions)

The chmod command changes the permissions of a file or directory. It’s useful for controlling who can read, write, or execute a file.

Example:

chmod 400 script.sh

Output:

No output, just changes the permissions of the file script.sh to read only. You can type ll or ls -l to verify the permissions as shown below

10. `sudo` (Admin privileges)

The sudo command in Linux allows a permitted user to execute a command as the superuser or another user, as specified by the security policy. It stands for "Superuser Do" and is commonly used to perform tasks that require administrative or root privileges.

Example:

sudo apt update

Output:

sudo apt update refreshes the list of available packages and versions. sudo is needed because this command updates system files that require administrative privileges without which permission will be denied as shown on the screenshot below.

11. `cat` (Concatenate and Display File Content)

The cat command displays the content of a file. It can also concatenate multiple files and display them together.

Example:

cat file_name.txt

Output:

Displays the contents of file_name.txt.

12. `grep` (Search Text)

The grep command searches for a specific pattern of text within files.

Example:

grep "search_term" file2.txt

Output:

Displays the lines containing the string search_term in file2.txt.

13. `echo` (Display a Line of Text)

The echo command outputs the specified text or variables to the terminal.

Example:

echo "Hello, World!"

Output:

Displays the text Hello, World! in the terminal.

14. `find` (Search for Files and Directories)

The find command searches for files and directories within a specified directory.

Example:

find /home/username/Documents -name "*.txt"

Output:

Lists all .txt files in /home/username/Documents and its subdirectories. A clear example is shown on the image below.

15. `top` (Display System Processes)

The top command provides a real-time view of running processes and system resource usage.

Example:

top

Output:

Displays a dynamic view of system processes.

Conclusion

These basic Linux commands are essential tools for anyone working in a Linux environment. Mastering them will make navigating and managing your system more efficient. As you become more comfortable with these commands, you'll be able to perform more complex tasks with ease. Remember to use the man command to explore the full capabilities of each command.

Resources

1.How to write a high quality post on DEV

2.Youtube Linux Operating System - Crash Course for Beginners by freecodecamp

Dev post formatting markdown cheat sheet on github

hello devs!!

benny fmo — Tue, 13 Aug 2024 11:03:51 +0000

echo hello world