DEV Community: yadanaresh

Securing Your AWS Terraform Deployments: 5 Authentication Methods

yadanaresh — Sun, 03 Dec 2023 23:38:06 +0000

Introduction

In the rapidly evolving landscape of cloud infrastructure management, securing access to AWS resources is a foundational concern. One critical aspect is the secure authentication of Terraform, a popular infrastructure-as-code tool, to your AWS environment. In this blog post, we will delve into five distinct authentication methods, providing comprehensive explanations and practical demonstrations for each. By the end, you'll be well-equipped to make informed decisions aboåut the most suitable authentication approach for your specific use case.

Prerequisites

Before diving into the practical demonstrations, it's essential to ensure that the following prerequisites are satisfied:

Procure Bastion Server:

Acquire a Bastion server with SSH port open.
Ensure that HTTP and HTTPS ports are open to facilitate necessary communications.

Install Terraform on Bastion Server:

Terraform should be installed on the Bastion server, serving as the central point for infrastructure provisioning.

Reference: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

Install AWS CLI on Bastion Server:

AWS CLI is a crucial component for several authentication methods. Install it on the Bastion server to facilitate secure interactions with AWS services.

Reference: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Note: The following scenarios and practical demonstrations are explained using RHEL OS. Ensure that your Bastion server is running RHEL, and adapt commands accordingly based on the specific Linux distribution you are using.

Scenario 1: Direct Values in Code

Explanation:
The most straightforward yet insecure method involves hardcoding AWS secret and access keys directly into Terraform code or configuration files. While this approach might be convenient for testing or training purposes, it poses a significant security risk in a production environment. Anyone with access to the codebase can retrieve sensitive information, making it a less-than-ideal choice for real-world scenarios.

Embedding AWS secret and access keys directly into Terraform code is generally considered a bad practice due to security concerns. When secret keys are exposed in Terraform code, it poses a significant security risk as anyone with access to the Terraform code can potentially access sensitive AWS resources, make unauthorized changes, or compromise the security of your infrastructure. It's crucial to follow best practices for handling secrets to avoid these risks.

Configuration File:
Terraform configurations are often stored in files with a .tf extension. These files contain the infrastructure code and configuration details. In some cases, AWS secret and access keys might be directly embedded in these files.

Usage in Resources:
The variables are then used in the AWS resource definitions within the Terraform code. For example:

provider "aws" {
  access_key = "AKIATDG3VPEUQ2HE5NEY"
  secret_key = "bU2BBXqBhXeDvtsgta41mmUzbX5eV+IJCxGIkk/0"
  region     = "us-west-2"
}

resource "aws_instance" "senario1" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
}

Exposing Secrets:

Unfortunately, when secrets are directly embedded like this, they become visible to anyone who has access to the Terraform code. This includes version control systems, shared repositories, and collaborators on the project.
Security Implications:

Exposing AWS secret and access keys in Terraform code is a security vulnerability. If the Terraform code is inadvertently shared or exposed, unauthorized individuals may gain access to AWS resources, leading to potential data breaches, unauthorized usage, or even financial losses.

  provider "aws" {
      access_key = "AKIATDG3VPEUQ2HE5NEY"
      secret_key = "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"
      region     = "us-west-2"
    }

    resource "aws_instance" "senario1" {
      ami           = "ami-0c55b159cbfafe1f0"
      instance_type = "t2.micro"
    }

commands to execute
terraform init
terraform plan
terraform apply

Scenario 2: Environment Variables

Explanation:
Storing AWS credentials as environment variables in Terraform improves security over direct code embedding but introduces risks. While protecting against code exposure, environment variables are susceptible to command history leaks. Manual export before Terraform execution and limited access control granularity make them less suitable for certain production scenarios. Organizations should consider advanced secrets management solutions, automate secure variable configuration, and regularly rotate credentials to enhance security. In production, combining environment variables with AWS IAM roles or dedicated key management services provides a more robust and secure approach.

Exporting AWS Secret Keys as Environment Variables:

In this demonstration, you are likely using commands in the terminal to export AWS secret keys as environment variables. This is a more secure approach than embedding the keys directly in Terraform code, as it separates sensitive information from the infrastructure configuration.

    export AWS_ACCESS_KEY_ID="AKIATDG3VPEUQ2HE5NEY"
    export AWS_SECRET_ACCESS_KEY="bU2BBXqBhXeDvtsgta41mmUzbX5eV+IJCxGIkk/0"

Terraform Configuration:

In your Terraform code, you then reference these environment variables instead of directly embedding the secret keys.

 provider "aws" {
  region = "us-east-1" #always region is us east
}
resource "aws_instance" "senario2" {
  ami           = "ami-023c11a32b0207432" # Specify ami details
  instance_type = "t2.micro"
}

Security Breach in Bash History:

The vulnerability arises from the fact that commands entered in a terminal, including the export commands for AWS secret keys, are often logged in the bash history file. If someone gains access to the history file, they can see the exported secret keys.

For example, if a user runs history or looks into the .bash_history file, they can find:

  history |grep -i export
   22  export AWS_ACCESS_KEY_ID="AKIAZ3QDJN4PLRT7OUGC"
   23  export AWS_SECRET_ACCESS_KEY="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"

This creates a potential security risk, as anyone with access to the system could view these commands and extract sensitive information.

Mitigation:

To mitigate this risk, you should take steps to secure the bash history file. You can configure the shell not to log certain commands or use tools that help manage and encrypt the history file.

To mask sensitive commands in Bash history in Linux, you can use the HISTCONTROL environment variable with the value ignorespace or ignoreboth. This prevents commands preceded by a space from being saved to the history. Here's how you can set it:

Copy code
echo "HISTCONTROL=ignorespace" >> ~/.bashrc
source ~/.bashrc
Now, when you precede a command with a space, it won't be stored in the Bash history. For example:

[root@ip-172-31-33-196 AWS-Terra]#  export AWS_ACCESS_KEY_ID="AKIAZ3QDJN4PLRT7OUGC"
[root@ip-172-31-33-196 AWS-Terra]# 
[root@ip-172-31-33-196 AWS-Terra]#  export AWS_SECRET_ACCESS_KEY="IGFhfJaQ7ZGyt1ZGSJPfuchhNREiJDcdiaTzWdoY"
[root@ip-172-31-33-196 AWS-Terra]# 
[root@ip-172-31-33-196 AWS-Terra]# history |grep -i export
   22  export AWS_ACCESS_KEY_ID="AKIAZ3QDJN4PLRT7OUGC"
   23  export AWS_SECRET_ACCESS_KEY="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"
   50  history |grep -i export
   51  history |grep -i export

Practical Demonstration:

commands

terraform init
----------------
Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.29.0...
- Installed hashicorp/aws v5.29.0 (signed by HashiCorp)

terraform plan
---------------
`Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

terraform apply
----------------
Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.senario2: Creating...
aws_instance.senario2: Still creating... [10s elapsed]
aws_instance.senario2: Still creating... [20s elapsed]
aws_instance.senario2: Still creating... [30s elapsed]
aws_instance.senario2: Creation complete after 31s [id=i-005cb70b463c82be3]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Scenario 3: AWS CLI with Config Files

Explanation:
Leveraging the AWS CLI for AWS credential management offers an improved security approach by storing access and secret keys in a dedicated file (~/.aws/secure-credentials). This method introduces an additional layer of protection, addressing concerns related to embedding credentials directly in Terraform code or using environment variables. The confidential file ensures that sensitive information is isolated and encrypted, enhancing security measures. However, it is imperative to enforce strict access controls to the AWS CLI and associated credential files. Limiting access to authorized teams or individuals helps prevent unauthorized use and ensures that only trusted personnel can interact with AWS resources. In summary, integrating the AWS CLI with secure credential storage practices is a robust security strategy, combining convenience with heightened protection against credential exposure and misuse.

Practical Demonstration:

let's walk through the steps to configure the AWS CLI, store credentials in a secure file, and integrate them into Terraform for enhanced security.

Before we test scenario 3, let's unset the environment variable using the following commands.

[root@ip-xxx-xx-xx-xxx AWS-Terra]# unset AWS_ACCESS_KEY_ID
[root@ip-xxx-xx-xx-xxx AWS-Terra]# unset AWS_SECRET_ACCESS_KEY
[root@ip-xxx-xx-xx-xxx AWS-Terra]# echo $AWS_SECRET_ACCESS_KEY

[root@ip-xxx-xx-xx-xxx AWS-Terra]# echo $AWS_ACCESS_KEY_ID

[root@ip-xxx-xx-xx-xxx AWS-Terra]#

1. Configure AWS CLI:

aws configure

Follow the prompts to enter your AWS access key, secret key, region, and preferred output format. This information will be stored in the AWS CLI configuration file (~/.aws/credentials).

2. Store Credentials in a Secure File:

Create a dedicated file to store your AWS credentials securely. For example:

echo "[default]" > ~/.aws/credentials
echo "aws_access_key_id = YOUR_ACCESS_KEY" >> ~/.aws/credentials
echo "aws_secret_access_key = YOUR_SECRET_KEY" >> ~/.aws/credentials

Secure the file's permissions to make it readable only by the owner:

chmod 600 /AWS-Terra/.aws/credentials

3. Configure Terraform to Use AWS CLI Credentials:

In your Terraform configuration, set the AWS provider block to use the AWS CLI credentials file:

provider "aws" {
  shared_credentials_files = ["/AWS-Terra/.aws/credentials"]
  region                 = "us-east-1"
}

4. Enhanced Security:

By default, only authorized users should have access to the AWS CLI configuration and credentials. Ensure that access to the AWS CLI is restricted to authorized teams or individuals.

provider "aws" {
  shared_credentials_files = ["/AWS-Terra/.aws/credentials"]
  region = "us-east-1" #always region is us east
}
resource "aws_instance" "senario3" {
  ami           = "ami-023c11a32b0207432" # Specify ami details
  instance_type = "t2.micro"
}

commands

terraform plan
--------------
aws_instance.senario2: Refreshing state... [id=i-005cb70b463c82be3]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # aws_instance.senario2 will be destroyed
  # (because aws_instance.senario2 is not in configura 


Plan: 1 to add, 0 to change, 1 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.


terraform apply
----------------
aws_instance.senario2: Refreshing state... [id=i-005cb70b463c82be3]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # aws_instance.senario2 will be destroyed
Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.senario2: Destroying... [id=i-005cb70b463c82be3]
aws_instance.senario3: Creating...
aws_instance.senario2: Still destroying... [id=i-005cb70b463c82be3, 10s elapsed]
aws_instance.senario3: Still creating... [10s elapsed]
aws_instance.senario2: Still destroying... [id=i-005cb70b463c82be3, 20s elapsed]
aws_instance.senario3: Still creating... [20s elapsed]
aws_instance.senario2: Destruction complete after 30s
aws_instance.senario3: Still creating... [30s elapsed]
aws_instance.senario3: Still creating... [40s elapsed]
aws_instance.senario3: Still creating... [50s elapsed]
aws_instance.senario3: Creation complete after 52s [id=i-0e214b3b0fb488a6d]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Scenario 4: EC2 Role-Based Authentication

Explanation:
Securing AWS infrastructure through IAM roles and their attachment to EC2 instances is a best practice that enhances security. This approach minimizes security breaches by providing temporary and scoped permissions, ensuring resources are accessed based on the instance's role. IAM roles offer dynamic and flexible access control, reducing the risk associated with long-lived credentials. In practical terms, setting up EC2 role-based authentication involves creating roles with specific permissions and attaching them to instances. This not only simplifies credential management but also strengthens security measures, aligning with the principle of least privilege. Leveraging IAM roles showcases AWS's robust security model and is essential for securing cloud-based environments effectively.

IAM Role Permissions for EC2 Bastion Instance Creation:

IAM Console:

Navigate to the IAM Console in the AWS Management Console.
Select "Roles" and click "Create role."
Choose "AWS service" as the trusted entity and EC2 as the use case.

Permissions:

In the "Permissions" step, attach the following policies to the role:
AmazonEC2FullAccess: Provides comprehensive EC2 permissions.
AmazonSSMFullAccess: Allows interaction with EC2 instances using AWS Systems Manager.

Note:

It's essential to follow the principle of least privilege. If you only need to create and manage EC2 instances, the above policies are generally sufficient. However, if you plan to perform additional tasks (e.g., creating VPCs, modifying security groups), you may need to attach additional policies accordingly.

Create Role:

Complete the role creation process and make note of the Role ARN, which you will use when launching your EC2 instance.

Instance Launch Configuration:

Launch a new EC2 instance or select an existing one.
In the instance configuration, choose the IAM role created earlier in the "Configure Instance Details" section.

Verify Role-Based Authentication:

Connect to the EC2 instance using SSH or any other preferred method.
Confirm the role-based authentication by checking the AWS CLI configuration on the instance using aws configure list.

Terraform code

provider "aws" {
  region = "us-east-1"  # Set your desired AWS region
}

resource "aws_iam_role" "ec2_role" {
  name = "ec2_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_instance_profile" "ec2_instance_profile" {
  name = "ec2_instance_profile"
  role = aws_iam_role.ec2_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_full_access" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2FullAccess"
  role       = aws_iam_role.ec2_role.name
}

resource "aws_instance" "Bastion_ec2_role" {
  ami           = "ami-023c11a32b0207432"
  instance_type = "t2.micro"
  key_name      = "terraform-infra"

  iam_instance_profile = aws_iam_instance_profile.ec2_instance_profile.name

  tags = {
    Name = "bastion-role"
  }
}

terraform init
----------------
Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v5.29.0...
- Installed hashicorp/aws v5.29.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

terraform plan
----------------
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.senario4 will be created
  + resource "aws_instance" "senario4" {

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

terraform apply
-----------------
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_instance.senario4 will be created
  + resource "aws_instance" "senario4" {
      + ami                                  = "ami-023c11a32b0207432"

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_instance.senario4: Creating...

aws_instance.senario4: Still creating... [4m30s elapsed]
aws_instance.senario4: Creation complete after 4m33s [id=i-01c26968753921b24]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

After creating the Bastion EC2 instance with the 'ec2_role,' which has the 'AmazonEC2FullAccess' permission, I logged into the Bastion server. Subsequently, I executed a Terraform script to create an additional EC2 instance without passing any AWS access and secret keys explicitly. The EC2 instance was successfully created based on the IAM role policy associated with 'ec2_role.' This demonstrates Scenario 4, where there is no need to provide AWS access and secret keys explicitly; EC2 instances can be launched securely using IAM roles.

Scenario 5: HashiCorp Vault Integration

Explanation:
Integrating Terraform with HashiCorp Vault represents the apex of contemporary cloud security practices. This synergy centralizes the administration of sensitive information, specifically AWS secret and access keys, ensuring their secure storage. The dynamic retrieval of these credentials during Terraform execution adds an extra layer of security. Exploring the nuances of Vault integration reveals its robust security features, establishing it as a reliable safeguard for critical information. If you harbor a keen interest in securing AWS secrets, integrating them with HashiCorp Vault emerges as a preferred and effective method, offering a comprehensive approach to managing and protecting sensitive data in cloud environments.

Taking a hands-on approach

creating HashiCorp vault secrets

installation of vault on Linux server


sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install vault

commands

[-31-36-42 ~]# vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token hvs.IZnSVaRnskiqOgLbkKVgzJgY
token_accessor w1ggJYnsUZwl8f3on9h2A5Vj
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]

commands

[31-36-42 ec2-with-vault]# vault kv put secret/aws-creds access_key=AKIAUUSZO22P4DSSDUCWHVPKssf
------Secret Path ----
secret/data/aws-creds

------ Metadata -------
Key Value

created_time 2023-12-03T22:50:07.362873124Z
custom_metadata
deletion_time n/a
destroyed false
version 3
[root@ip-172-31-36-42 ec2-with-vault]# vault kv put secret/aws-sec secret_key=3wWrTb9qUm+OewewebBB44ElkqbO6fNxE82pWKGLIheJM
---- Secret Path -----
secret/data/aws-sec

--------- Metadata --------
Key Value

created_time 2023-12-03T22:50:47.626808901Z
custom_metadata
deletion_time n/a
destroyed false
version 1

provider.tf

provider "vault" {
  address = "https://127.0.0.1:8200"
  skip_tls_verify = true
  token = "xxxxxxxxxxxxxxxxxxx"
}

vault-secrect.tf

data "vault_generic_secret" "aws-creds" {
    path = "secret/data/aws-creds"
}
data "vault_generic_secret" "aws-sec" {
   path = "secret/data/aws-sec"
}

 provider "aws" {
  region = "us-east-1" #always region is us east
}
resource "aws_instance" "ec2-with-vault" {
  ami           = "ami-023c11a32b0207432"
  instance_type = "t2.micro"
  key_name      = "terraform-infra"

  # Other instance configurations...

  tags = {
    Name = "ec2-with-vault"
  }
}

Conclusion

Choosing the right authentication method for your Terraform deployments is pivotal in maintaining a robust and secure AWS environment. By understanding the strengths and weaknesses of each scenario and following the practical demonstrations, you can make informed decisions aligned with your organization's security requirements. Stay tuned for the upcoming sections, where we will provide detailed step-by-step instructions and insights for each authentication method.

AWS Setting up VPC Transit Gateway and EC2 Instances With Terraform

yadanaresh — Thu, 09 Nov 2023 02:25:41 +0000

Introduction

In the dynamic landscape of cloud infrastructure, managing network communication and deploying resources across multiple VPCs can be a complex task. AWS provides a solution in the form of the VPC Transit Gateway, streamlining the connectivity between VPCs and on-premises environments. Automating these setups becomes easier and more efficient using Terraform, an Infrastructure as Code tool. In this blog post, I'll guide you through the process of setting up a VPC Transit Gateway and deploying EC2 instances across distinct VPCs, interconnected via the gateway, all orchestrated using Terraform.

In this guide, I aim to provide a comprehensive walk-through for two scenarios simultaneously: performing each task manually in the AWS Management Console and, in parallel, furnishing the corresponding Terraform script for automating each individual task. This dual approach will offer readers a clear understanding of executing actions within the AWS Console while simultaneously illustrating the equivalent infrastructure setup in Terraform for automation.

Task 1: Creating VPCs, Subnets, Route Tables and Internet Gateways

Virtual Private Clouds (VPCs) form the foundation of network isolation within AWS, allowing segmentation of resources. We'll define VPCs with unique CIDR blocks, create subnets spread across multiple availability zones to ensure high availability, and establish Internet Gateways for external connectivity.

Creating VPC with different CIDR IP Range
AWS Management Console:

Log in to the AWS Management Console.
Navigate to the VPC service by either searching for "VPC" in the AWS Management Console's search bar or finding it under the "Networking & Content Delivery" section.
Once in the VPC dashboard, click on "Your VPCs" from the left-hand navigation pane.
Click on the "Create VPC" button.
Provide a name vpc1 for the VPC and enter the CIDR block range, for instance, "10.1.0.0/16".
Choose an availability zone, for example, "us-west-2".
Click "Create" to set up the first VPC.
Follow the same steps for other 2 VPC (vpc2,vpc3) creation as per first VPC, but ensure that you use distinct CIDR blocks for the second and third VPCs. For example, "10.2.0.0/16" and "10.3.0.0/16" respectively, while selecting the same availability zone ("us-west-2" in this case)

Creating 3 Subnets Associated with Different VPCs

In the VPC dashboard, select "Subnets" from the left-hand navigation pane.
Click on "Create Subnet".
Choose the appropriate VPC in the "VPC" dropdown list.
Provide a name subnet1 for the subnet and select the availability zone.
Enter the CIDR block for the first subnet, ensuring it falls within the CIDR block range of the chosen VPC.
Click "Create" to generate the first subnet.
Repeat the same process, but for each new subnet, select a different VPC in the "VPC" dropdown list, name the subnet, select an availability zone, and define unique CIDR blocks for each of the second and third subnets.
Ensure each subnet belongs to its respective VPC and the CIDR blocks don't overlap with other subnets or VPCs.
Click "Create" to generate the additional subnets.

Creating Internet Gateways and Attaching to VPCs:

In the VPC dashboard, select "Internet Gateways" from the left-hand navigation pane.
Click on "Create Internet Gateway".
Provide a name igw1 for the Internet Gateway and click "Create" to generate the first Internet Gateway.
Attaching the First Internet Gateway to a VPC:
After creating the Internet Gateway, select it from the list.
From the "Actions" menu, click on "Attach to VPC".
Choose the VPC to which you want to attach this Internet Gateway and click "Attach".

Creating the Second igw2 and Third igw3 Internet Gateways:

Repeat the above steps to create two additional Internet Gateways, providing unique names for each gateway.
Attaching Each Internet Gateway to Its Respective VPC:
Select each newly created Internet Gateway.
From the "Actions" menu, click "Attach to VPC".
Choose the corresponding VPC for each Internet Gateway and click "Attach".

Creating Route Tables and Associating Subnets:

In the VPC dashboard, select "Route Tables" from the left-hand navigation pane.
Click on "Create Route Table".
Provide a name for the route table and select the VPC with which this route table will be associated.
Click "Create" to generate the first route table. Associating the First Route Table with Subnet:
Under the "Route Tables" section, select the newly created route table.
Click on the "Subnet Associations" tab.
Click on "Edit subnet associations".
Choose the first subnet to associate with this route table and click "Save". Creating the Second and Third Route Tables
Repeat the above steps to create two additional route tables, each associated with a different VPC.
Make sure you select unique names for the route tables and associate them with their respective VPCs.
Associating Subnets with the Remaining Route Tables:
For each additional route table, go to the "Subnet Associations" tab and click "Edit subnet associations".
Choose the corresponding subnet to associate with each route table and save the changes.

Terraform script:

provider "aws" {
  region = "us-west-2" # Change to your preferred AWS region
}

# Create VPCs
resource "aws_vpc" "vpc1" {
  cidr_block = "10.1.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
}

resource "aws_vpc" "vpc2" {
  cidr_block = "10.2.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
}

resource "aws_vpc" "vpc3" {
  cidr_block = "10.3.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
}

# Create Internet Gateways
resource "aws_internet_gateway" "igw1" {
  vpc_id = aws_vpc.vpc1.id
}

resource "aws_internet_gateway" "igw2" {
  vpc_id = aws_vpc.vpc2.id
}

resource "aws_internet_gateway" "igw3" {
  vpc_id = aws_vpc.vpc3.id
}

resource "aws_route_table" "route_table1" {
  vpc_id = aws_vpc.vpc1.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "aws_internet_gateway.igw1.id"  # Replace with the actual internet gateway ID for VPC 1
  }
}
resource "aws_route_table" "route_table2" {
  vpc_id = aws_vpc.vpc2.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "aws_internet_gateway.igw2.id"  # Replace with the actual internet gateway ID for VPC2
  }
}
resource "aws_route_table" "route_table2" {
  vpc_id = aws_vpc.vpc3.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "aws_internet_gateway.igw3.id"  # Replace with the actual internet gateway ID for VPC3
  }
}

# Create Subnets and associating with route tables
resource "aws_subnet" "subnet1" {
  vpc_id     = aws_vpc.vpc1.id
  cidr_block = "10.0.1.0/24"
  availability_zone = "us-west-2a"
  route_table_id = aws_route_table.route_table1.id
}

resource "aws_subnet" "subnet2" {
  vpc_id     = aws_vpc.vpc2.id
  cidr_block = "10.1.1.0/24"
  availability_zone = "us-west-2b"
  route_table_id = aws_route_table.route_table2.id
}

resource "aws_subnet" "subnet3" {
  vpc_id     = aws_vpc.vpc3.id
  cidr_block = "10.2.1.0/24"
  availability_zone = "us-west-2c"
  route_table_id = aws_route_table.route_table3.id

# Create Internet Gateway Attachments
resource "aws_vpc_attachment" "igw_attachment1" {
  vpc_id = aws_vpc.vpc1.id
  internet_gateway_id = aws_internet_gateway.igw1.id
}

resource "aws_vpc_attachment" "igw_attachment2" {
  vpc_id = aws_vpc.vpc2.id
  internet_gateway_id = aws_internet_gateway.igw2.id
}

resource "aws_vpc_attachment" "igw_attachment3" {
  vpc_id = aws_vpc.vpc3.id
  internet_gateway_id = aws_internet_gateway.igw3.id
}

Task 2: Provisioning EC2 Instances in Different VPCs**

EC2 instances play a pivotal role in hosting applications and services within VPCs. With Terraform, we'll automate the deployment of EC2 instances across these VPCs, setting up Apache web servers using user data. This automated configuration simplifies the initialization process, ensuring that each instance is ready for use upon launch.

AWS Management Console:
Navigate to the EC2 dashboard.

Click on "Launch Instance".
Choose an Amazon Machine Image (AMI) based on your requirements.
Select the instance type and configure the instance details, including the VPC and subnet.
Add storage, configure security groups, and define user data scripts for installation or configurations.
Launch the instance.

Terraform Script:

# Create EC2 Instances
resource "aws_instance" "ec2_instance1" {
  ami           = "ami-12345678" # Replace with RHEL desired AMI ID
  instance_type = "t2.micro" # Change as needed
  subnet_id     = aws_subnet.subnet1.id
  user_data     = <<-EOF
    #!/bin/bash
sudo yum update -y
sudo yum install -y httpd
sudo service httpd start
sudo chkconfig on
sudo chmod -R 755 /var/www/

cat <<HTML > /var/www/html/index.html
<!DOCTYPE html>
<html>
  <head>
    <title>Server Details</title>
  </head>
  <body>
    <h1>Server Details</h1>
    <p><strong>Hostname:</strong> hostname</p>
    <p><strong>IP Address:</strong> $(hostname -I | awk '{print $1}')</p>
  </body>
</html>
HTML
  EOF
}

# Repeat the above code for other 2 aws_instance with different subnet selection.

Task 3:Implementing VPC Transit Gateway**

Implementing VPC Transit Gateway and Route Table Associations

In the Transit Gateway dashboard, select "Transit Gateways" from the left-hand navigation pane.
Click on "Create Transit Gateway".
Provide a name for the Transit Gateway and configure other settings as needed.
Click "Create" to generate the Transit Gateway.
Attach VPC,Subnet in transit gateway attchment

Terraform Script

# Create Transit Gateway
resource "aws_ec2_transit_gateway" "my_transit_gateway" {
  description = "My Transit Gateway"
}

# Create Transit Gateway Route Tables
resource "aws_ec2_transit_gateway_route_table" "route_table1" {
  transit_gateway_id = aws_ec2_transit_gateway.my_transit_gateway.id
}

# Associate Route Tables with Attachments
resource "aws_ec2_transit_gateway_route_table_association" "route_table_association1" {
  transit_gateway_attachment_id = aws_ec2_transit_gateway_vpc_attachment.vpc_attachment_ec2_1.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.route_table1.id
}

# Repeat the above code for other Transit Gateway route tables and associations

Conclusion
Setting up a VPC Transit Gateway and deploying EC2 instances across VPCs with Terraform simplifies infrastructure management, ensuring scalability and security. This blog post aimed to guide you through the process and explain the importance of using Infrastructure as Code for AWS setups. We encourage further exploration into Terraform's capabilities for automating and managing AWS infrastructure.

A Step-by-Step Guide to AWS EC2 Auto Scaling

yadanaresh — Tue, 31 Oct 2023 18:05:16 +0000

Amazon Web Services (AWS) Elastic Compute Cloud (EC2) is a powerful and flexible cloud computing service that allows you to run virtual machines in the cloud. However, managing the capacity of your EC2 instances manually can be a daunting and time-consuming task. This is where AWS EC2 Auto Scaling comes into play. In this step-by-step guide, we will explore the concept of auto scaling and show you how to set up and configure AWS EC2 Auto Scaling to ensure that your application can handle varying workloads efficiently.

What is AWS EC2 Auto Scaling?

AWS EC2 Auto Scaling is a feature that allows you to automatically adjust the number of EC2 instances in your fleet to match your desired capacity. It is designed to help you maintain high availability and cost efficiency by automatically adding or removing instances based on the defined scaling policies and conditions. This ensures that your application can handle varying levels of traffic without manual intervention.

Prerequisites

Before you begin, make sure you have an AWS account and are logged in to the AWS Management Console.

Sign in to AWS Console

To get started with AWS EC2 Auto Scaling, sign in to your AWS Management Console. If you don't have an AWS account, you can sign up for one here.

Step 1: Create a Virtual Private Cloud (VPC)

Go to the AWS Management Console.
Navigate to the VPC service.
Click on "Create VPC."
Provide a name for your VPC and specify the IP address range.
Click "Create VPC."

Step 2: Create Two Subnets

In the VPC service, go to "Subnets."
Click "Create Subnet."
Choose your VPC.
Specify a name for the subnet.
Select the availability zone (e.g., us-east-1a).
Set the IP range for the subnet.
Click "Create Subnet."
Repeat the process for the second subnet (e.g., us-east-1b).

Step 3: Create an Internet Gateway

In the VPC service, go to "Internet Gateways."
Click "Create Internet Gateway."
Provide a name for the internet gateway.
Click "Create Internet Gateway."
Select the newly created internet gateway and attach it to your VPC.

Step 3.1: Create a Route Table

In the VPC service, go to "Route Tables."
Click "Create Route Table."
Provide a name for the route table.
Choose your VPC.
Click "Create Route Table."

Step 3.2: Associate Subnets with Route Table

Select the newly created route table.
In the "Subnet Associations" tab, click "Edit subnet associations."
Associate both of your subnets with the route table.

Step 3.3: Edit Route with Destination

In the "Routes" tab of the route table, click "Edit routes."
Add a route with a destination of "0.0.0.0/0" and select the internet gateway you created.
Click "Save routes."

Step 4: Create a Target Group

Go to the EC2 service.
Under the "Load Balancing" section, select "Target Groups."
Click "Create target group."
Provide a name for the target group.
Specify the protocol and port your application uses.
Choose your VPC.
Configure health checks as needed.
Click "Create."

Step 5: Create an Application Load Balancer

In the EC2 service, under the "Load Balancing" section, select "Load Balancers."
Click "Create Load Balancer."
Choose "Application Load Balancer."
Configure listeners and routing as necessary.
Select the two subnets you created.
Configure security groups to allow HTTP traffic and open it to the world.
Attach the existing target group.
Click "Create."

Step 6: Create Security Groups

In the EC2 service, navigate to "Security Groups."
Create two security groups: one for the Application Load Balancer and another for the EC2 instances launched by Auto Scaling.
Allow HTTP (port 80) for the Application Load Balancer security group and allow it to be accessed from anywhere.
For the EC2 instances, allow HTTP (port 80) and SSH (port 22) access, and make them accessible to everyone.

Step 7: Create a Launch Template

In the EC2 service, go to "Launch Templates."
Click "Create launch template."
Select the Amazon Machine Image (AMI) you want to use (e.g., RHEL 9).
Configure the instance type (e.g., t2.micro).
Ensure that you enable "Auto-assign Public IP."
Attach the security group you created for EC2 instances.
Add your SSH key pair.
In the user data section, add any necessary startup scripts or configurations.

Step 7.1: Create an Auto Scaling Group

In the EC2 service, under the "Auto Scaling" section, select "Auto Scaling Groups."
Click "Create Auto Scaling group."
Choose the launch template you created.
Configure network settings by selecting the two subnets you created.
Attach the existing load balancer and target group.
Enable health checks and monitoring.
Set the desired, minimum, and maximum group sizes (e.g., 2, 1, 3).
Leave scaling policies as "none" for this basic setup.
Click through the remaining options and create the auto scaling group.

Step 8: Test and Monitor

After configuring auto scaling, it's crucial to thoroughly test it to ensure that it behaves as expected. You can simulate traffic spikes or resource failures to see if the auto scaling group responds correctly.

Additionally, use AWS CloudWatch to monitor your instances and scaling activities. Create custom CloudWatch alarms to trigger actions based on specific metrics.

Fine-Tune and Optimise

Once your auto scaling group is up and running, periodically review your scaling policies and configurations to optimize cost and performance. Adjust the scaling thresholds, instance types, and other settings as needed based on your application's usage patterns.

"For your convenience, I have prepared a Terraform script that fulfills the aforementioned requirements. This script orchestrates the setup of an AWS infrastructure, incorporating essential components like a VPC, subnets, an internet gateway, security groups, load balancers, launch templates, and an auto-scaling group for EC2 instances."

Here is Git repo: https://github.com/yadanaresh/aws-ec2-autoscale.git

Conclusion

AWS EC2 Auto Scaling is a powerful tool for managing the capacity of your EC2 instances automatically. By following this step-by-step guide, you can set up and configure auto scaling to ensure your application can handle varying workloads efficiently, improve availability, and optimize costs. Remember that the key to successful auto scaling is proper planning and monitoring to align your application's performance with your business needs.

Automating S3 File Upload Email Notifications with AWS Lambda and SNS

yadanaresh — Sat, 21 Oct 2023 02:28:23 +0000

Introduction
In the fast-paced world of cloud computing, automation is the key to efficiency and reliability. One common scenario is the need to receive notifications whenever files are uploaded to an Amazon S3 bucket. In this blog post, we'll explore how to set up an AWS Lambda function to automatically send SNS notifications whenever a file is uploaded to an S3 bucket.

Prerequisites
Before we dive into the technical details, here are some prerequisites to get started:

An AWS account.
An S3 bucket where you will upload files.
An SNS topic for sending notifications.
Basic knowledge of AWS services and Python.
Setting Up the AWS Lambda Function
Creating the Lambda Function

To create an AWS Lambda function in Python that sends a push notification using Amazon SNS whenever a file is uploaded to an S3 bucket, you can follow these steps.

Create an S3 Bucket:

You can follow on-screen guidance to create an S3 bucket. After successfully creating the bucket, make a note of the bucket name. You will need to update the lambda function code with this bucket name.

Bucket Name: s3-lambda-sns1
AWS Region : US East(N. Virginia) us-east-1 (Note, your Lambda,SNS services should be part of same region)

The rest of the options should remain as their default settings, there is no need for any additional changes.

Click on Create Bucket to create the bucket.

Create an SNS Topic:

Create an SNS(Simple Notification Service) topic that will be used to send notifications. Note the topic ARN to update in lambda code [YOUR_SNS_TOPIC_ARN].

The rest of the options should remain as their default settings, there is no need for any additional changes.

SNS is services created.

create Subscription to receive the email alerts

Note: Once you enter your email address, Amazon will send a confirmation email to the provided address. you should verify it, as email alerts will not be triggered until the email address is verified.

Create an IAM Role:

Create an IAM role for your Lambda function that allows it to interact with S3 and SNS. Attach policies for AmazonS3FullAccess and AmazonSNSFullAccess.

Create the Lambda Function:

Create a Lambda function with the following Python code.

code for your reference:

import json
import boto3

s3 = boto3.client('s3')
sns = boto3.client('sns')

def lambda_handler(event, context):
    # Retrieve information about the S3 event
    records = event.get('Records', [])

    for record in records:
        if record['eventName'] == 'ObjectCreated:Put':
            # Extract the bucket and object key from the S3 event
            bucket_name = record['s3']['bucket']['name']
            object_key = record['s3']['object']['key']
# S3 trigger is added to the Lambda function, S3 values will be automatically loaded
            # Send a notification using SNS
            message = f"File '{object_key}' has been uploaded to the S3 bucket '{bucket_name}'."
            subject = "S3 File Upload Notification"

            sns.publish(
                TopicArn='YOUR_SNS_TOPIC_ARN',
                Subject=subject,
                Message=message
            )

    return {
        'statusCode': 200,
        'body': json.dumps('Notification sent successfully!')
    }

Testing:

You can test the Lambda function by uploading a file to the specified S3 bucket. You should receive an email notification through the SNS topic when the Lambda function is triggered.

Email alert is triggered after file uploaded to S3

Remember to set up proper error handling, logging, and security best practices in your Lambda function to ensure it works reliably in a production environment. Also, ensure that the IAM role assigned to the Lambda function has appropriate permissions for S3 and SNS.

Conclusion
In this blog post, we've demonstrated how to automate the process of sending SNS notifications whenever a file is uploaded to an S3 bucket using AWS Lambda. This approach is a valuable addition to your cloud automation toolkit, providing real-time alerts for file uploads.

By following the steps outlined in this blog post, you can easily set up your own S3 file upload notification system and adapt it to your specific use case. Automation not only saves time but also enhances the reliability and responsiveness of your AWS infrastructure.

Stay tuned for more AWS and cloud-related tips and tutorials!