DEV Community: Revathi Joshi

How to retrieve DynamoDB items using secrets stored in AWS Secrets Manager with AWS Lambda - 2

Revathi Joshi — Wed, 13 Mar 2024 21:05:51 +0000

Please read my previous article - How to retrieve DynamoDB items using secrets stored in AWS Secrets Manager with AWS Lambda - 1

Let’s get started!

Please visit my GitHub Repository for DynamoDB articles on various topics being updated on constant basis.

Pre-requisites:

AWS user account with admin access, not a root account.
Create an IAM role

Resources Used:

What is Amazon DynamoDB?

What is AWS Secrets Manager?

What is AWS Lambda?

Steps for implementation to this project:

Part 2 6. Create a Secret Manager to Store Access key and Secret Access keys

Next
Next
Store

7. Write a Lambda code to create DynamoDB Items by retrieving the access keys from Secrets Manager.

Click on Functions at the left side and select the Function you created.
Select the Code tab under the lambda myFunction
Copy the file3, replace it with the existing code and and change the Secret ARN in file3

import boto3
import json
import base64

def lambda_handler(event, context):
    secret_name = "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:mySecret-q7slUY"
    # Create a Secrets Manager client
    secretClient = boto3.client(
        service_name = 'secretsmanager',
        region_name = 'us-east-1'
    )

    get_secret_value_response = secretClient.get_secret_value(
        SecretId=secret_name
    )
    secret = get_secret_value_response['SecretString']
    Table_name = 'myTable2'

    print('DynamoDB Table creation started.')

    dynamodb = boto3.resource(
        'dynamodb',
        aws_access_key_id = json.loads(secret).get('Access Key'),
        aws_secret_access_key = json.loads(secret).get('Secret Access Key'),
        region_name = 'us-east-1'
    )

    student_table = dynamodb.create_table(
        TableName = Table_name,
        KeySchema = [
            {
                'KeyType': 'HASH',
                'AttributeName': 'StudId'
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'StudId',
                'AttributeType': 'N'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 2,
            'WriteCapacityUnits': 2
        }
    )

    # Wait until the Table gets created
    student_table.meta.client.get_waiter('table_exists').wait(TableName = Table_name)
    print('DynamoDB Table Creation Completed.')

    print('Insert Student data to table started.')
    # Insert 1st item into DynamoDB table
    table = dynamodb.Table(Table_name)
    table.put_item(
    Item = {
            'StudId': 100,
            'FirstName': 'Rev1',
            'LastName': 'Joshi1',
            'Dept': 'Science',
            'Age': 11
        }
    )



    # Insert 2nd item into DynamoDB table
    table.put_item(
    Item = {
            'StudId': 200,
            'FirstName': 'Rev2',
            'LastName': 'Joshi2',
            'Dept': 'Science',
            'Age': 22
        }
    )



    # Insert 3rd item into DynamoDB table
    table.put_item(
    Item = {
            'StudId': 300,
            'FirstName': 'Rev3',
            'LastName': 'Joshi3',
            'Dept': 'Science',
            'Age': 33
        }
    )
    print('Insert Student data to table Completed.')

Deploy
Test
Output

8. View DynamoDB Table created in console.

Select the table myTable2 and click on Explore table items Button in the right side

9. Write a lambda code to view the table items using a secret manager.

Click on Functions at the left side and select the Function you created.
Select the Code tab under the lambda myFunction
Copy the file4, replace it with the existing code and change the Secret ARN in file4

import boto3
import json
import base64

def lambda_handler(event, context):
    secret_name = "arn:aws:secretsmanager:us-east-1:xxxxxxxxxxxx:secret:mySecret-q7slUY"
    # Create a Secrets Manager client
    secretClient = boto3.client(
        service_name = 'secretsmanager',
        region_name = 'us-east-1'
    )

    get_secret_value_response = secretClient.get_secret_value(
        SecretId=secret_name
    )

    secret = get_secret_value_response['SecretString']
    Table_name = 'myTable2'

    # Create a DynamoDB table
    print('DynamoDB Table creation started.')

    dynamodb = boto3.resource(
        'dynamodb',
        aws_access_key_id = json.loads(secret).get('Access Key'),
        aws_secret_access_key = json.loads(secret).get('Secret Access Key'),
        region_name = 'us-east-1'
    )

    # Connect to table & Scan the entire table
    table = dynamodb.Table(Table_name)
    response = table.scan()

    print('---------------------------------------')
    print('------------STUDENT DETAILS------------')
    print('---------------------------------------')
    for item in response['Items']:
        print('Student Id : ', item['StudId'])
        print('Student Name : ', item['FirstName'], ' ', item['LastName'])
        print('Student Department : ', item['Dept'])
        print('Student Age : ', item['Age'])
        print('_______________________________')
    print('---------------------------------------')

Deploy
Test
Output

Cleanup

Delete Lambda Function
Delete DynamoDB tables
Delete Secrets

What we have done so far

Successfully retrieved items from DynamoDB tables using secrets stored in AWS Secrets Manager with AWS Lambda function.

How to retrieve DynamoDB items using secrets stored in AWS Secrets Manager with AWS Lambda - 1

Revathi Joshi — Mon, 11 Mar 2024 22:28:44 +0000

AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, Encryption keys, SSH keys and other secrets throughout their lifecycles.

You replace hard-coded credentials with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them. And AWS Secrets Manager eliminates the need to hardcode sensitive information in plain text.
It provides default encryption to your secrets stored in AWS Secrets Manager.
Secrets Manager offers pay as you go pricing.

AWS DynamoDB is a fast and fully managed NoSQL database designed for applications that need consistent, single-digit millisecond latency at any scale.

It is a fully managed database and it supports both document and key value data models.
It has a very flexible data model. This means that you don't need to define your database schema upfront. Yet it provides fast, reliable and predictable performance.
DynamoDB tables consist of:
Items (Similar to a row of data in a table).
Attributes (Similar to a column of data in a table).
Supports key-value and document data structures.
Key = the name of the data. Value = the data itself.
Document can be written in JSON, HTML or XML.

AWS Lambda is a compute service that lets you run code without provisioning or managing servers.

With Lambda, you can run code for virtually any type of application or backend service.

Let’s get started!

Please visit my GitHub Repository for DynamoDB articles on various topics being updated on constant basis.

Objectives:

I have divided this article into 2 parts for understanding this process better.

Part 1

1. Create an IAM Role

2. Create a lambda Function

3. Write a lambda hard-code access keys to create DynamoDB tables and Items.

4. View DynamoDB Table created in console.

5. Write a lambda code to return the table data.

Part 2

6. Create a Secret Manager to Store Access key and Secret Access keys

7. Write a Lambda code to create DynamoDB Items by retrieving the access keys from Secrets Manager.

8. View DynamoDB Table created in console.

9. Write a lambda code to view the table items using a secret manager.

Pre-requisites:

AWS user account with admin access, not a root account.
IAM role

Resources Used:

What is Amazon DynamoDB?

What is AWS Secrets Manager?

What is AWS Lambda?

Steps for implementation to this project:

1. Create an IAM Role

Create role

2. Create a lambda Function

Create function

Select configuration tab in lower side and then click on Edit tab

3. Write a lambda hard-code access keys to create DynamoDB tables and Items.

Copy the code from file1and replace with existing code.
Note : change the AWS_Access_Key and AWS_Secret_Access_Key in file1.

import json
import boto3

def lambda_handler(event, context):
    # Input values
    Table_name = 'myTable1'
    AWS_Access_Key = 'xxxxxxxxxxxxxxxxxxxx'                
    AWS_Secret_Access_Key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

    # Create a DynamoDB table
    print('DynamoDB Table creation started.')

    dynamodb = boto3.resource(
        'dynamodb',
        aws_access_key_id = AWS_Access_Key,
        aws_secret_access_key = AWS_Secret_Access_Key,
        region_name = 'us-east-1'
    )

    student_table = dynamodb.create_table(
        TableName = Table_name,
        KeySchema = [
            {
                'KeyType': 'HASH',
                'AttributeName': 'StudId'
            }
        ],
        AttributeDefinitions=[
            {
                'AttributeName': 'StudId',
                'AttributeType': 'N'
            }
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 2,
            'WriteCapacityUnits': 2
        }
    )  

    # Wait until the Table gets created
    student_table.meta.client.get_waiter('table_exists').wait(TableName = Table_name)
    print('DynamoDB Table Creation Completed.')

    print('Insert Student data to table started.')
    # Insert 1st item into DynamoDB table
    table = dynamodb.Table(Table_name)
    table.put_item(
    Item = {
            'StudId': 100,
            'FirstName': 'Rev1',
            'LastName': 'Joshi1',
            'Dept': 'Science',
            'Age': 11
        }
    )



    # Insert 2nd item into DynamoDB table
    table.put_item(
    Item = {
            'StudId': 200,
            'FirstName': 'Rev2',
            'LastName': 'Joshi2',
            'Dept': 'Science',
            'Age': 22
        }
    )



    # Insert 3rd item into DynamoDB table
    table.put_item(
    Item = {
            'StudId': 300,
            'FirstName': 'Rev3',
            'LastName': 'Joshi3',
            'Dept': 'Science',
            'Age': 33
        }
    )
    print('Insert Student data to table Completed.')

Click on the Test button to run the code.
Output

4. View DynamoDB Table created in console.

Select the table and click on Explore table items Button in the right side

5. Write a lambda code to return the table data.

Click on Functions at the left side and select the Function you created.
Select the Code tab under the lambda myFunction
Copy the file2 and replace it with the existing code.

import json
import boto3

def lambda_handler(event, context):
    # Input values
    Table_name = 'myTable1'
    AWS_Access_Key = 'xxxxxxxxxxxxxxxxxxxx'
    AWS_Secret_Access_Key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

    # Create a DynamoDB table
    print('DynamoDB Table creation started.')

    dynamodb = boto3.resource(
        'dynamodb',
        aws_access_key_id = AWS_Access_Key,
                  aws_secret_access_key = AWS_Secret_Access_Key,
        region_name = 'us-east-1'
    )

    # Connect to table & Scan the entire table
    table = dynamodb.Table(Table_name)
    response = table.scan()

    print('---------------------------------------')
    print('------------STUDENT DETAILS------------')
    print('---------------------------------------')
    for item in response['Items']:
        print('Student Id : ', item['StudId'])
        print('Student Name : ', item['FirstName'], ' ', item['LastName'])
        print('Student Department : ', item['Dept'])
        print('Student Age : ', item['Age'])
        print('_______________________________')
    print('---------------------------------------')

Deploy
Test
Output

How to move an Amazon RDS DB instance from an Amazon Virtual Private Cloud (Amazon VPC) to a new VPC

Revathi Joshi — Sat, 09 Mar 2024 23:12:24 +0000

In this article, I am going to show you how to migrate an Amazon Relational Database Service (Amazon RDS) DB instance from one virtual private cloud (VPC) to another in the same AWS account. This is useful especially when you want to place your application stack and database in different VPCs for security reasons.

But this sort of approach has few limitations.

When you move the RDS DB instance to a new network and configure the new VPC, the DB instance reboots. So, change the VPC during a planned change window that is outside the RDS weekly maintenance window.
The DB instance you’re migrating must be a single instance with no standby. It must not be a member of a cluster.
Amazon RDS must not be in multiple Availability Zones. Convert the DB instance to a single AZ, and then convert it back to a Multi-AZ DB instance after moving to the new VPC.
Amazon RDS must not have any read replicas. Remove the read replicas, and then add read replicas after the DB instance is moved to the new VPC.
The subnet group created in the target VPC must have subnets from the Availability Zone where the source database is running. If the AZs are different, then the operation fails.

Let’s get started!

Please visit my GitHub Repository for RDS articles and GitHub Repository for VPC articles on various topics being updated on constant basis.

Objectives:

1. Create RDS Database Instance

2. Create a new VPC, Subnets, Route Tables, VPC Security Group

3. Create a new DB subnet group

4. Modify the Amazon RDS DB instance to use the new DB subnet group

Pre-requisites:

AWS user account with admin access, not a root account.
an IAM role with permissions required for the VPC, subnets, and Amazon RDS console

Resources Used:

Amazon RDS documentation
AWS IAM Documentation
Amazon VPC documentation

Steps for implementation to this project:

1. Create RDS Database Instance

2. Create a new VPC, Subnets, Route Tables, VPC Security Group

Create a new VPC

Create new Subnets

Create subnet

Create a new Route Table

Create a new VPC Security Group

Create security group

3. Create a new DB subnet group.

Create

4. Modify the Amazon RDS DB instance to use the new DB subnet group

Continue 12

When the migration to the target VPC is complete, the target VPC's default security group is assigned to the Amazon RDS DB instance. 15

choose Events in the left navigation pane.
Confirm that the process moved the DB instance to the target VPC 17

Cleanup

delete RDS instance
delete VPC and its components

What we have done so far

I have successfully moved an Amazon RDS DB instance to a new VPC, by first changing its subnet group.

How to minimize your cloud spending on AWS resources using Systems Manager Automation documents

Revathi Joshi — Fri, 08 Mar 2024 23:50:10 +0000

An easy way to accomplish unneccessary spending and to minimize infrastructure when it’s not under heavy use, is - by turning off Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Relational Database Service (Amazon RDS) instances for workloads outside of business hours. For workloads that cannot be turned off (due to dependencies on other systems), downsizing instance types is a good alternative.

Applying these measures you can save companies up to 70% in infrastructure costs.

In this post, I will show you how to use AWS Systems Manager Automation documents to turn off your Amazon EC2 and Amazon RDS instances. These Automation documents can then be scheduled for known low usage periods such as nights, holidays, and weekends.

You can use pre-defined Automation documents (prefixed with AWS) or define your own. These can be invoked on a schedule or via a trigger. For more information, check Systems Manager automation.

Let’s get started!

Please visit my GitHub Repository for EC2 articles on various topics being updated on constant basis.
Please visit my GitHub Repository for RDS articles on various topics being updated on constant basis.

For my article, I am going to use pre-defined Automation documents:

AWS-StartEC2Instance
AWS-StopEC2Instance
AWS-StartRdsInstance

Objectives:

1. Create an RDS database

2. Create EC2 Instance

3. Create an IAM Role

4. Add an Inline Policy to the IAM Role to allow Systems Manager to manage RDS Instance

5. Create an Association - Scheduling through State Manager to stop EC2 Instance

6. Create an Association - Scheduling through State Manager to stop RDS database

Pre-requisites:

AWS user account with admin access, not a root account.
Create an IAM role

Resources Used:

AWS Systems Manager Automation
AWS Systems Manager
State Manager Associations
IAM
EC2
RDS

Steps for implementation to this project:

1. Create an RDS database

2. Create EC2 Instance

3. Create an IAM Role

Next 4

Create role

4. Add an Inline Policy to the IAM Role to allow Systems Manager to manage RDS Instance

Find out the ARN of your RDS database

Choose the role that you just created.
In the Permissions tab, choose Add inline policy, select the JSON tab, and replace the JSON content with the following code.
Make sure you replace both resource parameters with one or more ARNs of your databases

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "rds:StopDBInstance",
                "rds:StartDBInstance"
            ],
            "Resource": "arn:aws:rds:us-east-1:xxxxxxxxxxxx:cluster:database-1"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "rds:DescribeDBInstances",
            "Resource": "arn:aws:rds:us-east-1:xxxxxxxxxxxx:cluster:database-1"
        }
    ]
}

Create policy

5. Create an Association - Scheduling through State Manager to stop EC2 Instance

select the SSM Automation document named AWS-StopEC2Instance.
press Enter

For a single EC2, choose Simple execution.
in the Input parameters section enter the EC2 instance id
In the AutomationAssumeRole box, pick the role EC2toSystemsManager.

In the Specify schedule section, choose On Schedule and CRON schedule builder.
Under Association runs, choose the last option, then choose Day, enter 18 and 00. You should now have the following: “Every Day at 18:00”. (that means Stop the EC2 instance at 6:00 pm or 18:00).
To ensure that the association doesn’t run upon creation, choose the Apply association only at the next specified cron interval check box.

Create association
you should now see your association in the list.

After waiting for 10-15 min, stops the EC2 instance

6. Create an Association - Scheduling through State Manager to stop RDS database

Follow all the steps as you did for AWS-StopEC2Instance

Use AWS-StopRDSInstance

In the Specify schedule section, choose On Schedule and CRON schedule builder.
Under Association runs, choose the last option, then choose Day, enter 18 and 20. You should now have the following: “Every Day at 18:20”. (that means Stop the EC2 instance at 6:20 pm or 18:20).
To ensure that the association doesn’t run upon creation, choose the Apply association only at the next specified cron interval check box.

After waiting for 10-15 min, stops the RDS database

Cleanup

delete RDS Database
delete EC2 instance
delete Association

What we have done so far

I showed you how to cut off your cloud spending for stopping EC2, and Amazon RDS instances based on a schedule.

How to run commands remotely on an EC2 Instance with AWS Systems Manager

Revathi Joshi — Thu, 07 Mar 2024 23:14:13 +0000

Systems Manager is a good choice when you need to view operation data for groups of resources, automate operational actions, understand and control the current state of your resources, manage hybrid environments, and maintain security and compliance.

Using the run command, one of the automation features of Systems Manager, you can simplify management tasks by eliminating the need to use bastion hosts, SSH, or remote PowerShell.

In this article, I am going to show you how to update the packages on an EC2 instance. At first, you will create an Identity and Access Management (IAM) role, enable an agent on your instance that communicates with Systems Manager, then follow best practices by running the AWS-UpdateSSMAgent document to upgrade your Systems Manager Agent, and finally use Systems Manager to run a command on your instance.

Please visit my GitHub Repository for EC2 articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Create an Identity and Access Management (IAM) role

2. Create an EC2 instance

3. Update the Systems Manager Agent

4. Run a remote shell script

Pre-requisites:

AWS user account with admin access, not a root account.
Create an IAM role

Resources Used:

IAM
EC2
AWS Systems Manager

Steps for implementation to this project:

1. Create an Identity and Access Management (IAM) role

Create role

2. Create an EC2 instance

Attach a Systems Manager role to Amazon Elastic Compute Cloud (Amazon EC2) instances to make them managed nodes.

3. Update the Systems Manager Agent

Now that you have an EC2 instance running the Systems Manager agent, you can automate administration tasks and manage the instance. In this step, you run a pre-packaged command, called a document, that will upgrade the agent. It is best practice to update the Systems Manager Agent when you create a new instance.

type in AWS-UpdateSSMAgent
then press Enter
select the radio button on the left of AWS-UpdateSSMAgent.
This document will upgrade the Systems Management agent on the instance

you will see a page documenting your running command, and then overall success in green.

4. Run a remote shell script

Now that your EC2 instance has the latest Systems Manager Agent, you can upgrade the packages on the EC2 instance.

In this step, you will run a shell script through Run Command.

On the Run a command page, click in the search bar and
select, Document name prefix,
then click on Equals,
then type in AWS-RunShellScript
then press Enter

On the Command Parameters panel and insert the following command in the Commands text box:

sudo yum update

While your script is running remotely on the managed EC2 instance, the Overall status will be In Progress.

Then the Overall status will turn to Success.
When it does, scroll down to the Targets and outputs panel and select the Instance ID of your instance.
Your Instance ID will be different than the one pictured.

select the header of the Output panel to view the output of the update command from the instance.

Cleanup

Delete EC2 Instance

What we have done so far

Successfully created an EC2 instance and remotely run a command using AWS Systems Manager. You first set up a correct role/permissions through IAM. Next you launched an Amazon Linux instance that was preinstalled with the Systems Manager agent. Finally, you used Run Command to update the agent and remotely perform a yum update.

How to scan thru Amazon RDS for SQLServer error logs using metric filters

Revathi Joshi — Thu, 07 Mar 2024 01:48:22 +0000

SQL Server is a relational database management system developed by Microsoft. Amazon RDS for SQL Server makes it easy to set up, operate, and scale SQL Server deployments in the cloud. With Amazon RDS, you can deploy multiple editions of SQL Server including Express, Web, Standard and Enterprise, in minutes with cost-efficient and re-sizable compute capacity.

We can use Amazon CloudWatch metrics and Amazon Relational Database Service (Amazon RDS) event notifications to monitor different metrics and events.

In this post, I have created to receive Amazon Simple Notification Service (Amazon SNS) notifications for your RDS SQL Server instances - online or offline.

Please visit my GitHub Repository for RDS articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Create RDS for SQL Server

2. Publish Amazon RDS for SQL Server error logs to CloudWatch.

3. Create filter patterns for offline or online databases

4. Create alarms for the filtered metrics

5. Test the solution

Pre-requisites:

AWS user account with admin access, not a root account.
IAM Role

Resources Used:

Amazon RDS for Microsoft SQL Server
CloudWatch
Amazon Simple Notification Service

Steps for implementation to this project:

1. Create RDS for SQL Server

1

2

Wait for 4-5 min to finish creating the database.

2. Publish Amazon RDS for SQL Server error logs to CloudWatch.

Click on database-1

Click on Modify

In the Log exports section, choose the logs that you want to start publishing to CloudWatch.
Select Error log

3. Create filter patterns for offline or online databases

Open the CloudWatch console and from the Logs section, choose Log groups.
Select the Amazon RDS for SQL Server error log of your DB instance (/aws/rds/instance/database-1/error).

Under Actions, Choose Metric filters and choose Create metric filter.

For Filter pattern, type OFFLINE.

You can Test pattern using following steps:
Select log data to test from the dropdown in Test pattern section (in our case it is database-1.node1)
Click Test pattern
Test pattern will only work if one of the databases is in OFFLINE state and corresponding entry is available in the error log.
In our case, the database-1 is online, so nothing in Results section
Next
For Filter name, Metric namespace, and Metric name, enter OFFLINE Database(s).
For Metric value, enter 1.

Choose Next.
Choose Create metric filter.

Follow the same steps to create a metric filter for ONLINE databases. Adjust the filter pattern ONLINE, filter name Database(s) are ONLINE, metric namespace Database(s) are ONLINE, and metric name Database(s) are ONLINEaccordingly.

Click Next
Choose Create metric filter.

4. Create alarms for the filtered metrics

To create alarms for our filtered metrics, complete the following steps:
Select the filter OFFLINE Database(s) and choose Create alarm

Enter a metric name OFFLINE Database(s)
Choose Minimum for Statistic and set Period to 30 Seconds

For Threshold type, select Static.
Select Greater and enter 0.
Next

Select In alarm for Alarm state trigger.
For Send a notification to the following SNS topic, select Create new topic and enter a topic name OFFLINE-SQLDB-SNS-TOPIC.
Enter a valid email for receiving the notifications.
Choose Create topic, then
choose Next.

For Alarm name enter a name OFFLINE-ALARM.
Review the configuration, then choose Create alarm.

Now that the SNS topic is created, make sure you confirm the subscription by choosing the Confirm subscription link in the email.

Repeat the same steps to create alarms for ONLINE databases and adjust the conditions accordingly.
ONLINE-ALARM created

5. Test the solution

Cleanup

delete CloudWatch log group
delete RDS database instance
delete SNC topic and subscription

What we have done so far

I have demonstrated how to use metric filters to scan through Amazon RDS for SQL Server error logs and set up alerts as per your requirements.

How to troubleshoot Region Declaration and Outputs errors in Terraform

Revathi Joshi — Thu, 19 Oct 2023 22:12:43 +0000

How do you troubleshoot Terraform Configuration errors?

The primary method for interacting with Terraform is the HashiCorp Configuration Language (HCL).
When Terraform encounters an error in your configuration, it will report an error including line numbers and the type of issue found in the configuration.

I am going to show you how to troubleshoot the Terraform configuration that contains intentional errors which deploys an EC2 instance with underlying networking.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create 4 files - main.tf, variables.tf, outputs.tf and terrafprm.tfvars.

4. Fix all the Configuration Errors

5. Modified main.tf, variables.tf, and outputs.tf files, after fixing the configuration errors

6. Deploy Your Resources

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
Interpolation Syntax
Troubleshoot Terraform - Correct a variable interpolation error
Troubleshoot Terraform - Correct your outputs
learn-terraform-troubleshooting

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create 4 files - `main.tf`, `variables.tf`, `outputs.tf` and `terrafprm.tfvars`.

1. main.tf

terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region  = var.region
}

data "aws_ami" "linux" {
   most_recent = true
   owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  ami           = data.aws_ami.linux.id
  availability_zone = var.az_1a
  instance_type = var.instance_type
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = $var.name-mywebapp
  }
}

2. variables.tf

variable "regions" {
  description = "region"
}

variable "name" {
  description = "Value of the Name tag for the EC2 instance"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

3. outputs.tf

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.web_app.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.web_app.public.ip
}

output "instance_name" {
  description = "Tags of the EC2 instance"
  value       = aws_instance.web_app.tag.Name
}

4. terrafprm.tfvars

name = "rev"
region = "us-east-1"

4. Fix all the Configuration Errors

Errors

1. Correct the string variable interpolation error

cd terraform_files

Terraform format

terraform fmt

╷
│ Error: Invalid character
│ 
│   on main.tf line 42, in resource "aws_instance" "web_app":
│   42:     Name = $var.name-mywebapp
│ 
│ This character is not used within the language.
╵

╷
│ Error: Invalid expression
│ 
│   on main.tf line 42, in resource "aws_instance" "web_app":
│   42:     Name = $var.name-mywebapp
│ 
│ Expected the start of an expression, but found an invalid expression token.

in main.tf Update line 42 as follows to correct the variable interpolation error
replace

Name = ${var.name}-my webapp

with

Name = "${var.name}-my webapp"

terraform fmt

successful

Initiate the working directory

terraform init

2. Correct the Region Declaration Error
Validate the configuration

terraform validate

╷
│ Error: Reference to undeclared input variable
│ 
│   on main.tf line 14, in provider "aws":
│   14:   region = var.region
│ 
│ An input variable with the name "region" has not been declared. Did you mean "regions"?
╵

in variables.tf, Update line 1 as follows to correct the Region Declaration Error
replace

variable "regions" {

with

variable "region" {

3. Correct the Outputs Error
Validate the configuration

terraform validate

╷
│ Error: Unsupported attribute
│ 
│   on outputs.tf line 8, in output "instance_public_ip":
│    8:   value       = aws_instance.web_app.public.ip
│ 
│ This object has no argument, nested block, or exported attribute named "public".
╵
╷
│ Error: Unsupported attribute
│ 
│   on outputs.tf line 13, in output "instance_name":
│   13:   value       = aws_instance.web_app.tag.Name
│ 
│ This object has no argument, nested block, or exported attribute named "tag". Did you mean "tags"?

in outputs.tf, on line 8
replace

value       = aws_instance.web_app.public.ip

with

value       = aws_instance.web_app.public_ip

in outputs.tf, on line 13
replace

value       = aws_instance.web_app.tag.Name

with

value       = aws_instance.web_app.tags.Name

5. Modified `main.tf`, `variables.tf`, and `outputs.tf` files, after fixing the configuration errors

1. main.tf

terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region = var.region
}

data "aws_ami" "linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  ami               = data.aws_ami.linux.id
  availability_zone = var.az_1a
  instance_type     = var.instance_type
  user_data         = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = "${var.name}-mywebapp"
  }
}

2. variables.tf

variable "region" {
  description = "region"
}

variable "name" {
  description = "Value of the Name tag for the EC2 instance"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

3. outputs.tf

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.web_app.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.web_app.public.ip
}

output "instance_name" {
  description = "Tags of the EC2 instance"
  value       = aws_instance.web_app.tag.Name
}

6. Deploy Your Resources

terraform validate

terraform plan

terraform apply

enter yes to confirm deployment

EC2 instances - rev-mywebapp

Cleanup

terraform destroy

What we have done so far

We have successfully troubleshooted a string variable interpolation error, Region Declaration Error and output errors when formatting our configuration before we have deployed our infrastructure.

How to fix a Terraform Cycle Error

Revathi Joshi — Wed, 18 Oct 2023 22:11:33 +0000

What are Cycle errors?

Cycle errors are instances of circular logic in the Terraform dependency tree. Terraform analyzes the dependencies between resources in your infrastructure configuration to determine the order to perform your operations.

In this article, I am going to show you how Cycle error occurs when aws_security_group resources reference one another in their security_groups attributes. AWS cannot create the security groups because their configurations each reference the other group, which would not exist yet.

How to fix the Cycle error?

The fix for this Cycle error is to remove the mutually dependent security group rules in your configuration, leaving the two group resources without ingress attributes.
Instead of including the rules in the aws_security_group configuration, use the aws_security_group_rule resource and reference the security group IDs instead.
This avoids a cycle error because the provider will have AWS create both of the aws_security_group resources first, without interdependent rules.
It will create the rules next, and attach the rules to the groups last.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create 4 files - main.tf, variables.tf, outputs.tf and terrafprm.tfvars.

4. Initialize Your Working Directory

5. Fix the Cycle Error

6. Deploy Your Resources

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
Troubleshoot Terraform - Correct a cycle error
learn-terraform-troubleshooting

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create 4 files - `main.tf`, `variables.tf`, `outputs.tf` and `terrafprm.tfvars`.

1. main.tf
substitute vpc_id with your own VPC



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region  = var.region
}

data "aws_ami" "linux" {
   most_recent = true
   owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  for_each               = local.security_groups
  ami           = data.aws_ami.linux.id
  availability_zone = var.az_1a
  instance_type = var.instance_type
  vpc_security_group_ids = [each.value]
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = "${var.name}-mywebapp-${each.key}"
  }
}

 resource "aws_security_group" "sg_ping" {
   name = "Allow Ping"
   vpc_id = "<DUMMY VALUE>"

  ingress {
    from_port       = -1
    to_port         = -1
    protocol        = "icmp"
    security_groups = [aws_security_group.sg_8080.id]
  }
}

 resource "aws_security_group" "sg_8080" {
   name = "Allow 8080"
   vpc_id = "<DUMMY VALUE>"

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.sg_ping.id]
  }
}

locals {
  security_groups = {
    sg_ping   = aws_security_group.sg_ping.id,
    sg_8080   = aws_security_group.sg_8080.id,
  }

2. variables.tf



variable "region" {
  description = "region"
}

variable "name" {
  description = "Value of the Name tag for the EC2 instance"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

3. outputs.tf



output "instance_id" {
  description = "ID of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.id]
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.public_ip]
}

output "instance_name" {
  description = "Tags of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.tags.Name]
}

4. terrafprm.tfvars



name = "rev"
region = "us-east-1"

4. Initialize Your Working Directory



cd terraform_files

Terraform format



terraform fmt

Initiate the working directory



terraform init

Validate the configuration



terraform validate

get a cycle error for the security groups sg_8080 and sg_ping.

This cycle error you get due to the security groups co-dependent on each other, expecting the other to be already created and must both exist at the same time for successful creation.

5. Fix the Cycle Error

in main.tf - delete the ingress sections from the sg_ping and sg_8080 resource blocks.
Beneath the resource block of sg_ping, create a separate security group rule of type ingress named allow_ping



resource "aws_security_group_rule" "allow_ping"  {
     type = "ingress"
     from_port = -1
     to_port = -1
     protocol = "icmp"
     security_group_id = aws_security_group.sg_ping.id
     source_security_group_id = aws_security_group.sg_8080.id
}

Beneath the resource block of sg_8080, Create another security group rule of type ingress named allow_8080



resource "aws_security_group_rule" "allow_8080" {
     type = "ingress"
     from_port = 8080
     to_port = 8080
     protocol = "tcp"
     security_group_id = aws_security_group.sg_8080.id
     source_security_group_id = aws_security_group.sg_ping.id
}

main.tf looks like this



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region = var.region
}

data "aws_ami" "linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  for_each               = local.security_groups
  ami                    = data.aws_ami.linux.id
  availability_zone      = var.az_1a
  instance_type          = var.instance_type
  vpc_security_group_ids = [each.value]
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = "${var.name}-mywebapp-${each.key}"
  }
}

resource "aws_security_group" "sg_ping" {
  name   = "Allow Ping"
  vpc_id = "vpc-0da931f5deb73c9e2"
}

resource "aws_security_group_rule" "allow_ping"  {
     type = "ingress"
     from_port = -1
     to_port = -1
     protocol = "icmp"
     security_group_id = aws_security_group.sg_ping.id
     source_security_group_id = aws_security_group.sg_8080.id
}

resource "aws_security_group" "sg_8080" {
  name   = "Allow 8080"
  vpc_id = "vpc-0da931f5deb73c9e2"
}

resource "aws_security_group_rule" "allow_8080" {
     type = "ingress"
     from_port = 8080
     to_port = 8080
     protocol = "tcp"
     security_group_id = aws_security_group.sg_8080.id
     source_security_group_id = aws_security_group.sg_ping.id
}

locals {
  security_groups = {
    sg_ping = aws_security_group.sg_ping.id,
    sg_8080 = aws_security_group.sg_8080.id,
  }
}

Validate the configuration:



terraform validate

You should receive a success message stating the configuration is valid.

6. Deploy Your Resources

Create the Terraform plan



terraform plan

Create the resources



terraform apply

enter yes to confirm deployment

EC2 instances - rev-mywebapp-sg_8080 and rev-mywebapp-sg_ping

Cleanup



terraform destroy

What we have done so far

We have successfully fixed the cycle error and deployed our resources.

How to fix a Terraform String Variable Interpolation Error

Revathi Joshi — Tue, 17 Oct 2023 00:03:48 +0000

In this article. I am going to show you how fix the string variable interpolation error.

Embedded within strings in Terraform, whether you're using the Terraform syntax or JSON syntax, you can interpolate other values. These interpolations are wrapped in ${}, such as ${var.name}.

The interpolation syntax is powerful and allows you to reference variables, attributes of resources, call functions, etc.

Use the var. prefix followed by the variable name. For example, ${var.name} will interpolate the name variable value.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create 4 files - main.tf, variables.tf, outputs.tf and terrafprm.tfvars.

4. Interpolation Error

5. Fix Interpolation Error

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
Interpolation Syntax
Troubleshoot Terraform - Correct a variable interpolation error

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create 4 files - `main.tf`, `variables.tf`, `outputs.tf` and `terrafprm.tfvars`.

1. main.tf



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region  = var.region
}

data "aws_ami" "linux" {
   most_recent = true
   owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_server" {
  ami           = data.aws_ami.linux.id
  instance_type = var.instance_type
  availability_zone = var.az_1a
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF

  tags = {
    Name = $var.name-myweb
  }
}

2. variables.tf



variable "region" {
  description = "region"
}

variable "name" {
  description = "Value of the Name tag for the EC2 instance"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

3. outputs.tf



output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.web_server.id
}

output "instance_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.web_server.public_ip
}

output "instance_name" {
  description = "Name of the EC2 instance"
  value       = aws_instance.web_server.tags.Name
}

4. terrafprm.tfvars



name = "rev"
region = "us-east-1"

4. Interpolation Error

cd terraform-files
An error is expected after running the terraform fmt command.



terraform fmt

5. Fix Interpolation Error

main.tf
Edit and correct the tags block section by adding this:
From



tags = {
    Name = $var.name-myweb
  }



tags = {
  Name = "${var.name}-myweb"

Check the format



terraform fmt

Initialize Terraform



terraform init

Check syntax



terraform validate

Preview changes



terraform plan

Execute changes
Type yes to confirm.



terraform apply

The apply should be completed successfully.

Cleanup



terraform destroy

What we have done so far

We have successfully fixed the string variable interpolation error and deployed our resources.

How to fix a Terraform for_each Error

Revathi Joshi — Thu, 12 Oct 2023 21:28:35 +0000

What is a for_each error?

"Terraform's for_each attribute allows you to create a set of similar resources based on the criteria you define."

"When you need to create a set of similar instances, each assigned to a different security group. Terraform cannot parse aws_security_group..id in this attribute because the splat expression () only interpolates list types, while the for_each attribute is reserved for map types.

A local value can return a map type.
Define the local value in your main.tf file. This converts the list of security groups to a map.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create 4 files - main.tf, variables.tf, outputs.tf and terrafprm.tfvars.

4. Initialize Your Working Directory

5. Fix the for_each Error

6. Deploy Your Resources

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
Troubleshoot Terraform - Correct a for_each error
learn-terraform-troubleshooting

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create 4 files - `main.tf`, `variables.tf`, `outputs.tf` and `terrafprm.tfvars`.

1. main.tf
substitute vpc_id with your own VPC



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region  = var.region
}

data "aws_ami" "linux" {
   most_recent = true
   owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  for_each               = aws_security_group.*.id
  ami           = data.aws_ami.linux.id
  availability_zone = var.az_1a
  instance_type = var.instance_type
  vpc_security_group_ids = [each.id]
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = "${var.name}-mywebapp"
  }
}
 resource "aws_security_group" "sg_ping" {
   name                      = "Allow Ping"
   vpc_id                    = "<DUMMY VALUE>"
}

resource "aws_security_group" "sg_8080" {
   name                      = "Allow 8080"
   vpc_id                    = "<DUMMY VALUE>"
}

 resource "aws_security_group_rule" "sg_ping" {
   type                      = "ingress"
   from_port                 = -1
   to_port                   = -1
   protocol                  = "icmp"
   security_group_id         = aws_security_group.sg_ping.id
   source_security_group_id  = aws_security_group.sg_8080.id
}

 resource "aws_security_group_rule" "sg_8080" {
   type                     = "ingress"
   from_port                = 8080
   to_port                  = 8080
   protocol                 = "tcp"
   security_group_id        = aws_security_group.sg_8080.id
   source_security_group_id = aws_security_group.sg_ping.id
}

2. variables.tf



variable "region" {
  description = "region"
}

variable "name" {
  description = "Value of the Name tag for the EC2 instance"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

3. outputs.tf



output "instance_id" {
  description = "ID of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.id]
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.public_ip]
}

output "instance_name" {
  description = "Tags of the EC2 instance"
  value       = [for instance in aws_instance.web_app: instance.tags.Name]
}

4. terrafprm.tfvars



name = "rev"
region = "us-east-1"

4. Initialize Your Working Directory



cd terraform_files

Terraform format



terraform fmt

Initiate the working directory



terraform init

Validate the configuration



terraform validate

get a number of errors in the configuration file

5. Fix the for_each Error

Errors

in main.tf - Review the details of all the error messages
- 1. Correct the variable interpolation error - On line 33, there is an invalid for_each attribute reference to the aws_security group
- The error is being produced because the * expression in the aws_security_group.*.id value is not supported by the for_each attribute.



Error: Invalid reference
│ 
│   on main.tf line 33, in resource "aws_instance" "web_app":
│   33:   for_each               = aws_security_group.*.id
│ 
│ A reference to a resource type must be followed by at least one attribute access, specifying the resource name.

in main.tf - Review the details of the Invalid "each" attribute error message
- 2. On line 37, an invalid "each" object that is missing the vpc_security_group_ids attribute
- The error is being produced because the [each.id] value is dependent on the for_each attribute



Error: Invalid "each" attribute
│ 
│   on main.tf line 37, in resource "aws_instance" "web_app":
│   37:   vpc_security_group_ids = [each.id]
│

1. Fixing the errors

at the bottom of main.tf, declare local variables for the security groups being created in the configuration file:



locals {
  security_groups = {
    sg_ping = aws_security_group.sg_ping.id,
    sg_8080 = aws_security_group.sg_8080.id,
  }
}

2. Fixing the errors

On line 33, replace the aws_security_group.*.id value with local.security.groups
replace



for_each               = aws_security_group.*.id

with



for_each               = local.security_groups

3. Fixing the errors

On line 37, replace the [each.id] value with [each.value]
replace



vpc_security_group_ids = [each.id]

with



vpc_security_group_ids = [each.value]

4. Fixing the errors

On line 44, for the tag Name attribute
replace the value



${var.name}-mywebapp

with



${var.name}-mywebapp-${each.key}

main.tf looks like this



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  region = var.region
}

data "aws_ami" "linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "web_app" {
  for_each               = local.security_groups
  ami                    = data.aws_ami.linux.id
  availability_zone      = var.az_1a
  instance_type          = var.instance_type
  vpc_security_group_ids = [each.value]
  user_data              = <<-EOF
              #!/bin/bash
              echo "Hello, World" > index.html
              nohup busybox httpd -f -p 8080 &
              EOF
  tags = {
    Name = "${var.name}-mywebapp-${each.key}"
  }
}
resource "aws_security_group" "sg_ping" {
  name   = "Allow Ping"
  vpc_id = "vpc-0da931f5deb73c9e2"
}

resource "aws_security_group" "sg_8080" {
  name   = "Allow 8080"
  vpc_id = "vpc-0da931f5deb73c9e2"
}

resource "aws_security_group_rule" "sg_ping" {
  type                     = "ingress"
  from_port                = -1
  to_port                  = -1
  protocol                 = "icmp"
  security_group_id        = aws_security_group.sg_ping.id
  source_security_group_id = aws_security_group.sg_8080.id
}

resource "aws_security_group_rule" "sg_8080" {
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  security_group_id        = aws_security_group.sg_8080.id
  source_security_group_id = aws_security_group.sg_ping.id
}

locals {
  security_groups = {
    sg_ping = aws_security_group.sg_ping.id,
    sg_8080 = aws_security_group.sg_8080.id,
  }
}

Validate the configuration:



terraform validate

You should receive a success message stating the configuration is valid.

6. Deploy Your Resources

Create the Terraform plan



terraform plan

Create the resources



terraform apply

enter yes to confirm deployment

EC2 instances - rev-mywebapp-sg_8080 and rev-mywebapp-sg_ping

Cleanup



terraform destroy

What we have done so far

We have successfully fixed the for_each error and deployed our resources.

How to debug logging in Terraform setting it to Trace

Revathi Joshi — Wed, 11 Oct 2023 23:21:37 +0000

Logging has to be turned on locally if you want to debug the issues during terraform plan or apply.

Terraform depends on two environment variables being configured. These two variables are TF_LOG and TF_LOG_PATH. I will be setting my TF_LOG environment variable to the TRACE log level and the TF_LOG_PATH environment variable to the logs/terraform_logs.txt file. You can set your TF_LOG environment variable to DEBUG, INFO, WARN, or ERROR.

We will then deploy our resources and check to see if the log is created.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create 3 files - main.tf, variables.tf, and outputs.tf.

4. Enable Debug Logging

5. Capture Logging

6. Terraform deployment

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
Debugging Terraform
Implement logging-View all Terraform log output

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create 3 files - `main.tf`, `variables.tf`, and `outputs.tf`.

1. main.tf



terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

resource "aws_instance" "app_server" {
  ami           = var.instance_ami
  instance_type = "t3.micro"
  subnet_id     = var.subnet_id

  tags = {
    Name = var.instance_name
  }
}

2. variables.tf



variable "instance_ami" {
  description = "Value of the AMI used for the EC2 instance"
  type        = string
  default     = "<DUMMY_VALUE_AMI>"
}

variable "instance_ami" {
  description = "Value of the AMI used for the EC2 instance"
  type        = string
  default     = "ami-0bb4c991fa89d4b9b"
}



variable "subnet_id" {
  description = "Value of the subnet id used for the EC2 instance"
  type        = string
  default     = "<DUMMY_VALUE_SUBNET_ID>"
}

variable "subnet_id" {
  description = "Value of the subnet id used for the EC2 instance"
  type        = string
  default     = "subnet-05f279c5812013c5e"
}



variable "instance_name" {
  description = "Value of the Name tag for the EC2 instance"
  type        = string
  default     = "MyInstance"
}

3. outputs.tf



output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

output "instance_name" {
  description = "Name of the EC2 instance"
  value       = aws_instance.app_server.tags.Name
}

4. Enable Debug Logging

For one time session where you need detailed log information
1. Set an environment variable for TF_LOG:

For PowerShell



$env:TF_LOG=TRACE

For Bash



export TF_LOG=TRACE

2. Create the logs directory:



mkdir logs

3. Set an environment variable for TF_LOG_PATH to the logs directory:

For PowerShell



$env:TF_LOG_PATH=terraform.txt

For Bash



export TF_LOG_PATH=logs/terraform_logs.txt

This outputs your logs to the terraform.txt file.

4. to verify that it worked.

For PowerShell



> echo $env:TF_LOG
TRACE
> echo $env:TF_LOG_PATH
terraform.txt

For Bash



echo $TF_LOG



echo $TF_LOG_PATH

5. Capture Logging

Check the formatting of your resources: ```

terraform fmt


- Initialize your working directory:

terraform init


- This should have generated some logs.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8iici3uju3ho5rsfhb72.png)


- Confirm the Log was Written to the Directory
- After your working directory is initialized, check your terraform_logs.txt file to confirm it contains logs:

tail -50 logs/terraform_logs.txt


- You should see that logs are successfully output to terraform_logs.txt.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dq1yze1hauo76irk1mqb.png)


## 6. Terraform deployment

- Check which version of Terraform you're running
- This is useful if you're submitting a bug report to troubleshoot a Terraform issue.

terraform version


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p34gb6rt8gjeglk6sxv4.png)


- Plan your Terraform deployment:

terraform plan



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vsc2cuhfgmwgw0lwvktf.png)


- Apply your Terraform deployment:
- When prompted to Enter a value, enter yes.

terraform apply



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/no5igimrsk4fwkhietb4.png)


- After the apply completes successfully, check your terraform_logs.txt file to confirm it now contains logs related to your apply:

tail -50 logs/terraform_logs.txt


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/avbu3lnwxugyzaxekqml.png)

- MyInstance


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yubm20absdk67xivy2gj.png)


# Cleanup

terraform destroy


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gud3c27gpb7iya5lywmj.png)



# What we have done so far

We have successfully enabled debug logging by setting the `TF_LOG` environment variable to the `TRACE` log level and setting `TF_LOG_PATH` environment variable to the `logs/terraform_logs.txt` file. Aftrwards, we have deployed our resources and then checked to see if the log was created.

Using refresh-only mode to sync Terraform state File

Revathi Joshi — Mon, 09 Oct 2023 22:37:01 +0000

The -refresh-only option for terraform plan and terraform apply was introduced in Terraform v0.15.4. For prior versions you must use terraform refresh. It reads the current settings from all managed remote objects and updates the Terraform state to match. Automatically applying the effect of a refresh is risky. If you have misconfigured credentials for one or more providers, Terraform may be misled into thinking that all of the managed objects have been deleted, causing it to remove all of the tracked objects without any confirmation prompt.

The -refresh-only command presents an interactive prompt for you to review the changes that Terraform has detected before committing them to the state.

Please visit my GitHub Repository for Terraform articles on various topics being updated on constant basis.

Let’s get started!

Objectives:

1. Login to AWS Management Console

2. Create infrastructure for resources block

3. Under terraform_files resources directory - Create subfolder - S3.

Under S3 sub-folder - Create dynamodb.tf and s3-bucket.tf.

4. Deploy Your Backend Resources

5. Under terraform_files resources directory - Create sub-folder - Terraform.

Under Terraform sub-folder, Create 5 files - backend.tf, main.tf, variables.tf, outputs.tf and terrafprm.tfvars.

6. Create Your Resources

Under Terraform sub-folder, Create 5 files - backend.tf, main.tf, variables.tf, outputs.tf and terraform.tfvars.

7. Troubleshoot Error

Pre-requisites:

AWS user account with admin access, not a root account.
Cloud9 IDE with AWS CLI.

Resources Used:

Terraform documentation.
Terraform documentation for AMI.
What is Amazon S3?.
What is Amazon DynamoDB?
Use refresh-only mode to sync Terraform state

Steps for implementation to this project:

1. Login to AWS Management Console

Make sure you're in the N. Virginia (us-east-1) region

2. Create infrastructure for resources block

Let’s create the following organizational structure as shown below.

3. Under terraform_files resources directory - Create subfolder - S3.

Under S3 sub-folder - Create dynamodb.tf and s3-bucket.tf.

dynamodb.tf

resource "aws_dynamodb_table" "dynamodb-terraform-state-lock" {
  name = "backend-state-lock-dynamo"
  hash_key = "LockID"
  read_capacity = 20
  write_capacity = 20

  attribute {
    name = "LockID"
    type = "S"
  }
}

Dynamo DB Name - backend-state-lock-dynamo

s3.tf

provider "aws" {
  region     = "us-east-1"
}

resource "aws_s3_bucket" "revmybucket" {
    bucket = "revbucket123456"
}

bucket name - revbucket123456

4. Deploy Your Backend Resources

Now run these Terraform commands in the following order:

Run terraform version displays the current version of Terraform and all installed plugins.
Run terraform fmt to format the code.

Run terraform init to initialize Terraform.

Run terraform validate to check for any syntax errors.

Run terraform plan to see what resources will be created.

Run terraform apply executes the actions proposed in a terraform plan and type yes when prompted.
Run terraform apply --auto-approve flag will prevent Terraform from prompting you to enter yes explicitly before it deploys the code.

s3 bucket - revbucket123456

dynamodb table - backend-state-lock-dynamo, no backend state file created

terraform.tfstate file

5. Under terraform_files resources directory - Create sub-folder - Terraform.

Under Terraform sub-folder, Create 5 files - backend.tf, main.tf, variables.tf, outputs.tf and terraform.tfvars.
1. backend.tf
substitute your bucket name and dynamodb table name

terraform {
  backend "s3" {
    encrypt = true
    bucket = "revbucket123456"
    dynamodb_table = "backend-state-lock-dynamo"
    key    = "backend.tfstate"
    region = "us-east-1"
  }
}

2. main.tf

terraform {

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.23"
    }
  }

  required_version = ">= 0.14.9"
}

provider "aws" {
  profile = "default"
  region  = var.region
}

data "aws_ami" "linux" {
   most_recent = true
   owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}

resource "aws_instance" "app_server" {
  ami           = data.aws_ami.linux.id
  instance_type = var.instance_type
  availability_zone = var.az_1a

  tags = {
    Name = var.instance_name
  }
}

3. variables.tf

variable "region" {
  description = "region"
  type        = string
  default     = "us-east-1"
}

variable "az_1a" {
  description = "availability zone 1"
  type        = string
  default     = "us-east-1a"
}

variable "instance_type" {
  description = "Value of the Name tag for the EC2 instance type"
  type        = string
  default     = "t2.micro"
}

variable "instance_name" {
  description = "Value of the Name tag for the EC2 instance"
  type        = string
  default     = "MyInstance"
}

4. outputs.tf

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

output "instance_name" {
  description = "Name of the EC2 instance"
  value       = aws_instance.app_server.tags.Name
}

5. terraform.tfvars

name = "rev"
#region = "us-west-1"

6. Create Your Resources

cd terraform

Now run these Terraform commands in the following order:

Run terraform version displays the current version of Terraform and all installed plugins.
Run terraform fmt to format the code.

Run terraform init to initialize Terraform.

Run terraform validate to check for any syntax errors.

Run terraform plan to see what resources will be created.

Run terraform apply executes the actions proposed in a terraform plan and type yes when prompted.
Run terraform apply --auto-approve flag will prevent Terraform from prompting you to enter yes explicitly before it deploys the code.

s3 bucket - revbucket123456, file - backend.tfstate

dynamodb table - backend-state-lock-dynamo, backend state file created

7. Troubleshoot Error

"In Terraform, refreshing your state file updates Terraform's knowledge of your infrastructure, as represented in your state file, with the actual state of your infrastructure. Terraform plan and apply operations run an implicit in-memory refresh as part of their functionality, reconciling any drift from your state file before suggesting infrastructure changes."
You can update your state file without making modifications to your infrastructure using the -refresh-only flag for plan and apply operations.
This allows you to review any updates to your state file.
Uncomment the region value by removing the hashtag in terraform.tfvars

#region = "us-west-1"

region = "us-west-1"

terraform plan -refresh-only

Because you updated your provider for the us-west-1 region, Terraform tries to locate the EC2 instance with the instance ID tracked in your state file but fails to locate it since it's in a different region. Terraform assumes that you destroyed the instance and wants to remove it from your state file.
If the modifications to your state file proposed by a -refresh-only plan were acceptable, you could run a terraform apply -refresh-only and approve the operation to overwrite your state file without modifying your infrastructure.
However, in this tutorial, refreshing your state file would drop your resources, so do not run the apply operation.
Comment the region value by removing the hashtag in terraform.tfvars

region = "us-west-1"

#region = "us-west-1"

terraform plan -refresh-only

You should now see the following message, "No changes. Your infrastructure still matches the configuration."

Cleanup

delete the file - backend.tfstate from bucket

terraform destroy

What we have done so far

We have successfuly deployed backend and AWS resources. We reviewed the changes that Terraform has detected before committing them to the state using Terraform refresh-only command, thus demonstated the advantages of when to use that command.