DEV Community: Kshitij Sharma

TerraWeek Challenge – Day 5 : Terraform Modules: Writing Reusable Infrastructure

Kshitij Sharma — Thu, 16 Jul 2026 10:26:44 +0000

📅 Date: 16 July 2026

Welcome to Day 5 of my TerraWeek Challenge! 🚀

During the previous four days, I learned how to write Terraform configurations, provision AWS infrastructure, manage Terraform State, and configure Remote Backends.

Today's challenge focused on one of Terraform's most powerful features—Modules.

Until now, every infrastructure component was written directly inside the root configuration. While that approach works for small projects, it quickly becomes difficult to maintain as infrastructure grows.

Imagine creating the same EC2 configuration multiple times for different environments.

Instead of copying and pasting hundreds of lines of Terraform code, Modules allow us to package infrastructure into reusable building blocks.

This makes Infrastructure as Code:

Cleaner
Easier to maintain
Reusable
Consistent
Easier to test
Easier to version

Today's challenge introduced me to the concepts of:

Root Modules
Child Modules
Local Modules
Registry Modules
Git Modules
Module Version Pinning
Module Composition
Reusing Modules with for_each

By the end of today's challenge, I understood why almost every production Terraform project is built around reusable modules rather than large monolithic Terraform files.

📚 Learning Objectives

By the end of Day 5, I was able to:

Understand what Terraform Modules are.
Learn why Modules improve Infrastructure as Code.
Understand the difference between Root Modules and Child Modules.
Create my own reusable Local Module.
Pass values into Modules using Variables.
Read values from Modules using Outputs.
Understand why Modules should receive IDs as inputs.
Consume Modules from the Terraform Registry.
Understand Git-based Modules.
Learn different ways to pin Module versions.
Instantiate multiple Modules using for_each.
Explore Module Composition.

🤔 What is a Terraform Module?

A Terraform Module is simply a collection of Terraform configuration files grouped together to perform a specific task.

Instead of writing the same infrastructure repeatedly, we can package it into a reusable module and call it whenever needed.

For example:

Instead of writing an EC2 Instance configuration three different times:

Development
Staging
Production

we can create one reusable EC2 Module and use it across all environments.

Think of a Module as a reusable function in programming.

You write it once...

...and use it as many times as you want.

🏗️ Why Do We Need Modules?

As Terraform projects grow, the amount of Infrastructure as Code also grows.

Without Modules:

Code duplication increases.
Maintenance becomes difficult.
Bugs need to be fixed in multiple places.
Infrastructure becomes inconsistent.

Modules solve these problems by keeping infrastructure reusable and standardized.

Instead of updating ten different EC2 configurations, we update one Module.

Every environment automatically benefits from the change.

This significantly reduces maintenance effort.

🌍 Root Module vs Child Module

One of the most important concepts I learned today was the difference between the Root Module and a Child Module.

Root Module

The Root Module is the Terraform project that we execute directly.

This is where we run commands like:

terraform init

terraform plan

terraform apply

The Root Module usually contains:

Provider Configuration
Data Sources
Variables
Module Calls
Outputs

It acts as the entry point for the entire Terraform project.

Child Module

A Child Module is a reusable Terraform package that is called by another module.

Instead of creating infrastructure directly, it receives inputs from the Root Module and returns useful outputs.

In today's project, the reusable EC2 module is the Child Module.

This design keeps the infrastructure modular and much easier to maintain.

✨ Benefits of Using Modules

Modules provide several important advantages.

Some of the biggest benefits include:

Reusability
Consistency
Encapsulation
Versioning
Easier Testing
Better Team Collaboration
Cleaner Project Structure

Instead of maintaining hundreds of duplicated Terraform resources, Modules allow us to centralize infrastructure logic in a single place.

This is one of the main reasons why Modules are considered the backbone of scalable Terraform projects.

📂 Project Structure

Today's project consists of two parts.

The Root Module lives at the project root, while the reusable EC2 Module is stored inside the modules directory.

Day-05/
│
├── main.tf
├── outputs.tf
├── terraform.tf
│
├── modules/
│
│   └── ec2_instance/
│       ├── main.tf
│       ├── variables.tf
│       ├── outputs.tf
│       └── README.md
│
├── README.md
└── images/

The Root Module calls the Child Module whenever an EC2 instance needs to be created.

This separation keeps infrastructure reusable and organized.

📸 Screenshot

📁 Understanding the Module Structure

A well-designed Terraform Module typically contains four files.

`main.tf`

Contains the infrastructure resources.

Example:

EC2 Instance
Security Groups
IAM Roles

`variables.tf`

Defines every input the Module expects.

Instead of hardcoding values, the Module receives them from the Root Module.

`outputs.tf`

Exports useful information back to the Root Module.

Examples:

Instance ID
Public IP
Private IP

`README.md`

Explains:

Module purpose
Required Inputs
Outputs
Usage Examples

Documenting Modules is considered a best practice because other developers can quickly understand how to use them.

⚙️ Prerequisites

Before working with Terraform Modules, I ensured that the following tools were installed and configured correctly.

Terraform
AWS CLI
Git
Visual Studio Code
AWS Account

Since the module provisions AWS resources, valid AWS credentials are required before running Terraform.

🛠️ Verifying Terraform Installation

The first step was verifying the installed Terraform version.

terraform version

Terraform displayed the currently installed version.

This confirms that the required Terraform version is available before initializing the project.

📸 Screenshot

☁️ Verifying AWS CLI

Before provisioning infrastructure, I verified my AWS credentials.

aws sts get-caller-identity

Terraform uses these credentials to authenticate with AWS.

A successful response confirmed that my AWS CLI was configured correctly.

📸 Screenshot

🧩 Understanding the Root Module

In every Terraform project, there is always one Root Module.

The Root Module is the starting point of the entire Terraform configuration. It is responsible for initializing providers, fetching required data, calling reusable modules, and exposing useful outputs.

In today's project, the Root Module contains:

terraform.tf
main.tf
outputs.tf

Unlike previous days where the infrastructure was written directly inside main.tf, today's Root Module delegates the actual EC2 creation to a reusable Child Module.

Its responsibilities include:

Configuring the AWS Provider
Looking up the latest Amazon Linux AMI
Resolving required IDs (Subnet, Security Group)
Passing those values to the reusable module
Reading outputs from the module

This separation keeps the Root Module lightweight while the infrastructure logic remains encapsulated inside the Child Module.

📦 Understanding the Child Module

The reusable EC2 Module is located inside:

modules/
└── ec2_instance/

This module is responsible for creating a single EC2 Instance.

Instead of hardcoding infrastructure values, the module accepts everything as inputs, making it reusable across different environments, VPCs, AWS accounts, and regions.

The module itself contains:

main.tf
variables.tf
outputs.tf
README.md

Each file has a dedicated responsibility.

File	Purpose
`main.tf`	Creates the EC2 instance
`variables.tf`	Accepts input variables
`outputs.tf`	Exports useful values
`README.md`	Documents how to use the module

This is considered the standard structure for production-quality Terraform modules.

📥 Module Inputs

One of the key ideas behind Terraform Modules is Inputs.

Inputs make a module configurable.

Instead of embedding values inside the module, the Root Module supplies them when calling the module.

Typical inputs include:

Instance Name
AMI ID
Instance Type
Subnet ID
Security Group IDs
Environment
Tags

Example:

module "web_server" {

  source = "./modules/ec2_instance"

  name = "web"

  instance_type = "t2.micro"

  environment = "dev"

  ami = data.aws_ami.al2023.id

  subnet_id = local.subnet_id

  vpc_security_group_ids = local.security_group_ids

}

This approach keeps the module flexible and reusable.

📤 Module Outputs

Modules can also return useful information back to the Root Module.

Instead of searching AWS manually, Terraform exposes outputs from the Child Module.

Typical outputs include:

Instance ID
Public IP
Private IP

Example:

output "web_public_ip" {

  value = module.web_server.public_ip

}

Outputs make it easier for other modules, automation pipelines, or users to consume infrastructure information.

💡 Why Pass IDs Instead of Performing Lookups?

One of the most interesting design decisions in today's challenge was that the Child Module does not perform its own data lookups.

Instead, the Root Module resolves everything once and passes the values into the module.

For example:

ami = data.aws_ami.al2023.id

instead of placing the data "aws_ami" block inside every module.

This has several advantages:

Better performance
Avoids repeated API calls
Keeps the module reusable
Makes testing easier
Keeps the module cloud-agnostic

This is considered a Terraform best practice and is also the approach demonstrated in the assignment. :contentReference[oaicite:0]{index=0}

📍 Calling a Local Module

Terraform allows us to reference a module stored inside the current project.

Example:

module "web_server" {

  source = "./modules/ec2_instance"

}

The source attribute tells Terraform where the module is located.

Since today's module is stored locally, Terraform loads it directly from the filesystem.

During initialization, Terraform automatically discovers and prepares the module for use.

🌎 Using Modules from the Terraform Registry

Terraform also provides an official Terraform Registry, which hosts thousands of reusable modules maintained by HashiCorp and the community.

Instead of writing everything from scratch, we can use existing production-ready modules.

Example:

module "vpc" {

  source = "terraform-aws-modules/vpc/aws"

  version = "~> 5.0"

}

Using Registry Modules significantly reduces development time while promoting standardized infrastructure.

The assignment specifically encouraged exploring Registry Modules and pinning their versions. :contentReference[oaicite:1]{index=1}

🔗 Using Modules from Git

Modules can also be stored inside Git repositories.

Example:

module "network" {

  source = "git::https://github.com/example/terraform-modules.git//network?ref=v1.0.0"

}

Terraform downloads the module directly from the repository.

Git Modules are commonly used when organizations maintain their own internal module library.

🔒 Module Version Pinning

One of today's biggest lessons was understanding Module Version Pinning.

Without version pinning, Terraform always downloads the latest available version.

This can unexpectedly introduce breaking changes.

To avoid this, versions should always be pinned.

Some common strategies include:

Exact Version

version = "= 5.1.2"

Compatible Minor Versions

version = "~> 5.0"

Allows:

5.0.x
5.1.x
5.2.x

But blocks:

6.0

Version Range

version = ">= 5.0, < 6.0"

Useful when supporting multiple compatible releases.

Git Tag

source = "git::https://github.com/org/repo.git//module?ref=v1.2.0"

Git Commit SHA

source = "git::https://github.com/org/repo.git//module?ref=8d0f5d8b2..."

Using a full commit SHA provides maximum reproducibility because the module code can never change.

🎯 Why Version Pinning Matters

Version pinning offers several important benefits:

Reproducible deployments
Stable infrastructure
Protection from breaking changes
Consistent CI/CD pipelines
Easier debugging
Predictable upgrades

Professional Terraform projects almost always pin provider versions and module versions to ensure infrastructure behaves consistently across environments.

🔄 Modular Composition using `for_each`

Another powerful concept introduced today was Module Composition.

Instead of calling the same module repeatedly, Terraform allows us to instantiate multiple copies using for_each.

Example:

module "servers" {

  source = "./modules/ec2_instance"

  for_each = toset([
    "app",
    "worker",
    "cache"
  ])

  name = each.key

  instance_type = "t2.micro"

  environment = "dev"

  ami = data.aws_ami.al2023.id

  subnet_id = local.subnet_id

  vpc_security_group_ids = local.security_group_ids

}

With a single module definition, Terraform can provision multiple EC2 instances while keeping the configuration concise and maintainable. This pattern is one of the main practical tasks in the Day 5 assignment. :contentReference[oaicite:2]{index=2}

🚀 Deploying Infrastructure using Terraform Modules

With the Root Module and Child Module ready, it was finally time to deploy infrastructure using my own reusable Terraform Module.

Unlike the previous days where all infrastructure was written directly inside the root configuration, today's deployment was much cleaner.

The Root Module simply passed the required inputs to the Child Module, while the Child Module handled the actual EC2 creation.

This separation of responsibilities is one of the biggest reasons why Terraform Modules are considered a best practice for Infrastructure as Code.

The deployment workflow looked like this:

Write Root Module

        │

        ▼

Call Child Module

        │

        ▼

terraform init

        │

        ▼

Download Module

        │

        ▼

terraform plan

        │

        ▼

terraform apply

        │

        ▼

Verify EC2 Instance

        │

        ▼

terraform output

        │

        ▼

Reuse Module using for_each

🚀 Step 1 — Initialize Terraform

The first step was initializing the Terraform project.

terraform init

Unlike the previous days, Terraform now performed two tasks during initialization.

Downloaded the AWS Provider.
Initialized the Local Module.

Terraform automatically discovered the Child Module inside:

modules/ec2_instance

and prepared it for execution.

Once everything completed successfully, Terraform displayed:

Terraform has been successfully initialized!

This confirmed that both the provider and the reusable module were ready.

📸 Screenshot

📋 Step 2 — Review the Execution Plan

Before creating any infrastructure, I reviewed the execution plan.

terraform plan

Terraform evaluated:

Root Module
Child Module
Variables
Outputs
Resource Dependencies

Then it displayed every resource that would be created.

Since the infrastructure itself lives inside the Child Module, the execution plan clearly showed Terraform expanding the module before provisioning resources.

Example:

module.web_server.aws_instance.this

This demonstrated how Terraform internally resolves module resources.

Reviewing the execution plan before deployment helps prevent accidental infrastructure changes.

📸 Screenshot

🚀 Step 3 — Provision Infrastructure

Once I verified the execution plan, I deployed the infrastructure.

terraform apply

Terraform displayed:

Do you want to perform these actions?

After confirming with:

yes

Terraform provisioned the infrastructure by calling the Child Module.

Instead of creating resources directly from the Root Module, Terraform executed the reusable EC2 Module.

After a few moments Terraform displayed:

Apply complete!

This confirmed that the EC2 instance had been successfully provisioned using my own reusable Terraform Module.

📸 Screenshot

☁️ Verifying the EC2 Instance

After Terraform completed successfully, I opened the AWS Management Console.

Inside the EC2 Dashboard, I verified that the instance had been created successfully.

I confirmed:

Instance Name
Instance Type
Running State
Public IP Address
Availability Zone

Seeing the EC2 instance created entirely through a reusable Terraform Module reinforced the practical value of modular Infrastructure as Code.

📸 Screenshot

📤 Viewing Module Outputs

One of the biggest advantages of Terraform Modules is the ability to expose useful information through Outputs.

Instead of manually locating resource details inside AWS, Terraform displays them automatically.

Command:

terraform output

Example:

instance_id = "i-xxxxxxxxxxxxxxxx"

public_ip = "xx.xx.xx.xx"

private_ip = "172.xx.xx.xx"

These values are actually produced inside the Child Module and then returned to the Root Module.

This makes modules easy to integrate with other modules and automation pipelines.

📸 Screenshot

🔄 Creating Multiple EC2 Instances using `for_each`

One of today's most interesting tasks was learning that the same module can be instantiated multiple times.

Instead of writing three different EC2 resources, Terraform allows us to reuse a single module.

Example:

module "servers" {

  source = "./modules/ec2_instance"

  for_each = toset([
    "app",
    "worker",
    "cache"
  ])

  name = each.key

  instance_type = "t2.micro"

  environment = "dev"

  ami = data.aws_ami.al2023.id

  subnet_id = local.subnet_id

  vpc_security_group_ids = local.security_group_ids

}

When Terraform evaluates this configuration, it automatically creates:

App Server
Worker Server
Cache Server

without duplicating the module code.

This demonstrates the real power of Modules combined with Meta-Arguments.

📋 Reviewing the Plan with Multiple Module Instances

After adding for_each, I generated another execution plan.

terraform plan

Terraform now displayed multiple module instances similar to:

module.servers["app"]

module.servers["worker"]

module.servers["cache"]

This clearly showed that Terraform treats each module instance independently while still using the same reusable module definition.

🔍 Inspecting the Infrastructure

After deployment, I also explored Terraform's outputs and state to better understand how module resources are represented.

Useful commands included:

terraform output

terraform state list

Example output:

module.web_server.aws_instance.this

This naming convention makes it easy to identify which resources belong to which module.

🍫 Bonus Exploration

After completing the core module implementation, I explored a few advanced Terraform Module concepts that are commonly used in real-world DevOps projects.

These bonus tasks helped me understand how teams build reusable infrastructure libraries, maintain module quality, and safely evolve infrastructure over time.

Instead of focusing only on creating resources, these concepts emphasize writing modules that are reusable, maintainable, and production-ready.

⭐ Bonus 1 — Documenting the Module

One of the best practices recommended in today's challenge was documenting every reusable module.

A well-documented module allows other developers to understand:

What the module does
Required Inputs
Optional Inputs
Outputs
Usage Examples
Default Values

Instead of reading the module source code, developers can simply open the module's README.md.

My reusable EC2 module includes documentation describing its purpose, required inputs, optional parameters, outputs, and a usage example, making it much easier to consume in other projects. :contentReference[oaicite:0]{index=0}

⭐ Bonus 2 — Input Validation

Reusable modules should validate user inputs whenever possible.

Terraform allows us to define validation rules inside variables.tf.

Example:

variable "environment" {

  type = string

  validation {

    condition = contains(
      ["dev","staging","prod"],
      var.environment
    )

    error_message = "Environment must be dev, staging or prod."

  }

}

Input Validation prevents invalid values from entering the module and reduces configuration mistakes.

This makes reusable modules much safer for teams.

⭐ Bonus 3 — Publishing Modules using Git

Terraform Modules don't have to remain inside the same repository.

They can also be stored in a separate Git repository and reused across multiple projects.

Example:

module "web_server" {

  source = "git::https://github.com/yourusername/terraform-aws-modules.git//ec2_instance?ref=v1.0.0"

}

Using:

?ref=v1.0.0

pins the module to a specific Git tag.

Instead of downloading the latest code every time, Terraform always downloads the exact version.

This improves stability and reproducibility.

⭐ Bonus 4 — Module Composition

Another advanced concept introduced today was Module Composition.

Instead of building one large module containing everything, Terraform encourages connecting multiple small modules together.

For example:

Networking Module

        │

        ▼

EC2 Module

        │

        ▼

Load Balancer Module

The output of one module becomes the input of another.

Example:

module "network" {

  source = "./modules/network"

}

module "web" {

  source = "./modules/ec2_instance"

  subnet_id = module.network.public_subnet_id

}

This creates highly modular and reusable Infrastructure as Code.

Instead of duplicating networking logic inside every EC2 module, the networking module provides the required outputs.

⭐ Bonus 5 — Why Modular Infrastructure Matters

Today's challenge completely changed the way I think about Terraform projects.

Without modules:

Infrastructure becomes repetitive.
Code duplication increases.
Maintenance becomes difficult.
Bugs are fixed in multiple places.

With Modules:

Infrastructure becomes reusable.
Updates happen in one place.
Projects remain organized.
Teams can share common infrastructure.
Testing becomes much easier.

This is why almost every production Terraform repository relies heavily on reusable modules.

🧹 Cleaning Up the Infrastructure

After verifying that everything was working correctly, I removed all the infrastructure created during today's challenge.

Terraform makes cleanup just as simple as deployment.

Command:

terraform destroy

Terraform displayed a confirmation prompt.

Do you really want to destroy all resources?

After entering:

yes

Terraform removed every resource that had been created by the Root Module.

Since the infrastructure itself was provisioned through the Child Module, Terraform automatically traversed the module hierarchy and destroyed all managed resources safely.

Finally, Terraform displayed:

Destroy complete!

Destroying unused cloud resources is an important habit because it helps avoid unnecessary AWS costs.

📸 Screenshot

🎯 What I Learned

Day 5 introduced one of the most important Terraform concepts—Modules.

Some of the key takeaways from today's challenge are:

Learned what Terraform Modules are.
Understood the difference between Root Modules and Child Modules.
Built my own reusable Local Module.
Passed values into Modules using Variables.
Returned useful information using Outputs.
Learned why modules should receive IDs instead of performing their own lookups.
Explored Local Modules, Registry Modules, and Git Modules.
Understood different Module Version Pinning strategies.
Used for_each to instantiate multiple module instances.
Explored Module Composition.
Learned how documentation and validation improve module quality.

The biggest lesson from today was understanding that Modules are the foundation of scalable Infrastructure as Code.

Instead of writing the same infrastructure repeatedly, we can package it once and reuse it across projects, environments, and teams.

📂 Repository Structure

TerraWeek/
│
├── Day-01/
│   ├── README.md
│   └── ...
│
├── Day-02/
│   ├── README.md
│   └── ...
│
├── Day-03/
│   ├── README.md
│   └── ...
│
├── Day-04/
│   ├── README.md
│   └── ...
│
├── Day-05/
│   ├── main.tf
│   ├── outputs.tf
│   ├── terraform.tf
│   │
│   ├── modules/
│   │   └── ec2_instance/
│   │       ├── main.tf
│   │       ├── variables.tf
│   │       ├── outputs.tf
│   │       └── README.md
│   │
│   ├── README.md
│   └── images/
│
└── ...

🚀 Conclusion

Day 5 marked an important shift in my Terraform journey.

Until now, I had been focused on learning how to provision infrastructure.

Today, I learned how to design reusable infrastructure.

By separating the Root Module from reusable Child Modules, passing inputs through variables, exposing outputs, and organizing infrastructure into well-defined components, I gained a much better understanding of how production Terraform projects are structured.

Exploring Registry Modules, Git Modules, Module Version Pinning, and Module Composition also gave me insight into how large teams build consistent, maintainable Infrastructure as Code.

With five days of the TerraWeek Challenge completed, I now have a much stronger foundation in writing Terraform that is not only functional but also reusable and scalable.

I'm excited to continue the challenge and explore even more advanced Terraform concepts in the upcoming days.

🏷️ Tags

#Terraform #TerraformModules #InfrastructureAsCode #AWS #DevOps #CloudComputing #Automation #PlatformEngineering #CloudEngineering #LearnInPublic #TrainWithShubham #TerraWeekChallenge

🌱 TerraWeek Challenge – Day 4 : Terraform State & Remote Backends (Native Locking)

Kshitij Sharma — Wed, 15 Jul 2026 11:55:00 +0000

📅 Date: 15 July 2026

Welcome to Day 4 of my TerraWeek Challenge! 🚀

After spending the previous three days learning Terraform fundamentals, HCL, Variables, Providers, Resources, and provisioning cloud infrastructure on AWS, today's challenge introduced one of the most important concepts in Terraform—State Management.

Creating infrastructure is only one part of Infrastructure as Code.

Managing that infrastructure safely over time is where Terraform State becomes essential.

Today's challenge focused on understanding:

What Terraform State is
Why Terraform needs a State File
Why the State File is considered sensitive
Local vs Remote State
Remote Backends using Amazon S3
Native State Locking using use_lockfile
State Commands
Importing Existing Infrastructure into Terraform

One interesting thing I learned today is that almost every professional Terraform project stores its state remotely instead of keeping it locally.

Remote State makes collaboration easier, prevents accidental conflicts, and keeps infrastructure management much safer.

By the end of today's challenge, I understood why Terraform State is often called the brain of Terraform.

📚 Learning Objectives

By the end of Day 4, I was able to:

Understand what Terraform State is.
Learn why Terraform maintains a State File.
Understand the importance of State Management.
Learn why Terraform State should never be committed to Git.
Understand State Drift.
Explore Terraform State Commands.
Learn the difference between Local and Remote State.
Configure an Amazon S3 Backend.
Understand Native State Locking using use_lockfile.
Bootstrap backend infrastructure.
Migrate Local State to Remote State.
Import existing resources into Terraform State.
Explore advanced State Management features.

📂 Project Structure

Today's challenge was slightly different from the previous days.

Instead of working with a single Terraform project, the assignment was divided into two separate directories.

Day-04/
│
├── backend_demo/
│   ├── terraform.tf
│   └── resources.tf
│
├── backend_infra/
│   ├── terraform.tf
│   ├── resources.tf
│   ├── variables.tf
│   └── outputs.tf
│
├── README.md
└── images/

Each folder serves a completely different purpose.

📁 backend_infra

This directory is responsible for creating the infrastructure required to store the Terraform State.

It provisions:

Amazon S3 Bucket
Bucket Versioning
Bucket Encryption
Other backend-related resources

The important thing to remember is that this folder itself uses Local State because the backend must exist before Terraform can store its state remotely.

📁 backend_demo

Once the backend infrastructure is created, this directory demonstrates how Terraform uses that S3 bucket as its Remote Backend.

Instead of storing the state file locally, Terraform uploads it directly to Amazon S3.

This simulates how Terraform projects are managed in real production environments.

📸 Screenshot

⚙️ Prerequisites

Before starting today's challenge, I ensured that the following tools were installed and configured.

Terraform
AWS CLI
Git
Visual Studio Code
AWS Account
Amazon S3 Permissions

Since today's project interacts with AWS, the AWS CLI must already be configured with valid credentials.

Unlike previous days, this challenge also requires permissions to create and manage Amazon S3 buckets.

🛠 Verifying Terraform Installation

Before creating any infrastructure, I verified the installed Terraform version.

Command:

terraform version

Terraform displayed the currently installed version along with provider compatibility information.

Verifying the version beforehand helps ensure compatibility with newer features such as Native S3 State Locking, which is available in recent Terraform releases.

📸 Screenshot

☁️ Verifying AWS CLI Configuration

Since Terraform communicates with AWS through the AWS Provider, it is important to verify that AWS credentials are configured correctly.

I confirmed my identity using:

aws sts get-caller-identity

This command returns:

AWS Account ID
User ARN
User ID

Receiving a successful response confirms that Terraform will be able to authenticate with AWS.

It is always a good practice to verify authentication before provisioning infrastructure.

📸 Screenshot

🗄️ What is Terraform State?

Terraform State is a file that stores information about every resource Terraform manages.

Whenever Terraform creates infrastructure, it records important information inside the terraform.tfstate file.

This includes:

Resource IDs
Metadata
Dependencies
Current Configuration
Resource Attributes

Terraform uses this file to compare the current state of infrastructure with the desired state described in the Terraform configuration.

Without the State File, Terraform would have no way of knowing which resources already exist.

This is why the State File is considered the heart of every Terraform project.

🤔 Why Does Terraform Need State?

Imagine creating an EC2 Instance today.

Tomorrow, you modify the instance type.

How does Terraform know that the instance already exists?

The answer is simple.

Terraform checks the State File.

Instead of creating a brand-new EC2 instance every time, Terraform compares the desired configuration with the recorded state and determines exactly what needs to change.

Because of this comparison, Terraform can safely:

Create resources
Update resources
Destroy resources
Detect infrastructure changes

Without State, Terraform would behave as if every deployment were the first deployment.

🔐 Why is Terraform State Sensitive?

One of the most surprising things I learned today was that the Terraform State File may contain confidential information.

Depending on the infrastructure, it may store:

Passwords
Database Connection Strings
API Keys
Access Tokens
Resource IDs
Internal Metadata

Although Terraform marks sensitive variables in outputs, the actual values can still exist inside the State File.

Because of this, the State File should never be:

Edited manually
Shared publicly
Committed to Git
Stored in unsecured locations

Professional teams usually store Terraform State inside secure Remote Backends such as Amazon S3 with encryption enabled.

🌍 Local State vs Remote State

Terraform supports two approaches for storing the State File.

Local State

terraform.tfstate

The State File remains on the developer's local machine.

Advantages:

Simple
Easy to use
Great for learning

Disadvantages:

Difficult for teams
Easy to lose
No locking
No centralized storage

Remote State

The State File is stored inside a remote storage service.

Examples include:

Amazon S3
HCP Terraform (Terraform Cloud)
Azure Storage
Google Cloud Storage

Advantages:

Centralized storage
Team collaboration
Better security
Automatic backups
State locking

For production environments, Remote State is the recommended approach.

Today's assignment focused on using Amazon S3 as the Remote Backend with Native State Locking.

🗂️ Understanding the Terraform State File

In the previous section, I learned why Terraform State is one of the most critical components of every Terraform project.

However, simply knowing what the State File is isn't enough.

To work efficiently with Terraform, it's important to understand how Terraform interacts with the State File, how we can inspect it, and how we can safely manage it over time.

Terraform provides several built-in commands that allow us to inspect, move, rename, remove, and visualize resources stored inside the State File.

These commands become extremely useful while maintaining production infrastructure.

📄 What Does the State File Contain?

Every time Terraform provisions infrastructure, it records important information inside the terraform.tfstate file.

The State File stores details such as:

Resource IDs
Current Infrastructure
Dependencies
Metadata
Resource Attributes
Provider Information

Terraform uses this information to compare the desired configuration with the existing infrastructure before making any changes.

This comparison is what makes Terraform idempotent and reliable.

⚠️ Never Edit the State File Manually

One of the biggest lessons from today's challenge was understanding that the Terraform State File should never be edited manually.

Although it is stored as a JSON file, manually changing its contents can corrupt the state and cause Terraform to lose track of managed resources.

Instead of editing the file directly, Terraform provides dedicated commands to safely manipulate the State.

This ensures that the infrastructure remains consistent and predictable.

🌪️ Understanding State Drift

Infrastructure doesn't always change through Terraform.

Sometimes resources are modified manually through the AWS Console or another tool.

For example:

Someone changes an EC2 instance type manually.
A Security Group rule is updated directly in AWS.
An S3 bucket policy is modified outside Terraform.

These unexpected changes create a mismatch between the actual infrastructure and the Terraform State.

This situation is called State Drift.

Terraform detects these differences during:

terraform plan

terraform refresh

Detecting State Drift early helps prevent unexpected infrastructure changes during future deployments.

🛠️ Exploring Terraform State Commands

Terraform provides several commands for inspecting and managing the State File.

Instead of modifying the State manually, these commands should always be used.

Let's explore the most important ones.

📋 `terraform state list`

This command lists every resource currently managed by Terraform.

terraform state list

Example Output:

aws_s3_bucket.tf_state

aws_s3_bucket_versioning.versioning

aws_s3_bucket_server_side_encryption_configuration.encryption

This command is particularly useful when working with large Terraform projects because it provides a quick overview of all managed resources.

📸 Screenshot

🔍 `terraform state show`

Sometimes we need detailed information about a specific resource.

Terraform provides the following command:

terraform state show aws_s3_bucket.tf_state

This displays detailed information such as:

Bucket Name
ARN
Region
Encryption
Versioning
Tags

Unlike terraform show, this command focuses on a single resource.

📸 Screenshot

🔄 `terraform state mv`

As Terraform projects grow, resources are often renamed or reorganized.

Instead of destroying and recreating infrastructure, Terraform allows us to move resource addresses inside the State File.

Example:

terraform state mv aws_s3_bucket.old aws_s3_bucket.new

This command updates the Terraform State without modifying the actual AWS resource.

It is extremely useful while refactoring Terraform configurations.

❌ `terraform state rm`

Sometimes we want Terraform to stop managing a resource without deleting it.

For that purpose, Terraform provides:

terraform state rm aws_s3_bucket.tf_state

This command removes the resource from the State File only.

The actual AWS resource continues to exist.

This is useful when migrating infrastructure or handing resource management to another Terraform project.

📖 `terraform show`

Terraform also allows us to display the current infrastructure in a human-readable format.

Command:

terraform show

This command displays:

Resources
Attributes
Dependencies
Outputs

Unlike opening the JSON State File manually, terraform show presents the information in a much cleaner and easier-to-read format.

☁️ Understanding Remote Backends

Up until now, all Terraform State Files were stored locally.

While this works well for learning and personal projects, it creates several challenges in team environments.

Some common problems include:

Multiple developers maintaining different State Files.
Risk of accidentally deleting the State.
No centralized storage.
No State Locking.
Difficult collaboration.

To solve these issues, Terraform supports Remote Backends.

A Remote Backend stores the State File in a centralized location that can be accessed safely by multiple team members.

Some commonly used Remote Backends include:

Amazon S3
HCP Terraform (Terraform Cloud)
Azure Storage
Google Cloud Storage

Today's assignment focused on using Amazon S3 as the Remote Backend.

🪣 Why Amazon S3?

Amazon S3 is one of the most widely used Remote Backends for Terraform.

Some of its advantages include:

Highly Durable
Secure
Centralized
Versioning Support
Encryption Support
Easy Team Collaboration

By storing the State File in S3, every team member works with the same infrastructure state.

This significantly reduces deployment conflicts.

🔒 Native State Locking

One of the biggest updates introduced in recent Terraform versions is Native S3 State Locking.

Older Terraform versions required DynamoDB to prevent multiple users from modifying the same State File simultaneously.

Modern Terraform versions simplify this process using:

use_lockfile = true

This enables native locking directly within Amazon S3, eliminating the need for DynamoDB.

Whenever Terraform performs an operation such as apply, it temporarily creates a .tflock file inside the S3 bucket.

Once the operation finishes successfully, the lock file is automatically removed.

This prevents concurrent modifications to the same State File and makes team collaboration much safer.

🏗️ Bootstrapping the Backend Infrastructure

Before Terraform can store its State remotely, the Remote Backend itself must already exist.

This creates an interesting challenge:

How can Terraform store its State inside an S3 bucket that hasn't been created yet?

The solution is called Bootstrapping.

The backend_infra project is responsible for creating:

S3 Bucket
Versioning
Encryption

Since the bucket doesn't exist initially, this project temporarily uses Local State.

Once the S3 bucket is available, other Terraform projects can safely migrate their State into it.

⚙️ Configuring the S3 Backend

After creating the backend infrastructure, the next step is configuring the backend.

Example:

terraform {

  backend "s3" {

    bucket = "your-terraform-state-bucket"

    key = "day04/terraform.tfstate"

    region = "us-east-1"

    encrypt = true

    use_lockfile = true

  }

}

This configuration instructs Terraform to:

Store the State File in Amazon S3.
Encrypt the State File.
Enable Native State Locking.
Maintain a centralized State for the project.

Once configured, Terraform automatically offers to migrate the existing Local State to the Remote Backend during initialization.

🚀 Bootstrapping the Remote Backend

After understanding Terraform State and Remote Backends, it was finally time to implement everything practically.

Unlike the previous days, today's practical was divided into two separate Terraform projects.

The first project creates the backend infrastructure, while the second project uses that infrastructure as its Remote Backend.

The workflow looks like this:

backend_infra
      │
      ▼
Create S3 Bucket
      │
      ▼
Configure Remote Backend
      │
      ▼
Migrate Local State → Amazon S3
      │
      ▼
Verify Remote State
      │
      ▼
Explore State Commands
      │
      ▼
Import Existing Resources

This two-step approach is called Bootstrapping, and it's the standard way of creating Terraform backends in production.

🏗️ Step 1 — Create the Backend Infrastructure

The very first step was creating the infrastructure that would later store the Terraform State.

I navigated to the backend_infra directory.

cd backend_infra

This project is responsible for creating:

Amazon S3 Bucket
Bucket Versioning
Server-Side Encryption

At this stage, Terraform still uses Local State, because the backend doesn't exist yet.

🚀 Step 2 — Initialize Backend Infrastructure

Once inside the backend_infra directory, I initialized Terraform.

terraform init

Terraform downloaded the required AWS Provider and initialized the working directory.

During initialization, Terraform created:

.terraform/
.terraform.lock.hcl

Once the initialization completed successfully, Terraform displayed:

Terraform has been successfully initialized!

📸 Screenshot

📋 Step 3 — Create the S3 Backend

After initialization, I provisioned the backend infrastructure.

terraform apply

Terraform displayed the execution plan, listing all the S3-related resources it was about to create.

These resources included:

S3 Bucket
Bucket Versioning
Bucket Encryption

After reviewing the execution plan, I confirmed the deployment.

yes

Terraform created the backend infrastructure successfully.

Once completed, the S3 bucket was ready to store Terraform State.

📸 Screenshot

🪣 Verifying the S3 Bucket

After the deployment completed successfully, I opened the AWS Management Console to verify that the backend infrastructure had been created.

Inside the Amazon S3 Console, I confirmed:

The S3 Bucket was created.
Versioning was enabled.
Server-Side Encryption was enabled.

This bucket will now act as the centralized location for storing Terraform State.

📸 Screenshot

🔄 Step 4 — Configure the Remote Backend

Once the backend infrastructure was available, I moved to the second project.

cd ../backend_demo

Unlike backend_infra, this project contains the Remote Backend configuration.

The terraform.tf file points Terraform to the S3 bucket created earlier.

Example:

terraform {

  backend "s3" {

    bucket = "my-terraform-state-bucket"

    key = "day04/terraform.tfstate"

    region = "us-east-1"

    encrypt = true

    use_lockfile = true

  }

}

The most important addition here is:

use_lockfile = true

This enables Native S3 State Locking, eliminating the need for DynamoDB.

🚀 Step 5 — Initialize the Remote Backend

After configuring the backend, I initialized Terraform again.

terraform init

This time Terraform detected an existing Local State and asked whether it should migrate the State to Amazon S3.

Terraform displayed a message similar to:

Do you want to copy existing state to the new backend?

I selected:

yes

Terraform then uploaded the existing Local State to the S3 bucket.

This completed the migration from Local State to Remote State.

📸 Screenshot

☁️ Verifying Remote State

After the migration completed, I opened the Amazon S3 Console again.

Inside the bucket, I verified that Terraform had uploaded:

terraform.tfstate

This confirmed that Terraform was now using Remote State instead of Local State.

From this point onward, every Terraform operation updates the State File directly inside Amazon S3.

📸 Screenshot

🔒 Verifying Native State Locking

One of the biggest improvements in modern Terraform is Native State Locking.

Whenever Terraform executes commands such as:

terraform apply

Terraform temporarily creates a lock file inside the S3 bucket.

terraform.tfstate.tflock

The lock file prevents multiple users from modifying the State File simultaneously.

After Terraform completes the operation successfully, the lock file is removed automatically.

Watching the .tflock file appear and disappear during deployment demonstrated how Terraform safely prevents concurrent updates.

📸 Screenshot

📤 Viewing Terraform Outputs

Once the backend was fully configured, I viewed the output values generated by Terraform.

Command:

terraform output

Depending on the configuration, Terraform displayed useful information such as:

Bucket Name
Bucket ARN
Region

Outputs provide a convenient way to access infrastructure details without manually searching through the AWS Console.

📥 Importing Existing Resources

Another important feature introduced in today's challenge was the Import Block.

Sometimes cloud resources already exist because they were created manually.

Instead of recreating them, Terraform allows us to import those resources into the State File.

Example:

import {

  to = aws_s3_bucket.imported

  id = "my-existing-bucket"

}

Terraform then generates the configuration automatically using:

terraform plan -generate-config-out=generated.tf

The generated configuration can be reviewed and integrated into the existing project.

Import Blocks make it much easier to bring manually created infrastructure under Terraform management.

📸 Screenshot

🍫 Bonus Exploration

After completing the core tasks, I explored a few additional Terraform features that are commonly used in production environments.

Although these tasks were optional, they provided a much deeper understanding of how Terraform manages infrastructure safely and efficiently across teams.

These bonus exercises focused on improving state management, collaboration, and infrastructure refactoring.

⭐ Bonus 1 — Comparing Remote Backends

Terraform supports multiple Remote Backends for storing the State File.

Although I implemented Amazon S3 in this challenge, I also explored some other popular backend options.

Backend	Best Use Case	Locking Support	Notes
Amazon S3	AWS Workloads	✅ Native (`use_lockfile`)	Most widely used in AWS environments
HCP Terraform (Terraform Cloud)	Teams & Enterprises	✅ Built-in	Remote runs, policies, collaboration
Azure Storage	Azure Workloads	✅ Supported	Best for Microsoft Azure users
Google Cloud Storage (GCS)	Google Cloud	✅ Supported	Ideal for GCP-based projects

For AWS-based infrastructure, Amazon S3 remains one of the most popular choices because it is highly durable, secure, and integrates seamlessly with Terraform.

⭐ Bonus 2 — S3 Bucket Versioning

One of the biggest advantages of storing Terraform State in Amazon S3 is Bucket Versioning.

Instead of replacing the existing State File every time Terraform performs an operation, Amazon S3 preserves older versions automatically.

This provides several benefits:

Recovery from accidental deletion
Rollback to previous State versions
Better auditing
Improved disaster recovery

If the latest State File becomes corrupted or is deleted accidentally, an earlier version can be restored directly from the S3 Console.

This significantly improves the reliability of Infrastructure as Code projects.

⭐ Bonus 3 — Understanding the `moved` Block

As Terraform projects evolve, resources are often renamed or reorganized.

Normally, renaming a resource causes Terraform to destroy the old resource and create a new one.

To avoid this unnecessary recreation, Terraform provides the moved block.

Example:

moved {

  from = aws_s3_bucket.old_bucket

  to   = aws_s3_bucket.state_bucket

}

Instead of recreating the infrastructure, Terraform updates the resource address inside the State File.

This makes refactoring Terraform configurations much safer.

⭐ Bonus 4 — Understanding the `removed` Block

Sometimes we want Terraform to stop managing a resource without actually deleting it.

Terraform provides the removed block for this purpose.

Example:

removed {

  from = aws_s3_bucket.logs

}

Using a removed block removes the resource from Terraform State while leaving the actual infrastructure untouched.

This is particularly useful when:

Migrating resources to another Terraform project
Handing resource management to another team
Removing resources from Terraform without deleting production infrastructure

⭐ Bonus 5 — Continuous Health Checks using `check`

Modern Terraform also supports Check Blocks.

These blocks allow Terraform to validate infrastructure continuously.

Example:

check "bucket_versioning_enabled" {

  assert {

    condition = true

    error_message = "Bucket versioning must remain enabled."

  }

}

Unlike validation rules, Check Blocks evaluate infrastructure after deployment.

They help detect configuration drift and ensure important infrastructure requirements continue to be satisfied.

🧹 Cleaning Up the Infrastructure

After verifying that everything was working correctly, I removed all the infrastructure created during today's challenge.

Since two separate Terraform projects were used, cleanup also happened in two stages.

Step 1 — Destroy the Demo Infrastructure

First, I navigated to the backend_demo directory.

cd backend_demo

Then I destroyed the demo resources.

terraform destroy

Terraform displayed a confirmation prompt.

Do you really want to destroy all resources?

After entering:

yes

Terraform removed every resource managed by the demo project.

📸 Screenshot

Step 2 — Destroy the Backend Infrastructure

Once the demo resources were removed, I switched to the backend infrastructure project.

cd ../backend_infra

Since the S3 bucket contained multiple versions of the Terraform State File, I first ensured that the bucket was empty.

After clearing the bucket, I executed:

terraform destroy

Terraform successfully removed:

S3 Bucket
Bucket Versioning
Bucket Encryption Configuration
Other backend resources

This completed the cleanup process.

📸 Screenshot

🎯 What I Learned

Day 4 introduced one of the most important concepts in Terraform—State Management.

Some of the key takeaways from today's challenge are:

Learned what the Terraform State File is and why it exists.
Understood why Terraform State should never be edited manually.
Explored Terraform State Commands.
Learned the concept of State Drift.
Understood why Terraform State is considered sensitive.
Learned the difference between Local State and Remote State.
Bootstrapped backend infrastructure using Terraform.
Configured Amazon S3 as a Remote Backend.
Enabled Native State Locking using use_lockfile.
Migrated Local State to Remote State.
Imported existing infrastructure into Terraform State.
Explored advanced features such as moved, removed, and check blocks.

The biggest lesson from today's challenge was understanding that Infrastructure as Code isn't only about creating resources—it is equally about managing them safely throughout their entire lifecycle.

📂 Repository Structure

TerraWeek/
│
├── Day-01/
│   ├── README.md
│   └── ...
│
├── Day-02/
│   ├── README.md
│   └── ...
│
├── Day-03/
│   ├── README.md
│   └── ...
│
├── Day-04/
│   ├── backend_demo/
│   │   ├── terraform.tf
│   │   └── resources.tf
│   │
│   ├── backend_infra/
│   │   ├── terraform.tf
│   │   ├── resources.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   │
│   ├── README.md
│   └── images/
│
└── ...

🚀 Conclusion

Day 4 was one of the most valuable days of the TerraWeek Challenge because it shifted my focus from simply creating infrastructure to managing it safely over time.

Learning how Terraform stores infrastructure information, migrates Local State to Remote State, protects State using Native S3 Locking, and imports existing resources made me realize that Terraform is much more than an automation tool—it is a complete infrastructure management platform.

Understanding State Management is essential for working on real-world DevOps projects where multiple engineers collaborate on the same infrastructure.

With four days of the TerraWeek Challenge completed, I now have a much stronger understanding of both infrastructure provisioning and infrastructure management.

I'm looking forward to exploring Terraform Modules and more advanced Infrastructure as Code patterns in the upcoming days.

🏷️ Tags

#Terraform #AWS #TerraformState #RemoteBackend #AmazonS3 #InfrastructureAsCode #DevOps #CloudComputing #Automation #PlatformEngineering #LearnInPublic #TrainWithShubham #TerraWeekChallenge

🌱 TerraWeek Challenge – Day 3 : Providers, Resources & My First Cloud Infrastructure

Kshitij Sharma — Tue, 14 Jul 2026 07:53:25 +0000

📅 Date: 14 July 2026

Welcome to Day 3 of my TerraWeek Challenge! 🚀

After spending the first two days understanding the fundamentals of Infrastructure as Code (IaC), Terraform, HCL, Variables, and Expressions, today was my first step into provisioning real cloud infrastructure.

Until now, I had been working locally using providers like Docker. While those exercises helped me understand Terraform syntax and workflows, today's challenge introduced me to something much closer to real-world DevOps practices.

For the first time, I learned how Terraform communicates with cloud providers, how infrastructure is created inside AWS, and how different cloud resources work together to build a complete network.

The focus of today's challenge was understanding Providers, Resources, Data Sources, and Terraform's powerful Meta-Arguments such as count, for_each, depends_on, and lifecycle.

Along the way, I also learned the basics of AWS networking by provisioning a small infrastructure consisting of a VPC, Subnet, Internet Gateway, Route Table, Security Group, and an EC2 Instance.

By the end of today's session, Terraform no longer felt like just another Infrastructure as Code tool—it started feeling like a complete infrastructure automation platform.

📚 Learning Objectives

By the end of Day 3, I was able to:

Understand Terraform Providers
Learn why Provider Version Pinning is important
Configure the AWS Provider
Understand Resources and Data Sources
Learn the difference between creating and reading infrastructure
Understand the AWS networking fundamentals
Provision cloud resources using Terraform
Explore Terraform Meta-Arguments
Learn how Terraform manages resource dependencies
Understand Terraform Lifecycle Rules
Safely provision, update, and destroy infrastructure

📂 Project Structure

To keep the project organized, I separated the Terraform configuration into multiple files.

.
├── terraform.tf
├── main.tf
├── variables.tf
├── outputs.tf
├── README.md
└── images/

Each file has its own responsibility.

File	Purpose
`terraform.tf`	Terraform settings and AWS Provider configuration
`main.tf`	Infrastructure resources
`variables.tf`	Input variables
`outputs.tf`	Output values
`README.md`	Documentation
`images/`	Screenshots used in the README

Keeping the project modular improves readability and makes future maintenance much easier.

📸 Screenshot

⚙️ Prerequisites

Before provisioning any AWS resource, I ensured the following tools were installed and configured correctly.

Terraform
AWS CLI
Git
Visual Studio Code
HashiCorp Terraform Extension

Unlike Day 2, today's challenge required communicating with AWS.

Instead of storing credentials inside Terraform files, AWS CLI securely stores them in the local credentials file.

This is considered one of the best security practices while working with Terraform.

🛠 Verifying Terraform Installation

The first step was verifying that Terraform was installed correctly.

Command:

terraform version

This command displays the currently installed Terraform version.

Verifying the installation helps ensure that the correct version is available before initializing the project.

📸 Screenshot

☁️ Configuring AWS CLI

Since Terraform needs permission to communicate with AWS, the AWS CLI must be configured before creating any infrastructure.

I configured AWS credentials using:

aws configure

The AWS CLI prompted me for:

AWS Access Key ID
AWS Secret Access Key
Default Region
Output Format

Terraform automatically uses these credentials from the local AWS credentials file.

To verify everything was working correctly, I executed:

aws sts get-caller-identity

This command confirms that Terraform will be able to authenticate with AWS successfully.

📸 Screenshot

🌍 What is a Terraform Provider?

A Provider is a plugin that allows Terraform to communicate with a specific platform or service.

Without a provider, Terraform has no way of creating or managing infrastructure.

Different providers support different platforms.

Some of the most commonly used providers are:

AWS
Azure
Google Cloud
Docker
Kubernetes
GitHub
Cloudflare

Whenever Terraform executes an apply, it communicates with the provider, which in turn interacts with the cloud platform using its APIs.

This makes providers the bridge between Terraform and the infrastructure we want to manage.

⭐ Why Providers Matter

Imagine writing a Terraform configuration that creates an EC2 instance.

Terraform itself does not know how AWS works.

Instead, it delegates the task to the AWS Provider, which understands AWS APIs and knows how to create, update, or delete AWS resources.

Similarly:

Docker Provider manages Docker resources.
AzureRM Provider manages Azure resources.
Google Provider manages Google Cloud resources.

Changing the provider often allows the same Terraform workflow to be used across different cloud platforms.

This is one of the biggest reasons why Terraform is called a multi-cloud Infrastructure as Code tool.

📌 Provider Version Pinning

One of the first things I learned today was the importance of Version Pinning.

Terraform providers receive updates regularly.

Sometimes those updates introduce:

New features
Bug fixes
Performance improvements
Breaking changes

If we always download the latest version automatically, an existing project may suddenly stop working after an update.

To avoid this, Terraform allows us to pin provider versions.

Example:

terraform {

  required_providers {

    aws = {

      source  = "hashicorp/aws"

      version = "~> 6.0"

    }

  }

}

The ~> operator is called the Pessimistic Constraint Operator.

It allows Terraform to install compatible updates while preventing unexpected major version upgrades.

Version pinning makes projects stable, reproducible, and easier to maintain across teams.

🌎 Provider Alias

Terraform also supports configuring multiple instances of the same provider.

This is done using Aliases.

Example:

provider "aws" {

  region = "ap-south-1"

}

provider "aws" {

  alias  = "mumbai"

  region = "ap-south-1"

}

Provider aliases are useful when:

Deploying resources across multiple AWS Regions
Managing multiple AWS Accounts
Creating Disaster Recovery environments
Working with Multi-Region architectures

Although my project used a single provider configuration, understanding aliases gave me insight into how Terraform scales for enterprise deployments.

📦 Resources vs Data Sources

One of the most important concepts I learned today was the difference between Resources and Data Sources.

At first, both looked very similar because both use Terraform blocks.

However, their purpose is completely different.

Resources

Resources create or manage infrastructure.

Whenever Terraform creates something new, it is defined using a Resource block.

Example:

resource "aws_instance" "web" {

  instance_type = "t2.micro"

}

Resources are responsible for:

Creating infrastructure
Updating infrastructure
Destroying infrastructure

Terraform keeps track of Resources inside the state file.

Data Sources

Data Sources never create infrastructure.

Instead, they read information that already exists.

Example:

data "aws_ami" "amazon_linux" {

  most_recent = true

  owners = ["amazon"]

}

Instead of creating an AMI, Terraform simply retrieves information about the latest Amazon Linux image.

Data Sources are useful for reading:

Existing AMIs
Availability Zones
Existing VPCs
Existing Security Groups
Existing IAM Roles

Resources vs Data Sources

Resources	Data Sources
Create infrastructure	Read existing infrastructure
Managed by Terraform	Not managed by Terraform
Stored in Terraform State	Only fetch information
Can be updated or destroyed	Read-only

Understanding this distinction is essential because most production Terraform projects use a combination of both.

🏗 Understanding AWS Networking

Before creating infrastructure, today's assignment introduced a simple overview of AWS networking.

Initially, terms like VPC, Subnet, and Internet Gateway sounded confusing.

However, thinking of them as parts of a neighborhood made them much easier to understand.

VPC → Your own private neighborhood.
Subnet → A street inside that neighborhood.
Internet Gateway → The main gate connecting your neighborhood to the internet.
Route Table → Road signs that tell traffic where to go.
Security Group → A security guard deciding who can enter.
EC2 Instance → The actual house where your application lives.

Together, these resources form the foundation of a basic AWS network.

Understanding how these components connect before writing Terraform code made the implementation much easier.

⚙️ Configuring the AWS Provider

After understanding what Providers are and why they are important, the next step was configuring the AWS Provider inside Terraform.

A provider tells Terraform which platform it should communicate with and how that communication should happen.

Since today's challenge focused on AWS, I configured the HashiCorp AWS Provider with version pinning.

Example:

terraform {

  required_version = ">= 1.5.0"

  required_providers {

    aws = {

      source  = "hashicorp/aws"

      version = "~> 6.0"

    }

  }

}

provider "aws" {

  region = var.aws_region

}

Instead of hardcoding the region directly, I used a variable to make the configuration reusable across multiple environments.

This allows the same Terraform project to work with different AWS Regions by simply changing the input variable.

🌎 Understanding Resources

Resources are the heart of every Terraform project.

Whenever Terraform creates something in the cloud, it is defined using a Resource Block.

Examples of AWS Resources include:

VPC
Subnet
Internet Gateway
Route Table
Security Group
EC2 Instance
S3 Bucket

Each Resource block describes the desired state of the infrastructure.

Example:

resource "aws_vpc" "main" {

  cidr_block = "10.0.0.0/16"

}

Here, Terraform creates a new VPC inside AWS.

Once created, Terraform tracks this resource inside the terraform.tfstate file.

This state file helps Terraform understand what has already been created and what changes are required during future deployments.

📖 Understanding Data Sources

Unlike Resources, Data Sources do not create anything.

Instead, they fetch information that already exists.

For today's assignment, I used a Data Source to retrieve the latest Amazon Linux 2023 AMI.

Example:

data "aws_ami" "amazon_linux" {

  most_recent = true

  owners = ["amazon"]

}

This block searches AWS and returns the latest available Amazon Linux image.

Using a Data Source is much better than hardcoding an AMI ID because AMI IDs change over time.

Whenever a newer Amazon Linux image becomes available, Terraform automatically retrieves the latest one.

This makes the infrastructure more reliable and future-proof.

🎯 Why Use Data Sources?

Suppose I manually write an AMI ID.

ami-0abc123456789

After a few months, AWS may release a newer image.

If I continue using the old AMI, my EC2 instance won't use the latest updates.

Instead, using:

data "aws_ami"

Terraform always selects the latest matching image automatically.

This eliminates manual maintenance.

⚡ Meta-Arguments in Terraform

Today's challenge also introduced Meta-Arguments.

Meta-Arguments are special arguments that modify how Terraform creates resources.

Unlike normal arguments, Meta-Arguments work across many resource types.

The four Meta-Arguments explored today were:

count
for_each
depends_on
lifecycle

Let's understand each one.

1️⃣ count

The count Meta-Argument creates multiple identical resources.

Example:

resource "aws_instance" "web" {

  count = 2

  instance_type = "t2.micro"

}

Instead of creating one EC2 instance, Terraform creates two identical instances.

This is useful when all resources are exactly the same.

However, if one resource is deleted later, Terraform may need to renumber the remaining resources.

Because of this limitation, count is best suited for identical infrastructure.

2️⃣ for_each

Unlike count, for_each creates resources using unique keys.

Example:

resource "aws_security_group" "sg" {

  for_each = {

    web = 80

    ssh = 22

  }

}

Each resource gets its own identity.

Benefits:

Better readability
Easier updates
Stable resource names
No index shifting

Because of these advantages, for_each is generally preferred over count whenever resources have unique names.

3️⃣ depends_on

Terraform automatically detects dependencies.

However, sometimes explicit dependencies are required.

Example:

resource "aws_instance" "web" {

  depends_on = [

    aws_internet_gateway.main

  ]

}

This tells Terraform that the Internet Gateway must be created before the EC2 instance.

Using depends_on avoids race conditions during deployment.

4️⃣ lifecycle

Lifecycle rules control how Terraform manages resources.

Example:

lifecycle {

  create_before_destroy = true

  prevent_destroy = false

}

Some commonly used lifecycle rules include:

create_before_destroy
prevent_destroy
ignore_changes

Lifecycle rules provide greater control over infrastructure updates.

For production environments, these settings can help prevent downtime and accidental deletion of critical resources.

🔄 Terraform Resource Dependencies

One of the most interesting things I learned today is that Terraform automatically builds a dependency graph.

For example:

VPC

↓

Subnet

↓

Internet Gateway

↓

Route Table

↓

Security Group

↓

EC2 Instance

Terraform understands these relationships automatically by reading references inside the configuration.

As a result, resources are created in the correct order without requiring manual intervention.

Only when Terraform cannot determine the dependency do we use depends_on.

📋 Understanding Terraform State

Every time Terraform creates infrastructure, it records the details inside a State File.

terraform.tfstate

The State File stores information such as:

Resource IDs
Current Configuration
Metadata
Resource Relationships

Terraform compares this file with the configuration to determine what needs to be created, updated, or destroyed.

Without the State File, Terraform would have no way of tracking existing infrastructure.

📄 Listing Managed Resources

Terraform also provides a command to display every resource currently managed by the State File.

Command:

terraform state list

This command lists all resources Terraform is managing.

For today's project, the output included resources such as:

AWS VPC
Public Subnet
Internet Gateway
Route Table
Security Group
EC2 Instance

This command is extremely useful for debugging and understanding the current infrastructure.

💡 Best Practices Learned Today

While working on today's assignment, I also learned a few important best practices.

✔ Always pin provider versions.

✔ Never hardcode AWS credentials inside Terraform files.

✔ Use Data Sources instead of hardcoding AMI IDs.

✔ Prefer for_each over count whenever resources have unique identities.

✔ Use Lifecycle rules carefully to avoid accidental resource deletion.

✔ Let Terraform manage dependencies automatically whenever possible.

🚀 Provisioning My First Cloud Infrastructure

After understanding Providers, Resources, Data Sources, and Meta-Arguments, it was finally time to deploy my first real cloud infrastructure using Terraform.

Unlike the previous days where I worked with local resources, today's challenge focused on provisioning infrastructure inside AWS. This helped me understand how Terraform interacts with cloud providers and automates the creation of networking and compute resources.

The infrastructure created during this challenge included:

Virtual Private Cloud (VPC)
Public Subnet
Internet Gateway
Route Table
Route Table Association
Security Group
EC2 Instance

Each resource had its own responsibility, but Terraform managed all of them together as a single Infrastructure as Code project.

🔄 Terraform Workflow

Before creating any infrastructure, I followed Terraform's standard workflow.

Write Configuration

        ↓

terraform init

        ↓

terraform validate

        ↓

terraform plan

        ↓

terraform apply

        ↓

terraform output

        ↓

terraform state list

        ↓

terraform destroy

This workflow helps verify the configuration before deployment and ensures infrastructure changes remain predictable.

🚀 Step 1 — Initialize Terraform

The first command executed in every Terraform project is:

terraform init

This command initializes the working directory and downloads all required provider plugins.

During initialization Terraform also creates:

.terraform/
.terraform.lock.hcl

Once the initialization completes successfully, Terraform confirms that the project is ready for execution.

📸 Screenshot

✅ Step 2 — Validate the Configuration

Before provisioning infrastructure, I validated the Terraform configuration.

terraform validate

Terraform checked the configuration for:

Syntax errors
Missing references
Invalid arguments
Unsupported attributes

If everything is correct, Terraform displays:

Success! The configuration is valid.

Running validation before deployment is a good habit because it catches mistakes early.

📸 Screenshot

📋 Step 3 — Review the Execution Plan

Next, I generated an execution plan.

terraform plan

Terraform compared the desired configuration with the current infrastructure and displayed every action it planned to perform.

The plan included:

Creating a VPC
Creating a Public Subnet
Attaching an Internet Gateway
Creating a Route Table
Creating a Security Group
Launching an EC2 Instance

At the end of the execution plan Terraform displayed a summary similar to:

Plan: 6 to add, 0 to change, 0 to destroy.

The planning phase provides an opportunity to review infrastructure changes before they are applied.

📸 Screenshot

🚀 Step 4 — Apply the Configuration

After reviewing the execution plan, I deployed the infrastructure.

terraform apply

Terraform asked for confirmation before creating the resources.

Do you want to perform these actions?

After entering:

yes

Terraform began provisioning the AWS infrastructure.

The deployment included:

Creating the VPC
Creating the Public Subnet
Creating the Internet Gateway
Configuring the Route Table
Creating the Security Group
Launching the EC2 Instance

After a few moments, Terraform displayed:

Apply complete!

Watching an entire cloud environment being created from a few Terraform files was one of the biggest highlights of today's challenge.

📸 Screenshot

☁️ Verifying Resources in AWS

Once Terraform finished creating the infrastructure, I logged into the AWS Management Console to verify that every resource had been created successfully.

I confirmed the following resources were available:

VPC
Public Subnet
Internet Gateway
Route Table
Security Group
EC2 Instance

Verifying the deployment in AWS helped me connect the Terraform configuration with the actual cloud resources.

📤 Viewing Terraform Outputs

Terraform Outputs display useful information after the infrastructure has been created.

Instead of manually searching for resource IDs or IP addresses, Terraform automatically displays them.

Command:

terraform output

Example Output:

instance_id = "i-xxxxxxxx"

public_ip = "xx.xx.xx.xx"

vpc_id = "vpc-xxxxxxxx"

Outputs make Terraform configurations much easier to consume and can also be used by other modules or CI/CD pipelines.

📸 Screenshot

📦 Understanding Terraform State

One of the most important concepts I learned today was the Terraform State File.

Whenever Terraform creates infrastructure, it stores information inside:

terraform.tfstate

The state file contains:

Resource IDs
Current infrastructure
Metadata
Dependency information

Terraform uses this file to compare the existing infrastructure with the desired configuration and determine what changes are required.

Without the state file, Terraform would not know which resources it already manages.

📄 Listing Managed Resources

Terraform also provides a command to display every resource currently tracked inside the state file.

terraform state list

The output included resources such as:

aws_vpc.main
aws_subnet.public
aws_internet_gateway.main
aws_route_table.public
aws_security_group.web
aws_instance.web

This command is extremely useful for debugging and understanding the current infrastructure managed by Terraform.

🍫 Bonus Exploration

After completing the core tasks, I explored some additional Terraform features that are commonly used in real-world Infrastructure as Code projects.

Although these features are considered bonus tasks for today's challenge, they helped me understand how Terraform can automate deployments even further and simplify infrastructure management.

⭐ Bonus 1 — Automating Server Setup using User Data

One of the bonus tasks suggested either attaching an Elastic IP or using User Data to automatically install Nginx during the EC2 instance launch.

For this project, I chose the User Data approach because it demonstrates Infrastructure as Code more effectively.

Instead of manually connecting to the EC2 instance and installing software, Terraform executes a shell script automatically when the instance boots for the first time.

Example:

user_data = <<-EOF
#!/bin/bash
yum update -y
yum install -y nginx
systemctl enable nginx
systemctl start nginx
EOF

This script performs the following actions automatically:

Updates the package repository.
Installs the Nginx web server.
Enables the Nginx service.
Starts the web server immediately after launch.

Using User Data eliminates manual configuration and ensures every EC2 instance is configured consistently.

This is one of the key advantages of Infrastructure as Code.

⭐ Bonus 2 — Visualizing Resource Dependencies

Terraform automatically understands the relationships between resources.

To visualize these dependencies, Terraform provides the following command:

terraform graph > graph.dot

This generates a DOT file containing the dependency graph.

Using Graphviz, the DOT file can be converted into an image.

dot -Tpng graph.dot -o graph.png

The generated graph clearly illustrates how Terraform determines the order in which resources should be created.

For example:

VPC
   │
   ▼
Subnet
   │
   ▼
Internet Gateway
   │
   ▼
Route Table
   │
   ▼
Security Group
   │
   ▼
EC2 Instance

Understanding this dependency graph helped me appreciate how Terraform automatically manages resource creation without requiring manual sequencing.

📸 Screenshot

⭐ Bonus 3 — Understanding the `moved` Block

Terraform also provides the moved block, which allows developers to rename resources without destroying and recreating them.

Example:

moved {

  from = aws_instance.web

  to   = aws_instance.server

}

Normally, renaming a resource would cause Terraform to destroy the existing resource and create a new one.

The moved block updates Terraform's state instead, preserving the existing infrastructure while reflecting the new resource name in the configuration.

Although I didn't need to rename any resources in this project, learning about the moved block gave me a better understanding of how Terraform safely handles refactoring.

🧹 Destroying Infrastructure

Once I verified that everything was working correctly, I removed all the resources created during this challenge.

Command:

terraform destroy

Terraform displayed a confirmation prompt.

Do you really want to destroy all resources?

After entering:

yes

Terraform destroyed every managed resource in the correct dependency order.

This included:

VPC
Public Subnet
Internet Gateway
Route Table
Security Group
EC2 Instance

Finally, Terraform displayed:

Destroy complete!

Destroying unused cloud resources is an important best practice because it helps avoid unnecessary AWS charges and keeps the environment clean.

📸 Screenshot

🎯 What I Learned

Day 3 was my first experience provisioning real cloud infrastructure using Terraform, and it significantly expanded my understanding of Infrastructure as Code.

Some of the key takeaways from today's challenge are:

Learned how Terraform communicates with AWS using Providers.
Understood the importance of Provider Version Pinning.
Explored the difference between Resources and Data Sources.
Learned the basics of AWS Networking, including VPCs, Subnets, Internet Gateways, Route Tables, Security Groups, and EC2.
Used Data Sources to retrieve existing AWS information dynamically.
Explored Terraform Meta-Arguments such as count, for_each, depends_on, and lifecycle.
Learned how Terraform automatically manages resource dependencies.
Understood the role of the Terraform State File.
Explored bonus concepts such as User Data, Terraform Graph, and the Moved Block.

The biggest lesson from today was that Infrastructure as Code isn't just about provisioning cloud resources—it's about creating infrastructure that is automated, reusable, version-controlled, and easy to maintain.

📂 Repository Structure

TerraWeek/
│
├── Day-01/
│   ├── README.md
│   ├── example/
│   └── images/
│
├── Day-02/
│   ├── README.md
│   ├── example/
│   └── images/
│
├── Day-03/
│   ├── terraform.tf
│   ├── main.tf
│   ├── variables.tf
│   ├── outputs.tf
│   ├── README.md
│   └── images/
│
└── ...

🚀 Conclusion

Day 3 was an exciting milestone in my Terraform journey because it introduced me to real cloud infrastructure.

The previous two days helped me understand Terraform syntax and HCL, but today's challenge showed me how those concepts come together to provision an actual AWS environment.

From configuring Providers and using Data Sources to deploying an EC2 instance inside a custom VPC, every step reinforced the power of Infrastructure as Code.

I also explored advanced Terraform capabilities such as Meta-Arguments, User Data, Terraform Graph, and the Moved Block, which are commonly used in production environments.

With three days of the TerraWeek Challenge completed, I now have a much stronger understanding of how Terraform can automate cloud infrastructure in a consistent, repeatable, and scalable way.

I'm excited to continue the journey and explore even more advanced Terraform concepts in the upcoming days.

🏷️ Tags

#Terraform #AWS #InfrastructureAsCode #CloudComputing #DevOps #HashiCorp #EC2 #VPC #Automation #CloudEngineer #LearnInPublic #TrainWithShubham #TerraWeekChallenge

🌱 TerraWeek Challenge – Day 2 - HCL Deep Dive: Variables, Types & Expressions

Kshitij Sharma — Mon, 13 Jul 2026 12:24:01 +0000

📅 Date: 13 July 2026

Welcome to Day 2 of my TerraWeek Challenge! 🚀

After completing Day 1, where I learned the fundamentals of Infrastructure as Code (IaC) and deployed my very first infrastructure using Terraform, today's challenge focused on understanding the language behind Terraform itself—HashiCorp Configuration Language (HCL).

On Day 1, I learned how Terraform works.

On Day 2, I explored how Terraform configurations are actually written.

Instead of using fixed values inside Terraform files, I learned how to create reusable and dynamic configurations using Variables, Expressions, Locals, Outputs, and several built-in Terraform Functions.

To put these concepts into practice, I deployed an Nginx Docker container using the Docker Provider, where almost every configurable value was controlled through variables instead of being hardcoded.

By the end of today's challenge, I gained a much deeper understanding of how professional Terraform configurations are structured and why HCL plays such an important role in Infrastructure as Code.

📚 Learning Objectives

By the end of Day 2, I was able to:

Understand HashiCorp Configuration Language (HCL)
Learn the structure of Terraform configuration files
Differentiate between Blocks and Arguments
Work with HCL Expressions
Create reusable configurations using Variables
Explore all major Terraform Variable Types
Apply Variable Validation
Protect confidential information using Sensitive Variables
Use Locals for reusable values
Generate useful Outputs
Explore commonly used Terraform Functions
Understand Variable Precedence
Deploy an Nginx Docker container using Terraform
Explore advanced HCL concepts such as For Expressions and Conditional Expressions

📂 Project Structure

Before starting the implementation, I organized my Terraform project into multiple files. Instead of writing everything inside a single file, each file was given a specific responsibility, making the project much cleaner and easier to maintain.

.
├── terraform.tf
├── main.tf
├── variables.tf
├── outputs.tf
├── terraform.tfvars
├── README.md
└── images/

Each file serves a different purpose:

File	Purpose
`terraform.tf`	Configures the Terraform and Docker provider
`main.tf`	Defines Docker resources
`variables.tf`	Declares input variables
`terraform.tfvars`	Stores variable values
`outputs.tf`	Displays useful information after deployment
`README.md`	Project documentation

Organizing Terraform projects in this way improves readability and makes the codebase easier to manage as projects grow.

📸 Screenshot

⚙️ Prerequisites

Before working on today's challenge, I made sure that the following tools were installed and configured correctly.

Terraform
Docker Desktop
Git
Visual Studio Code
HashiCorp Terraform Extension

Since today's project uses the Docker Provider, Docker Desktop must be running before executing any Terraform commands.

🛠 Verifying Terraform Installation

The first step was to verify whether Terraform was installed correctly.

Command:

terraform version

This command displays the currently installed version of Terraform along with additional build information.

Verifying the installation beforehand helps avoid unnecessary troubleshooting later in the workflow.

📸 Screenshot

🐳 Verifying Docker Installation

Since the project deploys an Nginx container locally, Docker must also be installed and running.

I verified Docker by executing the following command:

docker version

Alternatively, the following command can also be used:

docker ps

If Docker is running correctly, it displays the client and server information (or the list of running containers).

Ensuring Docker is running before initializing Terraform prevents provider connection errors during deployment.

📸 Screenshot

📖 What is HCL?

HCL stands for HashiCorp Configuration Language.

It is a declarative language developed by HashiCorp specifically for writing Infrastructure as Code.

Unlike traditional programming languages where developers describe every individual step, HCL focuses on defining the desired state of the infrastructure.

Terraform reads the configuration, compares it with the current state, and automatically determines what changes are required to reach the desired state.

Because of this declarative approach, HCL configurations remain simple, readable, and easy to maintain.

🌍 Why Do We Need HCL?

Imagine deploying the same infrastructure for multiple environments such as:

Development
Testing
Staging
Production

If every configuration contained hardcoded values, we would have to edit the Terraform files before every deployment.

This approach quickly becomes difficult to manage.

HCL solves this problem by allowing configurations to become:

Reusable
Readable
Consistent
Scalable
Easy to Maintain
Version Controlled

Instead of changing the infrastructure code repeatedly, we simply provide different variable values and Terraform takes care of the rest.

This is one of the primary reasons why Terraform is widely adopted across the industry.

🏗 Understanding the Anatomy of an HCL Block

Every Terraform configuration is built using Blocks.

A block represents a specific component of the configuration, such as a provider, resource, variable, output, or module.

The basic syntax of an HCL block looks like this:

block_type "label_one" "label_two" {

  argument = value

}

Each part of the block has its own purpose.

Component	Description
Block Type	Specifies the kind of block
Label One	Resource or Provider Type
Label Two	Local Name
Arguments	Configuration values

Example:

resource "docker_container" "web" {

  name = var.container_name

}

Here:

resource is the block type.
docker_container specifies the type of resource.
web is the local name.
name is an argument that receives its value from a variable.

Understanding this structure is essential because almost every Terraform configuration follows this pattern.

📦 Understanding Blocks

Terraform provides different block types, each designed for a specific purpose.

Some commonly used blocks include:

terraform
provider
resource
variable
locals
output
module

For example, the Provider block tells Terraform which platform it should communicate with.

provider "docker" {

}

Similarly, a Resource block describes the infrastructure Terraform should create.

resource "docker_image" "nginx" {

  name = "nginx:latest"

}

A Variable block defines user input that can be reused throughout the project.

variable "container_name" {

  type = string

}

Although these blocks perform different tasks, they all follow the same HCL syntax.

📝 Arguments vs Blocks

One concept that initially confused me was the difference between Arguments and Blocks.

After experimenting with Terraform configurations, the distinction became much clearer.

Arguments

Arguments assign values to configuration settings.

Example:

name = "nginx"

More examples:

image    = "nginx:latest"
internal = 80
external = 8080

Arguments always follow the format:

key = value

Blocks

Blocks, on the other hand, group related configuration together.

Example:

ports {

  internal = 80
  external = 8080

}

Here, ports is a nested block that contains multiple arguments.

This structure keeps Terraform configurations organized and improves readability.

Arguments vs Blocks

Arguments	Blocks
Assign values	Group related configuration
Written as `key = value`	Written using `{ }`
Cannot contain nested configuration	Can contain arguments and nested blocks

Understanding this difference makes it much easier to read and write Terraform configurations.

🔗 Understanding HCL Expressions

Expressions allow Terraform configurations to become dynamic.

Instead of hardcoding values, Terraform can calculate values during execution.

Expressions can reference:

Variables
Resources
Outputs
Functions
Operators

Example:

var.environment

This expression retrieves the value stored inside the environment variable.

Expressions can also reference resource attributes.

docker_container.web.name

Terraform automatically reads the name attribute of the Docker container.

Another useful feature is String Interpolation.

Example:

"${var.environment}-container"

If the value of environment is:

dev

Terraform automatically produces:

dev-container

Expressions are one of the most powerful features of HCL because they allow infrastructure to adapt based on input values instead of relying on hardcoded configurations.

🔤 Understanding Terraform Variables

After learning the basics of HCL, the next step was understanding one of the most powerful features of Terraform—Variables.

When I first started writing Terraform configurations, I noticed that many values such as the container name, ports, environment, and image name were hardcoded inside the resource block.

Although this approach works for small projects, it quickly becomes difficult to manage when the same infrastructure needs to be deployed multiple times.

For example, suppose I want to deploy the same Docker container for different environments.

Development might use:

dev-nginx

Staging could use:

staging-nginx

Production might use:

prod-nginx

If every value is hardcoded, I would need to edit my Terraform configuration before every deployment.

Terraform Variables solve this problem.

Instead of changing the code, I only change the input values, and Terraform automatically provisions the required infrastructure.

This makes Terraform configurations:

Reusable
Flexible
Easier to Maintain
Environment Independent

📦 Creating Variables

Variables are declared using the variable block.

A simple variable looks like this:

variable "container_name" {

  description = "Name of Docker Container"

  type = string

}

Each variable usually contains:

Name
Description
Type
Default Value (Optional)
Validation (Optional)

Once declared, variables can be used anywhere inside the project.

Example:

resource "docker_container" "web" {

  name = var.container_name

}

Here, Terraform replaces the variable with the value provided during execution.

📘 Understanding Variable Types

Terraform supports multiple variable types depending on the kind of data we want to store.

For this assignment, I explored all the major variable types available in Terraform.

🔹 String

A String stores textual values.

Example:

variable "container_name" {

  type = string

}

Possible values:

nginx

terraweek-web

docker-container

String variables are commonly used for:

Names
Regions
Tags
Environment Names

🔹 Number

Number variables store numeric values.

Example:

variable "external_port" {

  type = number

}

Possible values:

Terraform automatically understands these values as numbers instead of text.

🔹 Boolean

Boolean variables contain only two values.

true

false

Example:

variable "enable_logging" {

  type = bool

}

These are commonly used to enable or disable optional features.

📚 Collection Types

Collection Types store multiple values together.

Terraform provides several collection types.

List

A List stores ordered values.

Example:

variable "ports" {

  type = list(number)

}

Example value:

[
80,
443,
8080
]

Lists preserve the order of values.

Map

Maps store values as Key → Value pairs.

Example:

variable "extra_labels" {

  type = map(string)

}

Value:

{

project = "terraweek"

owner = "aditya"

environment = "dev"

}

Maps are commonly used for metadata and labels.

Set

A Set is similar to a List but removes duplicate values automatically.

Example:

variable "allowed_ports" {

  type = set(number)

}

Example:

[
80,
80,
443,
8080
]

Terraform internally converts this into:

80
443
8080

This helps avoid duplicate entries.

🧱 Structural Types

Terraform also supports more advanced data structures.

Object

Objects group multiple related values together.

Example:

variable "container_config" {

  type = object({

    name = string

    port = number

  })

}

Example Value:

{

name = "nginx"

port = 8080

}

Objects make configurations cleaner by grouping related properties.

Tuple

A Tuple stores multiple values with predefined data types.

Example:

variable "sample_tuple" {

  type = tuple([

    string,

    number,

    bool

  ])

}

Example Value:

[
"nginx",
8080,
true
]

Unlike Lists, each position inside a Tuple has its own type.

✅ Variable Validation

Terraform allows us to validate user input before infrastructure is created.

This helps prevent accidental configuration mistakes.

For this assignment, I restricted the environment variable so that only three values are accepted.

variable "environment" {

  type = string

  default = "dev"

  validation {

    condition = contains(

      ["dev","staging","prod"],

      var.environment

    )

    error_message = "Environment must be dev, staging or prod."

  }

}

Now, if someone provides:

terraform apply -var="environment=test"

Terraform immediately throws an error instead of creating invalid infrastructure.

Validation makes configurations much safer and more reliable.

🔒 Sensitive Variables

Sometimes Terraform configurations contain confidential information like:

Passwords
API Keys
Tokens
Secrets

Displaying these values in logs or outputs would be a security risk.

Terraform provides the Sensitive Variable feature to protect confidential information.

Example:

variable "docker_password" {

  type = string

  sensitive = true

}

Whenever Terraform displays outputs, sensitive values are automatically hidden.

This prevents accidental exposure of credentials.

📍 Understanding Locals

Locals are reusable values calculated inside Terraform.

Instead of writing the same expression multiple times, we define it once inside a locals block.

Example:

locals {

  name_prefix = "tws-${var.environment}"

}

Now the same value can be reused anywhere.

local.name_prefix

Using Locals improves readability and reduces duplicate code.

🛠 Terraform Built-in Functions

Terraform provides many built-in functions for working with strings, collections, and numbers.

During this challenge, I explored some of the most commonly used functions.

upper()

Converts text into uppercase.

upper("terraweek")

Output:

TERRAWEEK

merge()

Combines multiple maps.

merge(
{project="terraweek"},
var.extra_labels
)

join()

Joins multiple strings using a separator.

join("-",["terra","week","2026"])

Output:

terra-week-2026

format()

Formats text.

format(
"http://localhost:%d",
var.external_port
)

Output:

http://localhost:8080

These functions make Terraform configurations much more dynamic and reusable.

💻 Exploring Terraform Console

Terraform Console is an interactive shell used to evaluate HCL expressions and test Terraform functions without modifying configuration files.

To start the console:

terraform console

I tested several expressions during this challenge.

> upper("terraweek")
"TERRAWEEK"

> merge({a=1},{b=2})
{
  "a" = 1
  "b" = 2
}

> join("-",["terra","week","2026"])
"terra-week-2026"

> length(["docker","terraform","hcl"])
3

Using the console helped me understand how functions and expressions behave before using them in actual Terraform code.

📸 Screenshot

🚀 Building a Real Terraform Project

After learning HCL, Variables, Locals, Functions, and Expressions, it was finally time to implement everything in a real Terraform project.

Instead of provisioning cloud resources on AWS or Azure, today's assignment uses the Docker Provider to deploy an Nginx container locally.

The advantage of using Docker is that we can practice all Terraform concepts without creating a cloud account or worrying about cloud costs.

Although the provider changes, the Terraform workflow remains exactly the same.

Whether Terraform creates an EC2 instance on AWS or an Nginx container using Docker, the configuration syntax and workflow remain identical.

Today's implementation helped me understand how Terraform combines HCL with providers to automate infrastructure deployment.

🔄 Terraform Workflow

Before executing any command, it's important to understand Terraform's workflow.

Terraform follows a simple sequence:

Write Configuration

        ↓

terraform init

        ↓

terraform fmt

        ↓

terraform validate

        ↓

terraform plan

        ↓

terraform apply

        ↓

terraform output

        ↓

terraform destroy

Following this workflow ensures that the infrastructure is created safely and consistently.

🚀 Step 1 — Initialize Terraform

The very first command executed in every Terraform project is:

terraform init

This command initializes the current working directory.

During initialization Terraform:

Downloads required providers
Creates the .terraform directory
Creates the provider lock file
Prepares the project for execution

If initialization is successful, Terraform displays a success message.

Running terraform init is mandatory before executing any other Terraform command.

📸 Screenshot

✅ Step 2 — Validate Configuration

Once Terraform is initialized, the next step is validating the configuration.

Command:

terraform validate

Validation checks the configuration for:

Syntax errors
Invalid references
Missing blocks
Configuration mistakes

If everything is correct, Terraform displays:

Success! The configuration is valid.

Running validation before deployment helps detect mistakes early.

📸 Screenshot

📋 Step 3 — Preview Infrastructure

Before creating infrastructure, Terraform allows us to preview every planned change.

Command:

terraform plan

The execution plan displayed the following actions:

Download the Nginx Docker image.
Create a Docker container.
Configure the required port mapping.
Generate output values.

At the end of the execution plan, Terraform summarized the changes.

Example:

Plan: 2 to add, 0 to change, 0 to destroy.

Reviewing the execution plan before deployment is considered one of Terraform's best practices because it allows us to verify the changes before they are applied.

📸 Screenshot

🚀 Step 4 — Apply Configuration

After verifying the execution plan, the infrastructure was deployed using:

terraform apply

Terraform displayed a confirmation prompt.

Do you want to perform these actions?

After typing:

yes

Terraform performed the following tasks:

Pulled the latest Nginx image.
Created the Docker container.
Exposed port 8080.
Generated output values.

After a few seconds, Terraform displayed:

Apply complete!

At this point, the infrastructure had been successfully created.

📸 Screenshot

🐳 Verifying the Running Container

After deployment, I verified that the Docker container was actually running.

Command:

docker ps

This command lists all currently running Docker containers.

The output confirmed that:

The Nginx image was downloaded.
The container was running successfully.
Port 8080 was mapped correctly.

Verifying the container is a good practice before testing the application in a browser.

📸 Screenshot

🌐 Accessing the Nginx Application

Once the container was running, I opened the following URL in my browser:

http://localhost:8080

The default Welcome to nginx! page appeared successfully.

This confirmed that Terraform had successfully deployed and configured the Docker container.

It was satisfying to see the infrastructure working exactly as expected.

📸 Screenshot

📤 Viewing Terraform Outputs

Terraform Outputs display useful information after infrastructure has been created.

Instead of manually checking the container configuration, Terraform automatically displayed the required values.

Command:

terraform output

Example Output:

container_name = "tws-web"

access_url = "http://localhost:8080"

image = "nginx:latest"

Outputs are extremely useful because they can also be consumed by other Terraform modules or CI/CD pipelines.

📸 Screenshot

📊 Understanding Variable Precedence

Terraform variables can receive values from multiple sources.

If the same variable exists in more than one location, Terraform follows a fixed precedence order.

The highest priority source always wins.

-var

      ↓

-var-file

      ↓

*.auto.tfvars

      ↓

terraform.tfvars

      ↓

TF_VAR_ Environment Variables

      ↓

Default Values

Understanding variable precedence is important because it allows the same Terraform configuration to be reused across multiple environments without modifying the code.

🍫 Bonus Exploration

After completing the required tasks, I explored a few advanced HCL features.

These features make Terraform configurations cleaner and more powerful.

Bonus 1 — For Expressions

Terraform provides For Expressions to transform collections.

Example:

locals {

  upper_labels = [

    for label in ["docker","terraform","nginx"] :

    upper(label)

  ]

}

Instead of transforming each value manually, Terraform automatically iterates through every element.

This feature becomes extremely useful while working with Lists and Maps.

📸 Screenshot

Bonus 2 — Conditional Expressions

Conditional Expressions work similarly to an if-else statement.

Example:

var.environment == "prod"

?

"large"

:

"small"

If the environment is Production, Terraform returns:

large

Otherwise:

small

Conditional expressions help create reusable configurations for different deployment environments.

📸 Screenshot

Bonus 3 — Optional Object Attributes

Terraform also supports optional attributes inside Object types.

Example:

variable "container_config" {

  type = object({

    name = string

    port = optional(number,8080)

  })

}

If the port value is not supplied, Terraform automatically assigns the default value of 8080.

This feature helps create flexible object definitions without forcing users to specify every attribute.

🧹 Destroying Infrastructure

Once the testing was completed, I removed all the resources created by Terraform.

Command:

terraform destroy

Terraform again displayed a confirmation prompt.

After typing:

yes

Terraform removed:

Docker Container
Docker Image (if configured)
Terraform-managed resources

Finally, Terraform displayed:

Destroy complete!

Destroying unused infrastructure is a good practice because it keeps the environment clean and avoids unnecessary resource usage.

📸 Screenshot

🎯 What I Learned

Day 2 helped me understand the actual language behind Terraform.

Some of the key takeaways from today's challenge are:

Learned how Terraform configurations are written using HCL.
Understood the purpose of Blocks, Arguments, and Expressions.
Explored all major Terraform Variable Types.
Applied Validation Rules to improve configuration reliability.
Protected confidential values using Sensitive Variables.
Used Locals to avoid duplicate code.
Worked with Terraform Outputs.
Explored useful built-in Functions.
Used Terraform Console to experiment with expressions.
Understood Variable Precedence.
Successfully deployed an Nginx Docker container using Terraform.
Explored advanced HCL concepts like For Expressions, Conditional Expressions, and Optional Object Attributes.

Overall, today's challenge gave me a much stronger understanding of writing reusable and production-ready Terraform configurations.

🚀 Conclusion

Day 2 marked an important milestone in my Terraform learning journey.

While Day 1 introduced the fundamentals of Infrastructure as Code, today's challenge focused on writing clean, reusable, and maintainable Terraform configurations using HCL.

From understanding variables and expressions to deploying a real Docker container and exploring advanced HCL features, every concept contributed to building a stronger foundation in Terraform.

I'm excited to continue the TerraWeek Challenge and dive deeper into modules, state management, and cloud infrastructure in the upcoming days.

🏷️ Tags

#Terraform #HashiCorp #InfrastructureAsCode #Docker #DevOps #HCL #IaC #TrainWithShubham #TerraWeekChallenge

Day 1 of #TerraWeek — From Zero to First `terraform init`

Kshitij Sharma — Sun, 12 Jul 2026 10:46:45 +0000

🌱 TerraWeek Challenge – Day 1

Introduction to Infrastructure as Code (IaC) & Terraform Basics

📅 Date: 12 July 2026

Welcome to Day 1 of my TerraWeek Challenge! 🚀

Today was all about understanding Infrastructure as Code (IaC), learning the fundamentals of Terraform, and provisioning my first resource locally. Although the project itself was simple, it helped me understand the workflow that powers modern infrastructure automation.

🎯 Learning Objectives

By the end of Day 1, I was able to:

Understand what Infrastructure as Code (IaC) is.
Learn why Terraform is one of the most popular IaC tools.
Install and configure Terraform.
Learn Terraform's core workflow.
Understand important Terraform concepts.
Create and destroy my first infrastructure.
Explore OpenTofu and Terraform's lock file.

📖 Why Infrastructure as Code?

Before writing any Terraform configuration, I wanted to understand why Infrastructure as Code exists.

Managing cloud infrastructure manually through a web console may work for a personal project, but it quickly becomes difficult as projects grow.

Some common problems with manual infrastructure are:

Human errors while configuring resources.
Difficulty recreating the exact same environment.
Poor collaboration among teams.
No version history for infrastructure changes.
Slow deployments.

Infrastructure as Code solves these problems by allowing infrastructure to be written, reviewed, and version-controlled just like application code.

Instead of remembering every click in a cloud console, everything is defined inside configuration files that can be executed repeatedly.

🌍 What is Terraform?

Terraform is an open-source Infrastructure as Code (IaC) tool developed by HashiCorp.

It uses HashiCorp Configuration Language (HCL) to describe the desired infrastructure, and Terraform automatically determines the changes required to reach that desired state.

One of Terraform's biggest strengths is that it is provider agnostic, meaning the same tool can manage resources across:

AWS
Azure
Google Cloud
Docker
Kubernetes
GitHub
Cloudflare
DigitalOcean
VMware
Hundreds of other providers

⭐ Why Terraform?

Terraform has become one of the industry's most widely adopted Infrastructure as Code tools because it offers:

Declarative configuration
Multi-cloud support
Huge provider ecosystem
State management
Reusable modules
Version-controlled infrastructure
Easy CI/CD integration

⚖️ Terraform vs Alternatives

Tool	Comparison
Terraform	Multi-cloud Infrastructure as Code using HCL
OpenTofu	Community-driven open-source fork of Terraform
Pulumi	Uses programming languages like Python and Go instead of HCL
CloudFormation	AWS-only Infrastructure as Code service
Ansible	Primarily used for configuration management rather than provisioning infrastructure

🛠 Installing Terraform

The first practical step was installing the latest version of Terraform.

After installation, I verified everything using:

terraform version
terraform -help

Terraform Version

Terraform Help

VS Code Extension

To improve the development experience, I installed the official HashiCorp Terraform Extension.

Features include:

Syntax Highlighting
IntelliSense
Auto Completion
Formatting
Validation

Screenshot

📘 Important Terraform Terminologies

Before creating any infrastructure, I learned the six core concepts that form the foundation of every Terraform project.

Provider

A provider is a plugin that allows Terraform to communicate with a specific platform.

Example:

provider "aws" {
  region = "ap-south-1"
}

Resource

A resource represents any infrastructure object managed by Terraform.

Example:

resource "local_file" "example" {
  filename = "hello.txt"
  content  = "Hello Terraform"
}

State

Terraform stores information about all managed resources inside a file named:

terraform.tfstate

The state file allows Terraform to understand what already exists and what changes need to be made.

Plan

Before applying any changes, Terraform generates an execution plan showing exactly what will happen.

terraform plan

This provides an additional safety check before modifying infrastructure.

HCL

HCL (HashiCorp Configuration Language) is the language used to write Terraform configurations.

Example:

resource "random_pet" "name" {
  length = 2
}

Module

A module is a reusable collection of Terraform configuration.

Instead of rewriting the same infrastructure repeatedly, modules allow configurations to be packaged and reused.

Example:

module "network" {
  source = "./modules/network"
}

🚀 My First Terraform Workflow

The challenge uses the local and random providers, allowing me to learn Terraform without needing a cloud account.

I followed the complete Terraform workflow.

Step 1 – Initialize Terraform

terraform init

Terraform downloaded the required providers and initialized the project.

Step 2 – Format Code

terraform fmt

Automatically formatted the Terraform configuration files.

Step 3 – Validate Configuration

terraform validate

Terraform checked the configuration for syntax errors.

Step 4 – Preview Changes

terraform plan

Terraform displayed the execution plan before creating any resources.

Step 5 – Apply Configuration

terraform apply

Terraform created the required resources.

Generated Output

cat greeting.txt

The local file was successfully created.

Step 6 – Destroy Infrastructure

terraform destroy

Terraform removed every resource it had created.

🔄 Terraform Workflow

Write Configuration
        │
        ▼
terraform init
        │
        ▼
terraform fmt
        │
        ▼
terraform validate
        │
        ▼
terraform plan
        │
        ▼
terraform apply
        │
        ▼
terraform destroy

🍫 Bonus Exploration

Enable CLI Autocomplete

terraform -install-autocomplete

OpenTofu

OpenTofu is a community-driven fork of Terraform that remains fully open source while maintaining compatibility with existing Terraform configurations.

tofu version

Understanding `.terraform.lock.hcl`

During initialization, Terraform generated a file named:

.terraform.lock.hcl

This file records:

Provider versions
Provider checksums
Dependency information

Tracking this file in Git ensures that every developer and CI/CD pipeline uses the exact same provider versions.

💡 Key Takeaways

Day 1 helped me understand that Terraform is much more than a provisioning tool.

The biggest lesson was learning how Infrastructure as Code brings consistency, automation, and version control to infrastructure management. Even with a simple local project, I experienced Terraform's complete lifecycle—from initialization to clean destruction of resources.

This foundation will make it much easier to build real cloud infrastructure in the upcoming TerraWeek sessions.

📂 Project Structure

Day-01/
│── README.md
│── example/
│── images/
│   ├── terraform-version.png
│   ├── terraform-help.png
│   ├── vscode-extension.png
│   ├── init.png
│   ├── fmt.png
│   ├── validate.png
│   ├── plan.png
│   ├── apply.png
│   ├── greeting.png
│   ├── destroy.png
│   ├── autocomplete.png
│   ├── opentofu-version.png
│   └── lock-file.png

🚀 Conclusion

Day 1 laid the foundation for my Terraform journey. I learned why Infrastructure as Code has become an essential DevOps practice, explored Terraform's core concepts, completed my first Terraform workflow, and understood how infrastructure can be managed in a consistent, repeatable, and version-controlled way.

Looking forward to Day 2, where I'll dive deeper into HCL, variables, expressions, and reusable configurations.

🏷️ Tags

#Terraform #InfrastructureAsCode #DevOps #HashiCorp #OpenTofu #GitHub #CloudComputing #TrainWithShubham #TerraWeekChallenge

Why is it important to make a good api rather than a clean api

Kshitij Sharma — Thu, 23 Apr 2026 09:25:56 +0000

Kshitij Sharma

Apr 23

When Your “Clean” REST API Becomes a Production Nightmare

#api #backend #systemdesign #webdev

4 min read

When Your “Clean” REST API Becomes a Production Nightmare

Kshitij Sharma — Thu, 23 Apr 2026 04:41:21 +0000

Everything looked perfect on paper.

Clean endpoints
Nice resource naming
Proper HTTP methods

Then production hit:

Clients started retrying aggressively
Data inconsistencies appeared
Versioning became a mess
One change broke 3 consumers

That’s when reality kicks in:

REST API design is not about elegance — it’s about survivability under change.

The Real Constraints of REST APIs in Production

You’re not designing endpoints.
You’re designing contracts under uncertainty.

What actually shapes your API:

Multiple clients (web, mobile, third-party)
Network unreliability
Backward compatibility pressure
Partial failures
Latency budgets
Data ownership boundaries

Ignoring these = brittle APIs that collapse under scale.

Resource Modeling Is Where Most People Fail

Everyone talks about /users and /orders.

That’s surface-level.

The real question:

What is the lifecycle of your resource?

Bad Design (naive CRUD mindset)

POST /orders
GET /orders/:id
PUT /orders/:id
DELETE /orders/:id

Looks fine. Completely wrong for real systems.

Why?

Orders aren’t freely mutable
State transitions matter (created → paid → shipped)
Business rules are ignored

Model State Transitions Explicitly

Better:

POST   /orders
POST   /orders/:id/pay
POST   /orders/:id/ship
POST   /orders/:id/cancel

Now:

You encode business logic in API
You prevent invalid transitions
You reduce client-side bugs

Idempotency: The Thing That Saves You From Chaos

Most APIs break under retries.

Reality:

Clients retry
Proxies retry
Load balancers retry

If your endpoint isn’t idempotent → duplicate operations.

Real Failure Case

Payment API:

POST /payments

Client times out → retries → duplicate charge.

Congrats, you just lost user trust.

Fix: Idempotency Keys

POST /payments
Idempotency-Key: 8f3a-xyz-123

Server logic:

if (exists(idempotencyKey)) {
  return previousResponse;
}

processPayment();
storeResult(idempotencyKey);

Partial Failure Handling (The Silent Killer)

Your API calls:

DB
Cache
External service

One fails.

Now what?

Most APIs:

Return 500 and pray.

That’s not a strategy.

Better Approach: Explicit Failure Semantics

Return partial success where valid
Use compensating actions
Log correlation IDs

Example:

{
  "status": "partial_success",
  "data": {...},
  "failed_dependencies": ["inventory-service"]
}

Versioning: Where APIs Go to Die

Naive approach:

/v1/users
/v2/users

Problem:

You now maintain 2 systems forever
Clients don’t migrate

Better Strategy: Evolution Over Versioning

Add fields, don’t remove
Use default values
Deprecate gradually

When Versioning Is Actually Needed

Breaking contract changes
Semantic shifts (not just fields)

Even then:

Prefer header-based versioning

Accept: application/vnd.myapi.v2+json

Overfetching vs Underfetching

Classic REST problem.

Overfetching

GET /users/:id

Returns:

Name
Email
Address
Preferences
Activity logs

Client only needs name.

Waste:

bandwidth
latency

Underfetching

Client needs:

user
orders
payments

Makes 3 calls.

Now latency multiplies.

Practical Fix: Controlled Expansion

GET /users/:id?include=orders,payments

Trade-off:

More complex backend
Better client efficiency

Implementation: What a Production-Ready API Looks Like

Express.js Example (Opinionated Structure)

const express = require('express');
const app = express();

// Middleware: request ID for tracing
app.use((req, res, next) => {
  req.id = crypto.randomUUID();
  next();
});

// Idempotency middleware
const store = new Map();

app.post('/payments', async (req, res) => {
  const key = req.headers['idempotency-key'];

  if (store.has(key)) {
    return res.json(store.get(key));
  }

  const result = await processPayment(req.body);

  store.set(key, result);
  res.json(result);
});

// Explicit state transition
app.post('/orders/:id/ship', async (req, res) => {
  const order = await getOrder(req.params.id);

  if (order.status !== 'paid') {
    return res.status(400).json({ error: 'Invalid state' });
  }

  await shipOrder(order);
  res.json({ status: 'shipped' });
});

Common Mistakes That Kill APIs

❌ Treating REST Like CRUD

You ignore:

business logic
state transitions
invariants

❌ Ignoring Timeouts and Retries

Your system works… until network instability hits.

❌ No Observability

No:

request IDs
structured logs
tracing

Debugging becomes guessing.

❌ Tight Coupling to DB Schema

Changing DB → breaks API

Fix:

API is a contract, not a reflection of your database

❌ Overusing HTTP Status Codes

People do:

200 OK (with error inside body)

Or:

500 for everything

Both are wrong.

Trade-offs You Can’t Escape

Flexibility vs Simplicity

Flexible APIs → harder to maintain
Simple APIs → limited use cases

Performance vs Consistency

Strong consistency → slower
Eventual consistency → complex

Versioning vs Evolution

Versioning → fragmentation
Evolution → constraints on change

Abstraction vs Control

High abstraction → easy usage
Low abstraction → better performance

What a Mature REST API Actually Looks Like

Explicit state transitions
Idempotent operations
Backward-compatible changes
Observability baked in
Controlled data fetching
Failure-aware responses

Final Reality Check

If your API:

breaks on retries
can’t evolve without versioning chaos
hides business logic
lacks observability

It’s not production-ready.

Key Takeaways

REST is not CRUD — it’s contract design under failure
Idempotency is non-negotiable
State transitions must be explicit
Versioning is a last resort, not default
Most failures come from network behavior, not code
API design is about handling bad conditions, not ideal flows

If you design APIs assuming everything works perfectly,
your system will fail the moment it doesn’t.

Understand the mechanism behind the api failing randomly

Kshitij Sharma — Wed, 22 Apr 2026 17:41:09 +0000

Kshitij Sharma

Apr 22

When Your API “Randomly” Starts Timing Out

#backend #webdev #networking #distributedsystems

4 min read

When Your API “Randomly” Starts Timing Out

Kshitij Sharma — Wed, 22 Apr 2026 17:30:48 +0000

You deploy a perfectly fine service. Load tests passed. Latency looked clean. Then production hits—and suddenly:

P95 latency spikes
Requests hang without logs
CPU is fine, memory is fine… but users are screaming

This isn’t a “bug.” This is you not understanding the actual HTTP request lifecycle beyond the textbook diagram.

If you don’t know what really happens between a client sending a request and your handler returning a response, you’re flying blind.

HTTP Request Lifecycle — What Actually Happens Under the Hood

Forget diagrams like Client → Server → Response. That’s marketing-level abstraction.

A real request goes through:

1. Connection Establishment

DNS resolution
TCP handshake (3-way)
TLS handshake (if HTTPS)
Connection pooling / reuse (keep-alive)

2. Kernel → User Space Transition

NIC receives packet → kernel buffer
Socket read readiness via epoll/kqueue
Data copied into user space buffers

3. HTTP Parsing

Raw bytes → protocol parsing (headers, method, path)
Chunked decoding / content-length validation
Header normalization

4. Routing & Middleware Chain

Path matching (often regex or trie-based)
Middleware execution (auth, logging, rate limiting)

5. Business Logic Execution

DB calls
External APIs
CPU-bound work

6. Response Construction

Serialization (JSON, protobuf, etc.)
Compression (gzip, brotli)

7. Write Back to Socket

Kernel send buffer
TCP congestion control
Potential partial writes

8. Connection Lifecycle Decision

Keep-alive reuse vs close
Idle timeout tracking

Miss any one of these layers, and you’ll misdiagnose production issues.

Where Systems Actually Break

Let’s cut the theory. Real failures:

🔴 1. Head-of-Line Blocking in Connection Pools

You think you're async, but your HTTP client pool is exhausted.

Result:

Requests queue waiting for a free connection
Latency explodes without CPU increase

🔴 2. Slow Clients = Resource Leaks

If a client reads slowly:

Your server keeps buffers open
Threads/event-loop slots remain occupied

This is classic slowloris territory.

🔴 3. Middleware Abuse

Stacking 10 middlewares sounds clean.

Reality:

Each adds latency
Each may block (logging, auth calls)
Hard to reason about ordering

🔴 4. TLS Handshake Overhead

Without reuse:

Every request pays ~1–2 RTT extra
CPU spikes due to crypto

🔴 5. Kernel Buffer Backpressure

Your app “sent” the response.

Kernel says:

Nope, buffer full. Try later.

If you ignore this:

Writes block
Event loop stalls
Throughput collapses

Architecture Decisions That Actually Matter

1. Thread-per-request vs Event Loop

Thread-per-request (e.g., classic Java)

Pros:

Simpler mental model
Blocking code is fine

Cons:

Context switching overhead
Memory per thread (~1MB stack)

Event-driven (Node.js, Netty, Go runtime hybrid)

Pros:

High concurrency
Efficient IO

Cons:

Blocking = catastrophic
Debugging harder

2. Reverse Proxy in Front (NGINX / Envoy)

You should not expose your app server directly.

Why:

Handles TLS termination
Absorbs slow clients
Better connection management

3. Connection Reuse Strategy

Bad:

New TCP per request

Better:

HTTP/1.1 keep-alive

Best:

HTTP/2 multiplexing

Trade-off:

HTTP/2 introduces head-of-line blocking at TCP layer
QUIC (HTTP/3) fixes it but adds complexity

Implementation: What This Looks Like in Code

Example: Minimal HTTP Server (Node.js — showing lifecycle touchpoints)

const http = require('http');

const server = http.createServer((req, res) => {
  // 1. Request received (already parsed by Node's HTTP parser)

  // 2. Middleware simulation
  const start = Date.now();

  if (req.headers['x-block']) {
    // simulate bad middleware
    while (Date.now() - start < 100) {}
  }

  // 3. Business logic
  setTimeout(() => {
    const responseBody = JSON.stringify({ ok: true });

    // 4. Response write
    res.setHeader('Content-Type', 'application/json');
    res.setHeader('Content-Length', Buffer.byteLength(responseBody));

    res.write(responseBody);

    // 5. End response (flush to kernel)
    res.end();
  }, 10);
});

// 6. Connection-level tuning
server.keepAliveTimeout = 5000;
server.headersTimeout = 6000;

server.listen(3000);

Where This Code Lies to You

You don’t see TCP
You don’t see kernel buffers
You don’t control backpressure explicitly
You don’t see partial writes

That abstraction is convenient—and dangerous.

Advanced Concern: Backpressure Handling

Most people ignore this. That’s why systems collapse under load.

Example (Node.js stream backpressure):

function writeResponse(res, data) {
  const canContinue = res.write(data);

  if (!canContinue) {
    // Kernel buffer full — wait
    res.once('drain', () => {
      console.log('Resumed writing');
    });
  }
}

If you ignore this:

Memory spikes
Latency spikes
Eventually crashes

Failure Case: Timeout Mismatch Hell

You configure:

Load balancer timeout: 60s
App server timeout: 30s
DB timeout: 10s

What happens?

DB times out → app retries
App still running → LB kills connection
Client retries → duplicate work

Result:

Cascade failure

Trade-offs You Can’t Avoid

Latency vs Throughput

Small buffers → lower latency, more syscalls
Large buffers → better throughput, worse tail latency

Simplicity vs Control

Frameworks hide complexity
But you lose control over:
- connection reuse
- backpressure
- parsing behavior

CPU vs Network Efficiency

Compression saves bandwidth
Costs CPU
Under load, CPU becomes bottleneck

Keep-Alive vs Resource Locking

Keep-alive reduces handshake overhead
But holds connections longer
Risk: connection pool exhaustion

Final System Design (What Actually Works in Production)

A sane architecture:

Client
  ↓
CDN (optional)
  ↓
Reverse Proxy (NGINX / Envoy)
  ↓
App Server (stateless, event-driven)
  ↓
Service Layer
  ↓
Database / Cache

Key rules:

Terminate TLS early
Enforce timeouts at every layer
Use connection pooling aggressively
Monitor queueing, not just CPU

Key Takeaways (No Fluff)

HTTP lifecycle is mostly not in your code—it’s in the kernel and network stack
Most latency issues are queueing problems, not computation problems
Backpressure is real; ignoring it will kill your system
Middleware is not free—treat it like production code, not decoration
Timeouts must be aligned across layers or you create cascading failures
Keep-alive and pooling are double-edged swords

If you still think HTTP is just “request comes in, response goes out,” you’re not ready to debug production systems.

DEV Community: Kshitij Sharma

TerraWeek Challenge – Day 5 : Terraform Modules: Writing Reusable Infrastructure

📚 Learning Objectives

🤔 What is a Terraform Module?

🏗️ Why Do We Need Modules?

🌍 Root Module vs Child Module

Root Module

Child Module

✨ Benefits of Using Modules

📂 Project Structure

📸 Screenshot

📁 Understanding the Module Structure

main.tf

variables.tf

outputs.tf

README.md

⚙️ Prerequisites

🛠️ Verifying Terraform Installation

📸 Screenshot

☁️ Verifying AWS CLI

📸 Screenshot

🧩 Understanding the Root Module

📦 Understanding the Child Module

📥 Module Inputs

📤 Module Outputs

💡 Why Pass IDs Instead of Performing Lookups?

📍 Calling a Local Module

🌎 Using Modules from the Terraform Registry

🔗 Using Modules from Git

🔒 Module Version Pinning

Exact Version

Compatible Minor Versions

Version Range

Git Tag

Git Commit SHA

🎯 Why Version Pinning Matters

🔄 Modular Composition using for_each

🚀 Deploying Infrastructure using Terraform Modules

🚀 Step 1 — Initialize Terraform

📸 Screenshot

📋 Step 2 — Review the Execution Plan

📸 Screenshot

🚀 Step 3 — Provision Infrastructure

📸 Screenshot

☁️ Verifying the EC2 Instance

📸 Screenshot

📤 Viewing Module Outputs

📸 Screenshot

🔄 Creating Multiple EC2 Instances using for_each

📋 Reviewing the Plan with Multiple Module Instances

🔍 Inspecting the Infrastructure

🍫 Bonus Exploration

⭐ Bonus 1 — Documenting the Module

⭐ Bonus 2 — Input Validation

⭐ Bonus 3 — Publishing Modules using Git

⭐ Bonus 4 — Module Composition

⭐ Bonus 5 — Why Modular Infrastructure Matters

🧹 Cleaning Up the Infrastructure

📸 Screenshot

🎯 What I Learned

📂 Repository Structure

🚀 Conclusion

🏷️ Tags

🌱 TerraWeek Challenge – Day 4 : Terraform State & Remote Backends (Native Locking)

📚 Learning Objectives

📂 Project Structure

📁 backend_infra

📁 backend_demo

📸 Screenshot

⚙️ Prerequisites

🛠 Verifying Terraform Installation

📸 Screenshot

☁️ Verifying AWS CLI Configuration

📸 Screenshot

🗄️ What is Terraform State?

🤔 Why Does Terraform Need State?

🔐 Why is Terraform State Sensitive?

🌍 Local State vs Remote State

Local State

Remote State

`main.tf`

`variables.tf`

`outputs.tf`

`README.md`

🔄 Modular Composition using `for_each`

🔄 Creating Multiple EC2 Instances using `for_each`

📋 `terraform state list`

🔍 `terraform state show`

🔄 `terraform state mv`

❌ `terraform state rm`

📖 `terraform show`

⭐ Bonus 3 — Understanding the `moved` Block

⭐ Bonus 4 — Understanding the `removed` Block

⭐ Bonus 5 — Continuous Health Checks using `check`