Terraform State File Management with Remote Backend

On day 4 of my learning journey I learnt how Terraform remembers what it creates, and how to manage that memory safely using a remote backend.

When running terraform apply, a file named terraform.tfstate is generated in the working directory.
This file is critical because it stores metadata that Terraform uses to track infrastructure resources.

Why Remote State Matters
When Terraform runs, it keeps track of infrastructure in a state file.

If that file stays on a local machine:

• It’s not shareable
• It’s not safe
• It can easily get corrupted or lost

StateFile Best Practices
Store StateFile to a Remote Backend: The problem with Statefile is it also contains all the important information like configuration, access keys and other sensitive information. So we shouldn't log that to a github or any other personal folders. It is better to keep that in a Remote Backend like S3 in AWS, Blob in Azure and GCP Cloud Staorage.

Do Not Update/Delete StateFile: StateFile will always be generated by Terraform. You should not make any manual changes to that file or shouldn't delete them.

State Locking: Just imagine there is a StateFile and 2 devops engineers tries to modify or create infra using that file with different changes. This will corrupt StateFile. So It should be locked in such a way until first user completes terraform apply, then it should be unlocked and second user should apply his changes after that.

Isolation of StateFile: StateFile should be isolated for multiple environments in a different folder. Not all StateFile should be combined with a same name or in a same folder.

Regular Backup: It is essential to take regular backup's of a StateFile as we cannot access them in case of Global outages. So Enable Versioning on AWS S3 Buckets and also setup policies to store older StateFiles to a different account or tar them to other location.

Backend Configuration Details

• The S3 bucket used for state storage must already exist before initializing Terraform.
• The bucket is not created by Terraform itself because it must be available to store the state file before Terraform runs.
• Creating the bucket manually can be done via AWS CLI, AWS Console, or CI/CD pipelines.
• Avoid including the bucket creation in Terraform resources to prevent circular dependencies.

Configured Remote Backend
I updated my Terraform configuration to use S3 as backend:

backend "s3" {
  bucket       = "my-terraform-state-bucket"
  key          = "day-04/dev/terraform.tfstate"
  region       = "us-east-1"
  use_lockfile = true
  encrypt      = true
}

Observing Remote State File Behaviour

• When running terraform plan or terraform apply, no state file is created locally except a minimal metadata file.
• The actual state file is stored inside the S3 bucket under the specified key (e.g.,dev/terraform.tfstate).
• The remote state file is a JSON file containing all resource details, including some encoded and sensitive information.
• This setup improves security by avoiding sensitive state information on local machines and centralising management.

Verified state in S3

Managing Terraform State

Terraform provides commands to manage state without manual JSON editing:

terraform state list Lists resources in the state file
terraform state show Shows detailed info about a specific resource
terraform state rm Removes a resource from the state file (safe method)
terraform state pull Fetches the current state file from backend

These commands help manage the state file programmatically and safely.

Below is the Youtube Video for reference: Tech Tutorials with Piyush — “Terraform StateFile Management with S3”