DEV Community

Cover image for Terraform State File Management
Anil KUMAR
Anil KUMAR

Posted on

Terraform State File Management

#terraform #tutorial #devops #aws

Today marks the Day 4 of 30 Days Terraform challenge and today we will deep dive into the Terraform State File management and Remote Backend using AWS S3.

What exactly is StateFile:

Statefile is like a brain to the Terraform. Without Statefile terraform cannot do anything and if you do not manage statefile properly, your entire infrastructure might be at risk.

Statefile is like a blueprint using which Terraform updates the environment in the AWS Cloud. Statefile is used to compare Infarstructure, create, delete and modify them.

The state file is a JSON file that contains:

  • Resource metadata and current configuration
  • Resource dependencies
  • Provider information
  • Resource attribute values

StateFile Best Practices:

  • Store StateFile to a Remote Backend: The problem with Statefile is it also contains all the important information like configuration, access keys and other sensitive information. So we shouldn't log that to a github or any other personal folders. It is better to keep that in a Remote Backend like S3 in AWS, Blob in Azure and GCP Cloud Staorage.

  • Do Not Update/Delete StateFile: StateFile will always be generated by Terraform. You should not make any manual changes to that file or shouldn't delete them.

  • State Locking: Just imagine there is a StateFile and 2 devops engineers tries to modify or create infra using that file with different changes. This will corrupt StateFile. So It should be locked in such a way until first user completes terraform apply, then it should be unlocked and second user should apply his changes after that.

  • Isolation of StateFile: StateFile should be isolated for multiple environments in a different folder. Not all StateFile should be combined with a same name or in a same folder.

  • Regular Backup: It is essential to take regular backup's of a StateFile as we cannot access them in case of Global outages. So Enable Versioning on AWS S3 Buckets and also setup policies to store older StateFiles to a different account or tar them to other location.

Remote Backend Benefits:

Collaboration: Team members can share state
Locking: Prevents concurrent state modifications
Security: Encrypted storage and access control
Backup: Automatic versioning and backup
Durability: Highly available storage

Eliminating DynamoDB

Earlier we have used DynamoDB for statelocking of StateFile mechanism, But recently S3 has now introduced inbuilt feature independently for State Locking of StateFile.

S3 native state locking:

When Terraform needs to acquire a lock, it attempts to create a lock file in S3
S3 conditional writes check if the lock file already exists
If the lock file exists, the write operation fails, preventing concurrent modifications
If the lock file doesn't exist, it's created successfully and the lock is acquired
When the operation completes, the lock file is deleted (appears as a delete marker with versioning)

Previous Method (DynamoDB):

Required separate DynamoDB table creation
Additional AWS service to monitor and maintain
More complex IAM permissions
Extra cost for DynamoDB read/write operations
DynamoDB state locking is now discouraged and may be deprecated in future Terraform versions

Code Block:

terraform {
  backend "s3" {
    bucket       = "<Bucket-Name>"
    key          = "dev/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true
    encrypt      = true
  }
}
Enter fullscreen mode Exit fullscreen mode
  1. bucket: write your bucket name in the bucket row.
  2. key: In the Key row, you can mention the folders present in that particular S3 bucket if that exists or you can leave that.
  3. region: AWS region for the S3 bucket
  4. use_lockfile: Enable S3 native state locking (set to true)
  5. encrypt: Enable server-side encryption for the state file

Make sure that the Bucket-Name is created before applying State Lock via manually or through terraform CLI or else you might be getting errors.

Commands:

tf plan: You cannot see any file getting generated as there is no existing infra.

tf apply: You can see a lock file getting generated in S3 Bucket with .lock extension. Also you could see a statefile in the local folder eventhough it doesn't contain any senesitive information.

terraform state list: Verify state is now remote.

terraform state show : Show detailed state information

terraform state rm : Remove resource from state (without destroying), terraform will no more manage that.

terraform state mv : Move resource to different state address

terraform state pull: Pull current state and display

Conclusion:

This is all you need to get started with StateFile Management and Remote Backend. Looking for more concepts to implement and play with going forward.
See you in the next blog.

Below is the Youtube Video for reference: Tech Tutorials with Piyush — “Terraform StateFile Management with S3”

Top comments (0)