Detecting Drift in Infrastructure as Code

The Problem: Drift Detection in Infrastructure-as-Code

In production environments, infrastructure-as-code (IaC) tools like Terraform are widely used to manage and provision infrastructure. However, a common issue that arises is drift between the declared state in the IaC configuration and the actual state of the infrastructure. This drift can occur due to manual changes made in the console, hotfixes applied directly to resources, or scaling values tuned by hand. If left undetected, drift can lead to configuration inconsistencies, security vulnerabilities, and ultimately, outages.

Technical Breakdown

To understand how drift occurs, let's consider a simple Terraform configuration for an AWS EC2 instance:

# File: main.tf
provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami           = "ami-0c94855ba95c71c99"
  instance_type = "t2.micro"
}

In this example, the Terraform configuration declares an AWS EC2 instance with a specific AMI and instance type. However, if an engineer manually changes the instance type in the AWS console to t2.small, the actual state of the infrastructure will diverge from the declared state. This drift can be difficult to detect, especially in large and complex infrastructure deployments.

The Fix / Pattern

To detect drift in IaC environments, automated drift detection tools and techniques can be employed. One approach is to use Terraform's built-in terraform state command to compare the declared state with the actual state of the infrastructure. For example:

# Run Terraform state command to detect drift
terraform state pull > declared_state.tfstate
terraform refresh-only -state=declared_state.tfstate

Alternatively, third-party tools like Terragrunt or Terraform Compliance can be used to detect drift and enforce compliance with the declared state.

To prevent drift from occurring in the first place, it's essential to establish a blameless culture and encourage engineers to make changes only through the IaC configuration. This can be achieved by implementing automated testing and validation of IaC changes, as well as providing clear documentation and training on IaC best practices.

Key Takeaway

Run automated drift detection on every environment daily and treat any difference between declared and actual state as a failing test, to prevent configuration inconsistencies and ensure infrastructure reliability.