Driftctl and Terraform, they're two of a kind!

Alexis — Wed, 30 Jun 2021 09:45:33 +0000

Driftctl is an Open Source project developed to help Operation team to maintain their infrastructure.

Terraform a brief introduction

Let's start by a reminder or an explanation on what is Terraform. Terraform is an Open Source software support by Hashicorp such as Vault, Packer, Consul and many other.

The aim of Terraform is to build, change and version your infrastructure. You will need to declare the infrastructure into Terraform files.

Terraform website

Why use Driftctl?

Infrastructure as code is awesome, but there are too many moving parts: codebase, state file, actual cloud state. Things tend to drift.

Drift can have multiple causes: from developers creating or updating infrastructure through the web console without telling anyone, to uncontrolled updates on the cloud provider side. Handling infrastructure drift vs the codebase can be challenging.

You can't efficiently improve what you don't track. We track coverage for unit tests, why not infrastructure as code coverage?

Driftctl tracks how well your IaC codebase covers your cloud configuration. Driftctl warns you about drift.

official documentation

What Driftcl does?

It read a tfstate file to catch how your project is build and will give an output to see if everything is correct.
That in order to avoid from scratch modification.

You can added these check into a CI to have a result at your convenience.

I will try in this post to show you with how to use it.
For the moment Drifctl works only on AWS but it is still under development and GCP will come :)

tfstate?

A Terraform State file represent your infrastructure and configuration. Terraform will use it to map real world resources to your configuration.

First of all! Deploy your infra.

I choose to deploy a LAMP (Linux, Apache, Mysql, PHP) infra with a bastion host, a web-server and a database.

provider.tf

provider "aws" {
  profile = "default"
  region  = "eu-west-3"
}

We can declare our instances:

instance.tf

resource "aws_instance" "nat" {
    ami = "ami-08755c4342fb5aede" #Red Hat Enterprise Linux 8 
    instance_type = "t2.micro"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.nat.id}"]
    subnet_id = "${aws_subnet.eu-west-3a-public.id}"
    associate_public_ip_address = true
    source_dest_check = false

    tags = {
        Name = "VPC NAT"
    }
}

resource "aws_instance" "web-1" {
    ami = "${lookup(var.amis, var.aws_region)}"
    instance_type = "t2.micro"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.web.id}"]
    subnet_id = "${aws_subnet.eu-west-3a-public.id}"
    associate_public_ip_address = true
    source_dest_check = false


    tags = {
        Name = "Web Server 1"
    }
}

resource "aws_instance" "db-1" {
    ami = "${lookup(var.amis, var.aws_region)}"
    instance_type = "t2.micro"
    key_name = "${var.aws_key_name}"
    vpc_security_group_ids = ["${aws_security_group.db.id}"]
    subnet_id = "${aws_subnet.eu-west-3a-private.id}"
    source_dest_check = false

    tags = {
        Name = "DB Server 1"
    }
}

Now we deploy your network

vpc.tf

resource "aws_vpc" "main_vpc" {
    cidr_block = "${var.vpc_cidr}"
    enable_dns_hostnames = true
}

resource "aws_eip" "nat" {
    instance = "${aws_instance.nat.id}"
    vpc = true
}

resource "aws_eip" "web-1" {
    instance = "${aws_instance.web-1.id}"
    vpc = true
}

resource "aws_route_table" "eu-west-3a-public" {
    vpc_id = "${aws_vpc.main_vpc.id}"

    route {
        cidr_block = "0.0.0.0/0"
        gateway_id = "${aws_internet_gateway.ig-main.id}"
    }

    tags = {
        Name = "Public Subnet"
    }
}

resource "aws_route_table_association" "eu-west-3a-public" {
    subnet_id = "${aws_subnet.eu-west-3a-public.id}"
    route_table_id = "${aws_route_table.eu-west-3a-public.id}"
}

resource "aws_route_table" "eu-west-3a-private" {
    vpc_id = "${aws_vpc.main_vpc.id}"

    route {
        cidr_block = "0.0.0.0/0"
        instance_id = "${aws_instance.nat.id}"
    }

    tags = {
        Name = "Private Subnet"
    }
}

resource "aws_route_table_association" "eu-west-3a-private" {
    subnet_id = "${aws_subnet.eu-west-3a-private.id}"
    route_table_id = "${aws_route_table.eu-west-3a-private.id}"
}

resource "aws_subnet" "eu-west-3a-public" {
    vpc_id = "${aws_vpc.main_vpc.id}"

    cidr_block = "${var.public_subnet_cidr}"
    availability_zone = "eu-west-3a"

    tags = {
        Name = "Public Subnet"
    }
}

resource "aws_subnet" "eu-west-3a-private" {
    vpc_id = "${aws_vpc.main_vpc.id}"

    cidr_block = "${var.private_subnet_cidr}"
    availability_zone = "eu-west-3a"

    tags = {
        Name = "Private Subnet"
    }
}

resource "aws_internet_gateway" "ig-main" {
    vpc_id = "${aws_vpc.main_vpc.id}"
}

Add rule to be sure that just yout bastion host is joinable by SSH for example

security_groups.tf

resource "aws_security_group" "nat" {
    name = "vpc_nat"
    description = "Can access both subnets"

    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }
    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
}

    vpc_id = "${aws_vpc.main_vpc.id}"

    tags = {
        Name = "NATSG"
    }
}

resource "aws_security_group" "web" {
    name = "vpc_web"
    description = "Allow incoming HTTP connections."

    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }
    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    egress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress { # MySQL
        from_port = 3306
        to_port = 3306
        protocol = "tcp"
        cidr_blocks = ["${var.private_subnet_cidr}"]
    }

    vpc_id = "${aws_vpc.main_vpc.id}"

    tags = {
        Name = "WebServerSG"
    }
}

resource "aws_security_group" "db" {
    name = "vpc_db"
    description = "Allow incoming database connections."

    ingress { # MySQL
        from_port = 3306
        to_port = 3306
        protocol = "tcp"
        security_groups = ["${aws_security_group.web.id}"]
    }

    ingress {
        from_port = 22
        to_port = 22
        protocol = "tcp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }
    ingress {
        from_port = -1
        to_port = -1
        protocol = "icmp"
        cidr_blocks = ["${var.vpc_cidr}"]
    }

    egress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    egress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }

    vpc_id = "${aws_vpc.main_vpc.id}"

    tags = {
        Name = "DBServerSG"
    }
}

Input variables serve as parameters for a Terraform module, allowing aspects of the module to be customized without altering the module's own source code, and allowing modules to be shared between different configurations.

Three of them are not used in this deployment but could be useful :

aws_access_key
aws_secret_key
aws_key_path

vars.tf

variable "aws_access_key" {
    default = "ACCESS"
}
variable "aws_secret_key" {
    default = "SECRET"
}
variable "aws_key_path" {
    default = "~/.ssh/id_rsa.pub"
}
variable "aws_key_name" {
    default = "ansible"
}

variable "aws_region" {
    description = "EC2 Region for the VPC"
    default = "eu-west-3"
}

variable "amis" {
    description = "AMIs by region"
    default = {
        eu-west-3 = "ami-08755c4342fb5aede" #Red Hat Enterprise Linux 8 
    }
}

variable "vpc_cidr" {
    description = "CIDR for the whole VPC"
    default = "10.0.0.0/16"
}

variable "public_subnet_cidr" {
    description = "CIDR for the Public Subnet"
    default = "10.0.0.0/24"
}

variable "private_subnet_cidr" {
    description = "CIDR for the Private Subnet"
    default = "10.0.1.0/24"
}

All files are available here: https://gitlab.com/aaurin/lamp

You can now launch your infrastructure managed by terraform,

Module initialization (aws here):

$ terraform init

If you want to see that Terraform will deploy you can plan your deployment (not mandatory in our case)

$ terraform plan

Then deploy your LAMP infra

$ terraform apply

...
...

aws_vpc.main_vpc: Creating...
aws_vpc.main_vpc: Still creating... [10s elapsed]
aws_vpc.main_vpc: Creation complete after 12s [id=vpc-0c45226d21c59070a]
aws_internet_gateway.ig-main: Creating...
aws_subnet.eu-west-3a-public: Creating...
aws_subnet.eu-west-3a-private: Creating...
aws_security_group.web: Creating...
aws_security_group.nat: Creating...
aws_subnet.eu-west-3a-private: Creation complete after 1s [id=subnet-04cd40add9de539dc]
aws_subnet.eu-west-3a-public: Creation complete after 1s [id=subnet-0d41d297b9ed3d325]
aws_internet_gateway.ig-main: Creation complete after 1s [id=igw-09faa5fd0eb66734d]
aws_route_table.eu-west-3a-public: Creating...
aws_route_table.eu-west-3a-public: Creation complete after 1s [id=rtb-0e182dc2f4256fd80]
aws_route_table_association.eu-west-3a-public: Creating...
aws_route_table_association.eu-west-3a-public: Creation complete after 0s [id=rtbassoc-0efc5f42b16f742ef]
aws_security_group.nat: Creation complete after 2s [id=sg-0f17e55a720cc32e9]
aws_instance.nat: Creating...
aws_security_group.web: Creation complete after 2s [id=sg-02b9aeddf0653d42c]
aws_instance.web-1: Creating...
aws_security_group.db: Creating...
aws_security_group.db: Creation complete after 2s [id=sg-01f95dcabb4f176b2]
aws_instance.db-1: Creating...
aws_instance.nat: Still creating... [10s elapsed]
aws_instance.web-1: Still creating... [10s elapsed]
aws_instance.db-1: Still creating... [10s elapsed]
aws_instance.nat: Still creating... [20s elapsed]
aws_instance.web-1: Still creating... [20s elapsed]
aws_instance.db-1: Still creating... [20s elapsed]
aws_instance.web-1: Creation complete after 23s [id=i-02658d8ceccbcd8e4]
aws_instance.nat: Creation complete after 23s [id=i-0a296ff78695c550a]
aws_eip.web-1: Creating...
aws_eip.nat: Creating...
aws_route_table.eu-west-3a-private: Creating...
aws_route_table.eu-west-3a-private: Creation complete after 1s [id=rtb-0215aaa607d739fce]
aws_route_table_association.eu-west-3a-private: Creating...
aws_eip.web-1: Creation complete after 1s [id=eipalloc-01a1f6f45b5de1323]
aws_eip.nat: Creation complete after 1s [id=eipalloc-0dd787a99bf35fdcf]
aws_route_table_association.eu-west-3a-private: Creation complete after 0s [id=rtbassoc-030ac110c46b505ff]
aws_instance.db-1: Still creating... [30s elapsed]
aws_instance.db-1: Still creating... [40s elapsed]
aws_instance.db-1: Creation complete after 43s [id=i-052fe5f0b1dd45dd6]

Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

once is ok, we can install drifctl.

drifctl installation

Download binary file using curl or wget command:

$ curl -L https://github.com/cloudskiff/driftctl/releases/latest/download/driftctl_linux_amd64 -o driftctl

# x86
$ curl -L https://github.com/cloudskiff/driftctl/releases/latest/download/driftctl_linux_386 -o driftctl

# macOS
$ curl -L https://github.com/cloudskiff/driftctl/releases/latest/download/driftctl_darwin_amd64 -o driftctl

Make it executable

$ chmod +x driftctl

Finally move it into your PATH:

$ sudo mv driftctl /usr/local/bin/

You can also use brew:

$ brew install driftctl

Or you can use a container:

$ docker run -t --rm \
  -v ~/.aws:/root/.aws:ro \
  -v $(pwd):/app:ro \
  -v ~/.driftctl:/root/.driftctl \
  -e AWS_PROFILE=non-default-profile \
  cloudskiff/driftctl scan

-v ~/.aws:/root/.aws:ro (optionally) mounts your ~/.aws containing AWS credentials and profile

-v $(pwd):/app:ro (optionally) mounts your working dir containing the terraform state

-v ~/.driftctl:/root/.driftctl (optionally) prevents driftctl to download the provider at each run

-e AWS_PROFILE=cloudskiff (optionally) exports the non-default AWS profile name to use

cloudskiff/driftctl:<VERSION_TAG> run a specific Driftctl tagged release

and run your first check:

driftctl scan --from tfstate://terraform.tfstate

...
...

Found 139 resource(s)
 - 15% coverage
 - 21 covered by IaC
 - 118 not covered by IaC
 - 0 missing on cloud provider
 - 0/21 changed outside of IaC

Let's explain this output:
% coverage -- coverage percent on your Infra as Code
covered by IaC -- number of resource managed by IaC
not covered by IaC -- number of resources found in IaC but not managed by IaC
missing on cloud provider -- number of resources found in IaC but not on remote
changed outside of IaC -- !! Drift !! number of changes on managed resources

We can see that I'm not alone on this project and my coverage is not full but I have no changed from my IaC.

Now I will make a change: add an SSH access to everywhere for the web security group

I run again the scan

driftctl scan --from tfstate://terraform.tfstate
...
Found changed resources:
  - Table: rtb-0215aaa607d739fce, Destination: 0.0.0.0/0 (aws_route):
    ~ InstanceOwnerId: "" => "836683081860" (computed)
    ~ NetworkInterfaceId: <nil> => "eni-0aea178a98f2cd0f5" (computed)
Found 140 resource(s)
 - 15% coverage
 - 21 covered by IaC
 - 119 not covered by IaC
 - 0 missing on cloud provider
 - 1/21 changed outside of IaC

We can now see the change and their it could be security vulnerability

Conclusion

I really like this tool, he could be very helpful for changes that terraform are not handle or to keep an eyes on your infra.
You can add it to your CI tool, and make workflow around the Driftctl output. find all CI compatible here

DEV Community: Alexis