DEV Community: driftctl

Security Group Rules : small changes, bitter consequences

driftctl — Wed, 05 May 2021 07:26:40 +0000

How a simple manual change in an AWS Security Group using the AWS Web Console can have bitter security consequences

In this article, we’ll show how a simple manual change in an AWS Security Group using the AWS Web Console can have bitter security consequences. This one is based on many first-hand experiences and user feedback, in a production context with infrastructure-as-code.

Allowing PgSQL (with Terraform)

For this article, we’ll take the example recently experienced by a team we know. They secured access to their PgSQL instances with this simple Terraform configuration, allowing only the web servers subnet:

// Secure the PgSQL RDS cluster using a dedicated SG
resource "aws_security_group" "pgsql" {
  name        = "PgSQL Security Group"
  description = "PgSQL Security Group"

  tags = {
    Name = "PgSQL Security Group"
  }
}

// PgSQL can only be accessed from the WWW network (10.0.0.0/8)
resource "aws_security_group_rule" "pgsql" {
  type              = "ingress"
  from_port         = 5432
  to_port           = 5432
  protocol          = "tcp"
  cidr_blocks       = ["10.0.0.0/8"]
  security_group_id = aws_security_group.pgsql.id
}

They had a scheduled terraform plan as a GitHub Action for the whole infrastructure, so if something changed, it was reported right away. The “GitOps” workflow also included some linters, a terraform plan, followed by a terraform apply after a merge on the main branch.

It worked well and satisfied the initial basic security requirements. Good job!

That one Security Group temporary change

A couple of years later, to win a new major customer, they agreed to allow a pentesting audit. Long story short, one auditor was pentesting from the internet, while the other worked from different internal networks of the company.

Oddly enough, they didn’t come back with the same results: the external audit came clean, while the internal one basically opposed a strict “no-go” to the commercial deal. Reason: the auditor reported he gained access to every customer database in a matter of minutes. With proof.

How could it happen?

One day, someone from the team needed to quickly test something on a customer database who experienced a bug. A single value to change. Going through the hassle of switching to the secure network, then connecting to the corporate VPN, then going through the Bastion using a smartcard for authentication, probably also having to explain soon to the security team why he connected and what he did…this was definitely too much. Especially when he could simply allow the company’s office outgoing IPv4 on the “PgSQL” security group! And remove it after the change. Moreover, he would necessarily remember about it, as all the Security Groups were 100% managed by Terraform on CI.

And it worked well! That team member fixed the customer issue in the database quickly.

And forgot to remove the security group rule, until it was discovered by the auditor, so that he could:

connect painlessly to the customer databases from the office
then easily find passwords using a tool like hydra against PostgreSQL.
profit.

How we ended up there?

Here’s the context in which this team worked:

a team member had AWS Web Console credentials large enough to make Security Group changes
they assumed Security Group Rules were represented similarly on AWS and Terraform
they wrote their Security Groups rules a certain way using Terraform

We can’t do much about the first issue: it’s the harsh reality of most companies today. Surely enough they need to adopt a proper full GitOps workflow, but most aren’t there yet (and won’t be for a while).

The lists are not what they seem

The second assumption is confusing because of what can the UI lead you to believe. As developers, we know that cidr_blocks = ["10.0.0.0/8"] is a list and that adding an element to it will make it different.

As a developer, I can also easily believe that such a design in the AWS Console would also generate a list, both IPs are listed under the same line, exactly as if I read ["10.0.0.0/8", "96.202.220.106/32"]:

The thing is, in the end, on the AWS side, they really are two elements of a list, but in the Terraform model, they end up being two distinct resources and not two elements. This difference in models between the cloud provider and Terraform can sometimes be misleading for users and can have major consequences.

The way you write your Terraform matters

The third and final situation here is the way the Terraform configuration was written.

Can you guess what happened in CI for months after the manual change wasn’t reverted? Nothing.

$ terraform apply
aws_security_group.pgsql: Refreshing state... [id=sg-0b6af725ba7e7691b]
aws_security_group_rule.pgsql: Refreshing state... [id=sgrule-3696232291]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

To be more precise, something happened. The manual Security Group rule change was added without anyone being notified about it in the terraform.tfstate right when the next terraform apply ran in CI:

"ingress": [
              {
                "cidr_blocks": [
                  "10.0.0.0/8"
                ],
                "description": "",
                "from_port": 5432,
                "ipv6_cidr_blocks": [],
                "prefix_list_ids": [],
                "protocol": "tcp",
                "security_groups": [],
                "self": false,
                "to_port": 5432
              },
              {
                "cidr_blocks": [
                  "96.202.220.106/32"
                ],
                "description": "",
                "from_port": 5432,
                "ipv6_cidr_blocks": [],
                "prefix_list_ids": [],
                "protocol": "tcp",
                "security_groups": [],
                "self": false,
                "to_port": 5432
              }
            ],

Yes, you will end up with a Terraform state with way more information and resources than you intended from the code, as the Terraform state job is not to represent your intention, but to map the real world with your config.

The way Security Group Rules were written couldn’t help discovering the change (not to mention notifying anyone). You could add a billion rules and it would be the same.

Fortunately, in this case, if you read Terraform’s documentation for the AWS provider (currently v3.36), you’ll find 2 options to configure Security Groups:

Use the aws_security_group resource with inline egress {} and ingress {} blocks for the rules.
Use the aws_security_group resource with additional aws_security_group_rule resources.

In this case, using the first option would have been better for this team, from a more DevSecOps point of view. See how the next terraform apply in CI would have had the expected effect:

$ terraform apply
aws_security_group.pgsql: Refreshing state... [id=sg-03a12201c68c38d92]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_security_group.pgsql will be updated in-place
  ~ resource "aws_security_group" "pgsql" {
        id                     = "sg-03a12201c68c38d92"
      ~ ingress                = [
          - {
              - cidr_blocks      = [
                  - "96.202.220.106/32",
                ]
              - description      = "Allow PgSQL from WWW"
              - from_port        = 5432
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 5432
            },
            # (1 unchanged element hidden)
        ]
        name                   = "PgSQL Security Group"
        tags                   = {
            "Name" = "PgSQL Security Group"
        }
        # (6 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Unfortunately, it’s often too late when you realize it, and this kind of option doesn’t exist for every resource out there. What was a DevOps discussion (“let’s automate those deployments and ship that on CI!”) can quickly become a DevSecOps discussion now (“how should we write our Terraform configuration according to our goals?”).

The driftctl option

That’s why using driftctl is always a good option. As the tool compares your Terraform state to the reality of your AWS account, you can deploy the tool at 2 important places of your workflow:

as a scheduled run (like an hourly cronjob), so you get reports when something changes
as a pre-condition to a terraform apply step in CI: so you ensure you’re working with an unmodified Terraform state (as by design the refresh part of the apply can modify it). You may also want to parallelize this driftctl step with a terraform plan.

Here’s a driftctl run when the activity is as expected on the AWS account:

$ driftctl scan
Scanned resources:    (54)
Found 2 resource(s)
 - 100% coverage
Congrats! Your infrastructure is fully in sync.
$ echo $?
0

Here’s a driftctl run with the above manual change:

$ driftctl scan
Scanned resources:    (55)
Found resources not covered by IaC:
  aws_security_group_rule:
    - Type: ingress, SecurityGroup: sg-0ce251e7ce328547d, Protocol: tcp, Ports: 5432, Source: 96.202.220.106/32
Found 3 resource(s)
 - 66% coverage
 - 2 covered by IaC
 - 1 not covered by IaC
 - 0 missing on cloud provider
 - 0/2 changed outside of IaC
$  echo $?
1

This output clearly shows:

a resource NOT on the Terraform state, of type aws_security_group_rule, for the Security Group sg-0ce251e7ce328547d, that allows TCP/5432 for 96.202.220.106/32. This should trigger an alarm!
some metrics for your own reference.

This output can be exported as JSON too, so you can easily manipulate it and integrated it into your other tools and notification systems. There’s a ton of other options and integrations you can use with driftctl, but that will be for another article!

Reference:

driftctl.com
driftctl GitHub
driftctl Documentation
Terraform AWS Provider resources documentation: aws_security_group_rule and aws_security_group

Codecov bash uploader security issue

driftctl — Tue, 04 May 2021 15:42:34 +0000

Codecov bash uploader security issue

How minimizing our attack surface with CircleCI Contexts helped us pass the Codecov bash uploader security issue unharmed

What is Codecov

For those who don’t know Codecov, it is an online platform for hosted code coverage reports and statistics founded in 2014. According to their website, over 29,000 organizations and more than 1 million developers use their tools. The list includes Google, Atlassian, The Washington Post, RBC, and Procter & Gamble Co.

Its main tool, the Bash Uploader, is used to send code coverage reports to the platform. Those reports help developers measure how well their source code is covered by tests. This is the main reason why it is widely used among developers and organizations.

What about the breach?

On April 15th, 2021, Codecov reported in a press release that someone had gained unauthorized access to its script and modified it without its permission.

“Our investigation has determined that beginning Jan. 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users’ continuous integration environments. This information was then sent to a third-party server outside of Codecov’s infrastructure” the company said.

According to Codecov, the hacker gained access to:

Any credential, token, or key that customers passed through their continuous integration runner that would be accessible when the Bash Uploader script was executed;
Any service, datastore, and application code that could be accessed with these credentials, tokens, or keys;
The git remote information (URL of the origin repository) of repositories using the Bash Uploader to upload coverage to Codecov in their continuous integration.

For those potential threats, Codecov is strongly encouraging developers to immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI process.

Why are we safe from that at CloudSkiff?

We use CircleCI to continuously build, test and deploy our open-source drift detection CLI, driftctl. Like most projects, we need to manipulate sensitive environment variables which could mean that this event could impact us.

But, CircleCI provides a simple and secure way to store and share those sensitive environment variables across projects and jobs (or pipelines, or steps in other CI), it is called a context. As an Organization in CircleCI, you can, in your settings, create and manage contexts.

You can use the context key in the workflows section of a project config.yml file to give any jobs access to the environment variables associated with the context.

version: 2.1

# Jobs declaration for build, test and deploy not displayed

workflows:
  build-test-deploy:
    jobs:
      - build
      - test
      - deploy:
          context: restricted-context
          requires:
            - build
            - test

In the example above, only the deploy job has access to the environment variables created in the restricted-context context. On the other hand, the build and test jobs don’t have any access to the sensitive variables in their execution.

That’s basically what we do as of today in our CircleCI config.yml file. Every time we need to call our test job to execute the Codecov Bash Uploader script (e.g. every push, pull request, and release workflows), since we don’t attach any context to it, the hacker has no access to our secret variables.

We try to keep our jobs as small as possible to reduce the attack surface by separating concerns, and CircleCI context is well suited to assist in this good practice.

We could have avoided that threat

One way we could have avoided that is the same way Codecov started investigating the attack. The company reported on the morning of April 1, 2021, that one customer was using the shasum that is available on its Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash.

Indeed, as stated on its documentation website, “it can be validated for correctness by calculating the shasum on download and comparing to the result we store in the GitHub repository”.

An example could be:

$ curl -s -o codecov https://codecov.io/bash \
&& VERSION=$(grep 'VERSION=\".*\"' codecov | cut -d'"' -f2) \
&& shasum -a 512 -c --ignore-missing <(curl -s [https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM)](https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM))

This simple script could be run on each test job before running the Bash Uploader. If we were doing it that way we could have seen sooner the hack, and we could have reported it sooner to Codecov.

Our final test job looks like this:

test:
docker:
  - image: golang:1.16
steps:
  - checkout
  - run: make install-tools
  - run: make test
  - run:
      name: Check Codecov
      command: |
        curl -s -o codecov https://codecov.io/bash \
          && VERSION=$(grep 'VERSION=\".*\"' codecov | cut -d'"' -f2) \
          && shasum -a 512 -c --ignore-missing <(curl -s https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA512SUM)
  - run:
      name: Codecov upload
      command: |
        chmod +x codecov
        ./codecov

Driftctl is a free and open-source CLI that tracks, analyzes, prioritizes, and warns of infrastructure drift

More info about us

Reference:

Our commitments to the community

driftctl — Thu, 18 Mar 2021 13:28:50 +0000

Here’s why we bring a new open source DevOps tool

As we celebrated the initial release of driftctl 3 months ago, we certainly hoped for some sort of acknowledgement or favorable opinion from those who would use it, but we were not prepared for such a warm welcome.

Reflecting on the first steps of the project, the fast moving star history and the contributions and interactions we have benefited from, we are both grateful and humbled by the support we receive from the open source community.

Why we launched this DevOps tool in the first place

At CloudSkiff, we are all fans of GitOps. We are big believers in the power of Infrastructure as Code and all its benefits.

We like it so much that we decided to build a tool to help DevOps, SREs, infrastructure teams and all cloud practitioners, collaborate over Infrastructure as Code. As a team, we’re a bunch of seasoned people and we all gathered a lot of experience on cloud infrastructures automation. But we wanted to know more. We wanted to make sure that what we were building was truly helping.

So we decided to talk to as many people as possible. We interviewed dozens of IaC practitioners across several geographies: teams of various sizes, levels of expertise, and cloud automation progress status. They all had one thing in common: because Ops life is what it is, and despite their best efforts and process, they had a hard time reconciling their infrastructure code with the reality of their IaaS accounts. They were facing issues with drift.

Drift can be driven by human input, poor configuration, applications making unwanted changes, etc. It has consequences on toil and efficiency, forces teams to put in place strict controls that decrease flexibility, and can have security and compliance impacts. Even as an experienced Terraform user, as your infrastructure team and codebase grows, it often becomes harder to track drift.

Of course, we knew about drift, having oftentimes faced it ourselves, but we were not aware that it was such a widespread issue. So we decided to try and bring a solution to this problem with driftctl, a free and open-source CLI that tracks, analyzes, prioritizes, and warns of infrastructure drift.

What OSS means to us and why we went open source in the first place.

Being part of the DevOps ecosystem, we are very close to a community of tools situated within the Cloud Native Computing Foundation’s orbit. Those of you who had a glimpse at its landscape know that the CNCF drives a thriving ecosystem of tools in its wake, some of them open source.

Major open source projects such as Kubernetes, OPA (Open Policy Agent), Chef, Puppet, Ansible, Istio… paved the way for us and it was only natural to be in keeping with this trend and give back to the community.

But beyond that, it is our firm belief that an open source first approach is key not only to the widest diffusion of a solution to an issue, but more importantly to the best and most comprehensive elaboration of this solution.

Being only 3 month old, driftctl already benefits from contributions from all over the world with major commits, issues and discussions from Asia, Europe, North America… Only truly open projects can federate such goodwill and efficiency.

This is why we released driftctl under the Apache 2.0 license, and we already gathered the best retributions out of it.

Our commitments to the open source community.

In concrete terms, what does it mean for us to be an open source project? Well, being OSS means first and foremost transparency and sharing:

1) Full transparency from master to release

Obviously, our code is open source. But it is important to us that the on-going developments are also made publicly available. Anyone who wants to know what’s cooking can access master, and see real “work in progress” on our code and not some copy/paste of a finished version waiting to be released from a private repo.

2) Live coding sessions

Because we like coding and sharing, we open live coding sessions on a regular basis. Almost daily, anyone on our community discord server and/or on Twitch can see someone from the engineering team live coding and sharing his screen. Feel free to come say “Hi!”.

3) Live release demos

Transparency also means showing and discussing what we do, which is why each major release comes with a live release demo where we showcase new features, discuss the most interesting issues we had, and talk about what is coming next on the roadmap. Those releases happen on Twitch and YouTube live and are available on replay.

4) Live chats within the engineering team

Being a remote first (and remote only) organization, we do not have any offices or headquarters, which means that any member from the Engineering team can work from any place of his/her liking and that sync and async communication is paramount.

While most of the asynchronous communication within the team goes through our GitHub repositories where we store and share all of our resources, we found that a simple engineering chat does the trick to ensure live exchanges within the team whenever we need to work together on something, or even often do some pair programming. Those chats are also publicly accessible to all on our community discord server.

5) Open issues and roadmap discussions

Our foreseeable roadmap is accessible to all in the ROADMAP.md file on GitHub. All issues (internal and external) are also listed and accessible on this account. Anyone can join in a discussion to propose or upvote a feature.

Indeed, we also try to prioritize items on the roadmap according to the feedback we get. Typically, so far driftctl supports both the AWS and the GitHub providers and we see a lot of discussions around which provider should come next (Azure or GCP). You are welcome to take part to the vote here.

6) Extended support time frame

In this early phase of the project, it is critical that we clearly understand what people expect from the tool, how they use it and what kind of issues they face (not to mention the usual early phase debugging period…). This is why we are extremely grateful for any feedback, any question, any suggestion from our users.

Being still a young organization with limited geographical outreach we decided to set up support on call rotation within the team to cover as many time zones across the globe as possible, and make sure we make the most out of each interaction we can get on our community discord server. This is why you are sure to have someone answering your questions from 7:00 am to 11:00 pm CET.

What’s coming next?

While we know that there are a lot more actions to take, so far we’ve sincerely tried to do our best to share our project with the open source community.

Raising awareness on the issue is one of our very strong focuses. Over the last months, we were lucky enough to be invited to talk about drift issues at some major conferences, such as Fosdem, Hashitalks or the CNCF London meetup and it was really great to share our views and experience on this topic. For those who would like to catch up, most of the talks are accessible on replay.

As we move forward in the process of building driftctl, we frequently discover new potential security threats due to forgotten or unnoticed changes on infrastructures. This is why our next efforts will be centered on demonstrating how teams need to complement their DevSecOps Day 2 toolbox and how ensuring the parity of their code and their infras is a prerequisite to shifting security to the left.

Infrastructure drifts aren’t like Pokemons… …You can’t catch ’em all

driftctl — Wed, 03 Mar 2021 10:01:15 +0000

This talk covers three major topics:

Infrastructure as Code: all the good intentions and the ideal world each of us expected when we started using it, and how it’s actually going in everyday’s Ops life. We will see that how it started is probably different from how it is going and from what we expected.
We will then “drift” together, using Terraform and AWS and share some war stories that we heard from infrastructure teams, and how things sometimes went really wrong for them.
We will finally present driftctl, our open source answer to infrastructure drift problems.

Infrastructure drift: definition

Let’s first agree on what is infrastructure drift: infrastructure drift is what happens when the reality of your infrastructure doesn’t match your expectations. This phenomenon is also known as **“Oh my god” **in production when things unexpectedly go wrong.

Infrastructure as code: an ideal world

How it all started

When it all started, we had this vision of an ideal world, and a bright future that Infrastructure as Code was to offer us, like Plato’s Republic was for society a while ago.

With Infrastructure as code, we were supposed to have one single source of truth (the state file in the case of Terraform).

We were supposed to have versioning of the code with git. The code was to be so great that it would be self-documenting.

In this ideal world, we would have testing everywhere: on the pipelines before and after tdd and bdd etc, like the rest of engineering. Collaboration would be a bliss between teams. Infrastructure would be immutable: we could scale it up and down and throw the rest away.

GitOps would be everywhere: CI/CD for infrastructure, no more locally running things. And basically all the credentials would be under control and multi-cloud would solve all our problems because you know what they say: “API everywhere”… and we would have the skills to handle all those APIs, and the time to do this greatly.

How it is actually going

Marcus Aurelius, the roman emperor a few centuries after Plato wrote to himself: “Do not expect Plato’s ideal Republic”.

As it happens we are not Roman emperors but our ideal Republic is not to be expected either. How it started is not exactly how we see it going.

In our team, many of us were quite experienced but we wanted more feedback on what we were building. So we just asked hundreds of infrastructure teams, SREs… etc around the world where they were standing in their Infrastructure as Code journey.

Basically what they told us was that changes were still happening outside of their Infrastructure as Code: mainly manual changes and scripts that are changing tags, or lambdas updating things after deployment, etc…

A lot of changes are still happening outside of their Infrastructure as Code scope, be it Terraform or Pulumi or whatever infrastructure tool they use, it doesn’t matter.

Do not expect Plato’s ideal Republic

Marcus Aurelius

A bunch of other things go wrong for them as well.

The scope of their changes is not very often fully committed because of broken processes.

People are still changing things manually on their AWS web console and then backporting the changes in Terraform until Terraform starts “screaming”, which means that people are still running Terraform locally with the right access keys on their own laptop. Terrible processes… and everybody knows that once you click everywhere on your console, a lot of other things are deployed on the background and so your Terraform code doesn’t reflect exactly what’s actually deployed online.

Many people we talked to were aware of the good practices about testing requirements. They knew frameworks like Terratest, etc… but most of them never had the time to either learn or write those tests. Sometimes they were just too costly to run for them.

The lack of communication obviously makes things worse: even in these Covid19 times, “Async” communication is still a major issue between teams.

Immutable infrastructure also is still a great struggle to achieve. A lot of the new projects are running more or less around this principle, but people also still have legacy to deal with, like thousands of VMs running every day.

Lots of people are still gaining console access. And guess what ? People with AWS console access tend to change things unless you lock them down completely, which is sometimes just not possible because they are your boss or your customer, or because they are “this very special co-worker” who needs to do “that special thing” manually on the web console… That’s where problems happen.

Many people are also facing skill issues. For example, if a full team very skilled on GCP has a new project that for some reason needs to run on Azure, as all the specifics of this provider are not immediately mastered, they often end up with bad practices in their Infrastructure as Code deployments.

And basically for lots of them, everything that has an API is still not under infrastructure code: we heard terrible stories about GitHub accesses, or DNS records or anything that can be used with Terraform that wasn’t yet by far deployed with it.

Drift happens

All those behaviors have something in common: they generate drift. While drift is pretty cool for cars, the discrepancy between intention and reality can lead to major issues, disasters, or security misconfiguration.

Drift can be a real major pain. To deal with it, most of the people that we met put something in place like terraform plan and apply in a cron tab or in CI/CD somewhere before and after some commits, but were still missing things.

$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed

We love Terraform and all Hashicorp products.

Terraform is an excellent provisioner but it’s not fair to use it for something it’s not meant to do. If anything is outside its scope, it’s just normal that Terraform doesn’t catch it. There are a lot of changes that cannot be seen by Terraform.

Many of us expected terraform plan in cron tabs or in CI/CD to report changes because some of the resources were managed by it, but unfortunately as we are going to see, it is not always the case.

Stories from the trenches

The following stories are based on real stories that we heard while interviewing infrastructure teams. We simplified them to make them easy to understand with some code.

How an intern with read-only access ended up with rogue administrative access and keys (without anyone noticing)

The first story is about an intern with read-only access, ending up with access keys controlled by no one. Those keys were linked to an administrative policy, which nobody noticed.

To gain a better understanding of this story, let’s look at this very simple Terraform configuration file:

We have a user with a key and a policy attachment for read-only.
Let’s apply this configuration:

$ terraform apply
Apply complete! Resources: 9 added, 0 changed, 0 destroyed

Now let’s move to the AWS console: here we can see again this intern user and the key that was created by our own code.

The story goes like this:

This intern went to the DevOps folks and complained that he couldn’t use his key for whatever reason. The team was in a rush, so they quickly logged in to the management console, they deactivated the first key and created a new one, perfectly sure they were going to remember it and add it to Terraform in just a few moments.

On top of that, Terraform was to notify them if they forgot anyway.

What do you think happened? Let’s check with a terraform apply on this user.

$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed

Basically nothing…

While this very simple use case may have led you to think that this specific key and user would be noticed and reported it was not the case.

So the story continues with this intern:

The intern comes back one hour later complaining that he still couldn’t do what he needed to do. As the infrastructure team did not have the time to look into it, they just gave him administrative access so that he could change whatever he needed to, thinking they would revert through his permissions quickly after.

What do you think Terraform found in this case with a terraform apply , as we have a policy attached to the IAM user and everything is managed by terraform?

This team was sure that this was to be reported in terraform plan or terraform apply. But it wasn’t the case because basically it’s out of the control of Terraform. It’s just an attached resource, it doesn’t work like that.

$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed

So the team forgot about it, they went on with the day and it was only weeks after that they realized they had an intern with an administrative key out there in the wild, uncontrolled by anyone, not even mentioned as “deactivated” on the code.

How an intern with rogue Administrative access opened everything to anyone on IPv4 and IPv6 (still without anyone noticing)

The next story starts with a process: as you know, you can have lambdas, scripts etcetera that have administrative access to some resources on a very limited scope.

In this case someone with an administrative access just opened everything to anyone on IPv4 and IPv6 and no one noticed anything. So how did it go?

Let’s go into the console > VPC > us-east-1 > Security groups

So we have our super secure security group that we created using Terraform, with a rule that is attached to it.

As we can see in the AWS console, we have our “super secure security group” and we have our ssh rule for allowing ssh to this local network

For the sake of simplicity we are doing this demo manually here, but someone or something could change this specific super secure security group and could allow anything anywhere on any kind of networks on IPv4 etc… and name it very, very carefully “temp” (you know, like those temporary rules that we’ll remember soon to change…)

So what happened here is that something, with the correct access rights allowed anything to anywhere on this network behind this security group.

This team expected any kind of rules that was added or modified to be reported on terraform and while it is true for a rule it is not true for a new rule and basically the support team went weeks or months with such a rule live on their system without any notification.

$ terraform apply
Apply complete! Resources: 9 added, 0 changed, 0 destroyed

How a scripting issue created a billing nightmare

Our last story is about a team that was building a lot of stuff on S3. They were putting a lot of their binaries there and removing them. So obviously they removed the versioning.

For some reason (long story short) someone activated versioning on all the buckets for the company including this one but they were pretty sure that it wasn’t going to be an issue because it’s the default on Terraform to not be versioned. But the code didn’t explicitly say it so.

Let log in on the AWS console and go to S3> bucket

Now we are on this bucket that we created using the terraform code above, and basically we are going do manually what they did through script: we are just going to enable bucket versioning.

So now Bucket versioning is enabled. Let’s check with a terraform apply what was reported:

$ terraform apply
Apply complete! Resources: 9 added, 0 changed, 0 destroyed

Nothing. So this poor team versioned for weeks hundreds and hundreds of terabytes a day of binaries until they received the bill…

Solutions at hand

There are a lot of solutions in our ecosystem to work around the kind of things that we just saw:

CI/CD Integration (Jenkins, Terraform Cloud, Atlantis, env0…)
Static Analysis (Checkov, TFLint, TFSec,…) are there to help you analyze your code before you actually push to your CI system (or if they are deploying the CI/CD systems, they are there to prevent you to push things like “allow anything to anyone” directly on the code)
Testing & Verification like Terratest (written in Go), InSpec (for the “older” of us, it’s written in Ruby and comes from the Chef days).
Policy & Compliance (Sentinel…) They are a very different set of tools, very cool and very protective.

Enters driftctl

That’s where drift control enters: we built driftctl to manage this drift and be alerted on all the discrepancies between the state and the reality on the AWS accounts.

Let’s just use the same terminal and the same session that we had previously. drift control will basically report you all the changes that you could not see with terraform plan or a terraform apply .

$ driftctl scan --from tfstate://./s3/terraform.tfstate --from tfstate://./vpc/terraform.tfstate --from tfstate://./iam/terraform.tfstate
Terraform provider initialized (name=aws, alias=us-east-1)
Terraform provider initialized (name=aws, alias=eu-west-3)
Terraform provider initialized (name=aws, alias=eu-north-1)

Found deleted resources:
 - Table: rtb-0c5a9bb483ee6eaaf, Destination: [10.0.1.0/24](https://www.google.com/url?q=http://10.0.1.0/24&sa=D&source=editors&ust=1614683869756000&usg=AOvVaw3mt8mKEET3obX8AAtRPp5D)

Found unmanaged resources:
  aws_security_group_rule:
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: ::/0
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: [0.0.0.0/0](https://www.google.com/url?q=http://0.0.0.0/0&sa=D&source=editors&ust=1614683869756000&usg=AOvVaw2umDoMvApX_SsgfP22bUtn)
  aws_iam_access_key:
  - AKIASBXWQ3AYRSNQN5EY (User: INTERN-8mszmf)

Found drifted resources:
  - AKIASBXWQ3AYWWYX72GX (User: INTERN-8mszmf) (aws_iam_access_key):
     ~ Status: "Active" => "Inactive" (computed)

Found 26 resource(s)
- 84% coverage
- 22 covered by IaC
- 3 not covered by IaC
- 1 deleted on cloud provider
- 1/22 drifted from IaC

Now, we can see the rules that allowed “anything to anywhere” on IPv4 and IPv6 and we know that there are rules that were not expected.
We also have the policy attachment of that intern we talked about.

So probably with your eyes of administrative people, DevOps, whatever you are, it doesn’t match. It should ring a bell that something is wrong here actually: what about this new access key? Even if you’re not a security person, if you have a new access key created out of nowhere, reported to you, this should ring a bell.

Let’s go ahead and create a new bucket and call it “Fosdem 2031”.

Obviously this bucket is not in Terraform because we just created live on the console. It’s perfectly normal for Terraform not to report you that this specific bucket was created right now. But this is the job of driftctl to report that you have both unmanaged resources that were created and deltas from managed resources.

$ driftctl scan --from tfstate://./s3/terraform.tfstate --from tfstate://./vpc/terraform.tfstate --from tfstate://./iam/terraform.tfstate
Terraform provider initialized (name=aws, alias=us-east-1)
Terraform provider initialized (name=aws, alias=eu-west-3)
Terraform provider initialized (name=aws, alias=eu-north-1)

Found deleted resources:
  aws_route:
  - Table: rtb-0c5a9bb483ee6eaaf, Destination: [10.0.1.0/24](https://www.google.com/url?q=http://10.0.1.0/24&sa=D&source=editors&ust=1614674194048000&usg=AOvVaw3D80Zc5czyA8iDpAvb61zo)

Found unmanaged resources:
  aws_s3_bucket:
  - fosdem2031
  aws_security_group_rule:
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: ::/0
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: [0.0.0.0/0](https://www.google.com/url?q=http://0.0.0.0/0&sa=D&source=editors&ust=1614674194048000&usg=AOvVaw00rswNHGCuK2ZVSP4oVZAl) 
  aws_iam_access_key:
  - AKIASBXWQ3AYRSNQN5EY (User: INTERN-8mszmf)

Found drifted resources:
  - AKIASBXWQ3AYWWYX72GX (User: INTERN-8mszmf) (aws_iam_access_key):
    ~ Status: "Active" => "Inactive" (computed)

Found 27 resource(s)
  - 81% coverage
  - 22 covered by IaC
  - 4 not covered by IaC
  - 1 deleted on cloud provider
  - 1/22 drifted from IaC

Let’s focus on the VPC here: if we change an existing rule…

… we can see it in the diffs in driftctl

$ driftctl scan --from tfstate://./s3/terraform.tfstate --from tfstate://./vpc/terraform.tfstate --from tfstate://./iam/terraform.tfstate

Terraform provider initialized (name=aws, alias=us-east-1) 
Terraform provider initialized (name=aws, alias=eu-west-3) 
Terraform provider initialized (name=aws, alias=eu-north-1)

Found deleted resources:
  aws_route: 
  - Table: rtb-0c5a9bb483ee6eaaf, Destination: 10.0.1.0/24

Found unmanaged resources: 
  aws_s3_bucket: 
  - fosdem2031
  aws_security_group_rule: 
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: ::/0 
  - Type: ingress, SecurityGroup: sg-0746ca93f95fcc338, Protocol: All, Ports: All, Source: 0.0.0.0/0
  aws_iam_access_key: 
  - AKIASBXWQ3AYRSNQN5EY (User: INTERN-8mszmf)

Found drifted resources:
  - AKIASBXWQ3AYWWYX72GX (User: INTERN-8mszmf) (aws_iam_access_key): 
    ~ Status: "Active" => "Inactive" (computed)
  aws_S3_bucket: 
    -fosdem2031

Found 14 resource(s)
- 50% coverage
- 7 covered by IaC 
- 4 not covered by IaC 
- 1 deleted on cloud provider 
- 1/14 drifted from IaC

The other things we can do with drift control is ignore some resources that you do not want to see reported for whatever reason you have.

So you can also filter using the same system as AWS CLI. For example, you can filter just for S3 buckets if you are not interested in the rest.

$ driftctl scan --filter "Type=='aws_s3_bucket'"

$ driftctl scan --from tfstate://./s3/terraform.tfstate --from tfstate://./vpc/terraform.tfstate --from tfstate://./iam/terraform.tfstate --filter "Type=='aws_s3_bucket'"
Terraform provider initialized (name=aws, alias=us-east-1)
Terraform provider initialized (name=aws, alias=eu-west-3)
Terraform provider initialized (name=aws, alias=eu-north-1)

Found unmanaged resources:
  aws_s3_bucket:
   — fosdem2031

Found 2 resource(s)
– 50% coverage
– 1 covered by IaC
– 1 not covered by IaC
– 0 deleted on cloud provider
– 0/1 drifted from IaC

It’s pretty cool for auditing purposes, for monitoring purposes etc…

And that’s a wrap!

Thank you very much for reading through all this talk transcript.
If you wish to gain a better understanding of driftctl, please visit our product page or our advanced tutorial. We also have a series of short demo videos that you can watch to get started in minutes.

For any questions, feel free to open a discussion on GitHub or visit us discord.

Driftctl is a free and open-source CLI that tracks, analyzes, prioritizes, and warns of infrastructure drift

More info about us

How to catch your infrastructure drift in minutes using driftctl.

driftctl — Mon, 08 Feb 2021 17:43:22 +0000

How to catch your infrastructure drift in minutes using driftctl.

Learn how to use driftctl in a real-life environment, with multiple Terraform states and output filtering.

Whether it’s a script gone wild, a bad API call from a trusted Lambda, or just the daily SNAFU, you want to know about the situation.
Driftctl will do just that. In this guide, you will learn how to use driftctl an open source tool that tracks and warns of infrastructure drift, in a realistic real-life environment, with multiple Terraform states and output filtering. We will demonstrate how manual changes can impact drift detection and how driftctl complements Terraform plan!

Requirements

We recommend using an AWS account dedicated to testing.

An AWS account with IAM keys (AWS Homepage). See this AWS CLI documentation for a quick start.
Terraform 0.14.x (download link)
Driftctl (github)
Example Terraform code for this lab (here)

Create a test AWS environment

Clone the example Terraform code and execute it with Terraform. This Terraform configuration will simply create a VPC and a basically locked down security group.

Disclaimer: we used simple Terraform resources using the AWS provider: we did not try to create the most advanced, useful or complete Terraform configuration.

$ git clone 
git@github.com:cloudskiff/driftctl-advanced-aws-tutorial.git
$ cd driftctl-advanced-aws-tutorial

Export your AWS variables (or AWS_ACCESS_KEY_ID / AWS_SECRET_KEY pair):

$ export AWS_PROFILE="your-profile"

Initialize both Terraform environments : app_env_1 & app_env_2

$ cd app_env_1
$ terraform init
[...]
Terraform has been successfully initialized!

Run Terraform:

$ terraform apply
[...]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
$ cd ..

Do the same for the second environment:

    $ cd app_env_2

    $ terraform init
    [...]
    Terraform has been successfully initialized!

    $ terraform apply
    [...]
    Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
    $ cd ..

Congrats, your AWS account now includes:

A VPC with a random name (from app_env_1)
A security group with a single rule (from app_env_2)

An overview of driftctl scanning options

As of this writing, driftctl can scan for AWS resources and complement Terraform in drift detection. More providers are on their way!

Using driftctl, you can:

Select one or multiple Terraform states, locally or on S3 (with --from <statefiles>)
Ignore some resources or fields by design (filling the .driftignore)
Filter the output to choose only some resources or tags (with --filter <expression>)
Format the output for readability or further processing (with --output <format>)

Driftctl with multiple local Terraform states

The small lab we created above is simulating 2 different “environments” (in real life it can be different applications, environments, or teams), with distinct Terraform states.

If we run driftctl with a single state, resources from the other state will be detected as drifts, or more precisely, unmanaged resources, which is not true and not what we want.

Here’s how to use driftctl with multiple states:

$ driftctl scan **--from** tfstate://./app_env_1/terraform.tfstate **--from** tfstate://./app_env_2/terraform.tfstate
Scanning AWS on region: us-east-1
Found 3 resource(s)
- 100% coverage

Create a Security Group drift and catch it

Let’s manually add a rule to the security group we created, so we can detect it using driftctl:

Go to the VPC Security Group in the AWS console.
Select the security group named “Super Secure Security Group”
Click on “Edit inbound rules” in the “Inbound rules” tab
Click on “Add Rule”

Select “All Traffic”, from “Anywhere”, add a random description
Click on “Save Rules”

Run Terraform from the folder where the security group is managed and confirm it doesn’t catch the manual change:

$ cd env_app_2
$ terraform apply
aws_security_group.supersecure: Refreshing state... [id=sg-0d04a4ce8ff6a74d3]
aws_security_group_rule.supersecure_sg_rule_1: Refreshing state... [id=sgrule-1254751605]
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Now run driftctl again:

$ cd ..
$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate
Scanning AWS on region: us-east-1
**Found unmanaged resources:
aws_security_group_rule:
- sgrule-619247160 (Type: ingress, SecurityGroup: sg-0d04a4ce8ff6a74d3, Protocol: All, Ports: All, Source: ::/0)
- sgrule-1309243877 (Type: ingress, SecurityGroup: sg-0d04a4ce8ff6a74d3, Protocol: All, Ports: All, Source: 0.0.0.0/0)
**Found 5 resource(s)
- 60% coverage
- 3 covered by IaC
- **2 not covered by IaC
**- 0 deleted on cloud provider
- 0/3 drifted from IaC

Holy cow, driftctl caught the drift!

Ignore resources with .driftignore

In many cases, you’ll want driftctl to ignore some resources forever, to stop being notified about their existence, and also stop lowering the coverage score. Those reasons can include the very IAM key you use to run this lab (by definition it can’t be in this Terraform repo), resources that can’t or won’t be managed by Terraform, resources managed from teams sharing the same AWS account but not using Terraform, some legacy resources, or simply manual tests running a bit longer than expected. This is exactly the same approach as the usual .gitignore : you simply add to this file all the resources you want to be ignored.

Create an unmanaged S3 bucket, catch it, ignore it.

To proceed with this step:

Go to the S3 dashboard
Manually create an S3 bucket that you’ll name however you like

Obviously, as this is completely done outside of its control, this bucket can’t be detected as a drift by Terraform, so we’re in the dark, as expected.

Confirm driftctl detects the manually created bucket:

$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate
Scanning AWS on region: us-east-1
Found unmanaged resources:
[...]
aws_s3_bucket:
- randomBucket514
[...]

Open the .gitignore file at the root of the repository, where we execute driftctl and add a line like aws_s3_bucket (in my case: aws_s3_bucket.randomBucket514) Now run driftctl again:

$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate
[...]

Now the manually created S3 bucket is ignored forever!

Output filters for better accuracy

It’s often helpful to be able to filter the output dynamically, to see results only for one type of resource (like only IAM users) or a specific tag (like only a specific environment).

Filtering in driftctl is implemented as in the AWS CLI: you won’t be lost! (hint: it’s JMESPath).

Let’s filter only VPC resources:

$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate **--filter "Type=='aws_vpc'"
**Scanning AWS on region: us-east-1
Found 1 resource(s)
- 100% coverage
Congrats! Your infrastructure is fully in sync.

Let’s now filter only for anything matching the “app_env_1” “Environment” tag on EC2:

$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate **--filter "Attr.Tags.Environment == 'app_env_1'"
**Scanning AWS on region: us-east-1
Found 1 resource(s)
- 100% coverage
Congrats! Your infrastructure is fully in sync.

You now can collect data only for the exact setup you want! Perfect for those reports

JSON output manipulation

We’ve covered driftctl human-readable output, but it’s also often useful to process it further. That’s why driftctl can output to JSON!

Here’s how to generate a JSON file directly:

$ driftctl scan --from tfstate://./app_env_1/terraform.tfstate --from tfstate://./app_env_2/terraform.tfstate **--output json://driftctl.json**

Take a look at the generated JSON file:

$ cat output.json
{
  "summary": {
  "total_resources": 5,
  "total_drifted": 0,
  "total_unmanaged": 2,
  "total_deleted": 0,
  "total_managed": 3
  },
[...]
$

Now you can process this file using a processor like jq, for example, to catch only the coverage percentage and maybe send it to a database, from which a graph can be generated:

$ jq '.coverage' **< driftctl.json
**60

The possibilities are now endless!

Shutting down the lab

Don’t forget to destroy the resources we created for this lab:

$ cd app_env_2; terraform destroy
$ cd ../app_env_1; terraform destroy

Delete the S3 bucket you randomly created and your AWS account is now as clean as before.

Recap

We covered a lot in this advanced tutorial:

We used driftctl in combination with multiple Terraform state files.
We detected manual drifts from Terraform-managed resources as well as entirely unmanaged resources.
We manipulated driftctl output to ignore and filter resources based on type or tags.
We used and processed driftctl JSON output.

Announcing Driftctl because drift matters

driftctl — Thu, 21 Jan 2021 14:21:39 +0000

We recently released the first versions of driftctl, a new open-source project for infrastructure developers, DevOps, SRE, and cloud practitioners, with the goal of helping manage all kinds of drifts.
Stephane Jourdan – CTO and founder

Why? Because infrastructure is a living thing and changes are risky.

In the team, we all managed cloud infrastructures at some level at our previous jobs, as developers with serverless function needs or DevOps managing large production systems. And we all knew that sometimes an emergency required a manual change, a customer tweaked a setting on the console, the boss activated something that had an impact later…or put more simply, that the whole infrastructure was just not totally under control by Terraform (or similar tool), just because ops life in production is rarely perfect and ideal.

And all those tiny changes sometimes got forgotten about, or partially implemented in infrastructure-as-code later on, unfortunately leading to security issues, unstable environments, failed deployments, unexpectedly important bills. And thus this makes us lose the advantages of IaC.

Keeping infrastructures secure & in sync with code.

Like many of you, we use and love infrastructure-as-code, and especially Terraform, and for years we relied on its features to ensure the coherence of our infrastructures (most of us ended up with terraform plan in crontabs or in CI, expecting that return code 0 would ensure a fully synchronized infrastructure!).

While Terraform is excellent at provisioning cloud infrastructures, telling you about changes in your AWS account is not where it shines, mainly because it’s simply not its job.

Until now, to be sure that no change happened in your infrastructure, it implied an excellent level of trust in your tools and dependencies, a very stable IaaS provider, every single resource in every region carefully described in Terraform, then the revocation of all admin credentials of your team (and we really think this is as bad for developers’ sense of ownership and empowerment as it is deeply against DevOps culture), most of the time strong CI/CD pipelines and processes, usage of compliance and policy tools, regular auditing, and probably more great ideas to slow down teams velocity.

But that was clearly not the reality we experienced daily out there! That was often either the goal to reach someday, or a naive belief from management when the team knew very well how to work their way around. But most of the time, credentials were passed around in the team, many resources were still not in Terraform, processes were not perfect, people sometimes forgot to push their locally applied changes, changes happened outside of Terraform, emergencies had to be dealt with quickly (and usually not in Terraform!), consultants came and went, projects were still handed over to customers and no one had time or money to implement complex policies (not to mention that not everyone requires the hassle of PCI/DSS compliance).

And yet, absolutely no one had a good overview of how good or bad things were, easily, so as to take action at some point.

Hope is not a strategy and…that’s why we built driftctl. Driftctl is like your IaC security belt!

Driftctl - a new open-source project to help you monitor IaC changes.

We built driftctl to help developers know when things change on their IaaS, whatever reason it does, and give them a good overview of their infrastructure-as-code coverage.

This tool is open-source at its core (Apache-2.0 license) and we decided to release driftctl very early, so we can gather feedback sooner than later and prioritize accordingly. Also, by building this tool publicly from the beginning, we hope that many users will contribute ideas, suggestions, and maybe some code early on.

The initial releases are focusing on detecting drift on AWS and on Terraform support. Future releases will add support for a lot more, but we didn’t want to wait to release this early. Those releases support what we thought were the most common services on AWS, and this was also backed with hours of interviews with DevOps practitioners around the world those last months. So we started with EC2, S3, IAM, RDS and Lambda, and we’re already working on supporting VPC, CloudFront, Aurora, DynamoDB, API Gateway, SNS/SQS, ECS/EKS/ECR, or KMS. The coming weeks will be exciting!

We also included in the first release a view of your cloud / IaC coverage, so you can quickly know with simple data if your IaaS account is more or less under control with Terraform. Also, if that percentage decreases, it means something changed outside of Terraform! The human-readable output can be switched to JSON if you want to process it.

Early on, we realized that for our own usage, this wasn’t enough: our own AWS accounts were filled with temporary tests, IAM roles, etc., and even our pretty much ideally locked down and mostly automated production account had a few manual resources (like the initial IAM user!). The output was deceptive as it included a lot of garbage we knew of but couldn’t do anything about. So we also added to the first release two different ways to ignore drifted resources, so you can start fresh with a clean output and start from there.

This is a community project, and this is the first step in our mission to build an open-source suite of tools to protect codified infrastructures, so definitely feel free to give us a shout to prioritize another service or resource: open a GitHub issue or discussion, and we’d love to chat with you on our Discord. Come say Hi!

Getting Started

Let me show you how to quickly get started with driftctl, once downloaded from https://github.com/cloudskiff/driftctl/releases . We’ll start with the most simple case where you have the Terraform state locally available (beware, usually not a good practice!).

First, export your AWS environment variables as usual:

$ export AWS_PROFILE=”my-aws-profile” 
$ export AWS_REGION=”us-east-1”
$ driftctl scan 
Scanning AWS on region: us-east-1
Found unmanaged resources:
  aws_iam_access_key:
    - AKIASBXWQ5ATWVVI7REW
[...]
Found 18 resource(s)
 - 22% coverage
 - 4 covered by IaC
 - 14 not covered by IaC
 - 0 deleted on cloud provider
 - 0/4 drifted from IaC

Now, you probably follow good practices and don’t store your state locally or on git but on S3. Here’s how it would work:

$ driftctl scan --from tfstate+s3://acmecorp/states/terraform.tfstate
[...]

Maybe, also as a good practice, you used tags extensively on your AWS resources, so you’d like driftctl to scan only for the Environment: Production key and value?

$ driftctl scan --filter "Attr.Tags.Environment == 'Production'"

Alternatively, you can exclude some resources directly in the .driftignore file (if you want to ignore the “terraform” IAM user, simply add a line containing aws_iam_user.terraform). Obviously, at any time, you can check the help by executing driftctl scan help.

This is only a quick introduction, a lot more details are included in the Github documentation.

Why you should take care of infrastructure drift

driftctl — Tue, 01 Dec 2020 16:34:24 +0000

When talking about infrastructure drift, you often get knowing glances and heated answers. Recording gaps in your infra between what you expected to be and the reality of what is, is a well known and wide spread issue bothering hundreds of DevOps teams around the globe. Interesting to note though, is that depending on their context, the exact definition they will give of drift will vary.

Facing impacts and consequences ranging from intensive toil to dangerous security threats, many DevOps team are keenly aware of the issue and actively looking for solutions.

We decided to look more closely into how they deal with it and conducted a study that will be released in the coming weeks. Here is a foretaste of this study, outlining some of the key facts we recorded.

How infrastructure drift is becoming a major issue : a combination of growing volume and automation
The ever growing amount of workloads running in the cloud has gone hand in hand with a similarly growing number of people interacting with infras, themselves working with several environments.

In this context, as Infrastructure as Code becomes widely adopted by users with heterogenous skillsets, the IaC codebases become larger, mechanically fueling daily occasions to create drift and making it harder to track.

The various definitions and core causes of infrastructure drift
Drift is a multi-faceted problem, let’s agree on a definition.

Indeed, depending on their context and process (or lack thereof) people will tend to stress one specific aspect of drift or another when prompted on a definition. Typically, some people will tend to relate drift to the lack of consistency between, say staging and production environments, which might be true, but incomplete.

Still, most of the people we talked to define infrastructure drift as an “unwanted delta” recorded between their Infrastructure code base and the actual state of their infras.

Two of the most common causes of drift are linked with process or workflow issues, like manual changes in a cloud console not being transposed as code, or changes applied to some environment but not propagated to others. Obviously those issues becomes more and more complex as the number of environments and team members grows. Some teams have dozens of environments that they need to keep updated.

Much more surprising though is the fact that what we call API-driven drift is a widespread issue. API-driven drift happens when an update from an application or a script calling a cloud provider API (or a Terraform provider), autonomously and directly impacts your actual state. In this case, you’ve just lost consistency with your codebase, and you won’t be aware of it until some random apply reveals a problem that might end up taking a couple of hours to fix. This happens even on the cloud provider’s side. One of the people we talked to told us of the time he suddenly got locked out of Azure because of a default machine authentication parameter updated from true to false.

The impacts of infrastructure drift
Maintaining a solid and reliable source of knowledge of how your cloud ressources are organized certainly is important, but at the end of the day you might simply think of infrastructure drift as a mere annoyance not worth the effort.

While drift mostly causes additional work, it also triggers severe security issues too. In one of our interviews, a DevOps lead analyzed the problem quite clearly: “every drift event causes uncertainty, a resolution time, and a potential security issue”. Willingly or not, a developer can do a lot with IAM keys and a SDK. Especially a junior one. Being able to catch bad decisions quickly and reverse your situation back to normal is crucial.

There are also more subtle effects of drift, especially on efficiency. To avoid excessive drift, some teams make significant adjustments to their workflows. In some teams, only the DevOps team lead is allowed access to production. In others where developers are not skilled at IaC, getting a small change to environments done goes through a painful and long ticketing system. In other words, drift causes issues, which leads to rigid / counterproductive processes, which leads to a decrease in speed and flexibility.

How teams solve drift so far
Whatever the maturity level of a team and the degree of automation of their cloud infrastructures, maintaining reliability and security while moving at developers speed implies tracking down inconsistencies, like infrastructure drift.

The faster it is detected, the easier it is to remediate drift, which is why many DevOps we interviewed had a terraform plan in a cron job.

Most teams will explain that to tackle drift, the first thing they did was restrict access to production to a few team members. That certainly contains the problem, but does not solve staging environments drift, as only a smaller part of teams restrict access to staging environments. Following through with that, deploying a full GitOps workflow will prevent some of the drift, but not all of it (not the API-driven events for example) and is still pretty hard to do by the book in practice when there is an urgent issue to solve on the production environment and MTTR is key for the business.

Why we think infrastructure drift should be addressed with a better solution
We recently caught an excellent twitter thread about the various impacts of fast and slow moving broken stuff. Infrastructure drift might not be considered a fast moving broken thing. This is maybe all the more dangerous as sometimes you just can’t get anyone to pay attention to slow-moving broken things because what harm could that iceberg do?”*

Analyzing what solutions are deployed against drift in over 100 teams led us to discover how poorly the topic is addressed. Stay tuned if you’d like to see how we intend to bring an open source solution to this issue.

*borrowed from @yvonnezlam