Secure Deployment: Three Security Highlights For Terraform on AWS

Coauthored with Ben de Haan

So, you want to build your infrastructure in AWS and use Terraform for your Infrastructure as Code? And you want to do it securely? In this blog we highlight three things you should pay attention to from a security perspective: your IAM roles and trust relationships, your Terraform state, and your detection & monitoring. Of course, each of these topics deserve their own in-depth blog, but we’ll start highlighting three key touchpoints.

If you have read our previous blogs on secure deployment, then you know by now that you can best deploy infrastructure using a secure CI/CD pipeline. If you followed up on that, then you already have a great starting point for your cloud-infrastructure. Hopefully you only deploy your production environment through infrastructure as code with protected branches and protected runners. So what’s next? Let’s talk about three key items: Identity and Access Management (IAM), the Terraform state, and monitoring.

IAM

AWS allows you to segregate your cloud infrastructure over multiple AWS accounts. An account is a natural boundary and you should absolutely use different accounts for e.g. production, staging, and development. The accounts can -and should- be managed using AWS Organizations. AWS Organizations is useful to glue the different AWS accounts together. This enables you to automatically provision them based on organizational structure and attach the appropriate guard rails.

Organization Account Access Role

When you create a new AWS account in the organization, it automatically gets the OrganizationAccountAccessRole. This role provides admin permissions to anyone who can assume it from the organizations root account. So far, so good.

The very next step you want to take is to set up separate roles for your cloud engineers or SRE team, DevOps team, data platform teams, and so on. Additionally, you’ll need roles for the CI runners that they use. Some of these roles might have to do some terraforming themselves.

It would be easy to just use the OrganizationAccountAccessRole and extend that trust relationship to SRE teams or pipelines. The risk here is that anyone who can assume such a role can do pretty much anything with the AWS account. Much like doing things with root, don’t use the OrganizationAccountAccessRole unless you really need to. Ensure you have an emergency process for when you do. Know who or what can assume administrative roles. With the emergency process in place, limit further use to your privileged CI/CD pipeline.

Terraform State

Your Terraform state is used by Terraform to keep track of real world resources. The state serves as a basis to make appropriate changes to your infrastructure.

What Could go Wrong in Terraform State

You (or an attacker) could manipulate the Terraform state by hand. If you do, you could lure Terraform into creating resources which your actual Terraform code never asked for. This can cause headaches in trying to find drift in the cloud account from your definitions in Terraform state.

Terraform state has its challenges when it comes to locking and such, which can provide security challenges of its own. More importantly: we often find secrets in Terraform state, such as “initial passwords” which often remain the same, as well as access-tokens used by custom providers. And how about the values that you want to store in AWS Secrets manager? They often end up in the Terraform state as well.
Essentially, there’s a few things to pay attention to when you think of a potential compromise of your Terraform state:

• unintended/unauthorized changes via state manipulation can cause untracked infrastructure changes;
• blast radius of states can extend across environments, meaning a compromise can potentially impact all environments at once;
• secret leaks via state provide additional entry points for attackers.

How to Secure Terraform State

• Try to prevent write access to the Terraform state apart from the privileged runners mentioned earlier as well as a “break glass” user for emergencies (which should be behind MFA).
• Split Terraform states by environment and/or other ways that make sense for your organization. Be sure to check out workspaces.
• Limit the ability to extract the state and do a local plan. Developers shouldn’t need direct access to production, as that is the task of the CI/CD pipeline. Use the aforementioned “break glass” role in case of emergency.
• Try to prevent storing secrets in Terraform state altogether by moving secrets management to a solution like HashiCorp Vault or something similar. Alternatively: make sure it ends up encrypted in Terraform state for the time being.

Detection & Monitoring

Even when you use infrastructure as code, you can easily make mistakes. You can assign overly broad permissions to roles or users, you can easily open up things like your S3 storage buckets too far, etcetera. Therefore:

• Always enable AWS Config and AWS Organizations’ service control policies (SCPs) to provide guard rails across the organization. For inspiration on SCPs, check out asecure.cloud’s SCP repository.
• Next, make sure you store your CloudTrail logs for a decently long period of time.
• Use the Access Analyzer to spot other possible mistakes.
• Make sure you enable access logging for your storage solutions, especially for those that keep the Terraform state.
• Finally: logging is nice, but don’t forget to make the information actionable. Set up an alerting system that alerts something and/or someone. Have it alert when highly privileged roles are assumed, the AWS root user is used, or when something else than your CI/CD pipeline accesses the Terraform state storage for production. Start small and extend based on risk – ideally you do regular threat modeling to discover detectable attack paths.

Next Steps

That concludes our three security highlights for Terraform on AWS. Now that we touched upon IAM, Terraform state, and Detection & Monitoring, it’s time for the next steps.

Want to catch things earlier? Try out tools like Terraform compliance, Prowler, Scoutsuite, or Inspec to inspect your security posture or even prevent misconfigurations from reaching production. You could also use driftctl to see whether there is any drift between your Terraform code and what’s in your environment. Want to have a deep dive on these topics? Let us know!