Directory Structure for Terraform Projects

Terraform doesn't concern itself with the directory structure of our project. It cares about state. We, as the users of the project, are the ones who benefit from a clean and easy-to-understand directory structure.

In this post, we'll explore basic directory structures used for Terraform projects.

Note: If you want a more in-depth discussion about the state and directory structure relationship, you might like my guide Meditations on Directory Structure for Terraform Projects.

• Common ways to structure your directories and files for Terraform projects
  • Per-project boundary
  • Per-service boundary
  • Using workspaces
• Final Thoughts

Common ways to structure your directories and files for Terraform projects

There are multiple ways you can structure your code, Terraform is very flexible about it, so there is no single answer to that question. But let's look at a few commong patterns.

Per-project boundary

A common pattern is structuring your code using the environment as the boundary for separating resources. The directory structure looks like the following:

── basic_project
    ├── environments
    │   ├── dev
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── prod
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   └── staging
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    ├── modules
    └── shared

This structure is clean and easy to understand at a glance. If you work in a specific environment, any change you make won't affect other environments. It has a couple of drawbacks. For example, we could easily add different resources to each environment, creating drift between the environments. It is good practice to keep your environments as similar as possible, so running tests in a lower environment will reflect the effects in the same way as running them in a production environment.

In a clean directory structure, the code inside the environment directories (i.e. dev, staging and prod) are primarily to reference modules. The idea is to reduce code duplication and use the modules as the source of truth of how resources should be configured and with sensible default behaviours.

Per-service boundary

When the number of resources is high, some teams break down the directory structure into smaller areas of interest. For example, split the directory structure by service. The file structure would look like the following:

── service_A
    └── environments
        ├── dev
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        ├── prod
        │   ├── main.tf
        │   ├── outputs.tf
        │   └── variables.tf
        └── staging
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

If you are observant, you'll notice that the directory structure is basically the same as the previous one. Breaking down into smaller areas is similar to breaking it down by project. The idea stays the same: create a boundary to hold certain resources.

Depending on the number of resources, the per-project structure might start causing your plan and apply commands to take a lot of time. The reason is that Terraform has to query the AWS API to check the state of the resources it is tracking. So, a larger number of resources will lead to more API calls.

There are other reasons why we might need to segregate the state:

• High Number of resources, we already mentioned this
• Using different AWS accounts per environment
• Breaking state per region. Projects sometimes segregate their state per region to:
  • Reduce blast radius if a region is down
  • Deployment strategies that would prefer to only deploy to certain regions for testing, i.e. canary deployments <!-- vim-markdown-toc Redcarpet -->

Using workspaces

Another common alternative for the per-project directory structure is using workspaces. This structure maintains the same code for all environments, reducing the possibility of differences between environments.

── single_code_multiple_var
    ├── infra
    │   ├── main.tf
    │   ├── provider.tf
    │   ├── outputs.tf
    │  └── variables.tf
    ├─── environments
    │   ├── env.dev.tfvars
    │   ├── env.staging.tfvars
    │  └── env.prod.tfvars
    └── modules

There are still cases where we would like different behaviours per environment. For example, using smaller instance types for lower-level environments and larger instance types for production. To reflect that difference, we use different values in tfvars files that will capture the differences. This directory structure and workflow are OK if you are OK with having the same backend for all the workspaces:

terraform {
  backend "s3" { <--- we can't use variable interpolation, so we can't do the following
    region = "${local.env[var.enviroment]}"
    bucket = "rderik-tfstate-${var.environment}"
    key = "${var.environment}/tfstate/terraform.tfstate"
    dynamodb_table = "rderik-tfstate-${var.environment}"
    encrypt = true
  }
}

If, for some reason, we need to use a different backend per environment, then we have to solve the problem with the backend, which has its own set of peculiarities.

Final Thoughts

The directory structure that we end up using for our project is highly dependent on how we want to manage the state of our project. Consider if you want to keep one state per project, service, region, etc. The key to deciding which directory structure to use in your project is for you to understand the relationship between:

• State
• Directory structure
• Backend
• Module versions

Once you clearly understand how you want to handle those aspects of your project, it'll be easier to choose the directory structure that will make your life easier. But don't worry too much about it. It is OK to start with one structure and later understand that you have outgrown that structure and migrate out of it. Believing you'll have all the answers from the beginning is a common mistake.

Software projects are iterative. You keep growing and you keep adapting that is the nature of our work.