Terraform: Standard Workflow and State Facilitation

Terraform facilitates managing compute resources, user accounts, certificates, secrets, and much more in different cloud environments. At its heart is a declarative specification language with which resources are expressed, and a rich CLI that executes various commands.

This article continues the introduction to Terraform. It details the Terraform workflow for defining, provisioning, modifying, and destroying infrastructure objects. At the core of this is the state - the representation of infrastructure objects managed by Terraform. You will also learn how to configure state backends and how state data is structured.

The technical context for this article is Terraform v1.4.6, but it is applicable to newer versions as well.

This article originally appeared at my blog admantium.com.

Terraform Workflow

In a Terraform project, you start with the initialization, and then iterate between writing, planning, and applying. Here is a short definition for each of these phases:

• Init: The first step is to create a file directory that includes the best-practic based files. As detailed in my earlier article, this will be files specifying the provider configuration and the projects' variables, output, data and resources. Typically, you will configure the providers in this phase so that a terraform init is successfull and all required providers are installed correctly.
• Write: In this phase you specify which resources you want to create, their attributes and dependencies. You also determine which input values are required. Non-sensitive data can be put in variable files, secrets should be defined as variables, but their concrete values should be provided as runtime arguments or envirionment variables for the Terraform binary. At the end of this phase, you will run terraform fmt which makes a syntax check of all files.
• Plan: Once all desired resources are described, you will check what modifications to the concrete infrastructure objects would result. Therefore, execute the command terraform plan to get a detailed plan which resources are created, modified, or destroyed with a great level of detail. Each resources will be prefixed with a symbol: -/+ means recreated,. + means created, - means deleted and ~ updated in place.
• Apply: The final step is to apply the changes. The command terraform apply prints the execution plan and asks the user to confirm the changes. If confirmed, the resources will be created, modified, or deleted. These conrete operations are at the same time constrained and facilitated by the provider. For example, the modification of an cloud servers operating system could mean to either stop the VM, change its HD configuration, and start it while keeping all other configuration items the same, or it cloud mean to destroy the VM entirely and create a new one.

The general workflow steps will be used iterativly until all managed infrastructure objects are controlled with Terraform.

Furthermore, additional commands complement the workflow for specific information needs. For example, you can run terraform refresh independent of the plan step to update the local state. Or you can perform an apply on selected infrastructure objects only with terraform apply -target.

For all commands, the log verbosity of the putput can be configured with the environment variable TF_LOG (values: TRACE, DEBUG, INFO, WARN or ERROR.) Log output can be persisted by writing to a log file: This needs to be defined with the environment variable TF_LOG_PATH.

State

All Terraform operations starting with the plan step require state. Fundamentally, state is an abtract representation of the infrastructure objects that should be managed by Terraform. There are several nuances to the term state in Terraform, so we need to define them upfront to avoid confusion.

State Terminology

• State backend: Defines where and how to state is saved. In its default operation mode, Terraform uses a local backend and a local state file. And when you configure another backend, then the state is stored remotely in whatever data format the backend defines.
• Local state: Refers to the state file created when no backend is configured.
• Remote state: Refers to the state data of a configured backend.
• Stored state: The resource object configuration contained in the state backend
• Actual state: The concrete resource object configuration returned by its provider plugin
• Intended state: The totality of all currently configured resources and their attributes
• State history: represents discrete, versioned states detailing all changes that occurred.

State Update and Modifications

With this, lets now investigate what happens when Terraform is used to update the state representation:

• Terraform builds a tree data structure of all resources contained in all configuration files
• For every item in this tree, Terraform requests data via the provider plugin (e.g. calling an REST API endpoint with an ID for a cloud server)
• If the item exists, Terraform create a state record of this resource, which includes all attributes that the provider plugin returned, and additional Terraform specific attributes
• If the item does not exist, nothing happens

The Terraform plan command can be used in two different modes. A speculative plan is created by executing terraform plan. All necessary changes are determined, but they are only shown as a textual representation. An executable plan is created by using terraform plan -out=FILE, which saves the concrete changes in a binary file. In both modes, the concrete steps are these:

• update the stored state: The actual state is checked, and any changes are recorded in the stored stage. At this point in time, the stored state absolutely matches the actual state
• determine update operation: For each item of the intended state, the differences between the stored state and the actual state are determined
  • If the intend state resource does not exist, create it
  • If the intend state resource differs from the actual state, determine if the provider plugin allows its modification. If yes, an update operation is planned, if not, a destroy and create operation is planned
  • If a stored state resource exists, but it is not defined in the intended state, a delete operation is planned

Local Backend

If no backend is configured, Terraform defaults to the local backend.

Configuration

You can provide this optional configuration, but as stated, it is not necessary:

terraform {
  backend "local" {}
}

State File

The local state file is stored at the location .terraform/terraform.tfstate. Its data format is JSON, and contains a complete record of the managed infrastructure, a history of their changes.

It’s important to note that this file contains all information about the output variables and resources in clear text, even sensitive data like IP addresses, password, SSH keys. Storing this information more securely can only be achieved with a remote backend.

Let’s detail the file contents. Overall, its structured into three distinguished parts - Terraform metadata, output variables, and resources - and has the following overall structure:

{
  // metadata
  "version": 4,
  //...
  "outputs": {},
  "resources": [ ]
}

State File Content: Terraform Metadata

This block details the Terraform version used and how many state updates have been recorded.

{
  "version": 4,
  "terraform_version": "1.4.6",
  "serial": 127,
  "lineage": "bdf149a7-9215-2c29-123a-f52841071353",
  //...
}

State File Content: Output Variables

All output variables and their versioned values are stored.

{
  //...
  "outputs": {
    "controller_ip": {
      "value": "95.216.143.196",
      "type": "string"
  }
}

State File Content: Resources

An array of objects that represents the current state resource objects, distinguished by their type and containing the combined attributes of the provider plugin and additional Terraform specific attributes.

• Terraform attributes: The following attributes determine if the resource is already managed by Terraform (e.g. it was created with a Terraform command), which concrete resource type it is, its name, and with which provider and terraform version it was created

- "mode": "managed",
  "type": "hcloud_server",
  "name": "controller",
  "provider": "provider[\"registry.terraform.io/hetznercloud/hcloud\"]",
  "terraform_version": "1.4.6"

• Resource attributes: The complete configuration of the resource, with attributes that are specific to it. In this case, a cloud server, it contains its datacenter, whether backups should be made, its firewall configuration, IP addresses and much more.

"instances": [
  {
    "index_key": "controller",
    "schema_version": 0,
    "attributes": {
      "allow_deprecated_images": false,
      "backup_window": "",
      "backups": false,
      "datacenter": "hel1-dc2",
      "delete_protection": false,
      "firewall_ids": [
        571098,
        571099,
        571100
      ],
      "id": "24908450",
      "ignore_remote_firewall_ids": false,
      "image": "debian-11",
      "ipv4_address": "95.216.143.196",
    }
  }
]

Remote Backend

State representation in a remote backend serves the same purpose of the local state file: To obtain a representation of the current state for planning resource objects states. In addition, it can have two additional features: Encryption and state locking. These features must be implemented by the concrete backend; the Terraform documentation details if that is the case.

• Encryption: this means that the remote backend saves its data not in plain text. This helps to protect secrets and other sensitive data
• State locking: helps to keep the state data persistent when multiple team members work with the same Terraform project. Whenever a member, or an automated processes e.g., in a CI/CD tool updates the state, no other updates, started by a different process, can happen until this operation is completed. This ensures that the state is persistent, and it prevents the simultaneous modification of remote resources that could lead to errors.

At the time of writing, for Terraform v1.13, the following remote backend types are supported:

• remote - Hashicorp proprietary backend
• azurem - Blog Storage
• consul - Consul KKV Store
• cos - Tencent Cloud Object Storage
• gcs - Google Cloud Storage
• http - State Storage with a REST client
• kubernetes - Kubernetes secrets objects
• oss - Alibaba Cloud OSS
• pg - Postgres Database
• s3 - Amazon S3 bucket or Dynamo DB

Lets see a concrete example by using the Postgres DB.

Configuration Example: Postegres

A Postgres DB backend is configured as follows:

terraform {
  backend "pg" {
    conn_str = "postgres://user:[e-mail]/terraform_backend"
  }
}

However, you should not store the concrete value for conn_str attribute because it contains sensitive data. Instead, it should be passed to the Terraform command as follows:

terraform init -backend-config="conn_str=postgres://user:[e-mail]/terraform_backend"

State Data Example: Postegres

The particular data representation is unique for each backend. In principle, it will provide the very same information as a local state file: Terraform metadata, a list of attributes with metadata and concrete resource attributes. The state is also versioned.

When using Postgres DB, the state data is a JSON document stored inside a table:

SELECT * FROM terraform_remote_state.states

ID  NAME      DATA
1   default   //json

The DATA is this:

{
  "version": 4,
  "terraform_version": "1.4.6",
  "serial": 1,
  "lineage": "c760c743-e9c7-6b0e-4ae4-9bfc3b5bb11c",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "hcloud_server",
      "name": "controller",
      "provider": "provider[\"registry.terraform.io/hetznercloud/hcloud\"]",
      "instances": [
        {
          "index_key": "controller",
          "schema_version": 0,
          "attributes": {
            "allow_deprecated_images": false,
            "backup_window": "",
            //...
          }
        }
      ]
    }
  ],
  "check_results": []
}

Conclusion

In this article, you learned about the Terraform workflow and state representation. A Terraform project starts with initialization, where you create a file structure and configure the necessary providers and a state backend. Then, you iterate over three phases: a) Writing resource definitions and doing as syntactic checks, b) planning where you update the state storage to match the actual state and then see which changes to the actual state will be performed, and c) applying the changes. In all these steps, state is essential. You learned several nuanced concepts of state, understood that applying consumes the state data to create operations so that the desired state is realized, and learned about the local and remote backend features and how they represent data.