Terraform - How to prevent sensitive data from ending up in the .tfstate

TL;DR

By default, Terraform stores all secrets in plaintext within its state file (.tfstate). The traditional sensitive = true argument only hides the data in the CLI, but does not protect it in the state. The definitive solution to securely inject secrets at runtime without leaving a trace in the state is to combine ephemeral = true variables with the write-only arguments of each provider.

Intro

Terraform is one of the most popular IaC (Infrastructure as Code) tools among DevOps teams. And it's no surprise; it's portable, has a wide catalog of providers, and is relatively easy to use.

With Terraform, we can deploy infrastructure, keep track of its configuration, and finally, delete it when it is no longer needed, all with just a few commands.

To properly work, Terraform needs to know the resources it has deployed as well as their configuration. To achieve this, it relies on a state file with the .tfstate extension that acts as a resource inventory.

This file is automatically managed by Terraform and reflects the state of our infrastructure. By default, the state is saved in plaintext, so anyone with access to it can read it without problems.

This represents a security problem for two reasons:

• It reveals the design of the environment. It allows knowing the infrastructure, the resources, how they connect to each other, as well as their configuration.

• It implies potential access to secrets.

We know that Terraform needs to store the value of these secrets to check if they change, but is there a safer way to manage them and prevent them from ending up exposed in the state?

What Does the Documentation Say?

As good engineers, our first instinct is to check the official documentation (or interrogate our trusted AI), which is quite clear about this:

If you add secret values directly to your configuration, Terraform stores those secrets in its state and plan files.

Fortunately, the documentation also points us to two key arguments:

• sensitive: Tells Terraform to hide the value of a variable or output in the CLI.

• ephemeral: Allows handling values at runtime without persisting them in the state.

It should be noted that both arguments are not mutually exclusive, so you can use them together to ensure that a secret neither appears in the terminal nor leaves a trace in the state.

But, if we don't store the secret's value, how does Terraform know when to update it? When using ephemeral, write-only arguments defined by each provider are also typically used. These arguments allow passing secrets, which are only available during runtime, to the resources without them becoming part of the state.

write-only arguments usually end with the suffix _wo and have their corresponding version argument _wo_version. The latter are used by Terraform to know when to update the secrets, so we just have to increment the value of that version for Terraform to detect the change and apply the new secret.

True to the motto of "Trust, but verify", let's move from theory to practice to check it for ourselves.

Scenario 1: Hardcoded Secret

The first scenario is to hardcode secrets directly into the Terraform configuration. As we have already mentioned, this causes them not only to be stored in the state, but also allows anyone with access to the configuration itself to read the secret effortlessly.

As an example, let's create a parameter within Parameter Store in AWS. These parameters can contain anything, since what they store is a string, and those things can include a secret, like an API key to access some platform.

resource "aws_ssm_parameter" "payment_api_key" {
  name        = "/production/payment-gateway/api-key"
  description = "API Key for processing payments securely"
  # "SecureString" ensures AWS encrypts this value at rest
  type        = "SecureString"
  # The value is passed in securely via the variable
  value       = "MiSuperSecreto876#"
  tags = {
    Environment = "production"
    Application = "billing"
  }
}

This code creates a parameter named /production/payment-gateway/api-key of type SecureString (encrypted with KMS), applies two tags (Environment=production and Application=billing) to it, and defines the value of the secret: MiSuperSecreto876#.

If we run a terraform plan, we see how the secret appears masked and in its place we find value = (sensitive value).

But, didn't we mention earlier that, in order to hide a secret in the CLI, we had to explicitly use the sensitive = true argument?

That's correct. However, there is a default behavior that has our backs:

Terraform also treats any expressions that reference a sensitive variable or output as inherently sensitive.

Let's take a look at the state:

"id": "/production/payment-gateway/api-key",
"name": "/production/payment-gateway/api-key",
"region": "us-east-1",
"tags": {},
"tags_all": {},
"tier": "",
"type": "SecureString",
"value": "MiSuperSecreto876#"

As expected, Terraform has stored the secret in the state.

Scenario 2: The sensitive argument to the rescue (or almost)

For the following scenario, we're going to apply some best practices. Instead of leaving the secret exposed in the code for anyone to see, we will move it to an external variable and use the sensitive=true argument:

variable "payment_api_key_value" {
  description = "The secret value for the payment API key"
  type        = string
  sensitive   = true  
}

resource "aws_ssm_parameter" "payment_api_key" {
  name        = "/production/payment-gateway/api-key"
  description = "API Key for processing payments securely"
  type        = "SecureString"
  # 2. Reference the sensitive variable
  value       = var.payment_api_key_value
  tags = {
    Environment = "production"
    Application = "billing"
  }
}

Upon running a terraform plan, the terminal output will be identical to the first scenario: the secret will not be shown in the CLI and Terraform will mask it completely.

However, the state still contains the secret:

"id": "/production/payment-gateway/api-key",
"name": "/production/payment-gateway/api-key",
"region": "us-east-1",
"tags": {},
"tags_all": {},
"tier": "",
"type": "SecureString",
"value": "MiSuperSecreto876#"

It is proven then that sensitive only hides the value in the CLI, but Terraform still saves it in the .tfstate so it can track changes.

Scenario 3: Secrets in an external platform

One of the most highly recommended practices when using secrets in Terraform is storing them in an external manager like AWS Secrets Manager and consuming them at runtime.

For this scenario, I've used HashiCorp Vault locally with the same Terraform code as the previous scenario:

variable "payment_api_key_value" {
  description = "The secret value for the payment API key"
  type        = string
  sensitive   = true  
}

resource "aws_ssm_parameter" "payment_api_key" {
  name        = "/production/payment-gateway/api-key"
  description = "API Key for processing payments securely"
  type        = "SecureString"
  value       = var.payment_api_key_value
  tags = {
    Environment = "production"
    Application = "billing"
  }
}

And we will execute it using:

terraform apply -var="payment_api_key_value=$(vault kv get -address="http://127.0.0.1:8200" -field=password secret/myapp/database)"

Which produces a plan similar to the previous ones with the secret masked.

However, the secret is still visible in the state:

"attributes": {
    "description": "API Key for processing payments securely",
    "id": "/production/payment-gateway/api-key",
    "name": "/production/payment-gateway/api-key",
    "region": "us-east-1",
    "tags": {},
    "tags_all": {},
    "tier": "",
    "type": "SecureString",
    "value": "PasswordSuperSecretaDeVault#",
  }

Scenario 4: Using ephemeral for the win

In the last scenario, we are going to test how the ephemeral argument works. As we mentioned earlier, this argument prevents Terraform from storing the secret in the state.

To start, let's add ephemeral=true to our variable:

variable "payment_api_key_value" {
  description = "The secret value for the payment API key"
  type        = string
  sensitive   = true
  ephemeral   = true  
}

Afterward, we add the write-only arguments:

resource "aws_ssm_parameter" "payment_api_key" {
  name             = "/production/payment-gateway/api-key"
  description      = "API Key for processing payments securely"
  type             = "SecureString"
  value_wo         = var.payment_api_key_value
  value_wo_version = 1
  tags = {
    Environment = "production"
    Application = "billing"
  }
}

Running terraform plan, we see how the secret is hidden thanks to sensitive (just like before).

If we apply the changes with terraform apply, we see that there is no trace of the secret in the state:

"attributes": {
    "data_type": "text",
    "description": "API Key for processing payments securely",
    "has_value_wo": true,
    "id": "/production/payment-gateway/api-key",
    "name": "/production/payment-gateway/api-key",
    "region": "us-east-1",
    "tags": {},
    "tags_all": {},
    "type": "SecureString",
    "value": "",
    "value_wo": null,
    "value_wo_version": 1
  }

Conclusions

Marking a variable as sensitive is a necessary measure to hide secrets in the CLI and logs, but it is insufficient on its own, as the secrets remain exposed to anyone with access to the state.

Consuming secrets at runtime from platforms like AWS Secrets Manager or HashiCorp Vault mitigates the risk of exposing them in the source code, but Terraform will still save the final value in the .tfstate if ephemeral is not used.

ephemeral and write-only arguments prevent secrets from ending up in the state.