Oleksandr Kuryzhev

Posted on Oct 17 Originally published at kuryzhev.cloud

Terraform Vault Secrets: Keep Them Out of State and Git

#terraform #devops

Originally published on kuryzhev.cloud

The scenario

A team's Terraform repo needs a database password. The quickest way to get one into the config is a terraform.tfvars file that someone pasted into a CI variable. Handling Terraform Vault secrets properly means the password lives in HashiCorp Vault, Terraform fetches it at apply time, and nothing sensitive is committed. That is the goal, but it has one catch worth understanding up front.

The classic approach is the vault_kv_secret_v2 data source. It works, but values read through data sources are persisted in Terraform state in plain text, and this is documented behavior. Marking a variable sensitive only hides it from CLI output. So the secret leaves Vault and ends up in whatever backend stores your state.

Newer Terraform and Vault provider releases close that gap with ephemeral resources and write-only arguments. These values exist during a run and are never written to plan or state. This guide builds that pattern end to end. It also shows the fallback when your versions or target resources don't support it yet.

Prerequisites

You need a reachable Vault server with a KV version 2 secrets engine mounted (the dev-mode default is secret/) and a way to authenticate non-interactively.

Ephemeral resources require Terraform 1.10 or later, and write-only arguments require 1.11 or later. The Vault provider needs a release that ships the vault_kv_secret_v2 ephemeral resource. Pinning the current 5.x line is a safe default, but confirm the exact minimum in the provider changelog. Verify the details in the Terraform ephemeral resources documentation and the Vault provider registry page, because they move between releases.

You also need a target resource that exposes a write-only argument for the secret. Many providers have added these gradually, for example password_wo on the AWS provider's database instance resource. Check your provider's resource documentation rather than assuming support.

Terraform 1.11+ (or 1.10+ if you only use ephemeral values in provider blocks and other ephemeral contexts)
A KV v2 secret, for example at secret/app/db with a password key
A Vault auth method for CI, such as AppRole or JWT/OIDC, not a long-lived root token
A remote state backend with encryption and restricted access, as a baseline regardless

If you already run keyless CI, the same idea applies here. The DevOps_DayS archive covers OIDC federation for CI, which pairs well with Vault's JWT auth.

Step 1: Configure the provider without hard-coded credentials

Keep the provider block free of tokens. The Vault provider reads VAULT_ADDR and VAULT_TOKEN from the environment. It also supports auth_login* blocks for AppRole, JWT, and other methods, which are listed in the provider documentation on developer.hashicorp.com.

The following provider block sets only the address and one behavior flag worth understanding.

terraform {
  required_version = ">= 1.11"
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "~> 5.0" # confirm the ephemeral-capable version in the registry
    }
  }
}

provider "vault" {
  address = var.vault_address # or set VAULT_ADDR in the environment

  # By default the provider requests a child token, which needs
  # permission on auth/token/create. This example skips that and uses
  # the supplied token directly; remove this line if your policy
  # allows token creation and you want a short-lived child token.
  skip_child_token = true
}

Watch out for the child token default. A typical failure is a 403 on auth/token/create when the CI policy is tightly scoped. You have two documented options. You can grant that capability and drop the flag, or keep skip_child_token = true as shown, in which case the CI token's own TTL is what limits exposure.

Step 2: Write a least-privilege Vault policy

The identity Terraform uses should read only the paths the configuration needs. With KV v2, the API path differs from the CLI path. The policy must target secret/data/..., not secret/....

# terraform-ci.hcl: apply with: vault policy write terraform-ci terraform-ci.hcl
path "secret/data/app/db" {
  capabilities = ["read"] # KV v2 reads go through /data/
}

path "secret/metadata/app/db" {
  capabilities = ["read"] # only needed if you read version metadata
}

Watch out for the missing data/ segment. A policy written for secret/app/db looks correct but still returns permission denied on KV v2. Bind the policy to your AppRole or JWT role. Keep token TTLs short enough that a leaked CI token expires quickly.

Step 3: Read Terraform Vault secrets as ephemeral values

An ephemeral block fetches the secret during plan and apply but never records it. You can reference it only in places that accept ephemeral values: write-only arguments, provider configuration, and other ephemeral contexts. Terraform rejects references from ordinary attributes at validation time, which is the safety net you want.

ephemeral "vault_kv_secret_v2" "db" {
  mount = "secret"
  name  = "app/db"
}

resource "aws_db_instance" "app" {
  identifier        = "app-db"
  engine            = "postgres"
  instance_class    = "db.t4g.micro"
  allocated_storage = 20
  username          = "app_admin"

  # Write-only: sent to the API, never stored in plan or state
  password_wo         = ephemeral.vault_kv_secret_v2.db.data["password"]
  password_wo_version = 1 # bump this number to trigger a password update
}

Because write-only values are not stored, Terraform cannot detect when they change. The companion _version argument is the documented mechanism: increment it when the secret rotates in Vault. Confirm the attribute names for your provider versions, since the ephemeral resource's output shape (data versus data_json) is version-dependent.

Step 4: Plan the fallback for unsupported resources

Not every resource has a write-only argument yet. When you must use the vault_kv_secret_v2 data source, the secret will be in state, so treat state as a secret store:

Use an encrypted remote backend.
Limit read access to the state to the same people who can read the secret.
If the backend is an object store such as S3 or GCS, enable versioning and access logging on the bucket.

Prefer dynamic secrets where possible. Vault's database and AWS secrets engines issue short-lived credentials, so a value captured in state expires on its own. That limits the damage even when the pattern isn't ideal.

# Fallback: value WILL be stored in state. Protect the backend accordingly.
data "vault_kv_secret_v2" "legacy" {
  mount = "secret"
  name  = "app/legacy"
}

# Reference as data.vault_kv_secret_v2.legacy.data["api_key"]

Also avoid managing the secret's value itself with vault_kv_secret_v2 as a resource unless you accept it in state. Recent provider releases offer a write-only data_json_wo argument for that case; check the resource documentation for your version.

Verify and test

Start by proving that the identity can read the path and nothing broader. Then confirm the secret is absent from state. Use a throwaway secret value for this test so a failure costs nothing.

# 1. Confirm the policy grants what you expect (note the /data/ path)
vault token capabilities secret/data/app/db

# 2. Validate and plan; a reference from a normal attribute fails here
terraform validate
terraform plan -out=tfplan

# 3. Apply, then search state and plan for the secret value
terraform apply tfplan
terraform state pull | grep -c "your-test-password"   # expect 0 with write-only args
terraform show -json tfplan | grep -c "your-test-password"

With the ephemeral and write-only path, both counts should be zero. A non-zero count means something is still reading the value through a data source or a regular attribute. Run terraform state list and look for data.vault_kv_secret_v2 entries.

Rotate the secret in Vault, bump password_wo_version, and confirm the plan shows an in-place update.
Revoke the CI token and confirm the run fails with a clear permission error rather than a silent default.
If an audit device is enabled, check Vault's audit log for the read. It gives you evidence of who fetched what and when.
Grep the repository and CI variables for leftover .tfvars files with the old password.

Where this leaves you

Getting Terraform Vault secrets right comes down to three habits:

Authenticate with short-lived identities.
Scope policies to exact KV v2 paths.
Prefer ephemeral resources with write-only arguments so values never reach state.

When a resource doesn't support that yet, treat the state backend as part of your secret store and favor dynamic credentials that expire. Ephemeral support and attribute names change between Terraform and provider releases. Before you upgrade, check the official Terraform documentation and the Vault provider registry page, and rerun the state-grep check after every provider bump.

DEV Community