Stop Minting Static Credentials

Introduction

Every static credential starts out reasonable. A pipeline needs to SSH into a
server, so someone generates a key pair and stores the private half somewhere
the pipeline can read it. A Terraform workspace needs to talk to AWS, so someone
creates an access key and pastes it into a workspace variable. An application
needs a database, so someone creates an application user, picks a password, and
puts it in a config file.

Each of those is a credential that nobody asked to expire. It works on day one,
and it keeps working long after the person who created it has moved on and
nobody remembers which systems would break if it were rotated. So it doesn't
get rotated.

The fix has the same shape every time. Stop storing a secret that proves who you
are, and start trusting an identity system to issue a short-lived one on demand.
This post walks through three of those substitutions, each done for real in my
lab environment, and the thing that bit on the way in for each one:

1. A Vault SSH certificate authority instead of distributing key pairs.
2. OIDC workload identity instead of long-lived cloud access keys, for AWS, Azure, and Google Cloud.
3. Vault-managed database credentials instead of a shared application user.

The Azure and Google Cloud trust configurations are in
terraform-dynamic-credentials,
verified against both HCP Terraform and a self-hosted Terraform Enterprise.
AWS dynamic credentials worked in my lab too, but an account admin created
the IAM side, and I don't have admin access to an AWS account to show every
step here. The AWS module in the repo is written from the HashiCorp docs and
passes terraform validate, but I haven't applied it myself. Code excerpts
elsewhere in the post are trimmed, with environment-specific names generalized.

TL;DR

• SSH: Vault's SSH secrets engine signs a certificate per job, good for thirty minutes. Hosts trust the CA's public key, baked into the image. There is no private key to distribute because there is no long-lived private key that grants access.
• Cloud: HCP Terraform and Terraform Enterprise sign a JWT for every plan and apply. AWS, Azure, and Google Cloud trade it for credentials that expire with the run. The workspace stores identifiers, not secrets.
• Databases: a Vault static role owns the application user's password and rotates it on a schedule. A dynamic role goes further and creates a new user per lease. Both work, and they fail in different ways.
• The gotchas are almost never in the happy path. They're in the second phase, the second week, or the second time you need to change something.

Problem 1: The SSH Key Pair Everyone Has a Copy Of

My lab's CI pipeline needed to configure a lab server over SSH. The original
setup was the common one: a private key stored in Vault's key/value engine, read
by the CI job, written to disk, and used directly.

secrets:
  SSH_PRIVATE_KEY:
    vault:
      engine:
        name: kv-v2
        path: secrets
      path: lab/lab-key
      field: private_key
    file: true
    token: $VAULT_ID_TOKEN

Storing the key in Vault beats storing it in the repo, but it doesn't change
what the key is. It's one private key, valid indefinitely, copied onto every
runner that executes the job, and rotating it means touching every host that
trusts it.

The Fix: Sign a Certificate per Job

OpenSSH has supported certificates for years. Instead of putting a user's public
key in authorized_keys on every host, you put one CA public key in
sshd_config, and any user certificate signed by that CA is accepted until it
expires. Vault's SSH secrets engine is that CA.

The Vault side is a mount, a CA, and a role that says what a certificate may
contain:

resource "vault_mount" "aap_ssh" {
  path = "aap-ssh"
  type = "ssh"
}

# generate_signing_key keeps the CA private key inside Vault. It is never
# returned by the API, so it can't land in Terraform state.
resource "vault_ssh_secret_backend_ca" "aap_ssh" {
  backend              = vault_mount.aap_ssh.path
  generate_signing_key = true
  key_type             = "ed25519"
}

resource "vault_ssh_secret_backend_role" "aap_role" {
  name    = "aap-role"
  backend = vault_mount.aap_ssh.path

  key_type                = "ca"
  allow_user_certificates = true

  allowed_users = "ansible"
  default_user  = "ansible"

  allowed_extensions = "permit-pty,permit-port-forwarding"
  default_extensions = {
    "permit-pty" = ""
  }

  ttl                 = "1800" # 30m
  max_ttl             = "7200" # 2h
  not_before_duration = "30"
}

The host side lives in the image pipeline. CI reads the CA's public key from
Vault, and Packer drops it into the image along with one line of sshd_config:

CA_DEST=/etc/ssh/vault-ssh-ca.pub
cp /tmp/vault-ssh-ca.pub "$CA_DEST"
chmod 0644 "$CA_DEST"

if grep -q '^TrustedUserCAKeys' /etc/ssh/sshd_config; then
  sed -i "s|^TrustedUserCAKeys .*|TrustedUserCAKeys $CA_DEST|" /etc/ssh/sshd_config
else
  echo "TrustedUserCAKeys $CA_DEST" >> /etc/ssh/sshd_config
fi

On the client side, the lab server is now configured by Ansible Automation
Platform rather than by the CI job. AAP has a HashiCorp Vault Signed SSH
credential type, and linking it to a Machine credential makes AAP request a
freshly signed certificate every time a job runs:

- name: Ensure signed-SSH input source on Machine credential
  ansible.controller.credential_input_source:
    input_field_name: ssh_public_key_data
    target_credential: "{{ machine_credential_name }}"
    source_credential: "{{ signed_ssh_lookup_name }}"
    metadata:
      secret_path: aap-ssh
      role: aap-role
      auth_path: approle
      valid_principals: ansible
      public_key: "{{ ssh_public_key }}"
    state: present

The end state: the lab's instances launch with no EC2 key pair at all, port 22
only accepts connections from AAP's egress range, and the static key is gone
from the pipeline. The key pair AAP signs still exists, but on
its own it opens nothing. A certificate is what gets you in, and each one is
dead within two hours at most.

Gotcha: The CA Is Not a Normal Resource

I originally built the CA by hand, as RSA 4096, and brought it under Terraform
later. That's where the first surprise was. Every argument on
vault_ssh_secret_backend_ca forces replacement, because the provider
implements create, read, and delete but no update. And the provider's read only
returns public_key and backend, so after an import the other arguments are
null in state. A configuration that matched the live CA exactly still planned a
replacement.

There was no way to adopt the existing CA without regenerating it. So instead of
importing it as RSA and rotating to ed25519 later, which would have regenerated
it twice, I changed the key type in the same apply and regenerated it once.

Any change to the CA resource produces a new CA, and every host trusting the old
public key stops accepting certificates. In a setup where the key is baked into
images, rotation isn't a Vault operation. It's
a Vault change followed by rebuilding every image. generate_signing_key = true
makes this sharper: the private key never leaves Vault, which is exactly why it
isn't in state, and also why losing Vault means a new CA and a full image
rebuild. I think that's the right trade, but it should be a decision, not a
discovery.

Gotcha: Write the TTLs in Seconds

The first version of the role said ttl = "30m". It applied cleanly, and then
every plan after it showed a change. The API returns TTLs as integers in string
form, the provider doesn't normalize them, and "30m" is not equal to "1800".
Writing seconds made the plan idempotent.

Gotcha: The Public Key Has a Space in It

The first image pipeline passed the CA public key to Packer's shell provisioner
as an environment variable, with this execute_command:

execute_command  = "sudo sh -c '{{ .Vars }} {{ .Path }}'"
environment_vars = ["VAULT_SSH_CA_PUBKEY=${var.vault_ssh_ca_pubkey}"]

An SSH public key is <type> <key>, with a space in the middle, and {{ .Vars }}
expands inside the single-quoted sh -c string. The quoting broke.

The fix was to never let the key pass through a shell at all. GitLab CI can
write a Vault secret to a temporary file instead of an environment variable:

VAULT_SSH_CA_PUBKEY:
  vault:
    engine:
      name: generic
      path: aap-ssh
    path: config/ca
    field: public_key
  file: true

Packer then uploads that file with a file provisioner, and the script reads it
from disk. The key never goes through an environment variable, a command-line
argument, or a shell expansion.

Problem 2: Cloud Keys in Workspace Variables

The classic Terraform setup for AWS is an IAM user with an access key, pasted
into the workspace as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. It works,
it never expires, and it's usually more privileged than anything else in the
account.

My lab had already moved past that, or so I thought. The workspace
authenticated to Vault with workload identity, then read AWS credentials from
Vault's AWS secrets engine, which created a fresh IAM user for each run:

data "vault_aws_access_credentials" "creds" {
  backend = "aws"
  role    = "packer"
}

On paper, those are dynamic credentials. In practice, the first AWS call failed
about 23 seconds after the credentials were minted, with
403 InvalidClientTokenId from STS. Adding skip_credentials_validation moved
the same 403 to iam:ListRoles. Adding skip_requesting_account_id and a
30-second time_sleep moved it again, this time to a 401 AuthFailure from the
first EC2 call, 56 seconds after minting. The error kept moving later instead of
going away. A brand new IAM user takes a while to be recognized everywhere in
AWS, and there was no sleep long enough to fix that reliably.

The HashiCorp docs say not to do exactly this: "data sources that
use secrets engines to generate dynamic secrets must not be used with Vault
dynamic credentials." The documented route through Vault is Vault-backed dynamic
credentials, which even has a TFC_VAULT_BACKED_AWS_SLEEP_SECONDS setting "to
mitigate eventual consistency issues in AWS when using the iam_user auth
type."

I went the other way and dropped Vault out of the AWS path entirely. With the
platform's native dynamic credentials, the same apply created a key pair, a
security group, an EC2 instance, and an Elastic IP with no delay at all.

The Fix: Let the Cloud Trust the Platform

HCP Terraform and Terraform Enterprise act as an OIDC identity provider. For
every plan and every apply, the platform signs a JWT describing the run. The
claim that matters is sub:

organization:my-org:project:my-project:workspace:my-workspace:run_phase:apply

You configure the cloud to trust tokens from the platform's issuer URL, and to
accept only tokens whose claims match your workspace. At run time, the provider
trades the token for short-lived cloud credentials. The workspace holds a role
ARN or a client ID, which are identifiers rather than secrets.

The docs are blunt about what happens if you skip the matching: "If you don't
match against at least the organization name, any organization or workspace on
HCP Terraform will be able to access your cloud resources!"

Something still needs admin rights to create the trust in the first place. The
difference is that it happens once, from a person's own login, rather than being
stored in every workspace.

AWS

The trust side is an IAM OIDC provider for the platform's hostname and a role
whose trust policy checks the audience and the subject:

resource "aws_iam_role" "this" {
  name = var.name

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = local.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${var.hostname}:aud" = "aws.workload.identity"
        }
        StringLike = {
          "${var.hostname}:sub" = "organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:run_phase:*"
        }
      }
    }]
  })
}

Note StringLike on the subject. The * after run_phase: is only a wildcard
under StringLike, and the docs call this out explicitly for when plan and
apply share a role. The workspace then needs two environment variables,
TFC_AWS_PROVIDER_AUTH=true and TFC_AWS_RUN_ROLE_ARN, and the provider block
should contain nothing but region.

In my lab, I specified the role and an account admin created it, so I can't
publish the live trust policy as tested. The module in the repo is written from
the docs and passes terraform validate, but I haven't applied it myself.
Azure and Google Cloud I built and ran end to end.

Azure

Azure's version is an app registration, its service principal, a role
assignment, and federated identity credentials that tell Entra ID which tokens
to accept:

resource "azuread_application_federated_identity_credential" "this" {
  for_each = toset(["plan", "apply"])

  application_id = azuread_application.this.id
  audiences      = ["api://AzureADTokenExchange"]
  display_name   = "${var.name}-${each.key}"
  issuer         = "https://${var.hostname}"
  subject        = "organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:run_phase:${each.key}"
}

The workspace gets TFC_AZURE_PROVIDER_AUTH=true, TFC_AZURE_RUN_CLIENT_ID,
ARM_SUBSCRIPTION_ID, and ARM_TENANT_ID.

The gotcha is the for_each. A federated credential's subject is an exact
string match, no wildcards, and the token's subject ends in the run phase. So
every workspace needs two credentials. I deleted the apply one to see what
happens: the plan ran fine, because data sources authenticate during plan, and
then the apply failed:

AADSTS700213: No matching federated identity record found for presented
assertion subject 'organization:my-org:project:dynamic-credentials-demo:workspace:dynamic-credentials-demo:run_phase:apply'.
Check your federated identity credential Subject, Audience and Issuer against
the presented assertion.

That's a nasty failure mode, because a green plan is what gets approved. Combine it with Entra ID's limit of 20 federated credentials per app
registration, and one app registration can serve at most ten workspaces.

Microsoft has a preview feature, flexible federated identity credentials, that
allows wildcard matching on the subject. I tested it both ways. On HCP
Terraform, a single credential matching
claims['sub'] matches '...:run_phase:*' covered both plan and apply. On
Terraform Enterprise, Entra ID refused to create it:

Expression is not supported for applications in this cloud 'Public' using
issuer 'https://<tfe-hostname>'.

Microsoft's docs list only https://app.terraform.io and its EU equivalent as
supported Terraform issuers. If you run Terraform Enterprise, two credentials
per workspace is the design.

Google Cloud

Google Cloud uses a workload identity pool, an OIDC provider in that pool, and a
service account the workspace impersonates:

resource "google_iam_workload_identity_pool_provider" "this" {
  project                            = var.project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.this.workload_identity_pool_id
  workload_identity_pool_provider_id = var.pool_provider_id

  attribute_mapping = {
    "google.subject"                        = "assertion.sub"
    "attribute.terraform_organization_name" = "assertion.terraform_organization_name"
    "attribute.terraform_project_name"      = "assertion.terraform_project_name"
    "attribute.terraform_workspace_name"    = "assertion.terraform_workspace_name"
    "attribute.terraform_run_phase"         = "assertion.terraform_run_phase"
  }

  attribute_condition = "assertion.sub.startsWith(\"organization:${var.organization}:project:${var.project}:workspace:${var.workspace}:\")"

  oidc {
    issuer_uri = "https://${var.hostname}"
  }
}

That startsWith condition covers both run phases, so Google Cloud doesn't have
Azure's two-credential problem. It carries more weight than it looks like it
does, though. The service account binding grants roles/iam.workloadIdentityUser
to every identity in the pool:

member = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.this.name}/*"

So the condition is the only thing checking that the token came from your
workspace and not from someone else's. When I changed it to a different
organization, the plan failed immediately:

oauth2/google: status code 400: {"error":"unauthorized_client",
"error_description":"The given credential is rejected by the attribute condition."}

The workspace gets TFC_GCP_PROVIDER_AUTH=true,
TFC_GCP_WORKLOAD_PROVIDER_NAME, and TFC_GCP_RUN_SERVICE_ACCOUNT_EMAIL.

If You Run Terraform Enterprise

Everything above works the same on Terraform Enterprise, with the issuer set to
your own hostname instead of app.terraform.io. Two things change.

First, the cloud has to be able to reach you. To verify a token, the cloud
fetches /.well-known/openid-configuration and /.well-known/jwks from your
Terraform Enterprise hostname. An instance that's only reachable on a private
network can't do native dynamic credentials to a public cloud. Vault-backed
dynamic credentials exist partly for this case.

Second, the certificate. HashiCorp's Azure docs say "Custom and self-signed
certificates are not supported due to restrictions in Azure," and the Google
Cloud docs say dynamic credentials "do not work if your Terraform Enterprise
instance uses a custom or self-signed certificate." My instance uses a publicly
trusted certificate, so neither applied, but an instance behind an internal CA
would have hit both.

Problem 3: The Shared Application User

The database version of a static credential is an application user whose
password was set once, lives in a config file or a Kubernetes secret, and is
shared by every instance of the application. Vault's database secrets engine
offers two ways out, and my lab has both configured against the same Postgres
database for Terraform Enterprise.

The Fix, Part One: A Static Role

A static role is a one-to-one mapping between a Vault role and an existing
database user. Vault stores the user's password and rotates it on a schedule.
Terraform Enterprise itself runs as tfe_app, whose password Vault rotates
weekly:

resource "vault_database_secret_backend_static_role" "tfe_static" {
  backend         = vault_mount.tfe_psql.path
  name            = "tfe-static"
  db_name         = "tfe-psql"
  username        = "tfe_app"
  rotation_period = 604800 # 7 days
}

Two things to know before you create one. Vault rotates the password the
moment the role is created, so the password you created the user with stops
working immediately. And the Vault docs are explicit that the user Vault itself
connects as should never be a static role: "Do not manage the same root database
credentials that you provide to Vault in config/ with static roles." In my lab,
Vault connects as tfe, and the application runs as tfe_app.

Terraform Enterprise runs on Kubernetes, so the Vault Secrets Operator (VSO)
delivers the password as a Kubernetes secret and restarts the deployment when it
changes:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: tfe-database
spec:
  vaultAuthRef: tfe-auth
  mount: tfe-psql
  path: static-creds/tfe-static
  allowStaticCreds: true
  refreshAfter: 24h
  destination:
    name: tfe-database
    create: true
  rolloutRestartTargets:
    - kind: Deployment
      name: terraform-enterprise

Gotcha: The First Rotation Is the Real Test

The first version of that manifest didn't have allowStaticCreds. It deployed
fine and Terraform Enterprise ran fine, for a week. Then Vault rotated the
password on schedule, the Kubernetes secret still held the old one, and Terraform
Enterprise started logging:

failed SASL auth: FATAL: password authentication failed for user "tfe_app" (SQLSTATE 28P01)

Without allowStaticCreds, VSO 1.4.0 treated the static credential response as
a non-renewable lease, concluded there was nothing left to refresh, and never
read it again. Its log said as much: "Vault secret does not support periodic
renewal/refresh via reconciliation." The VSO API reference describes the flag as
something that "should be set when syncing credentials that are periodically
rotated by the Vault server, rather than created upon request."

A rotation setup that hasn't survived its first rotation isn't tested yet. If
your rotation period is a week, you find out a week later, which is usually
long after the change that caused it has been merged and forgotten.

The Fix, Part Two: A Dynamic Role

A dynamic role goes further. Instead of rotating one user's password, Vault
creates a brand new database user for each lease and drops it when the lease
ends. Every instance of the application gets its own username, which the Vault
docs point out makes auditing easier, because you can trace a query to a
specific instance.

My lab has a dynamic role on the same database:

resource "vault_database_secret_backend_role" "tfe_psql" {
  backend = vault_mount.tfe_psql.path
  name    = "tfe-psql-role"
  db_name = "tfe-psql"

  creation_statements = [
    "CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';\nGRANT ALL PRIVILEGES ON DATABASE tfe TO \"{{name}}\";"
  ]

  default_ttl = 3600  # 1 hour
  max_ttl     = 86400 # 24 hours
}

Nothing in the lab actually consumes it. It has only ever been read by hand. So
before writing about it, I reproduced it against a throwaway Vault and Postgres
17, with the same creation statement and a table owned by a separate user, and
tried to use it the way an application would.

Gotcha: GRANT ALL on a Database Doesn't Reach the Tables

The generated user could log in, and that was all:

ERROR:  permission denied for table widgets

In Postgres, the privileges on a database are about connecting to it and
creating things in it, not about the tables inside it. GRANT ALL PRIVILEGES ON
DATABASE reads like it grants everything, and it grants none of the access an
application needs. The example role in Vault's own Postgres docs uses
GRANT SELECT ON ALL TABLES IN SCHEMA public instead.

Gotcha: A User That Owns Something Can't Be Dropped

The second problem is worse, because it shows up when the lease ends. I granted
the dynamic user permission to create tables, had it create one, and revoked the
lease. Vault queued the revocation, and the user was still there. The Vault log
explained why:

[ERROR] expiration: failed to revoke lease: lease_id=db/creds/dyn/h3AscHe4Ki5S9HC096qXdMsL
error="failed to revoke entry: resp: (*logical.Response)(nil) err: ERROR: role
\"v-token-dyn-hOoD3GJA9yCYjC7PX4uC-1789066250\" cannot be dropped because some
objects depend on it (SQLSTATE 2BP01)" attempts=1 next_attempt=21.012025154s

Vault retries, and it keeps failing. Any application that runs its own schema
migrations will create objects owned by whichever user ran them, which is a
user that's supposed to disappear.

The fix that worked was to give ownership to a role that never logs in, and
make every dynamic user act as that role:

CREATE ROLE app_schema NOLOGIN;
ALTER TABLE widgets OWNER TO app_schema;
GRANT CREATE ON SCHEMA public TO app_schema;

Then the creation statement makes each generated user a member of app_schema
and sets it as their default role:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}'
  VALID UNTIL '{{expiration}}' IN ROLE app_schema;
ALTER ROLE "{{name}}" SET ROLE app_schema;

With that in place, the generated user could read the table, a table it created
was owned by app_schema, and revoking the lease dropped the user cleanly.

Static or Dynamic?

They fail differently, and that's the useful way to choose. A static role keeps
one username and changes its password, so the risk is a consumer that doesn't
pick up the new one. A dynamic role changes the username every lease, so the
risk is anything tied to the old username: permissions granted to it, and
objects it owns.

Terraform Enterprise in my lab uses the static role. tfe_app owns the tfe
database and its public schema, so the ownership problem never comes up, and
the only thing that has to be right is the rotation. For an application that
doesn't own its schema, or where each instance should have its own identity,
a dynamic role is the stronger option, as long as its creation statement grants
access to the tables and not just the database.

Conclusion

Every substitution here follows the same pattern: remove the stored secret,
trust an identity system instead, and make whatever it issues expire.

• SSH certificates instead of key pairs - Vault signs a certificate per job with a thirty-minute TTL, and hosts trust the CA through TrustedUserCAKeys. Treat the CA resource as immutable: changing it means rebuilding every image that trusts it.
• Workload identity instead of access keys - the platform signs a token per run phase, and the cloud trades it for credentials that expire with the run. On AWS, use StringLike for the wildcard. On Azure, create a federated credential for plan and apply. On Google Cloud, the attribute condition is the only check that the token came from your workspace.
• Terraform Enterprise changes two things - the cloud has to reach its OIDC endpoints, and Azure and Google Cloud won't accept a custom or self-signed certificate.
• Vault-owned database credentials instead of a shared user - static roles rotate one user's password, and nothing proves the delivery works until the first rotation. Dynamic roles create a user per lease, and need a creation statement that grants table access and keeps ownership off the ephemeral user.
• Test past the first run - none of these failures showed up there. The Azure apply failed after a clean plan, the VSO outage waited a week for the first rotation, and the stuck revocation only appeared when the lease ended.

The Azure and Google Cloud configurations, with a workload you can run against
your own HCP Terraform or Terraform Enterprise workspace, are in
terraform-dynamic-credentials.