A new engineer asked, in her first week, why the RDS password was visible in the state file. I said it wasn't, opened the file to show her, and searched for the parameter name. It was there. Plain text, in a JSON blob, in an S3 bucket that fourteen people could read.
This is documented behaviour, not a bug. Terraform stores the full resolved attributes of every resource it manages, and a password argument is an attribute like any other. It doesn't matter that you passed it in from a secret store, or marked the variable sensitive, or never printed it in a plan. Sensitive marking hides it from output. It does not remove it from state. Ours had been sitting there since the cluster was created two years earlier, and every terraform apply any of us had ever run had downloaded a copy to a local temp file.
We had a second version of the same problem in a module that generated a service account key and passed it to a Kubernetes secret. Two resources, two copies of the key in state, plus a copy in whatever CI runner had last applied it.
The remediation was less exciting than the discovery. We rotated everything that had ever been in state, which took a day because some of it was consumed by services with no hot-reload. Then we changed the pattern: Terraform provisions the database and creates an empty secret, and the actual credential is generated by the secret manager itself and never passes through Terraform at all. Where that isn't possible, the resource is created with a throwaway value that is rotated out of band immediately after.
Everything else was hygiene. Bucket encryption on, versioning on with a lifecycle rule so old states expire, read access cut to a role rather than a user list, and access logging enabled so we would at least know who had looked. State locking through DynamoDB we already had, for a different reason.
The general point I'd make to anyone running Terraform: treat the state file with exactly the same care you'd give the credentials it manages, because that's what it is. It is not metadata. It is a copy of your infrastructure, secrets included.
– Sergey Shinder
Top comments (0)