In March our order service spent nine minutes failing every write with cannot execute INSERT in a read-only transaction. A pull request merged that morning had pointed it at the reporting replica. Two of us had approved the Terraform plan for that change.
The service runs on ECS, and its task definition is built in Terraform: a list of container definitions passed through jsonencode, environment variables included. One of those variables was the API key for our address lookup provider, read from a Terraform variable marked sensitive. The pull request was a tidy-up that moved shared settings into locals, and during the move the order service's DATABASE_URL was copied from the reporting worker's block a few lines further down.
Terraform carries sensitivity through expressions. Anything derived from a sensitive value is sensitive too, and jsonencode returns one string, so a single secret anywhere inside it marked the entire string. The plan for the change said container_definitions = (sensitive value), with forces replacement next to it. That is also exactly what it says for every image update, which is most of the plans anyone ever sees for that service. We approved the line we always approve. The diff we needed was behind it, masked by the same feature that keeps the key out of our CI logs.
We reverted the commit and applied, which is where the nine minutes went. Then we took the secret out of the values Terraform handles. The task definition now uses the secrets field with the ARN of an entry in Secrets Manager, and ECS fetches the value when the task starts. An ARN is not sensitive, so the plan shows the container definitions in full again, as a structured diff of the JSON, and a changed database host is a red line like any other. A policy check on the plan JSON fails the pipeline if any attribute of a production resource is masked entirely. And the order service checks at startup that its database is a primary, by asking pg_is_in_recovery, and refuses to report ready if it is not, so the next wrong host stops a rollout instead of nine minutes of orders.
Sensitive marking is there to hide a value. It hid every value next to it too, and a review can only approve what it is shown.
– Sergey Shinder
Top comments (0)