DEV Community

Sergey Shinder
Sergey Shinder

Posted on

We told Terraform to ignore one field and it hid a firewall change

Somewhere around two years ago, a colleague added four lines to a security group resource so that our applies would stop fighting with a legacy provisioning tool that managed one ingress rule. The lifecycle block ignored changes to ingress. It was the right call that week and nobody ever came back to it.

What that means in practice is that Terraform stopped having an opinion about who could reach that group on any port at all. The whole block was excluded, not the one rule. For most of those two years it did not matter. Then somebody debugging a connectivity problem from a laptop added an ingress rule for port 5432 from 0.0.0.0/0, intending to remove it after lunch, and did not.

Every apply after that was clean. Plan showed no changes. Our drift dashboard was green, and green meant, as far as anyone reading it was concerned, that the account matched the code. It did not. It matched the code plus whatever anyone had done by hand to a field we had told the tool to stop looking at. The open rule was found four months later by a scheduled security scan, not by us.

The fix for the resource itself was straightforward. The exception rule became its own aws_security_group_rule resource, managed explicitly, and the lifecycle block came out entirely. Ignoring a specific attribute is sometimes defensible; ignoring a block that contains your firewall is not.

Then we went looking. Eleven ignore_changes blocks across the estate, four of them covering whole nested blocks, two referencing a tool that had been decommissioned. Every one is now either deleted or carries a comment with a ticket reference and a review date, checked by a lint rule that fails the plan if either is missing.

The thing I actually value most is smaller. A nightly job runs plan with refresh-only against every workspace and posts the diff, and separately we now query the cloud for security group rules and compare them against what the code declares, independent of Terraform's own view. That second check is the only one that could ever have caught this.

A clean plan is not a statement about your infrastructure. It is a statement about the part of your infrastructure you are still looking at.

– Sergey Shinder

Top comments (0)