We had Terraform for everything. We were proud of it. Then production went down for two hours and the root cause was a security group rule that existed in AWS but not in any of our code.
Someone had SSH-troubleshooting access one night, added an ingress rule by hand to unblock a demo, and never told anyone. It worked. The demo went fine. Six months later a routine terraform apply reconciled state, saw a rule it didn't own, and removed it. The service that quietly depended on that rule fell over, and we spent two hours staring at Terraform diffs that all looked clean because the drift had already been erased.
That incident changed how I think about IaC. Writing the code is the easy 20%. The hard 80% is making sure the code is the only way anything changes.
We did three things. First, we turned on drift detection as a scheduled job — a nightly terraform plan that posts to Slack if the plan is non-empty. If prod drifts from code, we know the next morning, not six months later. Second, we pulled console write access for humans in production accounts. You get read-only by default; changes go through a PR. Emergency break-glass access exists, but using it fires an alert and creates a ticket automatically. Third, we started treating a non-empty plan on main as a failure, not a Tuesday.
The pushback was real. "I just need to fix one thing quickly" is a powerful argument at 2am. But every manual fix is a landmine you're planting for your future self, and you never remember where you buried them.
Infrastructure as Code isn't about the tool. Terraform, Pulumi, CloudFormation, doesn't matter. It's a discipline: the repository is the source of truth, and reality is expected to match it. The moment you allow a second source of truth — a helpful engineer with console access — you don't have IaC anymore. You have a very detailed diagram of what production used to look like.
Detect drift or drift will detect you, usually during an outage.
– Sergey Shinder
Top comments (0)