The plan output was four hundred lines long. Buried around line two hundred and ten, between a tag change and a security group rule, was a resource marked for replacement. It was the managed database. Somebody approved the plan in about eleven seconds, because the change description said "update instance tags," and it did do that too.
We caught it because the apply took longer than a tag change should and someone looked. We got lucky. The immutable field that triggered the replacement was a subnet group attribute a colleague had changed in a shared module three weeks earlier, and it had been sitting harmlessly in the module until the next plan happened to pick it up. The person who wrote it never saw a plan for this environment. The person who approved the plan never saw the module change.
Reading a plan is a skill nobody teaches and everyone is assumed to have. Four hundred lines of diff, mostly noise from computed attributes that will resolve at apply time, and the one line that matters is formatted exactly like the ones that don't. Humans are bad at this. We are especially bad at it at 5pm on a change we believe is trivial.
So we stopped relying on reading. Plans now go through a policy check before they reach a human. Any destroy or replace of a resource in a protected set fails the pipeline outright and requires a separate, explicitly labelled change with a second approver. Stateful resources carry prevent_destroy lifecycle blocks, which is blunt and occasionally annoying and has now stopped this twice. The plan is saved as a file and the apply runs against that exact plan, so what got approved is what gets executed.
The other change was about summaries. The pipeline posts a short header on every plan: counts of add, change, destroy, and the names of anything being replaced, at the top, before the diff. If the destroy count is not zero, that number is the first thing you see.
Your automation should not require a tired person to spot one word in four hundred lines. Make the dangerous case loud, and make it a different kind of change.
– Sergey Shinder
Top comments (0)