Internal audit took a sample of twenty production changes from a three month period and asked us to produce the approval record for each. Fourteen were straightforward. Six had no record at all, and the six had something in common: no person had performed them. A pipeline had.
They were not unauthorised in any meaningful sense. A firewall rule updated from a definition in version control, reviewed and merged by two engineers. A scaling policy adjusted by a scheduled job. A certificate renewed and deployed automatically, which is precisely the behaviour we spent a year building after a certificate expiry took a service down. Every one of them was safer than the manual equivalent it replaced. None of them appeared anywhere the auditor was entitled to look, because our change management process was written around a form that a human fills in.
The obvious response is to make the pipeline raise a change ticket. We tried that for a month and it was theatre. Several hundred records a week, approved by nobody in particular because they were already merged, in a queue that no reviewer could meaningfully read. It satisfied the letter of the control while destroying whatever value the control had.
What worked was moving the evidence rather than the approval. The approval already existed: it was the review on the merge request, the automated policy checks that ran against it, and the identity of the person who accepted it. So the pipeline now writes the change record itself, at the moment it acts, carrying the commit, the reviewers, the checks that passed, what it changed and where. Those changes are a defined pre-approved class, and the thing we approved once, formally, is the method rather than each instance.
The part that took longer was the exceptions. When an engineer runs a job by hand with elevated rights, that produces a different record, deliberately noisier, and those are reviewed weekly. There were more of them than anyone expected.
Automation does not remove the need for a control. It moves the control earlier, into the review and the definition, and it leaves the organisation with a governance process still watching the place where the work used to happen.
– Serguey Shinder
Top comments (0)