The job had been deleting temporary artefacts every night for about two years without anyone noticing it existed. That is normally a compliment. Then somebody renamed a storage prefix during a tidy-up, the filter that had been selecting temporary files stopped matching anything temporary, and the job deleted a little over four hundred thousand objects that were not temporary at all.
The script did not fail. It did not warn. From its own point of view it had an excellent night: it found more candidates than usual and processed them all successfully, and it reported success, because success was defined as completing without an exception. Our monitoring agreed. The alert we had was for the job not running, which is the alert almost everyone writes, and it is the wrong one for anything destructive.
The lesson I took from it is about what we choose to encode. We had encoded the action carefully. Retries, batching, rate limits against the storage API, structured logging, all reasonably professional. We had encoded nothing about expectation. Nobody had ever written down that this job normally removes somewhere between two and five thousand objects, so nothing in the system could recognise four hundred thousand as absurd. The knowledge existed in my head and in nobody's code.
Every destructive automation we run now carries three things it did not carry before. A bound, so the job refuses and alerts if the work it is about to do is wildly outside the normal range, rather than proceeding confidently. A dry run that is not optional in the deployment pipeline, whose output is compared against the previous run. And a delay between marking and deleting, so there is a window in which a human mistake is still recoverable, which in our case would have turned a very bad week into a shrug.
The uncomfortable truth is that this automation was not badly written. It was well written and narrowly scoped, and it did precisely what its author intended under the conditions its author imagined. Automation does not go wrong by disobeying you. It goes wrong by obeying you in a situation you never described to it, at a speed no human error could match.
– Serguey Shinder
Top comments (0)