Maintaining the Automation Became a Job of Its Own

We started automating operational tasks about three years ago, for the usual sound reasons. Today we run a little over sixty automated jobs: provisioning, certificate renewal, user onboarding, environment refreshes, report generation, housekeeping. Each one saved real time and none of them was a mistake.

What we did not plan for is that we now own sixty pieces of software.

They have dependencies that go out of support. They authenticate against systems whose interfaces change, and half the failures we see are not logic errors but a credential rotation or an API version that moved underneath a script written two years ago. They encode assumptions about the environment that stop being true. They need testing, and the ones that touch production most usefully are the hardest to test safely. Roughly one working day a week across the team goes to keeping the automation running, and that day is invisible in every plan we publish, because it is filed under operations rather than development.

The failure mode this creates is specific. Automation degrades quietly. A job that runs weekly and fails silently is discovered a month later by the person who needed its output, and by then the manual fallback has also rotted, because nobody has performed the task by hand since 2024. We had exactly this with an access review export: it had not produced a file for seven weeks and the first sign was an auditor asking for it.

We have started treating the automation estate as an estate. Every job has a named owner and a service it belongs to, which means unowned ones can be found and either adopted or deleted. Every job reports success and failure to the same monitoring we use for services, so silence is an alert rather than an absence. We review the inventory quarterly and retire the jobs whose original purpose has gone, which last time was seven of them. And we have become more willing to say that a task performed three times a year is not worth automating, because the maintenance will cost more than the task.

Automation moves work rather than removing it. That is still a good trade, but only if you budget for where the work went.

– Serguey Shinder