Our month end close touches four systems and takes two days, and for most of the past decade it has been held together by scripts written by one engineer. He retired in March, with generous notice, a proper handover period and the goodwill of everybody concerned.
In June the close finished eleven hours late, because of a script none of us knew existed.
The inventory we built afterwards found sixty-three automated jobs running somewhere in the estate. Twenty-two sat in the scheduler we manage and were documented. The remainder lived on individual servers, in personal task schedulers, inside database agents, and in one case as a mailbox rule that has been moving a file between two folders since 2017. Nine ran under a leaver's account that we had disabled years ago and hurriedly re-enabled when something unidentified stopped working.
The handover covered what he knew he did. No one can hand over a script written on a Tuesday afternoon four years earlier to solve a problem that has long since stopped feeling like a problem, because by then it is not a thing you do, it is simply how things are.
There is an easy operational conclusion and a harder organisational one. The easy conclusion is that automation needs an asset record exactly as a server does, with an owner, a purpose, a schedule and an account behind it. Ours now has one, reviewed quarterly, and anything unattributed is stopped in a controlled way instead of being left running out of nervousness.
The harder conclusion concerns how it all arrived. Every one of those scripts was written by somebody solving a genuine problem quickly, in a department that offered no supported way to do small automation and no time in which to do it. Forbidden, they would have carried on and stopped mentioning it. So we built a modest sanctioned route, with version control, a service account and a registration form that takes five minutes, because a rule people can follow in the time available beats a policy they will route around.
Of the sixty-three jobs, nine were achieving nothing whatsoever. It still took a fortnight to find anybody willing to be the person who switched them off.
– Serguey Shinder
Top comments (0)