Every night a small script on one of our servers collects price files from our suppliers, checks them and loads the new costs into our trading system, where they feed the prices our branches charge. One of my engineers wrote it in 2021. It was a good script, and it ran every night for more than three years.
In March our timber suppliers put their prices up by around six percent. Nine days later a branch manager asked why the cost of a common size of treated timber on his screen did not match the invoice in front of him. The price files had arrived every night. Nothing had loaded them since the second of March.
The script was fine. The problem was in how it had been scheduled. When you create a task in Windows Task Scheduler, the default setting is to run it only when the user is logged on. The engineer had created the task under his own administrator account while working on the server through a remote desktop connection, and he had never signed out. He closed the window instead, which leaves the session disconnected but still running, and as far as Windows was concerned he was logged on. For three years the task ran inside that forgotten session.
On the second of March the server restarted after its monthly updates. The session ended, and the task stopped running. It did not fail, because it never started. Task Scheduler showed it as ready, which is exactly how a healthy task looks, and our monitoring only alerted when a job reported an error.
Putting the fix in was quick. The task now runs whether anyone is logged on or not, under a managed service account rather than a person's. Then we searched every server for tasks set to run only when a user is logged on, and found eleven more. Three of them belonged to people who had moved to other teams.
The lasting change was to what we watch. Every scheduled job that matters now reports what it did, and we alert when that report has not arrived by a set time, as well as when it reports a failure.
A job that silently does not start looks the same as a job with nothing to do. Monitor for the result you expect, not just for errors.
– Serguey Shinder
Top comments (0)