For about a year my service desk had a bad week roughly once a quarter. Calls doubled on the Monday, the contact centre started the early shift short of agents, and by Thursday everything had calmed down again. Each review blamed something different and named nothing in particular.
It was a team leader in the contact centre who saw the pattern, because she kept a diary of the mornings she started short handed. Her bad mornings were thirteen weeks apart, almost to the day.
The cause was a project we had finished and celebrated. In the spring of 2024 we moved every account into a new directory over one weekend, and on the Monday everybody set a new password. Our policy expired passwords after ninety days. So two thousand three hundred passwords had been set on the same day, expired on the same day, and kept expiring together every quarter after that. At head office that meant reset calls. The contact centre had it worse. Agents changed the password at the desktop, but the softphone and the customer system each held the old one and kept retrying it, and within minutes the account locked. An agent who changed hers at five to eight was locked out by eight, and so was most of her row.
Each lockout was real and each reset was handled properly. What nobody saw was that one weekend had set a clock running on every account in the company.
We stopped expiring passwords on a schedule for accounts with a second factor, which is what most of the guidance we follow now recommends, and a password is changed when there is reason to think it is known. Self service reset works from the sign in screen for anyone enrolled. A lockout caused by an application retrying an old password raises an alert that names the application, and the softphone now signs in through single sign on, so there is no second copy to go stale.
The quarterly bad week has gone, and reset calls are down by about two thirds.
A project that touches every account on the same day can leave behind a timer nobody set on purpose. After a large migration I now ask what it has synchronised, as well as what it has changed.
– Serguey Shinder
Top comments (0)