Our severity definitions were written down and everybody had read them. Severity one meant complete loss of a critical service. Severity two meant significant degradation with no workaround. Severity three meant everything else. In practice almost everything we raised was a severity three, and the reason was not that our incidents were minor.
Raising a severity one woke up two managers, opened a bridge call, required customer communications to be drafted within the hour and triggered a formal review with a named owner. All of that landed on the person who chose the label. So an engineer at two in the morning, holding a half-diagnosed problem and hoping a restart would settle it, would classify it low and tell himself he could raise it later. Later never arrived, because by the time the picture was clear the incident was usually over. Severity only went up when somebody senior noticed independently and asked why nobody had told them.
The damage was not in the individual choices, which were all rational. It was that a year of incident records said we had almost no serious incidents. When we asked for investment in a platform that had been quietly falling over every fortnight, our own data argued against us.
We changed three things. Anyone can raise a severity two, but the incident manager on duty, not the person who found it, confirms or downgrades within fifteen minutes. Downgrading is normal and requires no justification, which removes the fear of overreacting. And the formal review is now triggered by measured impact and duration rather than by the label somebody picked while tired.
The count of severity one and two incidents roughly quadrupled in the following quarter. Nothing had got worse. We warned leadership before it happened, and that warning was the most important part of the whole exercise, because otherwise the honest numbers would have looked like a decline in service and someone would have asked us to make them smaller again.
A classification scheme that punishes people for using it honestly does not measure your incidents. It measures how much trouble they expect if they tell you the truth.
– Serguey Shinder
Top comments (0)