Our pricing lookup service stopped responding at around nine on a Saturday evening. The storefront fell back to cached prices, which were up to a week old, and stayed that way until somebody on the team noticed on Monday at twenty to nine. Fourteen trading hours of slightly wrong prices, and not one alert.
The outage itself was dull: the process had run out of memory and the container had not restarted because nothing was watching it. What the write up turned into was an inventory of everything this service was missing, eight months after it went live.
It was not in the paging rotation. It had no dashboard. Its logs were not shipped anywhere, so for the whole incident window we had nothing to read. Its base image was the one it had been built from in March, because our platform rolls base images by iterating a list in a repository and nobody had added it to the list, which also meant it had missed every security update since. Its small database had no backup job. It did have a pipeline, it did deploy cleanly, and by the only test anyone had applied, it was a fully working service.
None of that was carelessness. Every item on that list is a separate manual registration step, in a separate repository, described in a wiki page with fourteen numbered instructions. The team that built the service did nine of them. I would have done about nine.
Every repository now carries a service file: owner, tier, rotation, objectives, dependencies, data classification, backup requirement. The build refuses to publish an artifact from a repository that does not have a valid one, which is blunt and took a fortnight of grumbling to get through. The platform's rolling jobs iterate the registry built from those files instead of the hand written list, so being deployable and being maintained became the same fact rather than two facts that happen to agree. And a weekly job compares everything running in production against that registry and reports what it cannot match. The first run found two more.
We had automated the fifteen minutes it takes to ship a service and left the eight months of running it to a numbered list on a wiki page that people read once.
– Sergey Shinder
Top comments (0)