The cold start we had never practised

A planned electrical shutdown gave us a clean stop. Everything was drained, everything went down in order, and the building went quiet. Bringing it back took three hours and fifty minutes, and about three hours of that was us discovering what our services need in order to start.

Every component restarts constantly in normal life. Pods get rescheduled, hosts get patched, processes get redeployed several times a week. What none of them had ever done was start when nothing else was already running.

The configuration service reads its own settings from the database. The database credentials come from the secrets manager, which authenticates against an identity service that reads its configuration from the configuration service. In steady state this is invisible, because there is always a warm copy of something holding the loop together. From cold it is a circle with no entry point, and we broke it by hand with a value someone found in an old support ticket.

Behind that were the ordinary ones. Services that crash-loop when a dependency is absent rather than starting degraded and retrying, so the ones that could have come up were restarting faster than the ones they needed. A cache that the search service treats as optional in the documentation and as mandatory in the code. A message consumer group that, faced with nine hours of backlog and every consumer starting at once, rebalanced repeatedly and made no progress for twenty minutes.

We fixed the circle properly: bootstrap configuration is baked into the image as a fallback, and the identity service no longer depends on anything it also serves. Everything else got the same rule, which is that a missing dependency at startup means wait and retry with backoff, never exit.

Then we made it something we do rather than something we hope about. Twice a year we cold start a full copy of production from nothing, with a timer running. The first attempt took just under three hours. The third took thirty-five minutes, and most of what we removed was not technology, it was people finding out what order things go in.

You have tested restarting your components. That is not the same as having tested starting your system.

– Sergey Shinder