Our release was ninety-five percent automated. One pipeline, one button, artifacts built and promoted and deployed without anyone touching a server. The remaining five percent was a step that only worked when a specific colleague ran it, and in August he went to Portugal for two weeks and we could not ship.
The step was cache invalidation on a CDN, plus a config push to a legacy admin panel that had no API. He did it from a personal script on his laptop, using credentials issued to him personally, against a system whose access list had three names on it and no group. None of this was hidden. It was in the runbook, as a line reading "ask Marco to run the invalidation." Nobody had read that line as a risk, because for four years it had worked instantly every time.
What made it worse than a simple bus-factor problem was that we didn't know the shape of the step until we had to reproduce it. His script did four things, only two of which were documented, and one of those two had been changed since the doc was written. We reconstructed it from shell history over a video call while he sat in a cafe, which is not a release process, and then we deployed anyway because the alternative was a two week freeze.
Automating it took a day and a half. The credentials became a service account with a scoped token in the secret store rather than a person's login. The script moved into the repository, with the pipeline invoking it, so it gets reviewed and versioned like everything else. The legacy admin push turned out to be a form POST we could do with a request, once we stopped treating "no API" as final. Total effort: less than the outage it prevented.
The lesson isn't "automate everything," because some manual steps are deliberate and fine. It's that a manual step tied to a named individual is a different category of thing. Automation is not really about speed here. It's about making the process legible to people who weren't there when it was built.
If your release depends on a person rather than a role, you don't have a release process. You have a habit.
– Sergey Shinder
Top comments (0)