Six services failed to start within about forty minutes of each other on a Wednesday morning, all with the same error about a missing configuration key. None of them had been deployed. All of them had been rebuilt overnight by scheduled dependency updates, and all of them had picked up version 3.4.1 of our internal HTTP client library.
3.4.1 removed a configuration key. The author had renamed it, handled the old name for one release, and then dropped the compatibility shim. He had written a clear paragraph about it in the commit body. Our release automation derives the version bump from conventional commit prefixes, and the prefix was fix, so it published a patch. Every consumer had a caret range on that dependency, so every consumer took it without anyone deciding to.
It failed at startup rather than at build, which is why the tests all passed and the pipelines were green right up to the point where a container came up and read its config.
The first change was to stop trusting a prefix. A job in the library's release pipeline runs an API diff between the artifact about to be published and the currently published one. Removed public symbols, changed signatures and removed configuration keys, which we now declare in a schema file rather than reading ad hoc, all mark the change as breaking. If the derived version does not match, the release fails and asks a human what they meant. The commit message is a hint, not the source of truth.
Second, before publishing to our real registry, the pipeline publishes to a staging registry and builds three representative consumer services against it, including running their startup smoke tests. That caught two would-be incidents in the following month, both configuration related, both invisible to unit tests.
Third, the consumers stopped floating. Exact versions, with the update bot opening a pull request per bump, which someone merges. It is slightly more work and it means the upgrade happens at a moment somebody chose.
Semantic versioning is a promise about compatibility. Ours was being made by whoever typed the commit prefix, at speed, on the way to lunch.
– Sergey Shinder
Top comments (0)