For six days in July our release notes, our changelog and our status page all said that version 2.10 of the booking service was in production, including a fix for a double booking bug a large customer had been chasing us about. Production was running 2.9.0. We found out when support told that customer the fix was live and the customer sent back a screenshot of the bug.
Releases reach staging by exact version, straight from the pipeline that built them. Production is different: a promotion job, run on a schedule after a sign off, deploys the latest release. It worked out latest by listing the repository's tags and taking the last one. The listing was sorted as text. As text, v2.10.0 sorts after v2.1.0 and before v2.2.0, so the last tag in the list was still v2.9.0. The job deployed 2.9.0 over 2.9.0 every night, reported success every night, and a helm upgrade to the same version changes nothing, so nothing looked odd anywhere.
Two digit minor versions had simply never happened before. We had released 2.0 through 2.9 over eighteen months, and every sort in the world agrees on those.
Sorting the tags as versions was a one line change. It was also the least important one, because the real problem was the word latest. Promotion now takes an explicit version that was deployed to staging and signed off, recorded as an artifact of that sign off, and it refuses to run without one. After every deploy a check calls the service's version endpoint and fails the job if the running version is not the one it meant to ship. An alert fires if production has not matched the most recently signed off version within an hour of promotion. And the versioning library we do use has tests for the orderings that bite: 2.10 after 2.9, 1.0.0 after 1.0.0-rc.2, and 10.0 after 9.12.
The status page and changelog are now generated from what the version endpoint reports, rather than from what the pipeline intended.
Latest is not a version. It is a query, every query has a sort order, and ours had been chosen by default, by a function that thought it was sorting words.
– Sergey Shinder
Top comments (0)