Version 4.7.0 of our on premise product went out on a Thursday afternoon. On Friday morning support had twenty tickets from customers whose download link returned a 404, and two enterprise customers had opened a formal query asking which version they were supposed to be installing, because they had received two release emails naming two different numbers.
Our release workflow is seven steps: allocate the version from the last tag, build four platform artefacts, sign them, publish them to the customer portal one at a time, register the release with the licence server, email three hundred customers, update the documentation site. The portal returned a gateway error on the third upload. The job went red with two of the four artefacts already listed and downloadable.
There is one button, and it starts at step one. The engineer pressed it, which is what I would have done. The version step derives the next number from the newest tag, found 4.7.0 already there, and produced 4.7.1. Everything after that was internally consistent and wrong: the portal now offered a 4.7.0 with two platforms and a 4.7.1 with four, the licence server had two entries, the changelog described 4.7.0, and three hundred people got a second email. The artefacts themselves were fine. Nothing we shipped was defective. What we had was a release nobody could name.
Releases now begin by writing a row: a release id, the version allocated once and never derived again, and a status per step. Re running the workflow resumes at the first step that is not complete and is a no operation for the ones that are, so the button is safe to press and the version cannot move underneath it. The portal publish uploads to a staging prefix and then flips a manifest in one call, so a release is either listed with everything or not listed. Notifications are last, keyed by release id, and will not send twice.
A pipeline is a script and a release is a state. Ours existed nowhere except in the colour of the final job, which is why a failure halfway through left us with two half releases and no way to say what the truth was. Automation that reaches outside your own systems needs to know the name of the thing it is doing before it starts doing it.
– Sergey Shinder
Top comments (0)