Our orders, pricing and returns services live in one repository, and each has its own GitHub Actions workflow that deploys it to production when a change to its directory is merged. Last spring two deploys ran into each other on the cluster, so we gave all three workflows the same concurrency block, group: production with cancel-in-progress set to false. One deploy at a time, the others wait their turn. That was the idea.
On a Monday in October the pricing team merged a fix for the VAT rate on children's clothing. On Thursday finance found three days of invoices still charging the old rate. The pricing workflow had run, waited and been cancelled. Nobody had pressed anything.
A concurrency group is not a queue. While a run in the group is in progress, a new run waits in a pending state, but GitHub keeps at most one pending run per group. When another run joins, the one already waiting is cancelled and the new one takes its place. The documentation says this in one sentence, and the cancelled run's log says it was cancelled because a higher priority waiting request existed.
That Monday, an orders deploy started at ten past ten and took eleven minutes. Pricing merged at quarter past and went into pending. Returns merged at twenty past, and pricing was cancelled. Inside one workflow, this behaviour is what you want, because the newest commit contains the older ones. Across three workflows that deploy three different services, the newest run contained nothing of pricing's.
A cancelled run is grey, not red. Our notifications only went out on failure, and the pricing team saw their merge go green on the pull request and moved on.
Each workflow now has its own group, deploy-orders, deploy-pricing and deploy-returns, so a newer pricing change can only replace an older pricing change. The deploys no longer overlap on the cluster because the step that changes it takes a lock in the cluster itself and waits for it, and that wait is part of the run rather than a reason to cancel it. Every hour a job compares the commit each service is running with the last commit on main that touched its directory, and posts any service that has been behind for more than thirty minutes.
We had asked GitHub to make our deploys take turns, and it did, by keeping the most recent and quietly cancelling the rest.
– Sergey Shinder
Top comments (0)