In September we split the pipeline for our parcel tracking API in two. One workflow, CI, ran the tests on every push to main. A second, Deploy, built the image and released it to production, and it was triggered by workflow_run: when CI completes on main, run Deploy. The split let us rerun a deploy without rerunning forty minutes of tests, and for three weeks it looked like an improvement.
On a Thursday in October a change to how we parse carrier scan codes reached production, and for an hour and a half every tracking page for one carrier showed its parcels as unknown. CI had failed on that commit. Two tests had caught the bug exactly, the commit was marked red, and the deploy had gone out eleven minutes later anyway.
The trigger did what it said. A workflow_run event of type completed fires when the other workflow finishes, whatever its conclusion. Success, failure and cancelled are all completed. GitHub's documentation shows the check you are meant to add, a condition on github.event.workflow_run.conclusion, and our Deploy workflow did not have one. For three weeks every commit on main had been deployed whether its tests passed or not. We had simply not merged anything broken until that Thursday.
Reading the workflow again, we found a second assumption. A workflow started by workflow_run runs in the context of the default branch, and a plain checkout step fetches the latest commit on main, not the commit CI tested. Two merges close together could have shipped a commit whose tests had not even finished.
Deploy now has a condition on its first job requiring the conclusion to be success, and it checks out github.event.workflow_run.head_sha, so the commit it builds is the commit that was tested. That sha is written into the release record, and the job fails before building anything if it cannot find a successful CI run for it. Once a quarter we push a deliberately failing commit to a copy of the repository with the same workflows, to prove the deploy refuses it.
An event that tells you something has finished is not telling you how it finished. We had connected a release to a signal and never read the one field that carried the answer.
– Sergey Shinder
Top comments (0)