In March a customer opened an issue on our public SDK repository: the changelog announced version 3.8.0, the GitHub release page had notes for it, and npm said the version did not exist. Neither did 3.7.1 or 3.7.2. For nineteen days we had been announcing releases nobody could install.
Our release process is two workflows. The first runs on every merge to main, works out the next version from the commit messages, pushes a tag and creates a GitHub release. The second runs on release published, builds the package and publishes it to npm. For two years the first workflow authenticated with a personal access token stored as a secret, with repo scope across the whole organisation. In February an access review flagged it, rightly, and we replaced it with the GITHUB_TOKEN that every workflow run gets, with contents write permission and nothing else.
The first workflow kept working. Tags appeared, releases appeared, the run was green. The second workflow never ran again, and nothing anywhere said so. GitHub documents this: events caused by the GITHUB_TOKEN do not start new workflow runs, with the exception of workflow_dispatch and repository_dispatch. It is there to stop workflows triggering each other in loops. The personal token had been a person, as far as GitHub was concerned, and a person creating a release starts every workflow that listens for one.
A missing run leaves no trace. There was no failed job to look at, only an empty list on a page nobody had a reason to open.
We published the three versions by hand that afternoon, from their tags. Publishing is now a job in the same workflow as the release, joined with needs, so it cannot be skipped without the release run itself showing as failed. Where one workflow genuinely has to start another, it uses a short lived token from a GitHub App installed on that repository only, not a person's token. The last step of every release fetches the version from the npm registry and fails the run if it is not there. And a daily job compares every tag in the repository with the versions on npm and opens an issue for any tag that has no package.
Swapping a token is also swapping an identity. Ours was the same permission held by someone GitHub had been told not to listen to.
– Sergey Shinder
Top comments (0)