Our public Go client for the tracking API is a module on GitHub, released by pushing a version tag. In October we tagged v1.8.0, and forty minutes later a customer reported that it sent every request with an empty account header. The fix was two lines. The engineer who had made the release deleted the tag, merged the fix and pushed v1.8.0 again, on the new commit, so that nobody would have to remember a bad version.
For the next two days our issue tracker filled with two kinds of report. Some customers said v1.8.0 still sent the empty header. Others said their builds were failing with a checksum mismatch and the words SECURITY ERROR.
The first group were fetching through proxy.golang.org, the module proxy the go command uses by default. The proxy keeps a copy of every module version it serves, and it had fetched v1.8.0 at the original commit in the first forty minutes. Its documentation says plainly that a bad release may stay available from the mirror even after it has gone from the origin. As far as the proxy was concerned, v1.8.0 was the code it already had, and that was what it kept serving.
The second group fetched directly from GitHub, because their companies set GOPROXY to direct or ran their own mirror. They got the new commit. The go command checks every module it downloads against go.sum, and against the checksum database at sum.golang.org, which had recorded the hash of the original v1.8.0. A different hash for the same version is exactly what that check exists to catch, so it refused the download.
We put the tag back on the original commit, which made both groups agree again, and released the fix as v1.8.1. v1.8.1's go.mod has a retract directive for v1.8.0, so the go command will not choose it when it looks for the latest version, and go list -m -u shows it to anyone still using it as retracted.
Version tags matching v* are now protected by a ruleset that blocks updates and deletions for everyone, including administrators. The release checklist says that a bad version is fixed by the next version, and the release workflow checks that a tag it is about to create has never existed.
A Go version stops belonging to you as soon as someone downloads it. Moving the tag changed our repository and none of the copies.
– Sergey Shinder
Top comments (0)