We publish a command line tool that our partners use to book collections and download labels from our API. Version 2.7.0 went out on a Tuesday in September. By Wednesday afternoon support had forty tickets, all saying the same thing: every command answered with a message telling them their client was too old and to upgrade. They had all just upgraded.
The tool learns its own version at build time. The release workflow builds it with go build and an ldflags option, -X main.version followed by the tag, which tells the Go linker to set the string variable version in package main to that value. The variable defaults to dev, so a laptop build says so.
In 2.7.0 a tidy up moved build information out of main into a package of its own, internal/buildinfo, and the variable went with it. The release workflow still said main.version. The Go linker does not treat an -X naming a symbol that does not exist as an error, or even a warning. It ignores it. The build was green, and every binary we signed and published believed it was version dev.
The tool sends its version in a header on every request, and since the spring our API gateway has refused clients older than 2.4.0, because we removed an endpoint those versions depend on. The check parsed the header as a semantic version, and anything it could not parse it treated as 0.0.0. So anyone who stayed on 2.6 was fine, and everyone who did exactly what we asked was refused.
None of our tests could see it. The end to end suite runs the tool from source with go run, and no linker flag is involved there.
The release workflow now runs the built binary with its version command and fails unless the output matches the tag, character for character, before anything is signed. The ldflags line uses the full import path of the new package. The gateway no longer turns an unreadable version into an old one. It lets the request through and counts it, and a version of dev in production pages the release owner.
A default that looks like a real value will travel all the way to production without anyone noticing it. The build should have to prove what it stamped.
– Sergey Shinder
Top comments (0)