In July someone noticed that cloning our largest repository took ninety seconds of every pipeline run, and changed the checkout step to fetch only the latest commit. Builds got a minute and a half faster across eleven services. Nine days later a rollback in production installed a binary that was three weeks old.
Our versions come from git. A small script runs git describe, which walks back from the current commit to the nearest release tag and prints something like 4.12.0 plus the number of commits since. A checkout of one commit has no history and no tags, so git describe had nothing to walk back to and exited with an error. The script had a fallback for that case, written years earlier for people building on a laptop with no tags: print 0.1.0-dev and carry on. It printed a warning nobody reads, and every build of every service on main was now called 0.1.0-dev.
Our artifact repository allowed a development version to be overwritten, because development versions are meant to move. So each merge replaced the previous one under the same name. The deploy tool recorded which version it had installed, not which file. When the payments team rolled back, it asked the repository for the previous version, which was the same name as the current one, and got whatever had been uploaded last under it. That was a build of the reporting service. The pods failed their startup checks and the rollout stopped, which is the only reason this is a short story.
The checkout now fetches full history without file contents, which costs about four seconds and keeps every tag. The version script fails the build if it cannot find a tag, and the laptop fallback only applies when an environment variable says it is a laptop. The pipeline computes the version once, at the start, and passes it to every later step instead of letting each one ask git again. Release and main builds go to a repository that refuses to overwrite anything. And the deploy tool records the artifact's checksum, so a rollback installs the bytes that ran before, not a name.
A speed improvement removes something, and it is worth asking who was reading what you removed. Our version numbers had been reading the history all along.
– Sergey Shinder
Top comments (0)