DEV Community

Sergey Shinder
Sergey Shinder

Posted on

We moved our image build to the buildx action and every release for three weeks pushed the same old image

Our label service is built on a self-hosted runner, a long-lived virtual machine with its own Docker daemon. For two years the workflow ran docker build -t label-service:ci, a later step tagged label-service:ci with the commit SHA and pushed it to our registry, and the deploy job rolled out that tag.

In September we replaced the build step with docker/setup-buildx-action and docker/build-push-action, for the registry cache. We left the tag and push steps alone. Build, push and deploy all stayed green.

Three weeks later a fix for rotated carrier labels went out and the labels were still rotated. In the registry, the last fourteen SHA tags all pointed at the same digest.

setup-buildx-action creates a builder that uses the docker-container driver: BuildKit runs in a container of its own, and what it builds does not go into the Docker daemon's image store unless you ask. build-push-action has two inputs for that, load and push, and both default to false. The build log said so on every run: no output specified with docker-container driver, build result will only remain in the build cache.

So the next step, docker tag label-service:ci, went to the daemon and found an image. It was the one the old workflow had built on the morning we switched, still sitting there because the runner is never wiped. On a fresh hosted runner the tag step would have failed with no such image. On ours it succeeded fourteen times, and we shipped the same three-week-old build under fourteen new names.

The build step now tags with the SHA and pushes in the same step, with push: true, so there is no image left over for a later step to pick up by name. The deploy job takes the digest that step outputs rather than a tag. The image carries an org.opencontainers.image.revision label set to the commit, and the deploy refuses any image whose label does not match the commit it was asked to release. Every job on that runner now starts by pruning local images, so a leftover cannot stand in for anything again.

A long-lived runner remembers everything you ever built on it. That morning it answered for a build that had never happened.

– Sergey Shinder

Top comments (0)