In January a customer's security team emailed us a token. It began with npm_, it belonged to our private package registry, and they had found it in the agent image we ship for customers to run in their own clusters. They had simply run docker history.
Our agent's Dockerfile took the registry token as a build argument, wrote it into an .npmrc file, ran npm ci and deleted the file, all in one RUN instruction. Everyone who reviewed it, me included, had learned the same rule: a secret must never end up in a layer, so create it and remove it in the same step. The file never did end up in a layer. The rule was right and beside the point.
Docker records how every layer was made in the image's configuration, and for a RUN instruction that record includes the build arguments the command used. docker history --no-trunc printed NPM_TOKEN= and the full value in front of our shell command. Anyone who can pull an image can read its history. The pipeline had done that for fourteen months, into two hundred and twelve published tags, forty of them sitting in customer registries we cannot clean.
The token could read and publish every internal package. We revoked it within the hour and spent two days going through the registry's audit log for publishes we had not made. There were none.
The build now mounts the .npmrc as a BuildKit secret, which exists only while that one RUN executes and is never written to a layer or to the history. Dependencies are installed in a builder stage and the image we ship is a separate final stage, so nothing about how they were fetched travels with it. The token is no longer a long lived value in CI settings either: each job exchanges its OIDC identity for a read only registry token that expires after fifteen minutes. Docker's build checks run with warnings treated as errors, so an ARG whose name looks like a secret fails the build. And before any image is published, a step dumps its configuration and history and runs the scanner we already use on source code over them.
An image is not only its files. It carries a record of how it was built, and that record goes to everyone you give the image to.
– Sergey Shinder
Top comments (0)