You rotated the API key at 03:47. At 03:48, anyone with docker pull access could recover the old key from the layer you pushed six months ago. Rotation fixes the running container. It does not fix the registry.
Docker layers are SHA256 content-addressed blobs: immutable by design. Pushing an image to a registry turns every temporary secret into a permanent artifact, recoverable by anyone with pull access. Most remediation guides instruct teams to rotate credentials and rewrite the Dockerfile. That playbook solves the problem in the live container and ignores completely the inventory of layers already published to every registry that has ever run that image.
Private registries with strict access controls and automated image deletion policies limit the exposure window — an attacker needs access to the registry to pull the layers. The problem is that most organizations store the same secrets across public and private images, run scheduled CI builds that push to public Docker Hub for distribution, and cannot audit what was published in intermediate build stages two years ago.
Rotation Closes the Door After the Registry Already Has the Key
That immutability is what makes the remediation gap permanent: you can rebuild the running container, but every registry that cached that layer still holds the secret. Once pushed to a registry, a layer is independently addressable by its digest. The image tag is only a pointer; the layer persists indefinitely, regardless of what happens to the active image or the production container.
Adding RUN rm .env in a subsequent Dockerfile step creates a new layer that hides the file from the running container's filesystem. The original layer containing the .env file remains intact and recoverable via docker save. An attacker extracts that specific layer with a single tar -x command, without privileged access and without any exploit.
GitGuardian scanned 15 million Docker Hub images in Q4 2024 and found 100,000 valid and exploitable secrets at the time of disclosure. 60% of those secrets were created before 2024. The source images had likely been rebuilt multiple times during that period, but the old layers remained addressable in the registry. The SHA256 hash of a layer is permanent. Rotating the credential in production does not erase any of the records that already exist in registries that any pipeline, developer, or CI tool has ever pulled.
Three Commands Between a Pulled Image and a Compromised Credential
Extracting credentials from Docker layers requires no exploit and no privilege escalation. All it takes is docker pull access and three commands that ship with any standard Docker installation.
docker history --no-trunc <image> displays every RUN command and every ENV/ARG value embedded in the image at build time, unredacted, in the command field. docker save <image> -o image.tar && tar -xf image.tar extracts each layer as an individual tarball. tar -tf layer.tar | grep -i '.env\|credential\|secret' locates sensitive files without running the container and without any permission beyond reading the tarball.
Specialized tools eliminate even that manual effort. Dive (wagoodman/dive) renders per-layer filesystem diffs in a terminal interface, making it trivial to identify which layer introduced a specific file and inspect its contents. Secret Diver, from Cider Security, extracts layers independently and runs secret scanning on each one, with specific focus on intermediate build layers that are not part of the final image but were pushed alongside it. GitGuardian found 98% of secrets in image layers, not in manifests. Scanning limited to the manifest does not come close to the real attack surface.
Public Registries Are Searchable Credential Databases for Adversaries
At Docker Hub scale, the nature of the problem changes category. It is no longer an isolated development failure; it is structural. Docker Hub functions as an unauthenticated database of organizational credentials that any threat actor can query without a registered account.
RWTH Aachen University published in 2023 (arxiv:2307.03958, peer-reviewed) a study that analyzed 337,171 Docker Hub images. 8.5% contained secrets. The researchers recovered 52,107 private keys and 3,158 API secrets. 275,269 active hosts depended on the compromised private keys found in those images, and 22,082 certificates were using the exposed keys.
GitGuardian, in a scan of 15 million images in Q4 2024, found 1.2 million unique secrets in total. 170,000 images contained at least one valid secret. More than 7,000 were active AWS keys at the time of disclosure. An earlier scan by the same company on a sample of 200,000 images found 30,000 unique secrets in 19,000 images, corresponding to 10% of the sample. The exposure rate in public registries (9.0%) consistently exceeds that of private registries (6.3%), according to RWTH Aachen: the standard public publishing model structurally amplifies the problem.
The impact is distributed across sectors. Flare documented images from Fortune 500 organizations in pharmaceutical, energy, and defense among those affected. The exposure surface is not startups with immature practices; it is enterprises with established security programs that have never audited the backlog of already-published layers.
Codecov Showed That Intermediate Layers Are the Real Attack Vector
The Codecov breach compromised HashiCorp, Twilio, Rapid7, and Confluent. The initial access vector was not the production container: it was a credential extracted from an intermediate layer of a publicly accessible Docker image.
The official post-mortem, published in April 2021, documents that the attacker extracted a GCS HMAC key from an intermediate layer of the Codecov self-hosted image. That key granted write access to the GCS bucket where the bash upload script was hosted. The attacker silently modified the script to exfiltrate CI environment variables from every customer pipeline execution, without alerts and without any visible behavior change.
61 days passed before detection. Discovery was accidental: a single customer checksummed the download against the SHA256 published by Codecov and noticed a discrepancy. The compromised layer was not the final production image layer; it was a build stage artifact, never intended to be public. Even layers from discarded stages persist when the image is pushed without explicit stage controls, and that is the predominant pattern in pipelines without a documented build policy.
CI/CD Pipelines Publish the Most Dangerous Images
Automated build pipelines are the highest-risk publishing channel for secrets in layers. They inject secrets at build time systematically, frequently push intermediate artifacts to registries, and generate sufficient metadata to make extraction completely trivial for anyone with registry access.
The compromise of tj-actions/changed-files in March 2025 illustrates the scale of the problem. A poisoned GitHub Action caused CI/CD secrets, including AWS credentials, npm tokens, and Docker registry credentials, to be written to build logs across more than 23,000 repositories. Coinbase was the initial targeted victim; the attack expanded through the actions dependency chain to organizations with no direct relationship to the compromised repository.
The docker build --build-arg API_KEY=$API_KEY anti-pattern is prevalent and systematically misunderstood. ARG values appear in plain text in docker history output, even without BuildKit enabled. Multi-stage builds without explicit secret isolation still leak when a secret is copied into stage 1 and that stage's layers are pushed to the registry, a common situation in CI cache configurations that preserve intermediate stages to accelerate subsequent builds. The compromise of codfish/semantic-release-action in 2024 exploited exactly that configuration: a Docker container built from the repository's Dockerfile pushed intermediate layers to the registry, making them accessible via public pull.
Scanning What Is Already Out There Is the Missing Capability
Prevention controls such as BuildKit secrets and multi-stage builds protect new images. They do nothing for the backlog of already-published layers. Real remediation requires scanning historical layers in the registry, not just linting the Dockerfile.
BuildKit --mount=type=secret mounts the secret as a tmpfs file during the RUN step and never writes it to any layer. That is the correct architectural fix for new builds (requires Dockerfile syntax >= 1.2). Multi-stage builds keep secrets only in initial stages, excluded from the final image manifest, but only when it is the final stage that reaches the registry and the CI cache does not expose the intermediates.
For the existing backlog, three layer-scanning tools cover the real scope. ggshield secret scan docker <image> (GitGuardian) scans all layers including intermediate ones and integrates into CI pipelines as a gate before pushing to the registry. trivy image --scanners secret <image> (Aqua Security) runs per-layer secret detection with patterns for more than 150 secret types. trufflehog docker --image <image> scans each layer independently, including layers from deleted files that remain addressable by their original SHA256 digest. None of these tools scan the manifest: all of them go down to the layer level, which is where 98% of secrets are.
For continuous monitoring of public registry exposure across your organization's image namespace:
The intel.mago.team platform (MAGO team tool) offers continuous exposure monitoring in public registries. When an organization's Docker Hub namespace or specific image digests appear in threat intelligence feeds or secret scanning results, analysts receive alerts before adversaries can operationalize the credentials.
The right audit question is not "does our current Dockerfile hardcode secrets?" It is "which layers have we already pushed to registries that we cannot pull back?" That inventory, with layers identified by digest and cross-referenced against the organization's current secret inventory, is the real remediation scope. Start there.
Top comments (0)