Why a Secret Baked Into a Docker Image Never Really Goes Away

#ai #debugging #llm #machinelearning

A Dockerfile that copies a credential in, uses it, and removes it looks like it cleaned up after itself. Running the image confirms the file is gone. It is not gone; it is in the layer below, and the image ships both.

Why the delete does not delete

A container image is an ordered stack of read-only layers, each a tarball of the filesystem changes made by one build instruction. The runtime presents them through a union filesystem, so what you see inside a running container is the layers merged top-down. Nothing in that design lets an upper layer modify a lower one — layers are content-addressed and immutable, which is exactly what makes them shareable and cacheable across images.

So RUN rm /tmp/credentials.json cannot remove a file added by an earlier instruction. What it does is add a whiteout entry to the new layer: a marker — conventionally a file named .wh.credentials.json in the layer tar — that instructs the union filesystem to hide the lower entry. The merged view has no file. The layer below still contains the bytes, verbatim, and that layer is part of the image manifest, pushed to the registry, and pulled by everyone who pulls the image.

The same applies to a secret passed with --build-arg and to one set with ENV. Those do not even need a layer: build arguments are recorded in the image history and ENV values are recorded in the image config, which means they are readable without extracting anything.

Reading it back out

Demonstrating this to a colleague takes about ninety seconds and is the fastest way to end an argument about whether it matters. Everything below uses only the image; no access to the build host is required.

IMAGE=registry.example.com/inference-worker:1.4.2

# 1. The command history, including build args baked in by the builder.
docker history --no-trunc "$IMAGE"

# 2. The image config: ENV values, entrypoint, labels.
docker inspect --format '{{json .Config.Env}}' "$IMAGE"

# 3. The layers themselves.
docker save "$IMAGE" -o image.tar
mkdir -p unpacked && tar -xf image.tar -C unpacked
find unpacked -name '*.tar' -exec sh -c \
  'tar -tf "$1" | grep -i -E "credential|\.env|id_rsa|\.npmrc" && echo "  ^ in $1"' _ {} \;

The third step is the one that finds a deleted file, because it reads each layer tar independently rather than through the merged view. A secret scanner run against the layer tars rather than against a running container finds these; one run against the container does not, which is why an image can pass a scan and still carry a credential.

The three places it hides

A file layer. Anything added by COPY or ADD, or written by a RUN. The classic case is a COPY . . that sweeps up a .env file the author forgot about, followed by a cleanup instruction. A .dockerignore that lists .env, .git and *.pem prevents this entirely and is the highest-value three lines in most repositories.
The build history. docker history --no-trunc shows the command for each layer, so a credential passed on a RUN line or as a build argument is readable directly. There is no file to delete; the string is metadata.
The .git directory. Copied in by a broad COPY, it carries every credential ever committed and later removed from the working tree, because git keeps the objects. This is the one that survives a careful audit of the current source.

Two more places are adjacent and worth checking at the same time: a package manager config such as .npmrc or .pip/pip.conf holding a registry token, and a multi-stage build where the secret is used in the builder stage. The second is safe only if nothing copies it forward — the builder stage is not published, but COPY --from=builder /app /app will happily bring a credential along if it is inside /app.

What to do about an image already pushed

Rebuilding is not remediation, and neither is squashing. Both produce a new image; the old one is already in the registry, already in every node’s image cache that pulled it, and possibly already in a backup. Order the response accordingly.

Revoke the credential. First, before anything else, for the same reason as on an API key leaked into serverless logs. Every later step is cleanup; this is the only one that changes the exposure.
Establish the reach. Was the registry public or internal? Which tags share the affected layer — layers are shared by digest, so one bad layer can appear in every image built from the same base or the same cache. Who pulled it, from the registry’s access logs.
Delete the tags and the manifests, knowing that deletion in most registries removes the reference and leaves the blob until garbage collection runs, and that a node with the image cached still has it. Some registries offer immutable tags, in which case deletion is the only option and it is disruptive.
Rebuild without the secret and, since the layer digest changes, verify with docker history on the new image that the offending instruction is gone rather than assuming the Dockerfile edit was sufficient.

--squash flattens a new build into a single layer and does nothing for an image already pushed. It also loses layer caching and keeps the image history, so it is a poor answer to this problem even prospectively.

Building without ever writing it down

BuildKit’s secret mounts exist for exactly this case. A mounted secret is available as a file for the duration of one RUN instruction, is backed by a tmpfs, and is not committed to the layer or recorded in the history.

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# The token exists only inside this RUN, in a tmpfs mount.
RUN --mount=type=secret,id=hf_token \
    HF_TOKEN="$(cat /run/secrets/hf_token)" python download_weights.py

FROM python:3.12-slim
COPY --from=builder /app/weights /app/weights
COPY src/ /app/src/
CMD ["python", "-m", "app"]

# From a file, or straight from an environment variable.
docker build --secret id=hf_token,src=./hf_token.txt -t inference-worker:1.4.3 .
docker build --secret id=hf_token,env=HF_TOKEN     -t inference-worker:1.4.3 .

Three habits close the remaining gaps. Keep a .dockerignore that excludes .env, .git, *.pem and any local credential file, so a broad COPY cannot pick them up. Scan the layer tars in CI rather than the running container, since that is where a deleted secret lives. And supply runtime credentials at runtime — from a secret store or an injected file, as on injecting secrets into a Kubernetes pod — so the image is the same artefact in every environment and contains no credential to leak in any of them.