Container Images, Layers & Registries — An Image Is Not One Big File: One Changed Line, 150 MB Moved, and the Dockerfile Order That Makes It 2 MB

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

Container images, layers, and registries. Many people think that an image is one big file. It is not. It is a stack of smaller files, and each one is stored and sent on its own.

This article follows one image of 200 megabytes and one changed line of code. It reads two Dockerfiles, two small Python functions and one deploy file, line by line. At the end you know why the order of a Dockerfile decides how much is built and moved, and why a tag does not tell you what really runs.

---

You changed one line. Why did 150 MB move?

You run one service as three pods. A pod is one running copy of your service. Every deploy builds a new container image, and every server downloads it.

The image is two hundred megabytes. Today you change one line of code.

step	what happens
1	the build installs every dependency again
2	the build uploads 150 MB
3	every server downloads 150 MB
	your change was one line, in a file of a few kilobytes

So here is the question:

You changed one line. Why did one hundred and fifty megabytes move?

---

What this article covers (and what it does not)

The subject is DevOps, and one part of it: how a container image is built, stored and delivered.

An image is the packed file system that a container starts from.

This article covers three things:

• layers, which are the pieces of an image;
• the build cache, which decides what is built again;
• the registry, which is the server that stores images.

Left alone on purpose: the running process and its signals. That has its own episode.

Where does this live? It is built into your tools. You will not write this machinery yourself. It is built into Docker, Podman, containerd and Kubernetes. What you write is the Dockerfile, and the image name in your deploy file. Those two decide your build time, and what really runs in production.

---

A reprinted textbook and a thin clear sheet

Here are two ways to publish a correction to a thick textbook.

The first way: print the whole book again for every correction. Three corrections give you three thick books, and every reader must carry all of that paper home.

The second way: keep one book. Write each correction on a thin clear sheet, and lay the sheet over the page. To publish a correction, you send only the sheet. Every reader already has the book.

	reprint the book	lay a sheet over the page
three corrections	three thick books	one book, three thin sheets
what you send	the whole book, every time	only the new sheet
what the reader already has	nothing that helps	the book
in an image	one big file, copied whole	a base layer, plus thin layers

A container image works the second way. The book is the base layer. Each sheet is one layer on top of it.

---

The whole system, drawn once

Part	What it is	What it does
The build machine	1 CI runner, with 1 layer cache	reads your Dockerfile and builds the image
The registry	1 HTTP server	stores every layer under its digest
The nodes	2 servers that run your pods	pull layers and start containers

A digest is a hash of the bytes, so the same bytes always get the same name. A node is one server that runs your pods.

Two arrows join the parts. The build machine pushes layers to the registry. Each node pulls layers from the registry. Layer bytes move in one direction only: a node never sends a layer back.

Now one image for the whole article. Its name is shop.

layer	size
the base layer	50 MB
the dependencies	148 MB
your code	2 MB
the image	200 MB

---

The Dockerfile that most people write first

A Dockerfile is the list of steps that builds an image.

FROM python:3.12-slim                      #  50 MB
COPY . /app                                #   2 MB
RUN pip install -r /app/requirements.txt   # 148 MB
CMD ["python", "/app/main.py"]             # no layer

• Line one is FROM. It starts from a base image that already has Python. That is the 50 MB layer.
• Line two is COPY. It copies your whole project into the image. That is the 2 MB layer.
• Line three is RUN. It installs the dependencies. That is the 148 MB layer.
• Line four is CMD. It only records which command to start, so it adds no files.

step	layer	layer size
FROM	base image	50 MB
COPY . /app	your code	2 MB
RUN pip install	dependencies	148 MB

Three steps made three layers.

Remember the order: the code layer comes before the dependency layer.

---

The build cache: one key for each step

Now a small model of what the build tool does inside. It is not code for your service. It helps you reason about the tool.

from hashlib import sha256

def key(prev, step, files):
    text = prev + step + files
    return sha256(text.encode()).hexdigest()

def build(steps, cache):
    prev, rebuilt = "", []
    for step, files in steps:
        prev = key(prev, step, files)
        if prev not in cache:
            cache.add(prev)
            rebuilt.append(step)
    return rebuilt

steps = [("FROM base", ""), ("COPY . .", "app.py"), ("RUN pip install", "")]

• Every step gets a cache key. The key is a hash of three things: the key of the step before it, the text of the step, and the files that the step copies.
• The build function walks the steps in order.
• If the key is already in the cache, the tool reuses the old layer.
• If the key is new, the tool runs the step again.

Now change one line of code.

step	its key	what the tool does	size
FROM	the same	reuses the layer	50 MB
COPY . /app	new, because a copied file changed	runs the step again	2 MB
RUN pip install	new, because its key includes the key before it	runs the step again	148 MB

The COPY step gets a new key. The next step includes that key, so its key changes too.

One changed step rebuilds every step after it: 2 + 148 = 150 MB.

---

The fix: change the order

Put the steps that change rarely first, and the steps that change often last.

FROM python:3.12-slim
COPY requirements.txt /app/
RUN pip install -r /app/requirements.txt
COPY . /app
CMD ["python", "/app/main.py"]

• Line two now copies only one file: the list of dependencies.
• Line three installs them.
• Line four copies your code, last.

Now change one line of code again.

step	cache	layer size
FROM	reused	50 MB
COPY requirements.txt	reused	0 MB
RUN pip install	reused	148 MB
COPY . /app	REBUILT	2 MB

Line two copies the same file as before, so its key is the same. Line three has the same key too. The tool reuses both layers. Only line four runs again.

One line of code now rebuilds 2 MB, not 150.

---

The registry asks one question for every layer

The registry uses the same idea. Here is the model of a push.

def push(layers, registry):
    sent = 0
    for digest, size in layers:
        if digest not in registry:      # do you have this digest?
            registry[digest] = size     # no: upload this layer
            sent += size
    return sent

pull = push    # the same loop, against the disk of the node

• For every layer, the tool first asks the registry one question: do you already have this digest?
• If the answer is yes, the tool sends nothing.
• If the answer is no, the tool uploads that layer.
• A pull works the same way, against the disk of the node.

Now run both Dockerfiles after the one-line change.

order	rebuilt	pushed	pulled per node
code first	150 MB	150 MB	150 MB
code last	2 MB	2 MB	2 MB

Layers that did not change never move again.

---

The trap: three pods, one tag

Here is the same picture, one day later.

Your team pushes every build under the same tag: shop:prod. A tag is a readable name that points at one image.

day	what happens
Monday	nodes one and two pull shop:prod. They keep Monday's image on disk.
Tuesday	you push a new build under the same tag. The registry moves the tag to the new image.
Tuesday	the cluster adds node three. It has an empty disk, so it pulls the tag, and it gets Tuesday's image.
Tuesday	nodes one and two already have that tag on disk, so they pull nothing.

Here is the second question:

Three pods run the same tag. Are they running the same code?

---

The answer: a tag can move, a digest cannot

No. Two pods run Monday's code, and one pod runs Tuesday's code.

The cause is that a tag can move. A digest cannot. The digest of an image is a hash of its list of layers. If one byte changes, the digest changes.

So this is the guard. This is the deploy file that Kubernetes reads.

# by tag: a tag can move
image: shop:prod

# by digest: a hash of its list of layers, so it cannot move
image: shop@sha256:9f2c41d07b6e58a3c1f04e7d92ab35c68e10f7a4d3b925c06e8f1a7b4d2c93e5

• The first line names the image by tag.
• The second line names it by digest, after an at sign.

Now every node must run exactly those bytes. A node that holds a different image pulls the right one.

image named	node one	node two	node three
by tag	monday	monday	tuesday
by digest	tuesday	tuesday	tuesday

Give each build its own tag, and deploy by digest.

---

The catch: a layer only adds

There is one catch that surprises people.

A layer only adds changes. It never removes anything from the layers before it.

step	what it does	what the image holds
1	copies a password file into the image	a layer with the file
2	deletes that file	a new layer that says: this file is gone
	the final file system	does not show the file
	the earlier layer	still holds the file

Anyone who pulls the image can read that earlier layer. The image also did not get smaller.

So never copy a secret into an image.

---

The bill, because it is real

One deploy is one push and three pulls, one pull for each node. So four moves.

	code first	code last
one move	150 MB	2 MB
one deploy: 4 moves	600 MB	8 MB
ten deploys a day	6,000 MB	80 MB

6,000 divided by 80 is 75.

The same service, the same code, the same image: 75 times less to build and move.

And what you pay for it:

what you gained	what you pay
A one-line change rebuilds and moves 2 MB, not 150.	The order of the Dockerfile is now a rule that the team must keep.
Every node runs exactly the same bytes.	The deploy file names a long digest, so a tool must write it for each build.
Layers that did not change never move again.	A layer never forgets: one copied secret stays in the image.

---

For your next interview

What is a container image, and why does the order of a Dockerfile matter?

A container image is a stack of read-only layers, plus a small file that lists them. Tools like Docker and containerd store, push and pull each layer on its own.

Four words to remember:

• A layer: the files that one build step added.
• A digest: a hash of the bytes, which can never point at different bytes.
• A tag: a readable name, which can be moved.
• A registry: the server that stores layers by digest.

The order matters because one changed step rebuilds every step after it.

---

The one-line verdict

An image is a stack of layers that are cached, pushed and pulled one by one, so put the steps that change rarely first and your code last, and name the image by digest, because a tag can move.

One thing to go and look at tonight: open the Dockerfile of your main service. Find the line that copies your code, and the line that installs the dependencies. Check which line comes first.

---

References and further reading

On layers and digests

• Open Container Initiative, OCI Image Format Specification: Image Manifest — an image is a manifest that lists its layers in order, and each layer is named by its digest: github.com — image-spec, manifest.md
• Open Container Initiative, OCI Image Format Specification: Content Descriptors — a digest identifies content by a collision-resistant hash of its bytes, such as SHA-256: github.com — image-spec, descriptor.md

On the build cache and the order of the steps

• Docker, Docker build cache (Docker documentation) — once a layer changes, all the layers after it must be rebuilt as well: docs.docker.com — Build cache
• Docker, Optimize cache usage in builds (Docker documentation) — steps that change often belong near the end of the Dockerfile, with the dependency file copied and installed before the rest of the code: docs.docker.com — Optimize cache usage
• Docker, Build cache invalidation (Docker documentation) — the real rules behind the cache key; for COPY and ADD the builder computes a checksum from the files involved: docs.docker.com — Cache invalidation

On the registry, push and pull

• Open Container Initiative, OCI Distribution Specification — a client checks whether the registry already has a blob with a HEAD request on its digest, and a tag is a human-readable pointer to a manifest: github.com — distribution-spec, spec.md

On tags, digests and the pull policy

• The Kubernetes Authors, Images (Kubernetes documentation) — with the pull policy IfNotPresent the kubelet skips the pull when the image is already on the node; if a tag changes you can end up with a mix of Pods on old and new code, and a digest names one specific version of the image: kubernetes.io — Images

On the catch: a layer only adds

• Open Container Initiative, OCI Image Format Specification: Image Layer Filesystem Changeset — a removal is stored as a whiteout file in the new layer, so the earlier layer is not changed: github.com — image-spec, layer.md
• Docker, Build secrets (Docker documentation) — build arguments and environment variables are the wrong place for a secret, because they persist in the final image; a secret mount gives the build access without storing it: docs.docker.com — Build secrets

If a reference you would expect is missing, say so in the comments and I will add it.

---

Watch the full episode: Container images, layers & registries — Explained in Detail on YouTube