OverlayFS copy_up: Why Container Disks Quietly Grow

One of the most repeated sentences about containers goes like this: the layers are read-only, and the thin writable layer on top is ephemeral. The sentence is true. The problem is that "thin" is a hope, not a guarantee. On the night the disk fills up, nobody is discussing the elegance of layered architecture; everyone is trying to work out how a container born from an image of a few hundred megabytes ended up occupying tens of gigabytes.

The answer is almost always in the same place: copy_up. OverlayFS's file-granular copy logic writes traffic to disk that the application never intended. My argument is this: most container disk problems are not capacity problems, they are misunderstood mechanism problems. A limit set without knowing the mechanism doesn't solve anything — it only reschedules the explosion.

I have written before about a container turning into a resource monster from the CPU and memory side, and about logs quietly eating the disk from the symptom side. This piece is about disk, and the mechanism is different in kind. Throttle CPU and you get slow; fill the disk and you stop.

copy_up: the whole file, once per file

OverlayFS is built on two stacks. The lower layers (lowerdir) come from the image and are read-only; the upper layer (upperdir) belongs to the container itself. As long as a file is only read, nothing happens — the data stays in the lower layer.

The moment a write arrives, things change. In the kernel documentation's own words, when a file in the lower filesystem is accessed in a way that requires write access — "such as opening for write access, changing some metadata etc." — the file is first copied to the upper filesystem. The copy first creates the containing directory (and any parents as needed), then creates the object with the same metadata, then moves the data. To make sure a half-finished copy is never visible, OverlayFS calls fsync(2) on the upper file before completing the copy with rename(2) or link(2).

The operational translation of this sits in a single sentence in Docker's own documentation, and it deserves underlining: because OverlayFS works at the file level rather than the block level, "all OverlayFS copy_up operations copy the entire file, even if the file is large and only a small part of it's being modified."

So when you send a single UPDATE to a 2 GB SQLite database baked into the image, 2 GB gets written to the container's writable layer. From the application's point of view, one row changed; from the filesystem's point of view, an entire file moved. The same behaviour applies to .war archives or precomputed model files baked into the image.

The good news is that this price is paid once. In Docker's phrasing, copy_up only occurs the first time a given file is written to, and subsequent writes operate against the copy already copied up. So it is not a leak, it is an entry fee. But the entry fee is paid ten times when you start ten containers from the same image.

There is some consolation on the read side: OverlayFS supports page cache sharing, and multiple containers accessing the same file share a single page cache entry for it. Generous with memory, not with disk.

Deleting frees nothing

The second misunderstanding shows up in the cleanup reflex. You delete a large file inside the container, df keeps showing the same number, and you conclude the filesystem is mocking you.

It isn't. The Docker documentation is explicit: when a file is deleted within a container, a whiteout file is created in the container (upperdir), and the version of the file in the image layer (lowerdir) is not deleted, because the lowerdir is read-only. For directories, the same job is done with an opaque directory.

The physical representation on the kernel side is interesting too: a whiteout is created "as a character device with 0/0 device number or as a zero-size regular file with the xattr trusted.overlay.whiteout." An opaque directory is marked by setting the trusted.overlay.opaque xattr to y.

It's worth drawing the boundary carefully here, because it is easy to blur: a whiteout exists only to mask an object that has a counterpart in a lower layer. A file the application created at runtime already lives only in the upper layer; deleting it is an ordinary removal and it does free space. What doesn't free space is deleting a file that came from the image.

The consequence on the security side is uncomfortable. If a .env file or a private key accidentally made it into one of your image layers, a later RUN rm does not remove it from disk; it only makes it invisible. The secret is still there for anyone who can pull the image. This is one of the oldest and most stubborn traps in container security, and the fix is not deleting — it is using multi-stage builds so the file never enters the final image at all.

First ask this: which backend am I on?

Every command I write from here on depends on a single question. Docker's classic storage drivers (overlay2 and friends) and the snapshotters on the containerd side do the same job, but they don't do it in the same place.

Old posts and old Stack Overflow answers still circulate advice about aufs, devicemapper and overlay (v1). All of them are gone: aufs was deprecated in v19.03 and removed in v24.0, the legacy overlay driver was deprecated in v18.09 and also removed in v24.0, and devicemapper was deprecated in v18.09, disabled by default in v23.0 and removed in v25.0.

But "overlay2 is what's left, case closed" is no longer a valid conclusion. With Docker Engine 29.0, released on 10 November 2025, the containerd image store became the default on fresh installs, and in the documentation's own words overlay2 is now "a legacy storage driver that is superseded by the overlayfs containerd snapshotter." For upgraded systems the story is different: the documentation states that if you upgraded from an earlier version, your daemon continues using the legacy graph drivers (overlay2) until you enable the containerd image store.

So two different worlds run side by side today, and guessing which one you are in is not an option:

# Which backend? On the containerd snapshotter you'll see io.containerd.snapshotter.v1
docker info -f '{{ .DriverStatus }}'

Enabling it is done with the features.containerd-snapshotter key in /etc/docker/daemon.json, followed by a daemon restart. Two things are worth knowing before you switch: changing storage backends temporarily hides images and containers created with the other backend (the data stays on disk and reappears when you switch back), and containerd uses a separate storage path from Docker's data directory.

That last sentence touches the heart of operations: if your disk alert is built only on /var/lib/docker, you will never see growing image data on a fresh installation that uses the containerd image store.

The mechanism side is untouched by this split — copy_up, whiteout, EXDEV and metacopy are the behaviour of the overlayfs running underneath in both worlds. What changes are the commands, the paths and the limits. One limit lives on the classic side: overlay2 natively supports up to 128 lower layers. For Dockerfiles that generate hundreds of RUN lines, that ceiling is not theoretical.

Where to look: two different numbers

Before measuring, you need to know which number says what. docker ps --size produces two values: one is the amount of data on disk used for each container's writable layer, the other ("virtual size") is the total of the read-only image data used by the container plus the writable layer.

The first number is the one that matters for capacity planning. The second becomes misleading when summed per container, because containers born from the same image share that read-only data; add up ten containers' virtual size and you have counted the image ten times. If it were me, I would base alert thresholds on the writable layer alone and track the image side as a separate line item.

# Surface containers with the largest writable layers
docker ps --size --format '{{.Names}}\t{{.Size}}'

# Disk usage broken down by category (images / containers / volumes / build cache)
docker system df -v

The docker system df summary shows TYPE, TOTAL, ACTIVE, SIZE and RECLAIMABLE columns, and the verbose output splits the image side into two numbers. By the documentation's definition, SHARED SIZE is the space an image shares with another one, UNIQUE SIZE is the space used only by a given image, and SIZE is the virtual size of the image. UNIQUE SIZE is what tells you how much disk deleting an image will actually free; anyone who plans cleanup from SIZE never gets the relief they expected after the delete. The backend caveat applies here too: the 29.x release notes contain a run of fixes to image size reporting and to shared/unique size accounting, which means the meaning of these numbers is still settling on the containerd image store.

If a container's writable layer keeps growing, the question to ask is not "how big" but "which file." On the classic driver you can get that answer directly: ask where the container's upper layer lives on disk, then look there.

UPPER=$(docker inspect -f '{{.GraphDriver.Data.UpperDir}}' <container>)
sudo du -xh --max-depth=2 "$UPPER" | sort -h | tail -20

Those two lines turn "the disk is full" into "this container is growing this file." That difference decides how the night goes.

The culprit is usually one of three candidates: writing application logs to a file, writing to a data file baked into the image, or generating temporary files in a directory inside the image instead of /tmp.

A common monitoring mistake surfaces here as well. In most setups the disk alert is built on a single total usage figure. When that threshold fires, you have exactly one piece of information: "it's full." But whether it was filled by image layers, writable layers or volumes means three completely different interventions. The first needs image cleanup, the second needs the application's write pattern to change, the third turns into a question about retention policy and real capacity. If it were me, I would build the alert on those three line items rather than a single total; asking the right question at 3 a.m. is worth more than answering quickly.

The rename trap: EXDEV

This is the part nobody thinks about until they see the error message. In Docker's words, calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer; otherwise it returns EXDEV. The kernel documentation likewise notes that renaming a directory that is on the lower layer or merged can fail with EXDEV.

The kernel offers redirect_dir for this: when enabled, the directory is copied up (but not its contents) and the trusted.overlay.redirect xattr is set to the original location. With it off, the behaviour is plainly EXDEV.

Getting the scope right matters, because it is easy to attach this error to the wrong mechanism. The overlayfs restriction is specific to directories; moving an ordinary file from a temporary location into place does not produce EXDEV — copy_up is triggered and the operation completes. The second and far more common source of "invalid cross-device link" in the container world is something else entirely: the move crosses a volume or bind-mount boundary, where there really are two different filesystems. Same error message, two different stories. Hunting for mount points for hours without knowing which one you're in is one of the classic rites of passage in this line of work.

metacopy and data-only layers: why they aren't the default

"If the whole file gets copied, isn't there a mode that doesn't copy it?" There is: metacopy. When enabled, OverlayFS copies up only the metadata rather than the whole file when a metadata-specific operation such as chown/chmod is performed. In scenarios where image setup changes ownership on thousands of files, the saving is significant.

But set your expectations correctly: this is a deferral, not an exemption. In the documentation's words, the data will be copied up later when the file is opened for a write operation. So the moment you actually write, you pay the full copy_up price anyway.

The most visible reason it isn't the default is security. The kernel documentation warns directly: do not use metacopy=on with untrusted upper/lower directories, otherwise an attacker can create a handcrafted file with appropriate REDIRECT and METACOPY xattrs and gain access to the file on lower pointed to by REDIRECT. There is an incompatibility on top of that: the redirect_dir and nfs_export options conflict with metacopy=on.

The newer member of the same family is data-only layers. They are defined with a double-colon (::) separator, and the paths of files in those layers are not visible in the merged directories. According to the documentation, defining at least one data-only layer is enough to enable redirection of data to that layer without explicitly setting metacopy=on. Since kernel 6.8, the fsconfig-based mount API accepts layers as lowerdir+ and datadir+; kernel 6.15 added the override_creds mount option.

These are not knobs you turn by hand in day-to-day operations. But they explain why image build tools produce such different disk profiles from one another — and the day you read "we use metacopy" in a tool's release notes, you will know that isn't a free speed win but an assumption about trust.

Putting the limit in the right layer

Once you know the mechanism, the list of fixes gets short. I'm writing them in order of importance:

Move write traffic out of the writable layer. This is Docker's own recommendation: volumes provide the best and most predictable performance for write-heavy workloads, because they bypass the storage driver. Database files, uploaded content, large temporary files — all to volumes. For short-lived temporary files, --tmpfs /tmp goes one step further: the data is written to memory, so it spends your memory budget rather than your disk. On a node with swap there is no unconditional "it never touches disk" guarantee either; you pay the cost in memory instead of disk.

Don't write logs to files. Rotating logs inside the container may free space in the writable layer, but it doesn't make that space manageable: the rotated copies pile up in the same layer, they are invisible from the node side, and if the log path was baked into the image you paid the copy_up price on the first write too. Let the application write to stdout and leave rotation to the runtime's logging driver.

If you need a hard limit, prepare the infrastructure for it. docker run --storage-opt size=... constrains the writable layer, but it isn't free: in the documentation's words, for the overlay2 storage driver the size option is only available if the backing filesystem is xfs and mounted with the pquota mount option. You earn that limit when you build the server, not when you run the container. Since this is a graph-driver feature, verify it against your own setup before applying the same recipe on a daemon that moved to the containerd image store. If the infrastructure won't give you that, the fallback is cruder but works: put Docker's data directory on a separate block device, cap the logging driver by size, and define disk thresholds at the node level.

In Kubernetes the resource name is ephemeral-storage. The scope is wider than you might assume: what the kubelet measures is the writable layers of running containers, directories holding node-level logs and non-tmpfs emptyDir volumes; the Kubernetes documentation also counts the writable layer together with container images, the Pod's own logs (usually under /var/log/pods) and system files mapped into the Pod such as /etc/hosts. The consequence of exceeding the limit is, in the documentation's words, this: if a Pod is using more ephemeral storage than you allow it to, the kubelet sets an eviction signal that triggers Pod eviction. The subtlety is this: it is not a quota, it is an eviction trigger. The write isn't blocked; the Pod is killed.

resources:
  requests:
    ephemeral-storage: "1Gi"
  limits:
    ephemeral-storage: "2Gi"

Questions to ask your own setup

• What does docker info say: classic overlay2, or the containerd snapshotter? Are your alerts and limits built on that answer?
• Which container's writable layer grew in the last week? Is that number graphed anywhere, or do you find out when the disk fills?
• Is there a large file baked into the image that gets written to? If so, what would it cost to move it to a volume?
• Do application logs go to a file? Is rotation happening inside the container?
• Is the filesystem holding Docker's data xfs with pquota? If not, hard size limits are not an option for you today.
• On the Kubernetes side, are ephemeral-storage requests and limits defined on your Pods? If they are, does everyone on the team know that this is an eviction decision?
• Is there a secret that was deleted from the image at some point? None of those RUN rm lines cleaned it up.

What's ephemeral is almost never cheap

If I had to compress this piece into one sentence: a container's writable layer is not a filesystem, it is an accounting trick — and the bill for the trick is issued at file granularity.

OverlayFS isn't doing anything wrong here. On the contrary, this design is exactly what lets image layers be shared and hundreds of containers fit on the same disk. But the cheapness the design offers is on the read side; on the write side it promises you nothing. Nobody ever committed to "the writable layer will stay thin" — we assumed it.

I've written before about how disk exhaustion advances as a silent crisis; in containers that silence runs one layer deeper, because what's growing isn't even the application's data — it's a copy of the data the application touched. It's worth revisiting the piece on Docker storage battles with this in mind as well: every command in it now depends on the answer to "which backend am I on."

Which is why the best defence isn't an alert but a design decision: from day one, keep every byte that's meant to persist outside the writable layer.

Official Sources

• Linux Kernel — Overlay Filesystem
• Docker Docs — Use the OverlayFS storage driver
• Docker Docs — containerd image store
• Docker Docs — Docker Engine 29 release notes
• Docker Docs — Deprecated Engine Features
• Docker Docs — docker container run reference
• Docker Docs — docker container ls reference
• Docker Docs — docker system df reference
• Kubernetes — Local ephemeral storage
• Kubernetes — Resource Management for Pods and Containers