S M Tahosin

Posted on Oct 22

Docker Containers Aren't Virtual Machines: How Namespaces, cgroups, and OverlayFS Actually Work Under the Hood

#docker #devops #linux #architecture

Ask almost any developer to explain what a Docker container is, and you will usually get a diagram contrasting two towers:

On the left sits a Virtual Machine: an Operating System, a Guest Kernel, a Hypervisor, and virtual hardware drivers weighing several gigabytes.

On the right sits a Container: a slender box labeled "App and Dependencies" sitting directly on top of the "Host OS and Docker Engine."

The conclusion is invariably recited like gospel:

"Virtual machines virtualize the hardware. Containers virtualize the operating system. Containers are just lightweight virtual machines."

It is a convenient mental model for slide decks. It is also completely wrong.

There is no such thing as a "container" inside the Linux kernel. If you search the Linux kernel source code for a data structure named struct container, you will find nothing.

To prove it, open a terminal on any Linux server, launch an Nginx container with Docker, and run ps aux | grep nginx directly on the host machine:

$ docker run -d --name web -p 8080:80 nginx:alpine
3c5b8e919...

$ ps aux | grep nginx
root      4821  0.0  0.1  10240  4128 ?  Ss   14:20   0:00 nginx: master process nginx -g daemon off;
101       4865  0.0  0.0  10696  2856 ?  S    14:20   0:00 nginx: worker process

There is no hypervisor running. There is no guest kernel running.

Those Nginx processes are executing directly on your host CPU, scheduled by your host Linux kernel, using your host physical RAM.

A container is not a tiny virtual machine.

A container is an ordinary Linux process running in user space, wearing a pair of blinders (Namespaces), operating on a strict allowance (Control Groups), and looking at a layered illusion of files (OverlayFS).

When you run docker run, Docker does not boot an environment. It orchestrates three native Linux kernel features that have existed for decades, wraps them in a standardized configuration, and hands execution over to a low-level runtime.

Once you peel away the marketing buzzwords and inspect the mechanical nuts and bolts, container debugging stops feeling like black magic.

Here is what is really happening beneath your terminal when you execute a container.

1. The OCI Runtime Hierarchy: From `docker run` to `runc`

When developers say "Docker," they usually imagine a single monolithic application that downloads images, configures IP addresses, and runs processes.

In early versions of Docker (circa 2013), that was largely true. Everything lived in a giant root daemon called dockerd.

However, running production infrastructure on a monolithic daemon created severe stability issues. If the Docker daemon crashed or needed an update, every single running container on the host abruptly died with it. Furthermore, the industry needed open standards so vendors could build interoperable tooling.

Today, the Open Container Initiative (OCI) defines clear specifications for container images and runtimes. Docker has decomposed into a modular hierarchy of specialized layers:

[ Developer CLI: docker ]
           │
           │ (REST over UNIX Domain Socket: /var/run/docker.sock)
           ▼
[ Daemon: dockerd ] ── Image management, network plugins, volume drivers
           │
           │ (gRPC)
           ▼
[ Supervisor: containerd ] ── Image pull/unpack, lifecycle execution
           │
           │ (fork/exec)
           ▼
[ Shim: containerd-shim ] ── Decouples container process from daemon
           │
           ▼
[ OCI Runtime: runc ] ── Configures namespaces & cgroups, runs container
           │
           ▼
[ Your Application Process: e.g., nginx ]

Here is the exact sequence of events when you execute docker run:

Docker CLI translates your flags into an HTTP POST request and sends it over /var/run/docker.sock to the dockerd daemon.
dockerd verifies authentication, validates configurations, coordinates networking, and passes container creation down to containerd via a high-performance gRPC interface.
containerd pulls the container image if it does not exist locally, unpacks its tarball layers onto the disk using an OverlayFS snapshotter, and prepares a standard OCI config.json bundle.
containerd spawns a tiny helper process called containerd-shim.
containerd-shim invokes runc, passing the bundle path.
runc is a short-lived command-line tool written in Go. Its entire job is to interact directly with the Linux kernel: it calls system calls like clone(), unshare(), and pivot_root(), sets up cgroup limits, drops kernel privileges, and executes the target binary (e.g., nginx).
Once the container process starts, runc exits completely.

Why does containerd-shim exist between containerd and your application?

Without the shim, the container process would be a child of containerd. If containerd crashed or restarted during an upgrade, the kernel would send a SIGHUP signal to all child processes, terminating every running database and web server on your machine.

containerd-shim stays alive as the lightweight parent of your container process. It holds open the standard input/output file descriptors and reports exit codes back to containerd when the daemon restarts.

You can restart the Docker daemon without dropping a single active network connection on your containers.

2. The Illusion of Isolation: Linux Namespaces

If containers are just regular processes, why can't a container see processes running on the host machine or read files from other containers?

The answer is Linux Namespaces.

Introduced in Linux kernel 2.4.19 and expanded over the last twenty years, namespaces wrap global system resources in an isolated abstraction. Processes inside a namespace believe they own a dedicated instance of that resource.

Linux currently provides seven primary namespaces:

Namespace	Linux Flag	What It Isolates	The Mental Illusion
PID	`CLONE_NEWPID`	Process IDs	The container process believes it is PID 1 (init system)
NET	`CLONE_NEWNET`	Network devices, routing, iptables	Container gets its own loopback, eth0, and port space
MNT	`CLONE_NEWNS`	File system mount points	Container only sees its own root filesystem directory tree
UTS	`CLONE_NEWUTS`	Hostname and NIS domain	Container has its own independent hostname
IPC	`CLONE_NEWIPC`	System V IPC, POSIX message queues	Prevents shared memory crosstalk between containers
USER	`CLONE_NEWUSER`	User IDs and Group IDs	Container user UID 0 (root) maps to non-root UID on host
CGROUP	`CLONE_NEWCGROUP`	Control group hierarchy	Container only sees its own cgroup subgroup tree

Building a Container Manually (No Docker Required)

To prove that namespaces are standard Linux features rather than Docker creations, you can build a container by hand using a single Linux command: unshare.

If you run unshare with namespace flags on an Ubuntu host, you create isolated namespaces instantly:

# Create isolated UTS (hostname), PID (process), and MNT (mount) namespaces
$ sudo unshare --uts --pid --mount --fork /bin/bash

# Change the hostname inside this new shell
$ hostname my-custom-container
$ hostname
my-custom-container

# Exit back to the host shell
$ exit
$ hostname
prod-server-01

Changing the hostname inside the UTS namespace had zero effect on the host machine.

You can inspect the exact namespace IDs of any process on a Linux machine by reading the /proc virtual filesystem:

$ ls -l /proc/$$/ns
lrwxrwxrwx 1 root root 0 Oct 10 14:35 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Oct 10 14:35 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 root root 0 Oct 10 14:35 mnt -> 'mnt:[4026532224]'
lrwxrwxrwx 1 root root 0 Oct 10 14:35 net -> 'net:[4026531992]'
lrwxrwxrwx 1 root root 0 Oct 10 14:35 pid -> 'pid:[4026532225]'
lrwxrwxrwx 1 root root 0 Oct 10 14:35 uts -> 'uts:[4026532223]'

When two processes share the same inode number in /proc/<pid>/ns/, they live in the same namespace. When you run docker run --net=host, Docker simply attaches your container process to the host's network namespace inode instead of creating a new one.

3. Resource Walls: Control Groups (cgroups v1 and v2)

Namespaces determine what a process can see.

They do not determine what a process can use.

If you only use namespaces to isolate a process, a runaway while (true) loop in one container will consume 100 percent of your host CPU cores. A memory leak in another container will gobble up every gigabyte of physical RAM until the entire Linux machine freezes.

To prevent noisy neighbors from exhausting host resources, the Linux kernel provides Control Groups (cgroups).

Originally developed by Google engineers Paul Menage and Rohit Seth under the name "Process Containers" before being merged into Linux kernel 2.6.24, cgroups meter, throttle, and enforce resource limits on groups of processes.

In modern Linux distributions (such as Ubuntu 22.04+, Debian 11+, and Fedora), the Linux kernel has standardized on cgroups v2, which organizes all resources into a single unified hierarchy located at /sys/fs/cgroup/:

/sys/fs/cgroup/
├── cgroup.controllers
├── cgroup.procs
├── system.slice/
└── docker.slice/
    └── docker-3c5b8e919...scope/
        ├── cgroup.procs        # Contains the container PID (e.g., 4821)
        ├── cpu.max             # CPU quota and period limits
        ├── memory.max          # Hard memory ceiling
        ├── memory.current      # Current RAM consumption in bytes
        └── io.weight           # Disk I/O throttling weight

How CPU Throttling Works

When you set --cpus="0.5" in a Docker command, Docker does not assign half a physical CPU core to your process.

Instead, it configures the Linux Completely Fair Scheduler (CFS) using two values:

period: A fixed measurement interval (typically 100,000 microseconds, or 100ms).
quota: The total execution time your process is allowed to run within that interval.

For a 0.5 CPU limit, Docker calculates:

$$\text{Quota} = 0.5 \times 100{,}000 = 50{,}000\text{ }\mu\text{s}$$

It writes 50000 100000 directly into /sys/fs/cgroup/docker/<id>/cpu.max.

During each 100ms window, the Linux kernel lets your container process run on any available CPU core until it accumulates 50ms of CPU time. Once it hits that threshold, the kernel suspends the process until the next 100ms period begins.

This is why setting CPU limits on CPU-intensive workloads causes latency spikes and thread pauses rather than application crashes.

The Mystery of Exit Code 137: The OOM Killer

Memory behaves completely differently from CPU.

You can throttle CPU execution by making a thread wait. You cannot "throttle" RAM: when an application requests 10 megabytes of memory via malloc(), it needs those physical bytes immediately.

When you pass --memory="512m" to docker run, Docker writes 536870912 into the container's memory.max file.

If your application allocates memory beyond 512 megabytes, the Linux kernel invokes the Out-Of-Memory (OOM) Killer:

The kernel checks the container's oom_score.
It immediately dispatches an uncatchable SIGKILL (signal 9) directly to the application process.
The process terminates instantly without executing cleanup hooks or shutdown handlers.

In Docker and Kubernetes, the container exit status is calculated as:

$$\text{Exit Code} = 128 + \text{Signal Number} = 128 + 9 = 137$$

Whenever you see a container exit with code 137, you now know the exact kernel mechanism that killed it: cgroups strictly enforced memory.max and invoked the OOM killer.

4. The Layer Cake: OverlayFS and Copy-on-Write

We have seen how processes are isolated with namespaces and constrained with cgroups.

Now comes the biggest puzzle of all: the filesystem.

If you run twenty instances of an Ubuntu container on a single host machine, does Docker duplicate a 100-megabyte Ubuntu filesystem twenty times on your solid-state drive?

If it did, twenty containers would waste two gigabytes of storage, and launching each container would require copying gigabytes of data before the process could start.

Docker solves this using a union filesystem called OverlayFS.

An OverlayFS mount does not store files like a traditional disk partition. Instead, it combines multiple underlying directories into a single unified directory tree:

[ Unified View: merged ]  <--- What the container process actually sees
          ▲
          │ (Union Mount)
   ┌──────┴──────────────────────────┐
   │                                 │
[ upperdir ]                   [ lowerdir ]
Writable Container Layer       Read-Only Image Layers (Stacked)
(Created on docker run)        (Layer 3: App binary)
                               (Layer 2: Python runtime)
                               (Layer 1: Base Alpine/Debian)

The Four Directories of OverlayFS

Every OverlayFS mount is composed of four distinct directory paths managed inside /var/lib/docker/overlay2/:

lowerdir: One or more read-only directories containing the base image layers. These directories are immutable and shared across every container spawned from that image.
upperdir: A single read-write directory unique to the running container. Any file created, modified, or deleted while the container runs lives here.
workdir: An internal staging directory used by the Linux kernel to ensure atomic updates before writing to the upperdir.
merged: The mount point where the kernel projects the unified combination of lowerdir and upperdir. This is the exact root path passed to chroot or pivot_root during container initialization.

Copy-on-Write (CoW) in Action

When your container performs file operations, OverlayFS handles them using specific kernel heuristics:

Reading a file: The kernel searches upperdir. If the file is not there, it searches down through the lowerdir layers. As soon as it finds the file, it serves it directly from disk cache.
Creating a new file: The file is written straight into upperdir. The read-only base image layers are untouched.
Modifying an existing file: If an application opens a file from a read-only layer with write permissions (e.g., modifying /etc/nginx/nginx.conf), the kernel triggers Copy-on-Write. It copies the entire file from the read-only lowerdir up into upperdir, and applies the changes to the copy. The original file in the base image remains pristine.
Deleting a file: Because lowerdir is read-only, the kernel cannot physically erase the file from disk. Instead, it creates a special whiteout device (a character device with major and minor numbers set to 0,0) inside upperdir. When the container lists the directory, the whiteout device instructs OverlayFS to hide the underlying file from the merged view.

This design explains why starting a container takes milliseconds: Docker does not copy a filesystem; it simply mounts an OverlayFS view on top of existing read-only directories.

It also highlights a critical performance rule: never perform heavy disk writes to large files inside standard container layers.

If your application modifies a 500-megabyte file that exists in an image layer, OverlayFS must copy all 500 megabytes up into the upperdir before executing a single byte change. For heavy I/O operations, always use Docker Volumes, which bypass OverlayFS and write directly to host filesystem mounts.

5. Virtual Plumbing: Container Networking and iptables

How does a container get its own IP address, listen on port 80, and talk to external APIs without a physical network card?

Docker achieves this by combining Network Namespaces, Virtual Ethernet (veth) pairs, a Linux Bridge (docker0), and iptables NAT rules:

[ Host Physical Network Interface: eth0 (192.168.1.50) ]
                          │
                   (iptables NAT)
                          │
[ Linux Bridge: docker0 (172.17.0.1) ]
         │                               │
    (veth pair A)                   (veth pair B)
         │                               │
         ▼                               ▼
[ Container 1 Network NS ]      [ Container 2 Network NS ]
  eth0: 172.17.0.2                eth0: 172.17.0.3
  Default Gateway: 172.17.0.1     Default Gateway: 172.17.0.1

The Virtual Ethernet Cable (`veth` Pair)

When Docker starts a container in default bridge mode, it does not assign your physical network card to the container.

Instead, the Linux kernel creates a Virtual Ethernet pair (often abbreviated as veth).

Think of a veth pair as a bidirectional virtual fiber-optic cable with two ends:

One end (named something like veth3a9b1c) stays in the Host Network Namespace and plugs into a software virtual switch called docker0.
The other end is moved across the namespace boundary into the Container Network Namespace, where it is renamed to eth0 and assigned an IP address (like 172.17.0.2).

Any network packet entering one end of the virtual cable immediately emerges from the other end.

How Port Publishing Actually Works

When you run docker run -p 8080:80 nginx, how does traffic arriving on your host IP at port 8080 reach port 80 inside the container's isolated network namespace?

Docker does not write code to listen on port 8080 and forward raw TCP sockets.

Instead, Docker configures the Linux kernel packet filtering framework (Netfilter) by injecting rules into iptables:

$ sudo iptables -t nat -L -n -v
Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source       destination
    0     0 DNAT       tcp  --  !docker0 *     0.0.0.0/0    0.0.0.0/0    tcp dpt:8080 to:172.17.0.2:80

Notice the target: DNAT (Destination Network Address Translation).

When a client sends a TCP SYN packet to your host machine on port 8080:

The packet hits the host physical network interface (eth0).
The Linux kernel Netfilter hook intercepts the packet before routing.
The iptables rule rewrites the packet destination IP from 192.168.1.50:8080 to 172.17.0.2:80.
The host kernel looks at the new destination IP, consults its internal routing table, and forwards the packet onto the docker0 bridge.
The docker0 bridge transmits the packet down the veth pair into the container's network namespace, where Nginx receives it on port 80.

When the container replies, Netfilter uses Source NAT (MASQUERADE) to rewrite the source IP back to the host IP, ensuring the client receives a coherent response.

6. Three Production Gotchas Every Engineer Must Know

Now that you understand the underlying mechanisms (runc, namespaces, cgroups, OverlayFS, and veth networking), three common production anomalies suddenly make complete sense.

1. The PID 1 Zombie Apocalypse

In Linux, Process ID 1 (traditionally systemd or init) has two unique responsibilities:

It handles termination signals (like SIGTERM and SIGINT) and gracefully shuts down child services.
It acts as the foster parent for orphan processes, automatically harvesting zombie processes when children terminate.

When you configure your Dockerfile with:

# Naive approach
CMD ["npm", "start"]

Node or npm becomes PID 1 inside the container's PID namespace.

Most application runtimes (including Node.js, Python, and Java) were never designed to act as an operating system init system. They do not register default signal handlers for SIGTERM, nor do they reap orphaned zombie processes.

When you run docker stop, Docker sends a SIGTERM signal to PID 1. If your process ignores it, Docker waits ten seconds, gives up, and violently terminates the process with SIGKILL. Your database connections are severed, files remain half-written, and logs are truncated.

The Fix: Use a lightweight init supervisor like tini:

# Safe production approach
RUN apk add --no-cache tini
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]

tini assumes PID 1, handles signal forwarding correctly, reaps zombies, and ensures clean ten-millisecond shutdowns.

2. The Fallacy of `--privileged`

Because containers feel like virtual machines, developers struggling with permission errors often search online and find a dangerous suggestion:

"Just add --privileged to your docker run command."

Never do this in production.

When you pass --privileged:

Docker drops all AppArmor and seccomp system call filters.
It grants all 38 Linux kernel capabilities (including CAP_SYS_ADMIN).
It mounts all host device nodes directly into /dev inside the container.

Because the container shares the host Linux kernel, a process running as root inside a --privileged container can execute mknod or mount, access the host hard drive partition (/dev/sda1), overwrite the host /etc/shadow file, and escape into the host operating system in less than two seconds.

A container is not a sandbox. It is an isolated process sharing your kernel. Keep privilege boundaries tight.

3. Java and Node.js Core Count Misconceptions

Historically, runtimes like the JVM and Node.js queried hardware resources by reading /proc/cpuinfo and /proc/meminfo.

Because namespaces do not virtualize /proc/cpuinfo, an application running inside a container with --cpus="1" on a 64-core server would inspect /proc/cpuinfo, believe it had 64 available cores, and spawn 64 parallel garbage collection threads.

Those 64 threads would immediately collide with the 1-core cgroup quota, triggering severe CPU throttling and catastrophic latency spikes.

Modern runtimes (Java 11+, Node.js 16+, and Go 1.19+) have been rewritten to inspect /sys/fs/cgroup/ directly. Always verify that your base image runtime version is container-aware.

The Mental Shift

The next time you type docker run, take a moment to appreciate the engineering symphony unfolding beneath your keyboard:

You are not booting an operating system. You are not starting a hypervisor.

Your shell talks to a daemon. That daemon coordinates with a supervisor. The supervisor delegates to a focused CLI tool called runc.

runc asks the Linux kernel to create seven namespaces, allocates a cgroup slice, mounts a layered OverlayFS directory, plugs in a virtual network cable, and transforms into your application process.

The container is an illusion; the Linux kernel is the reality.

When you understand the primitives beneath the container, you stop treating Docker like a black box. You can diagnose OOM crashes in seconds, configure network routing without guessing, tune CPU quotas with precision, and architect production systems with confidence.

What was the most surprising container behavior you ever encountered in production? Did you ever watch an application get wiped out by the OOM killer or frozen by the PID 1 signal trap? I would love to hear your experiences and architecture patterns in the comments below.