DEV Community

Sreekanth Kuruba
Sreekanth Kuruba

Posted on

Docker Internals Deep Dive: What Really Happens When You Run docker run (2025 Edition)

#docker #devops #linux #architecture

Modern container platforms depend on predictable, modular behavior. Docker's architecture is a layered execution pipeline built around standard interfaces: REST, gRPC, OCI Runtime, and Linux kernel primitives. Understanding this flow eliminates ambiguity during debugging, scaling, or integrating with orchestration systems.

1. Core Architecture

CLI → dockerd (API + Orchestration) → containerd (Runtime mgmt)

→ containerd-shim (Process supervisor) → runc (OCI runtime)

→ Linux Kernel (Namespaces, cgroups, fs, net)

Docker CLI

  • User command interface
  • Converts flags to JSON
  • Talks to dockerd through /var/run/docker.sock

dockerd

  • REST API server
  • Container lifecycle orchestration
  • Network/volume management
  • Delegates image and runtime operations to containerd

containerd

  • High-level runtime manager
  • Manages snapshots, images, and content store
  • Pulls/unpacks layers
  • Creates OCI runtime specifications
  • Launches a containerd-shim for each container

Image Storage Detail:

Each layer is content-addressable via SHA256

Identical layers are deduplicated

OverlayFS uses hardlinks so layers are shared across containers

containerd-shim

  • Parent process for the container's workload
  • Keeps containers alive if dockerd/containerd restart
  • Manages IO streams (logs, attach)
  • Returns exit codes to containerd

runc

  • Implements the OCI runtime spec
  • Creates namespaces
  • Applies cgroup limitations
  • Mounts root filesystem
  • Executes the entrypoint
  • Exits immediately after container creation

Linux Kernel

  • Enforces process isolation (namespaces)
  • Resource control (cgroups)
  • Layered filesystems (OverlayFS)
  • Networking (veth, bridges, iptables/NAT)

✈️ The Airport Analogy: A Mental Model

Just as you don't need to know air traffic control to board a flight,
you don't need to understand all Docker components to run containers.
But when things go wrong, knowing the layers helps troubleshoot!

Component Airport Role Real-World Impact
Docker CLI Passenger Terminal You type docker run, check status
dockerd Airport Operations Center Manages all flights, gates, schedules
containerd Ground Control Loads luggage (images), assigns runways
containerd-shim Gate Agents Ensures plane stays ready even if Ops Center reboots
runc Pilot Actually flies the plane (executes container)
Kernel Air Traffic Control Manages airspace (resources), prevents collisions
Container The Actual Flight Your app running in isolated airspace

Use this mental model to remember component relationships during troubleshooting.

2. Execution Flow: docker run -d -p 8080:80 nginx

Step 1. CLI → dockerd

CLI parses command, constructs a JSON payload, and sends it over the Unix socket.

Step 2. dockerd Validation

Dockerd validates configuration, checks local images, and coordinates container creation.

Step 3. Image Pull (if needed)

containerd handles:

  • Registry authentication
  • Manifest resolution
  • Layer download and verification
  • Storage in the content store

Step 4. Filesystem Assembly

containerd prepares:

  • Snapshot
  • OverlayFS upper/lower directory layout
  • OCI bundle with metadata and runtime config

Step 5. Networking Setup

Dockerd configures the network namespace:

  • veth pair creation
  • Host end added to docker0
  • Container assigned IP (e.g., 172.17.0.2)
  • iptables DNAT for port-mapping
  • MASQUERADE rule for outbound traffic

Step 6. containerd → containerd-shim

containerd:

  • Spawns shim
  • Hands off the OCI spec
  • Delegates lifecycle supervision

Step 7. shim → runc

runc:

  • Creates namespaces
  • Mounts rootfs
  • Applies cgroup limits
  • Executes container entrypoint
  • Exits (shim remains as supervisor)

Step 8. Container Running

Container runs as an isolated Linux process:

  • shim maintains lifecycle
  • dockerd streams logs and reports state
  • kernel enforces isolation

Docker workflow

3. Component Responsibilities

Component Role Delegates
CLI User interface, request creation dockerd
dockerd API, orchestration, networking containerd
containerd Image mgmt, snapshots, lifecycle runc
containerd-shim Supervises container process kernel (via runc-created namespaces)
runc Creates container environment kernel
Kernel Isolation + resource control hardware

Related Architecture:
For Kubernetes, replace:

dockerd  →  kubelet → CRI → containerd
Enter fullscreen mode Exit fullscreen mode

Everything downstream (containerd → shim → runc → kernel) remains unchanged.

4. Key Clarifications

  • Containers are processes, not virtual machines.
  • runc does not stay resident; shim manages the lifecycle.
  • Docker's layered filesystem is copy-on-write for efficient storage.
  • Kubernetes removed dockerd and uses containerd directly for a simpler CRI pipeline.
  • Live-restore works because shim decouples containers from dockerd.

5. Debugging Guide (Ops-Ready Edition)

A structured, layered sequence for diagnosing container failures. Designed for SRE, DevOps, and runtime engineering teams.

Container exits immediately

Follow the layers from highest to lowest impact.

1. Application Layer

Severity: Low
Most failures originate here.

docker logs <container>
Enter fullscreen mode Exit fullscreen mode

Looks for: runtime exceptions, crash loops, missing configs, entrypoint failures.

2. Runtime Layer (containerd / OCI)

Severity: Medium
Issues here affect container creation, not app logic.

journalctl -u containerd
Enter fullscreen mode Exit fullscreen mode

Detects:

  • Invalid OCI specs
  • Snapshot/unpack errors
  • Permission issues
  • Image metadata failures

3. Kernel Layer

Severity: High
Kernel failures affect all containers on the node.

dmesg | tail -20
Enter fullscreen mode Exit fullscreen mode

Reveals:

  • Namespace creation failures
  • cgroup enforcement errors
  • LSM blocks (AppArmor/SELinux)
  • OverlayFS mount issues

Slow container startup

Pinpoint latency at the registry, storage, or runtime.

1. Image Pull / Unpack Latency 

journalctl -u containerd --since "2 minutes ago" | grep -Ei "pull|unpack"
Enter fullscreen mode Exit fullscreen mode

Finds slow remote pulls, layer unpack delays, decompression problems.

2. Host Storage Bottleneck 

iostat -dx 1 /var/lib/containerd
Enter fullscreen mode Exit fullscreen mode

Detects:

  • High I/O wait
  • OverlayFS backing store saturation
  • Slow disks or overloaded volumes

3. Registry / Network Slowness 

time docker pull alpine:latest
Enter fullscreen mode Exit fullscreen mode

Measures:

  • Round-trip latency
  • Download throughput
  • Registry auth or proxy delays

Network issues

Trace connectivity host → bridge → container.

1. Verify NAT / Port Forward Rules 

sudo iptables -t nat -L DOCKER -n -v
Enter fullscreen mode Exit fullscreen mode

2. Bridge & veth Topology 

ip addr show docker0
brctl show
Enter fullscreen mode Exit fullscreen mode

3. Container Namespace Networking 

docker exec <id> ip addr show
Enter fullscreen mode Exit fullscreen mode

Common Error Patterns

A quick pattern-matching cheat sheet.

Error Message Likely Cause
no such file or directory Missing entrypoint or wrong working dir
permission denied User namespace restriction, volume permissions
address already in use Host port collision
exec format error Architecture mismatch (amd64 vs arm64)
layer does not exist Corrupted image store, partial pull
failed to setup network namespace Kernel lacking required capabilities

Recovery Actions

Map root cause to corrective steps.

1. Image Pull Failures

  • Check registry auth tokens
  • Verify proxy/SSL configuration
  • Test connectivity to registry endpoints

2. OCI Spec / Runtime Errors

  • Ensure Docker + containerd + runc versions are compatible
  • Validate custom seccomp or AppArmor profiles
  • Recreate corrupted snapshots

3. Kernel Namespace / Cgroup Failures

  • Check kernel version supports required features
  • Validate cgroup v1/v2 mode
  • Inspect sysctl overrides affecting namespaces

DEBUGGING TREE IMAGE

6. Summary

A docker run invocation travels through a disciplined, modular execution path. Each component accepts a small, well-defined piece of responsibility and hands off cleanly to the next, forming a predictable control flow:

  • Dockerd parses intent and translates it into runtime instructions.
  • Containerd orchestrates container lifecycle through stable gRPC APIs.
  • Containerd-shim isolates the container’s process management from daemon restarts.
  • runc materializes the OCI Runtime Spec into Linux primitives.
  • The kernel provides the final enforcement layer through namespaces, cgroups, and filesystem drivers.

These boundaries are governed by open standards (REST → gRPC → OCI Spec → syscalls), ensuring compatibility, reliability, and deep observability across layers.

Isolation, resource governance, and performance efficiency emerge directly from native Linux constructs—no hidden hypervisor, no extra abstraction. As a result, containers start fast, run lean, and scale predictably.

Operational Note:
Because process ownership is delegated to containerd-shim, both dockerd and containerd can be restarted without disrupting running containers. This design supports safe daemon upgrades, node maintenance, and high-availability workflows that do not interrupt workloads.

  1. Core Architecture
  2. Execution Flow
  3. Component Responsibilities

  4. Key Clarifications

  5. Debugging Guide (Ops-Ready Edition)
    └── [DEBUGGING TREE IMAGE]
    └── Container exits immediately
    └── Slow container startup
    └── Network issues
    └── Common error patterns
    └── Recovery actions

  6. Summary

Drop your thoughts in the comments below! 👇
Follow me for more deep dives into fundamental CS concepts made approachable!

Top comments (0)