DEV Community: Muktadir M Aashif

Modern Linux Systems: gRPC, REST, and Process Communication

Muktadir M Aashif — Wed, 29 Apr 2026 16:54:37 +0000

In today’s cloud-native landscape, understanding how systems communicate—and how they’re built—is essential for engineers. This post synthesizes key concepts around gRPC vs. REST, Linux process communication, and the role of Rust in the modern Linux kernel, clarifying common misconceptions and highlighting real-world architectures.

gRPC vs. REST: Choosing the Right Protocol

At the heart of microservices lies a fundamental design decision: how should services talk to each other?

Core Differences

REST (Representational State Transfer) is resource-oriented, using HTTP verbs (GET, POST, PUT) over JSON or XML. It’s human-readable, widely supported, and ideal for public APIs and browser clients.

gRPC, by contrast, is action-oriented. Built on HTTP/2 and Protocol Buffers (protobuf), it enables:

Binary serialization (60–80% smaller payloads than JSON)
Native streaming (server, client, and bidirectional)
Strong typing via compile-time code generation
Lower latency and higher throughput

Recent 2026 benchmarks show gRPC delivering 107% higher throughput and 48% lower latency than REST in high-load scenarios [[9]]. Another study found 45% average latency reduction with gRPC in internal service meshes [[10]].

When to Use Which?

Use Case	Recommended
Public APIs, browser clients	✅ REST (or GraphQL)
Internal microservices	✅ gRPC
Real-time telemetry, chat, live updates	✅ gRPC (streaming)
Simple CRUD with broad tooling needs	✅ REST

Many organizations adopt a hybrid model: expose REST externally via an API gateway while using gRPC internally for performance-critical paths.

How Docker and containerd Use gRPC (and What They Don’t)

A frequent point of confusion: Does the docker CLI use gRPC?

No. The Docker client communicates with the dockerd daemon via a RESTful API over a Unix socket (/var/run/docker.sock) [[26], [31]]. This is standard HTTP/JSON—not gRPC.

However, beneath Docker lies containerd—and that’s where gRPC shines.

The containerd gRPC Architecture

containerd is the industry-standard container runtime powering both Docker and Kubernetes. It exposes a full gRPC API over a Unix socket (/run/containerd/containerd.sock) [[17], [19], [25]].

Key services include:

Tasks: Manage container lifecycle (create, start, kill)
Images: Pull, push, and manage container images
Snapshots: Handle layered filesystems

Each running container is managed by a containerd-shim process, which also communicates with containerd via gRPC [[18], [22]]. This design allows containers to survive daemon restarts—a critical reliability feature.

Similarly, BuildKit, Docker’s modern builder, uses gRPC for advanced features like parallel builds, remote caching, and secret injection [[25]].

💡 Takeaway: gRPC is used within the container stack, not by the Docker CLI itself. It’s a deliberate choice for performance and streaming in infrastructure components.

Linux Process Communication: Sockets ≠ gRPC

Linux provides several Inter-Process Communication (IPC) mechanisms:

Unix domain sockets
TCP/UDP sockets
Pipes and FIFOs
Shared memory
Signals

These are kernel-provided primitives—low-level building blocks. gRPC is not one of them.

gRPC is a user-space framework that can run over a Unix socket or TCP connection—but so can REST, Thrift, or a custom binary protocol. The socket is just the transport; the protocol is the language spoken on top.

For example:

sshd uses a custom text-based protocol over sockets
systemd uses D-Bus (not gRPC)
nginx talks HTTP/JSON to upstream services

Only when an application explicitly chooses gRPC (like containerd or Kubernetes’ kubelet) does gRPC appear in the communication flow.

Rust in the Linux Kernel: Safety Over Speed

As of 2026, Rust is now a permanent, non-experimental part of the Linux kernel [[1], [2], [6]]. But this shift isn’t about raw performance—it’s about memory safety.

Why Rust?

Historically, ~70% of kernel vulnerabilities stemmed from memory errors in C code: buffer overflows, use-after-free bugs, null pointer dereferences [[19]]. Rust’s ownership model eliminates these at compile time.

Early data shows up to 75% fewer memory-related bugs in Rust-written drivers compared to equivalent C modules [[1]].

Performance Impact?

Rust compiles to machine code as efficient as C. The “speed” gain is indirect:

Fewer crashes → higher system uptime
Less need for CPU-intensive mitigations (like KASLR)
Cleaner abstractions enable better optimization

Importantly, Rust in the kernel has nothing to do with gRPC. The kernel operates in privileged space; gRPC runs in user space and depends on kernel services (like sockets)—not the other way around [[12]].

🔒 Bottom line: Rust makes the kernel more secure and maintainable, not inherently “faster.” And it doesn’t change how processes communicate—unless those processes are rewritten in Rust and choose gRPC.

The Bigger Picture: A Layered Stack

Modern systems stack these technologies cleanly:

┌───────────────────────┐
│   Applications        │ ← May use gRPC or REST (user space)
├───────────────────────┤
│   Container Runtime   │ ← containerd (gRPC API over Unix socket)
├───────────────────────┤
│   Linux Kernel        │ ← Now includes Rust modules (memory-safe drivers)
└───────────────────────┘

gRPC is a communication choice for applications.
Rust is a language choice for safer systems code.
Sockets are the transport layer provided by the OS.

They coexist—but operate at different layers.

Final Thoughts

Understanding these distinctions empowers you to:

Design efficient service architectures (REST externally, gRPC internally)
Debug container runtimes with confidence
Appreciate why Rust matters for long-term system security
Avoid conflating transport (sockets) with protocol (gRPC)

As cloud-native systems grow more complex, clarity about what runs where becomes invaluable. Whether you’re debugging a slow API, securing a kernel module, or optimizing container startup times—knowing the stack is half the battle.

References

Rust in Linux kernel (2026): [[1]], [[2]], [[6]]
gRPC vs. REST benchmarks: [[9]], [[10]], [[12]]
containerd gRPC architecture: [[17]], [[19]], [[25]]
Docker daemon communication: [[26]], [[31]], [[32]]

This post reflects the state of systems engineering as of April 2026.

A Deep Dive into OverlayFS, Namespaces, and the Magic of {}

Muktadir M Aashif — Wed, 29 Apr 2026 16:46:45 +0000

If you’ve ever run docker run hello-world, you’ve witnessed magic. A container spins up in milliseconds, isolated from your host, with its own filesystem, network, and process tree. But what actually happens when you hit Enter?

As DevOps engineers, we often treat Docker as a black box. We write YAML, push images, and hope for the best. But when things break—when a container won’t start, when disk space vanishes, or when networking fails—knowing the internals isn’t just "nice to have." It’s survival.

In this deep dive, we’ll peel back the layers of Docker. We’ll explore how Linux kernel primitives like OverlayFS and Namespaces power containers, decode the mysterious {} syntax in docker inspect, and learn how to troubleshoot like a pro.

1. The Image: It’s Not a File, It’s a Stack

Most people think a Docker image is a single binary. It’s not. An image is a stack of read-only layers.

How It Works

Every instruction in your Dockerfile (RUN, COPY, ADD) creates a new layer. These layers are stored in a content-addressable storage system (usually under /var/lib/docker/overlay2/).

Union File System: Docker uses a Union File System (specifically OverlayFS) to merge these layers into a single view.
Content Addressability: Each layer is identified by a SHA256 hash. If two images share the same base (e.g., ubuntu:22.04), they share the same disk space. No duplication.

The JSON Truth

You can see this structure using docker image inspect. The output is a JSON object containing:

RootFS: A list of layer hashes.
History: The build steps that created each layer.

docker image inspect nginx:latest | jq '.[0].RootFS.Layers'

Key Insight: Images are immutable. You never change an image; you only create new ones with additional layers on top.

2. The Container: Just a Process, But Isolated

A running container is simply a process on your host Linux kernel. But it’s a special process, wrapped in isolation primitives.

The Linux Kernel Primitives

Docker doesn’t invent virtualization; it leverages existing Linux features:

Namespaces: Provide isolation.
- PID: The container sees only its own processes.
- NET: Its own network stack (IPs, ports).
- MNT: Its own filesystem view.
- UTS: Its own hostname.
Control Groups (cgroups): Limit resource usage (CPU, Memory, I/O). This prevents one container from starving the host.
OverlayFS Mount: The writable layer (the "container layer") is mounted over the read-only image layers.

Inspecting the Reality

When you run docker inspect <container_id>, you’re querying the Docker Daemon’s internal state database. The JSON output reveals the truth:

{
  "State": {
    "Status": "running",
    "Pid": 12345,
    "ExitCode": 0
  },
  "HostConfig": {
    "Memory": 536870912,
    "CpuShares": 1024
  }
}

Key Insight: docker run is essentially a wrapper around runc (the OCI runtime), which sets up namespaces, cgroups, and mounts, then executes your entrypoint.

3. Networking: Virtual Wires and Bridges

How does a container talk to the world? Through Linux Network Namespaces and Virtual Ethernet (veth) pairs.

The Default Bridge

By default, Docker creates a Linux bridge (docker0) on the host.

Each container gets a virtual interface (eth0) inside its network namespace.
This eth0 is connected via a veth pair to the docker0 bridge.
NAT (iptables): Outbound traffic is masqueraded so it appears to come from the host IP.

Inspecting Networks

docker network inspect bridge

This returns JSON showing which containers are attached, their IPs, and MAC addresses.

Key Insight: Containers on the same bridge can communicate via IP. For name resolution, Docker runs an embedded DNS server at 127.0.0.11.

4. The Power of `{{}}`: Mastering `docker inspect --format`

One of Docker’s most powerful but underused features is the --format (or -f) flag. It uses Go’s text/template engine to parse JSON output directly in the CLI.

Why Use It?

docker inspect returns massive JSON blobs. Often, you just need one field. Parsing JSON with jq is great, but --format is built-in and faster for simple queries.

Syntax Rules

{{ }}: Delimiters for template expressions.
.: Represents the current object.
{{.Field}}: Accesses a key.
{{.Parent.Child}}: Traverses nested objects.
{{index .Array 0}}: Accesses array elements (Go templates don’t support [0]).

Real-World Examples

Get Container IP:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-container

List Environment Variables:

docker inspect -f '{{range .Config.Env}}{{.}}{{"\n"}}{{end}}' my-container

Check if Running:

docker inspect -f '{{if .State.Running}}✅ Up{{else}}❌ Down{{end}}' my-container

Format Mounts Like JSON:

docker inspect -f '{{range .Mounts}}
{
  "Type": "{{.Type}}",
  "Source": "{{.Source}}",
  "Destination": "{{.Destination}}"
}
{{end}}' my-container

Pro Tip: Always wrap your template in single quotes ('{{...}}') to prevent the shell from interpreting special characters.

5. Storage Deep Dive: OverlayFS & Copy-on-Write

Where do files live? How does Docker save space? The answer is OverlayFS.

The Layer Cake

Imagine your image has 3 layers:

Base OS (ubuntu)
App Dependencies (nginx)
Your Code (app)

When you start a container, Docker adds a 4th layer: The Writable Layer.

Copy-on-Write (CoW)

Read: If a container reads a file from the image, it reads directly from the read-only layer. Fast.
Write: If the container modifies a file, Docker performs a copy-up:
1. Copies the file from the read-only layer to the writable layer.
2. Modifies the copy in the writable layer.
3. The original remains untouched.

⚠️ Performance Warning: CoW copies the entire file, not just changed blocks. Modifying a 1GB log file triggers a 1GB copy. This is why you should use Volumes for databases and heavy I/O.

Directory Structure

/var/lib/docker/overlay2/
├── <layer-hash>/
│   ├── diff/       # Files unique to this layer
│   ├── lower       # Link to parent layers
│   ├── merged/     # The unified view (mounted into container)
│   └── work/       # Internal workspace

6. Troubleshooting Like a Senior Engineer

When things go wrong, don’t guess. Inspect.

Step 1: Check Status

docker ps -a

Look at the STATUS and EXIT CODE.

0: Clean exit.
137: OOM Killed (Out of Memory).
125: Docker daemon error.

Step 2: Read Logs

Logs are stored in JSON files under /var/lib/docker/containers/<id>/<id>-json.log.

docker logs --tail 100 -f my-container

Step 3: Exec Into the Container

If the container is running but behaving strangely:

docker exec -it my-container /bin/sh

Now you can check files, network connectivity (curl, ping), and environment variables inside the namespace.

Step 4: Check Resources

docker stats

Real-time CPU, memory, and I/O usage. If a container is using 100% CPU, you’ll see it here.

Step 5: Disk Space

docker system df

Identify unused images, containers, and volumes. Clean up safely with:

docker system prune --dry-run  # Always dry-run first!

Conclusion

Docker is not magic. It’s a clever orchestration of Linux kernel features:

OverlayFS for efficient, layered filesystems.
Namespaces for isolation.
cgroups for resource control.
Go Templates for powerful CLI inspection.

Understanding these internals transforms you from a Docker user into a Docker engineer. You stop guessing why a container failed and start knowing exactly where to look.

Next time you run docker run, remember: you’re not just starting an app. You’re creating a namespace, mounting a union filesystem, and isolating a process—all in milliseconds.

📚 Further Reading

Happy Containerizing! 🐳

The Bandwidth Trap: Docker Registry VS Zot Registry

Muktadir M Aashif — Wed, 29 Apr 2026 16:37:19 +0000

Read Time: 8 Minutes

The Bandwidth Trap: Why Docker Registry Fails at Modern Artifact Distribution

In the cloud-native era, container registries are often treated as dumb storage buckets. We assume that if an image layer exists on disk, the registry will serve it efficiently. This assumption is dangerously outdated.

For years, the industry standard has been the Docker Distribution project (distribution/distribution). It works. But "working" is not the same as "optimized." As workloads shift toward edge computing, just-in-time scaling, and strict egress budgets, the architectural debt of the legacy Docker Registry becomes a tangible cost center.

This post dissects why Zot Registry, an OCI-native distribution engine, outperforms Docker Registry in two critical areas: mirror (pull-through caching) and on-demand serving. We’ll move beyond feature lists and examine the first-principles of bandwidth economics, concurrency safety, and storage integrity.

The Core Thesis: Storage Proxy vs. Content Distribution Engine

The divergence between Docker Registry and Zot isn’t about feature count; it’s about architectural intent.

Docker Registry is a legacy storage proxy designed for the Docker v2 era. It couples metadata tightly with filesystem paths. Its "proxy" mode was deprecated because it couldn’t handle race conditions, cache invalidation, or upstream rate limits reliably.
Zot Registry is an OCI-native artifact distribution engine built for cloud-native scale. It decouples storage from metadata, using content-addressable indexing to manage blobs intelligently.

When you need to mirror upstream repositories or serve images on-demand, you aren’t just moving files. You are managing data locality, concurrency, and upstream resilience. Zot solves these at the protocol layer. Docker Registry pushes them to your infrastructure team.

The Bandwidth Economics: Why Deduplication Matters

Bandwidth is not a network problem; it is a data locality problem. The most significant cost in container distribution is redundant upstream fetches.

The Scenario: Shared Layers

Consider a typical Kubernetes cluster pulling two images:

Image A: Layers 123, 456, 789
Image B: Layers 123, 456, 000, 001

Both images share the first two layers (123 and 456). In a perfect world, the registry should fetch these layers once and serve them locally for both images.

How Docker Registry Handles It

Docker Registry uses path-scoped caching. When Image A is pulled, layers 123 and 456 are stored. When Image B is requested, the registry checks the filesystem. If the blob exists, it serves it.

The Blind Spot: Under concurrent load, Docker Registry’s proxy mode spawns independent fetch goroutines. If ten nodes pull Image B simultaneously before the first fetch completes, the registry may trigger ten duplicate upstream fetches for layers 000 and 001. There is no atomic locking at the digest level. Furthermore, Docker Registry lacks native cross-repository deduplication awareness during the fetch phase, leading to wasted egress.

How Zot Handles It

Zot uses a content-addressable metadata index (backed by BoltDB, Redis, or DynamoDB).

Atomic Lookup: When Image B is requested, Zot checks the index for digests 123 and 456. It finds them instantly. Zero upstream traffic.
Concurrency Safety: For new layers 000 and 001, Zot locks the digest at the index level. The first request triggers the upstream fetch. Subsequent requests attach to the in-flight stream or wait for the write to complete. Guaranteed single upstream call per digest.
Cross-Repo Dedupe: If Layer 123 exists in any repository, Zot links it via hardlink or reference. No re-fetch.

The Result: In a cluster with 500 pods sharing common base images (e.g., alpine, distroless), Docker Registry can waste 3–5× the necessary upstream bandwidth on redundant fetches. Zot eliminates this waste at the protocol layer.

On-Demand Serving: Latency and Storage Efficiency

Modern workflows rely on lazy loading (e.g., estargz, nydus) and just-in-time pulls. This requires the registry to support efficient partial reads and non-destructive garbage collection.

The Garbage Collection Problem

Docker Registry: GC is destructive. It requires scanning the entire blob tree to identify unreferenced layers. During GC, the registry may lock or serve inconsistent data. Operators often disable GC, leading to storage bloat.
Zot: Uses reference counting in its metadata index. When a manifest is deleted, Zot decrements the refcount for each blob. GC only prunes blobs when refcount == 0. This is non-blocking, safe, and automated.

Lazy Pull Compatibility

Zot natively supports HTTP range requests for partial blob fetching, optimized for estargz and nydus formats. It serves only the requested byte ranges, reducing cold-start latency by 40–70% compared to full-layer downloads. Docker Registry supports range requests but lacks the indexing optimization to serve them efficiently under high churn.

Security and Policy Enforcement

In a zero-trust environment, you cannot serve an image without verifying its integrity.

Docker Registry: Basic TLS and auth. No native vulnerability scanning or signature verification. Relies on deprecated Notary or external sidecars. Policy enforcement happens after the pull, breaking the zero-trust model.
Zot: Native integration with Sigstore/cosign for on-demand signature verification. It can block unverified or vulnerable images before they reach the client. It also supports fine-grained RBAC, immutable tags, and full audit trails.

Pressure Testing: When Does Docker Registry Win?

Let’s be honest. Docker Registry isn’t useless. It wins in specific, narrow contexts:

Legacy Workflows: If you’re still using Docker schema v1 or private Notary, Zot won’t replicate those APIs. (You shouldn’t be using them anyway.)
Low-Concurrency, Pre-Warmed Caches: If your workload is sequential and all images are pre-pulled, the performance gap narrows.
Minimal SRE Capacity: If you lack the expertise to manage a new tool, sticking with the devil you know might feel safer. But remember: patching Docker Registry with custom Nginx/Lua proxies adds more complexity than migrating to Zot.

Actionable Migration Pathway

If you’re ready to optimize bandwidth and operational reliability, here’s how to proceed:

Deploy Zot as a Pull-Through Cache:

sync:
  onDemand: true
  dedupe: true
  pollInterval: 1h
  content:
    - prefix: "docker.io/library"
      tags:
        regex: "^(v[0-9]+\\.[0-9]+|latest)$"

Benchmark Concurrent Pulls:
Use crane or skopeo to pull shared-layer images in parallel. Measure upstream egress via VPC flow logs. Expect 40–80% reduction with Zot.

Enable Observability:
Monitor Prometheus metrics:

rate(zot_sync_cache_hits_total[5m])
rate(zot_sync_upstream_bytes_total[5m])

Enforce Signature Verification:
Integrate Sigstore/cosign to block unverified images at the registry layer.

Bottom Line

Docker Registry is a storage proxy patched for modern use. Zot is an OCI-native distribution engine. For mirror and on-demand workloads, Zot eliminates glue code, reduces operational overhead, and aligns with cloud-native security and performance requirements.

Bandwidth savings alone often pay for the migration within 30–90 days. But the real value is predictability. Zot guarantees single upstream fetches, safe garbage collection, and policy-driven serving. Docker Registry leaves you guessing.

Move deliberately. Test under load. But don’t delay. The architecture of your registry dictates the efficiency of your entire platform.