DEV Community: Satyaki

Image Volume Type GA in Kubernetes 1.36 — Finally Killing the Init Container Copy Pattern

Satyaki — Thu, 21 May 2026 05:49:06 +0000

For years, Kubernetes engineers have used the same awkward pattern whenever an application needed large read-only assets:

Init container
emptyDir
cp -r
Wait for startup
Duplicate storage usage

It worked, but it always felt like a workaround rather than a first-class Kubernetes primitive.

With Kubernetes 1.36, the image volume type is now GA, and it fundamentally changes how Pods can consume immutable file bundles.

Instead of:

pulling files into an init container,
copying them into an emptyDir,
and mounting them into the main container,

Kubernetes can now directly mount an OCI image filesystem into a container as a read-only volume.

That means:

no init-container copy step,
no duplicated bytes on disk,
faster startup times,
smaller artifact images,
and much cleaner Pod specs.

This becomes especially powerful for:

ML models
static websites
WASM modules
OPA bundles
language packs
Grafana dashboards
plugin distributions
and any independently versioned read-only asset bundle

Let me walk through a realistic end-to-end scenario.

The Scenario

Team A owns nginx (the web server).
Team B owns the website content (HTML/CSS/JS).
Team B ships new content 5x/day.
Team A ships nginx config changes maybe once a month.
They should not be coupled.

The static assets live in an OCI image:

registry.example.com/web/site-assets:v42

This image is:

scratch + files

No shell. No entrypoint. Just:

/site/index.html
/site/style.css
/site/app.js

The Old Way (Pre-1.36): Init Container + emptyDir

How You Built the Assets Image

# Dockerfile for site-assets

FROM busybox:1.36 AS source
COPY ./site /site

# Final image needs a shell because the init container will run `cp`
FROM busybox:1.36
COPY --from=source /site /site

Notice something important:

You cannot use FROM scratch here because the init container needs tools like:

cp
sh

So the image is bloated with BusyBox purely to enable the copy operation.

The Pod Manifest

apiVersion: v1
kind: Pod
metadata:
  name: web-old-way

spec:
  imagePullSecrets:
    - name: regcred

  initContainers:
    - name: load-assets
      image: registry.example.com/web/site-assets:v42

      command:
        - sh
        - -c
        - cp -r /site/* /shared/

      volumeMounts:
        - name: shared
          mountPath: /shared

  containers:
    - name: nginx
      image: nginx:1.27

      volumeMounts:
        - name: shared
          mountPath: /usr/share/nginx/html
          readOnly: true

  volumes:
    - name: shared
      emptyDir: {}

What's Actually Happening Under the Hood

1. Images are pulled

Kubelet pulls:

nginx:1.27
site-assets:v42

using the Pod's imagePullSecrets.

2. Kubelet creates the emptyDir

Kubernetes creates an actual directory on the node filesystem:

/var/lib/kubelet/pods/<pod-uid>/volumes/kubernetes.io~empty-dir/shared/

At this point it is completely empty.

3. Init container starts

The init container's root filesystem is the site-assets image.

The emptyDir gets bind-mounted into the init container at:

/shared

4. The copy operation happens

The init container executes:

cp -r /site/* /shared/

Every byte gets physically copied:

image layer -> emptyDir

5. Init container exits

Kubelet records successful completion.

6. nginx starts

The same emptyDir is mounted into nginx at:

/usr/share/nginx/html

nginx now serves the copied files.

Real Problems With the Old Pattern

1. Disk Usage Doubles

The files now exist in two places:

/var/lib/containerd/...

AND

emptyDir

A 2 GB ML model becomes:

2 GB image layer + 2 GB copy = 4 GB

per Pod.

2. Startup Latency

Large copy operations are expensive.

A 2 GB copy operation can easily add:

10–30 seconds

before the main container even starts.

3. You Need a Shell

You cannot use:

FROM scratch

because the image needs tooling like:

cp
sh

That means:

larger images
more CVEs
more attack surface

4. Verbose YAML

You need:

init containers
shared volumes
multiple mounts
copy commands

All just to move files around.

5. No Sharing Across Pods

Every Pod independently copies the same bytes into its own emptyDir.

Ten Pods on one node means:

10 independent copies

The New Way (Kubernetes 1.36): Image Volume Type

How You Build the Assets Image Now

FROM scratch

COPY ./site /site

That's it.

No shell.
No BusyBox.
No executables.

Just files.

The Pod Manifest

apiVersion: v1
kind: Pod

metadata:
  name: web-new-way

spec:
  imagePullSecrets:
    - name: regcred

  containers:
    - name: nginx
      image: nginx:1.27

      volumeMounts:
        - name: assets
          mountPath: /usr/share/nginx/html
          subPath: site

  volumes:
    - name: assets

      image:
        reference: registry.example.com/web/site-assets:v42
        pullPolicy: IfNotPresent

No init container.
No emptyDir.
No copy operation.

What's Actually Happening Under the Hood Now

1. Kubelet sees an image volume

Kubelet notices:

volumes:
  - image:

and asks the CRI runtime to mount the image.

2. Runtime pulls the image

containerd or CRI-O pulls:

site-assets:v42

using the normal image pull pipeline.

Exactly the same mechanism used for container images.

3. Runtime unpacks image layers

The image gets unpacked into the runtime snapshot store.

For example with containerd:

overlayfs snapshotter

No container is started.

Only the filesystem is materialized.

4. Filesystem is bind-mounted directly

The runtime bind-mounts the image filesystem directly into the nginx container:

/usr/share/nginx/html

Read-only by design.

5. nginx starts immediately

No copy step.
No waiting.

The files already exist.

The Mental Model Shift

The old model was:

"Start a helper container and copy files somewhere."

The new model is:

"Mount an OCI image filesystem directly as a volume."

That sounds subtle, but architecturally it's a major shift.

Kubernetes is effectively treating OCI images as generic immutable data artifacts — not just runnable containers.

What You Gain

No Data Duplication

The image is bind-mounted directly.

No copy operation.

Faster Startup

You eliminate the init-container copy phase entirely.

For large datasets or ML models, this is massive.

Smaller and Safer Images

You can now use:

FROM scratch

which means:

smaller images
fewer CVEs
reduced attack surface

Always Read-Only

Image volumes are immutable by specification.

The runtime enforces it.

Applications cannot modify the mounted content.

Shared Across Pods

Ten Pods mounting the same image on the same node share the same underlying bytes.

Huge improvement for large artifact distribution.

Cleaner YAML

The Pod spec now clearly expresses intent:

"Mount this image's filesystem here."

instead of implementing an entire file-copy workflow.

Important Caveats

Image Volumes Are Always Read-Only

If your application needs writable storage, use:

emptyDir
PVCs
ephemeral storage

instead.

subPath Is Extremely Useful

If your files live under:

/site

inside the image but you want them mounted directly into:

/usr/share/nginx/html

then:

subPath: site

solves that cleanly.

pullPolicy Works Exactly Like Container Images

You can use:

pullPolicy: Always

or:

pullPolicy: IfNotPresent

exactly as you already do with containers.

No Environment Variable Substitution

This does NOT work:

reference: ${ASSET_VERSION}

The field is literal.

Use:

Helm
Kustomize
templating

instead.

Running Pods Don't Automatically Update

If somebody pushes a new image to:

:v42

existing Pods continue using the old mounted bytes.

You must roll the Pod to pick up changes.

Which is good for reproducibility.

In production, pin image digests.

Runtime Support Matters

You need modern runtimes.

Roughly:

containerd >= 2.1
CRI-O >= 1.31

Older runtimes will fail with a clear unsupported feature error.

How imagePullSecrets Work

This is one of the nicest parts.

Image volumes automatically use the same authentication flow as normal container images.

That means Kubernetes automatically uses:

Pod imagePullSecrets
ServiceAccount imagePullSecrets
kubelet credential providers

No additional auth wiring required.

So this:

imagePullSecrets:
  - name: regcred

works for BOTH:

container images
image volumes

Multiple Private Registries

If assets and application images live in different registries:

imagePullSecrets:
  - name: app-registry-creds
  - name: assets-registry-creds

The runtime tries the secrets in order and uses whichever matches the registry hostname.

Quick Comparison

Aspect	Init Container + emptyDir	Image Volume
Pod complexity	Multiple containers and mounts	Single volume
Assets image	Needs shell/cp	`FROM scratch` works
Disk usage	Image + copied bytes	Image only
Startup time	Pull + copy	Pull only
Writable	Yes	No
Sharing across Pods	No	Yes
imagePullSecrets	Pod spec	Pod spec
Update without restart	No	No
Kubernetes support	Always	1.31 alpha → 1.33 beta → 1.36 GA

When You Should Actually Use It

Image volumes are ideal when you have:

large read-only assets
independently versioned bundles
OCI-distributed artifacts
data shared across multiple Pods

Examples include:

ML models
static websites
OPA bundles
plugins
WASM modules
Grafana dashboards
language packs

They're especially useful when:

the artifacts are too large for ConfigMaps
you want registry-native distribution
you want image signing/scanning/RBAC
the content must remain immutable

When NOT To Use It

Don't use image volumes when:

the application needs writable storage
the content is tiny text configuration
the data is stateful per Pod

In those cases:

ConfigMaps
PVCs
emptyDir

are still better fits.

Final Thoughts

The image volume type feels small on paper, but it removes one of the longest-standing operational hacks in Kubernetes.

For years, platform engineers built elaborate init-container copy workflows just to move immutable files into Pods.

Now Kubernetes finally has a native primitive for it.

If your workloads distribute:

large read-only assets
ML models
frontend bundles
policy packs
plugins
shared runtime data

this feature can significantly reduce:

startup latency
storage duplication
image complexity
YAML noise

More importantly, it aligns Kubernetes with a broader industry shift:

OCI images are no longer just executable containers.
They're becoming the standard distribution format for software artifacts in general.

And image volumes push Kubernetes one step further in that direction.

CPU Humbled Me — A Kubernetes Throttling Story Hidden Between Prometheus Scrapes

Satyaki — Fri, 15 May 2026 15:12:32 +0000

Memory is easy. CPU humbled me.

With memory, the rule is brutal but clear — cross the limit, the pod gets OOMKilled. Done.

CPU? CPU is sneaky. And I ignored it for the longest time… until it broke production.

Here's what happened 👇

We had an app running peacefully in-house. Then it went client-facing. Traffic surged, and suddenly ~15% of requests started timing out — most of them on DB calls.

I opened Grafana expecting a smoking gun. Nothing. CPU usage looked "fine." No throttling alerts screaming at me. Just confused timeouts.

The trap? Throttling happens in milliseconds. Prometheus scrapes every 15 seconds. Every bit of evidence was hiding between the scrapes.

Here was the setup:

resources:
  requests:
    cpu: 200m
    memory: 512Mi
  limits:
    cpu: 800m
    memory: 1.5Gi

Numbers from the incident (rough, but directionally honest):

Normal: 300 req/min → avg CPU ~180m
Surge: 1200 req/min → avg CPU ~650m, ~15% timeouts

So I sat down and actually did the math instead of guessing.

How CPU actually works

CPU is compressible. Memory isn't. When CPU runs out, your process doesn't die — it gets throttled. The Linux CFS scheduler slices time into periods (default: 100ms). Within each period, your container gets a quota based on its limit. Cross the quota mid-period? You wait for the next one. That wait is the latency you're seeing.

Walking through the numbers

Normal load:

300 req/min = 5 req/sec = 0.5 requests per 100ms
Avg CPU 180m = 18ms of CPU work per 100ms period
→ 18ms ÷ 0.5 req = ~36ms of CPU work per request

Surge load:

1200 req/min = 20 req/sec = 2 requests per 100ms
2 × 36ms = 72ms of CPU work needed per 100ms

But the limit was 800m → 80ms quota per 100ms. Looks fine on paper, right?

Here's the catch: avg CPU was 650m (65ms). The average hides the bursts. Some periods sat well below quota; others blew past the 80ms ceiling and got throttled. Average everything out across 15s scrapes and the dashboard whispers "all good" while users get timeouts.

That's the lesson. Average CPU is a liar in bursty workloads. Throttling lives in the gaps your monitoring can't see.

What to actually look at

Stop staring at container_cpu_usage_seconds_total. Look at:

container_cpu_cfs_throttled_periods_total
container_cpu_cfs_throttled_seconds_total

The ratio of throttled periods to total periods tells you the truth.

Remediations (in order of maturity, not just "increase the limit")

Right-size first. Requests and limits should reflect real workload behavior, not guesses copy-pasted from a template.
Load test before going client-facing. Running an app in-house ≠ serving real traffic.
VPA recommendations to understand what the app actually wants.
HPA so bursts get distributed across replicas instead of crushing one pod.
Then, if needed, raise the limit — with intent, not panic.

Bumping the limit is the easiest fix and the most expensive habit. Every patch carries a hidden cost — node capacity, bin-packing, cluster bills, blast radius. Understand the why before you reach for the YAML.

This one incident taught me more about Kubernetes resource management than months of reading docs. If you're running anything client-facing, please don't wait for a production incident to learn this.

CPU isn't just a number on a dashboard. It's a time budget — and your users feel every millisecond you overspend.

Have you been burned by CFS throttling? What metric finally gave it away for you? Drop it in the comments — I'd love to compare notes.

Understanding Kube-proxy & CoreDNS in Kubernetes no bluff

Satyaki — Thu, 22 Jan 2026 15:23:21 +0000

🛠 Setting the Stage: A Kind Cluster

Kubernetes is full of magic, but one of its most fascinating components is kube-proxy. It’s the silent operator that ensures traffic hitting a Service gets distributed across the right Pods. Under the hood, kube-proxy leverages Linux iptables to make this happen. Let’s peel back the layers and see it in action.

For this demo, I spun up a 3-node Kind cluster. On top of it, I deployed a simple nginx Deployment exposed via a ClusterIP Service.

Here’s the deployment and service in action:

📜 Peeking into iptables

Now comes the fun part. I logged into one of the nodes where a Pod is running and listed the NAT rules in the KUBE-SERVICES chain:

Notice the entry for our nginx-deployment Service. The destination IP here is the ClusterIP of the Service. This is kube-proxy’s starting point for redirecting traffic

🔀 Diving into the Service Chain

Every Service gets its own chain. For nginx, that’s KUBE-SVC-WRNOD73BKRQH4VVX. Let’s inspect it:

And here’s the magic:
When traffic hits the ClusterIP, kube-proxy rewrites it to one of the Pod IPs backing the Deployment.
The rules show a probability ratio — in this case, 50/50. That means half the traffic goes to one Pod, and the other half to the second Pod.
This is how kube-proxy achieves load balancing using nothing more than iptables.
So, what did we just see?

ClusterIP → Pod IPs translation via iptables.
Masquerading ensures the source IP is rewritten correctly.
Probability rules distribute traffic evenly across endpoints

🌐 How DNS Works in the Cluster

So far, we’ve seen how kube-proxy handles traffic routing and load balancing. But how does your application even know where to send requests? That’s where CoreDNS comes in.
CoreDNS acts as the nameserver inside Kubernetes, resolving Service names into their corresponding ClusterIPs. Let’s walk through it step by step.

🔍 Inspecting the kube-dns Service

In the kube-system namespace, you’ll find the kube-dns Service. This is essentially the front door to CoreDNS:

📄 The resolv.conf File

Inside Pods, the resolv.conf file contains the nameserver details and DNS search domains. This is how Kubernetes ensures that when you query something like nginx-deployment.default.svc.cluster.local, it knows how to resolve it.

🧪 Testing with nslookup

Let’s put it to the test. Logging into a node and running an nslookup shows the DNS resolution in action:

And it works exactly as expected — the Service name resolves to the ClusterIP, which kube-proxy then maps to the Pod IPs.

🎯 Wrapping It All Up

Between kube-proxy and CoreDNS, Kubernetes ensures that:

Traffic hitting a Service is load balanced across Pods.
Service names are resolved seamlessly into ClusterIPs.
Applications don’t need to worry about IP addresses — they just use DNS names. These two components are the backbone of Kubernetes networking. Without them, Services wouldn’t be discoverable or scalable.
🔥 And that’s the no-bluff walkthrough of kube-proxy and CoreDNS — two vital pieces of the Kubernetes puzzle. Next time you deploy an app, you’ll know exactly how the traffic finds its way to the right Pod.

Thats what kube-proxy does. Isnt it really cool ?