DEV Community

iapilgrim
iapilgrim

Posted on

Ephemeral Storage in AKS — A Practical Hands-On Lab

#azure #kubernetes #devops #tutorial

If you're running:

  • CI/CD pipelines
  • Batch jobs
  • Video transcoding
  • ETL pipelines
  • AI preprocessing

You do not need persistent storage.

In fact, persistent volumes can:

  • Increase cost
  • Add lifecycle complexity
  • Leak unused PVCs
  • Slow down I/O

Instead, Kubernetes offers ephemeral storage primitives that are:

✅ Fast
✅ Auto-cleaned
✅ Zero long-term storage cost
✅ Perfect for temporary workloads

This tutorial walks through a full working lab.

🧠 What Is Ephemeral Storage?

Ephemeral storage is storage that:

  • Exists only for the lifetime of a Pod
  • Is automatically deleted when the Pod is removed
  • Lives on node-local disk or memory

No manual cleanup.
No orphaned volumes.
No storage bills after the job completes.

🧰 What You’ll Learn

We’ll demonstrate:

1️⃣ emptyDir (disk-backed)
2️⃣ emptyDir (memory-backed)
3️⃣ Generic Ephemeral Volumes (dynamic PVC lifecycle)
4️⃣ Automatic cleanup behavior

1️⃣ Disk-Backed emptyDir

emptyDir is the simplest ephemeral storage mechanism.

It:

  • Is created when the Pod starts
  • Lives as long as the Pod runs
  • Is deleted when the Pod is deleted

Example 

apiVersion: v1
kind: Pod
metadata:
  name: emptydir-disk
spec:
  containers:
  - name: writer
    image: busybox
    command: ["/bin/sh", "-c"]
    args:
      - |
        dd if=/dev/zero of=/scratch/test.img bs=1M count=100
        sleep 600
    volumeMounts:
    - mountPath: /scratch
      name: scratch
  volumes:
  - name: scratch
    emptyDir: {}
Enter fullscreen mode Exit fullscreen mode

What Happens

  • A 100MB file is created inside /scratch
  • The data lives on node-local disk
  • Delete the Pod → data disappears

💡 Perfect for CI build artifacts or temporary data transforms.

2️⃣ Memory-Backed emptyDir (Ultra Fast)

You can mount emptyDir into RAM using medium: Memory.

volumes:
- name: memvol
  emptyDir:
    medium: Memory
Enter fullscreen mode Exit fullscreen mode

This creates a tmpfs mount.

Why It’s Powerful

  • Extremely fast
  • Zero disk I/O
  • Perfect for caching or scratch processing

⚠ Important:
It counts against container memory limits.

If memory is exceeded → Pod may be evicted.

3️⃣ Generic Ephemeral Volumes (Dynamic PVC)

This is where things get powerful.

Kubernetes can dynamically create a PVC that:

  • Is scoped to a Pod
  • Is automatically deleted when the Pod is deleted
  • Uses your StorageClass

Example 

volumes:
- name: eph
  ephemeral:
    volumeClaimTemplate:
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi
Enter fullscreen mode Exit fullscreen mode

When the Pod starts:

  • Kubernetes creates a PVC
  • A backing volume is provisioned

When the Pod is deleted:

  • The PVC is deleted automatically
  • The storage is released

✔ No PVC leaks
✔ No manual cleanup

Perfect for:

  • Data processing jobs
  • ML training scratch space
  • Large temporary workloads

🔥 What We Verified in the Lab

Feature Behavior
emptyDir Fast local disk storage
emptyDir (Memory) RAM-backed tmpfs
Generic Ephemeral Dynamic PVC lifecycle
Pod deletion Storage auto-cleaned
No persistence Zero long-term cost

⚖ Before vs After Using Ephemeral

❌ Traditional Approach (Persistent Volumes Everywhere)

  • PVC lifecycle management required
  • Risk of leaked volumes
  • Network storage latency
  • Ongoing storage cost

✅ Ephemeral Strategy

  • Faster local I/O
  • Automatic cleanup
  • No lifecycle overhead
  • Lower cloud cost
  • Better autoscaling behavior

🏗 Real-World Architecture Patterns

CI/CD Build Cluster

  1. Job starts
  2. Repo cloned into emptyDir
  3. Build artifacts generated
  4. Image pushed
  5. Pod exits
  6. Storage auto-deleted

Zero leftovers.

Video Transcoding System

  1. Download file
  2. Process locally
  3. Upload output
  4. Pod terminates

Scratch space disappears automatically.

ETL / Data Transformation

  1. Download dataset
  2. Transform locally
  3. Upload results
  4. Storage wiped

No persistent cost.

☁ Cloud Optimization (Advanced)

On managed Kubernetes like:

  • Microsoft Azure (AKS)
  • Amazon Web Services (EKS)
  • Google Cloud (GKE)

You can combine ephemeral volumes with:

  • Local SSD nodes
  • Ephemeral OS disks
  • Autoscaling node pools

This produces:

⚡ Extremely fast build clusters
💰 Reduced storage cost
🔄 Clean horizontal scaling

🚨 Best Practices

✔ Always set ephemeral-storage resource requests/limits
✔ Monitor node disk pressure
✔ Use separate node pools for build workloads
✔ Avoid storing logs in emptyDir (use log shipping)
✔ Use memory-backed volumes only when needed

🎯 When NOT to Use Ephemeral Storage

Avoid ephemeral storage if:

  • Data must survive Pod restart
  • You need shared access across Pods
  • You need backups or snapshots

Use PersistentVolumes in those cases.

🧩 Final Takeaway

Ephemeral storage is one of the most underused performance optimizations in Kubernetes.

For temporary workloads, it gives you:

  • Speed
  • Simplicity
  • Cost efficiency
  • Automatic cleanup

If you’re building:

  • CI/CD infrastructure
  • Batch processing platforms
  • Media pipelines
  • AI preprocessing clusters

Ephemeral storage should be your default choice.

Top comments (0)