Kubernetes Pod Churn & Node Thrashing: Ephemeral Storage Exhaustion and Eviction Cascades

When background worker pods continuously write temporary scratch files, un-rotated log buffers, or dynamic assets directly to container filesystems without explicit ephemeral storage limits, worker nodes quietly enter local disk pressure states. The kubelet then triggers aggressive pod evictions, sparking cluster-wide pod churn, Karpenter/Cluster Autoscaler thrashing, and AWS bill inflation. Here is an architectural deep dive into diagnosing local disk exhaustion and configuring resilient kubelet eviction boundaries.

In high-throughput B2B SaaS platforms running containerized microservices, background processing workloads (e.g., generating PDF reports, processing video/image assets, compiling code, or handling CSV batch imports) frequently rely on local container file systems for temporary scratch space. Under heavy traffic, un-capped ephemeral storage usage creates an invisible cluster-level failure mode: Cascading Node Eviction and Auto-Scaler Thrashing.

1. The Mechanics of OverlayFS and Ephemeral Storage Eviction

By default, when a container writes files to its own filesystem (outside of explicitly mounted Persistent Volumes), those writes land on the node's local disk through the container runtime's writable layer, typically the OverlayFS storage driver. Disk-backed emptyDir volumes and container logs consume the same node disk. If a pod uses significant local disk space without ephemeral storage requests and limits in its manifest, it triggers a chain reaction across the host node.

The Anatomy of a Node Eviction Cascade

• Unbounded Disk Growth: A burst of PDF generation worker tasks writes 15GB of temporary files to /tmp inside the container overlay filesystem.
• Kubelet Disk Pressure Breach: The node's disk crosses a kubelet hard eviction threshold (by default, nodefs.available below 10% or imagefs.available below 15%). The kubelet marks the node DiskPressure = True, the node is tainted, and the scheduler stops placing new pods on it.
• Aggressive Pod Eviction: The kubelet first garbage-collects dead containers and unused images, then starts evicting pods. It ranks pods by whether their disk usage exceeds their ephemeral storage request, then by pod priority, then by usage. With no requests set, every pod counts as over its request, so priority dominates, and lower-priority core services can be evicted before the high-priority worker that is actually filling the disk.
• Auto-Scaler Thrashing (Karpenter / Cluster Autoscaler): Rescheduled evicted pods flood neighboring nodes, pushing those nodes into disk pressure as well. The autoscaler provisions new EC2 instances for pending pods while consolidation removes others, so nodes churn continuously and compute spend can multiply for the duration of the incident.

2. Diagnosing Ephemeral Storage Bloat in Production

Pod objects do not report disk usage, so kubectl get pods cannot show it. The kubelet's stats summary endpoint does. Run the following diagnostic commands to find which pods are consuming node disk and driving instability (requires jq):

# Top 20 pods by ephemeral storage usage (MiB) across all nodes
for node in $(kubectl get nodes -o name | cut -d/ -f2); do
  kubectl get --raw "/api/v1/nodes/$node/proxy/stats/summary" \
    | jq -r --arg node "$node" '.pods[]
        | [$node, .podRef.namespace, .podRef.name,
           ((.["ephemeral-storage"].usedBytes // 0) / 1048576 | floor)]
        | @tsv'
done | sort -t$'\t' -k4 -rn | head -20

# Inspect recent eviction events
kubectl get events --all-namespaces --field-selector reason=Evicted | grep -i "ephemeral"

# Detailed check of Node Disk Pressure conditions
kubectl describe node <node-name> | grep -A 8 "Conditions:"

A pod status displaying Evicted with the message The node was low on resource: ephemeral-storage confirms your cluster is actively suffering from un-capped container write exhaustion.

3. Production-Ready Remediation: Strict Resource Limits & In-Memory emptyDir

Step 1: Enforce Ephemeral Storage Requests and Limits

Every Kubernetes manifest deployed to production must explicitly define ephemeral-storage resource boundaries alongside CPU and memory parameters. This lets the scheduler place pods according to available disk, and when a container exceeds its storage limit, the kubelet evicts only that pod instead of letting the whole node reach disk pressure.

# PRODUCTION-GRADE DEPLOYMENT MANIFEST WITH EPHEMERAL LIMITS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdf-generator-worker
  namespace: production
spec:
  replicas: 8
  selector:
    matchLabels:
      app: pdf-generator-worker
  template:
    metadata:
      labels:
        app: pdf-generator-worker
    spec:
      containers:
      - name: worker
        image: registry.internal.net/pdf-worker:v2.1.0
        resources:
          requests:
            cpu: "500m"
            memory: "2Gi"
            ephemeral-storage: "1Gi" # Scheduler reserves this much node disk
          limits:
            cpu: "2000m"
            memory: "4Gi"            # Includes up to 2Gi of tmpfs scratch below
            ephemeral-storage: "4Gi" # Hard ceiling: kubelet evicts this pod if exceeded
        volumeMounts:
        - mountPath: /tmp/scratch
          name: scratch-volume
      volumes:
      - name: scratch-volume
        emptyDir:
          medium: Memory # Back temporary writes with RAM (tmpfs)
          sizeLimit: 2Gi # Cap scratch space so it cannot consume all memory

Step 2: Redirect High-Frequency Scratch Writes to In-Memory tmpfs

For workloads requiring ultra-fast, short-lived file creation (e.g., image resizing or PDF rendering), mount an emptyDir volume backed by RAM (medium: Memory). Writes to this volume bypass the node disk entirely, removing them as a source of disk pressure and speeding up file I/O. One catch: files in a memory-backed emptyDir count toward the container's memory limit, not its ephemeral storage limit. Size the memory limit to cover the application plus the scratch volume, as in the manifest above, or a full scratch directory will get the pod OOM-killed.

4. Tuning Kubelet Eviction Thresholds and Karpenter Node Consolidation

Reduce premature evictions by tuning kubelet eviction settings in your node configuration (for example, the kubelet config in your EKS node group launch template or Karpenter EC2NodeClass user data):

# Kubelet Eviction Configuration Parameters (kubelet-config.json)
{
  "evictionHard": {
    "memory.available": "5%",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%",
    "imagefs.available": "15%"
  },
  "evictionSoft": {
    "nodefs.available": "15%"
  },
  "evictionSoftGracePeriod": {
    "nodefs.available": "2m"
  },
  "evictionMaxPodGracePeriod": 60
}

Then slow down Karpenter's consolidation so short traffic spikes don't cause node churn (Karpenter v1 API):

# Karpenter v1 NodePool with churn guardrails
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h # Rotate nodes every 30 days for patching
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m # Wait before consolidating so spikes can settle
    budgets:
      - nodes: "10%"     # Never disrupt more than 10% of nodes at once

• Soft Eviction Grace Periods: With a soft threshold, the kubelet only evicts if disk stays below 15% free for a full 2 minutes. Short spikes that clear on their own (a batch job finishing, logs rotating, image garbage collection) no longer trigger evictions. The hard threshold at 10% still protects the node.
• Karpenter Consolidation Guardrails: consolidateAfter delays consolidation so nodes are not removed the moment utilization dips, and disruption budgets cap how many nodes Karpenter can replace at once. expireAfter handles routine node rotation; it is not a churn control on its own.

5. Executive Summary & Systems Rules

• Always Specify ephemeral-storage Requests and Limits: Unbounded container writes will fill host node disks and trigger cascading evictions across unrelated pods.
• Use RAM-Backed emptyDir for Transient Files: Mount temporary scratch directories with medium: Memory to keep high-frequency file I/O off node disks, and raise the memory limit to cover the scratch size.
• Set Soft Kubelet Eviction Windows: Configure grace periods on nodes so brief disk spikes can clear before the kubelet evicts pods, and add consolidation delays and disruption budgets to your autoscaler.

---

Are pod evictions and node churn driving up your Kubernetes bill? I help SaaS teams right-size workloads, tune kubelet and Karpenter settings, and build clusters that stay stable under heavy batch traffic. Book a consultation →

Originally published on ctousman.com.