DEV Community

Cover image for Kubernetes 1.36: Breaking Free from Container-Level Resource Constraints
Pratheesh Satheesh Kumar
Pratheesh Satheesh Kumar

Posted on

Kubernetes 1.36: Breaking Free from Container-Level Resource Constraints

#kubernetes #devops #platform #performance

The Real Cost of One-Size-Fits-All Resource Allocation

You're running a machine learning training job in Kubernetes. Your main container needs exclusive CPU cores, NUMA alignment, and guaranteed memory—every microsecond counts. But your pod also runs three sidecars: a Prometheus exporter using 50m CPU, a log shipper, and a service mesh proxy. Before Kubernetes 1.36, you had two painful options:

  1. Allocate exclusive CPUs to every container, wasting resources on lightweight sidecars
  2. Give up on Guaranteed QoS class entirely, losing the performance guarantees your primary workload depends on

Pod-Level Resource Managers (alpha in 1.36) end this false choice. This is the kind of practical improvement that separates "works in dev" from "scales reliably in production."

How Pod-Level Resource Managers Actually Work

The enhancement extends the kubelet's CPU, Memory, and Topology Managers from a strict per-container model to a pod-centric allocation strategy. Enable the PodLevelResourceManagers and PodLevelResources feature gates, and you unlock hybrid resource allocation—where your primary container gets exclusive, NUMA-aligned resources while sidecars share the burstable pool.

The Architecture Shift

Previously, resource managers evaluated each container independently:

spec:
  containers:
  - name: ml-training
    resources:
      limits:
        cpu: "8"
        memory: "16Gi"
  - name: prometheus-exporter
    resources:
      limits:
        cpu: "100m"  # Still gets exclusive CPU logic applied
        memory: "128Mi"
Enter fullscreen mode Exit fullscreen mode

Now, with pod-level resources, you specify allocation intent at the pod level:

spec:
  resources:  # New pod-level field
    cpu: "8"
    memory: "16Gi"
    managedResources: [cpu, memory]  # Which managers handle this
  containers:
  - name: ml-training
    # Gets the pod-level resources
    resources:
      limits:
        cpu: "8"
        memory: "16Gi"
  - name: prometheus-exporter
    # Sidecar, uses shared resources
    resources:
      limits:
        cpu: "100m"
        memory: "128Mi"
Enter fullscreen mode Exit fullscreen mode

The kubelet now understands: "Allocate these resources to the pod as a unit, optimizing for the main workload, and let sidecars share what's available."

Real-World Impact: Three Scenarios Where This Matters

Scenario 1: High-Frequency Trading

Your order-matching engine needs 16 exclusive cores, zero latency variance, and strict NUMA binding. Historically, you'd have to waste 2 full cores on a sidecar collecting metrics. Now sidecars can share resources while the primary container gets guaranteed isolation.

Scenario 2: Database Workloads

PostgreSQL or RocksDB containers in Kubernetes need predictable page cache behavior. Pod-level allocation lets you assign exclusive CPUs to the database while keeping log collectors and health checkers lightweight and shared.

Scenario 3: ML Inference at Scale

Serving BERT or GPT models requires precise CPU allocation for tokenizer preprocessing and model serving. With pod-level resources, your GPU-powered model container gets exclusive CPU cores while sidecar authentication and request logging run with elastic resources.

Actionable: Testing This Today

If you're running 1.36, here's how to start experimenting:

  1. Enable the feature gates in your kubelet config: 
kubeletExtraArgs:
  feature-gates: PodLevelResourceManagers=true,PodLevelResources=true
Enter fullscreen mode Exit fullscreen mode
  1. Set the appropriate manager policies (static CPU policy required for NUMA alignment): 
kubeletExtraArgs:
  cpu-manager-policy: static
  memory-manager-policy: static
  topology-manager-policy: best-effort
Enter fullscreen mode Exit fullscreen mode

  1. Deploy a test workload with explicit pod-level resources and observe kubelet logs for allocation decisions.

  2. Monitor NUMA efficiency using tools like numastat on nodes—you should see better locality for resource-managed pods.

What's Still Alpha

This feature is alpha, so expect changes:

  • API shape for .spec.resources may evolve
  • Not all manager combinations are tested (CPU+Memory+Topology interaction especially)
  • Upgrade/downgrade paths aren't fully hardened
  • Documentation is still being written

Do NOT use this in production yet—but start testing in non-critical clusters now. Alpha feedback shapes the 1.37+ roadmap.

The Bigger Picture

Pod-level resource managers reflect a maturation in Kubernetes: moving from "one-size-fits-all per-container abstractions" to "workload-aware resource models." This follows the same evolution we've seen with pod disruption budgets, pod scheduling policies, and topology spread constraints.

The real win? You no longer have to choose between performance guarantees and resource efficiency. Your ML pipelines, databases, and trading systems can have both.

What's your biggest pain point with resource allocation in Kubernetes today—is it NUMA binding, sidecar overhead, or something else? Share in the comments; early adopter feedback is how features like this get refined.

Top comments (0)