Manu Shukla

Posted on Jul 23 • Originally published at ecorpit.com

Stateful AI agents on Kubernetes: the 2026 Agent Sandbox and GPU scheduling playbook

#kubernetes #aiagents #agentsandbox #gpuscheduling

Stateful AI agents on Kubernetes: the 2026 Agent Sandbox and GPU scheduling playbook

Summary. Kubernetes was designed in 2014 for stateless, replicated request handlers. AI agents are the opposite: long-running, stateful, bursty, and GPU-hungry. That mismatch is why the Kubernetes project shipped a new primitive. Agent Sandbox, previewed at KubeCon NA in November 2025 and generally available on Google Kubernetes Engine (GKE) since 20 May 2026, gives each agent a single stateful pod with a stable identity, and its warm pool can allocate 300 sandboxes per second per cluster, with 90% of allocations finishing in 200 milliseconds. The cost pressure is real: an on-demand NVIDIA H100 runs roughly $3.00 to $6.98 per GPU-hour across AWS, GCP and Azure as of April 2026, so an idle agent holding a GPU burns money. This guide covers the mismatch, the Agent Sandbox CRD, GPU scheduling with Dynamic Resource Allocation (GA in Kubernetes 1.34), and the deployment patterns that hold up in production.

Why Kubernetes fights AI agents

A Kubernetes Deployment assumes its pods are interchangeable. Any replica can serve any request, requests finish in milliseconds, and if a pod dies the load balancer sends the next request elsewhere. That model runs most of the web.

An agent breaks every one of those assumptions. It holds a conversation and a plan across many model calls. It spawns subprocesses, executes tool calls, and often runs untrusted code that a language model just wrote. A single agent task can run for minutes or hours and touch a dozen external systems, and it is a singleton: request 7 has to reach the same instance that handled requests 1 through 6, because that instance holds the state. Kill the pod and you lose the reasoning context, not just an in-flight request.

The Agent Sandbox maintainers put the motivation plainly in the project's documentation: Kubernetes handles stateless replicated applications through Deployments and numbered, stable sets through StatefulSets, but agent runtimes need "a long-running, stateful, singleton container with a stable identity." You can approximate that with a size-one StatefulSet plus a Service plus a PersistentVolumeClaim, but the project notes that this is cumbersome and has no built-in hibernation. When the maintainers of a platform build a dedicated primitive instead of telling you to compose existing objects, that is the clearest signal that the old model does not fit.

The autoscaling and cold-start problem

The mismatch shows up first in autoscaling. The Horizontal Pod Autoscaler reads CPU and memory, and GPU utilisation is not exported to the metrics server by default. A vLLM pod serving 50 concurrent requests can sit at 5 to 8% CPU while its GPU runs at 95%, per analysis from Spheron. The autoscaler sees a healthy, near-idle pod and does nothing, while users watch token latency climb past 30 seconds as the KV cache fills and requests queue behind it.

Cold starts make it worse. Loading model weights into VRAM takes 30 to 120 seconds. When a GPU pod restarts cold, the new replica spends 3 to 10 minutes pulling images, loading weights from storage, capturing CUDA graphs, and warming its KV cache; a full scale-from-zero can take 8 to 15 minutes for large models. Now put an agent on top. As InfoWorld noted in its July 2026 write-up on agentic compute patterns, provisioning a new environment takes 45 seconds to two minutes on a well-tuned cluster, but an agent's reasoning loop needs a fresh sandbox in under two seconds or the loop stalls waiting.

Property	Stateless microservice	Stateful AI agent
Request lifetime	Milliseconds to seconds	Minutes to hours
Instance affinity	None; any replica serves any request	Strict; the session must return to its pod
Scaling signal	CPU, RAM, requests per second	Queue depth, GPU memory, session count
Failure impact	One retried request	Lost reasoning state and context
Provisioning budget	Seconds are fine	Sub-second needed mid-loop
Right Kubernetes object	Deployment	Sandbox CRD or StatefulSet

Agent Sandbox: a Kubernetes primitive for agents

Agent Sandbox is an open-source project under Kubernetes SIG Apps, at kubernetes-sigs/agent-sandbox, that adds a Sandbox Custom Resource Definition and controller. A Sandbox is a declarative API for a single stateful pod with a stable hostname, persistent storage that survives restarts, and lifecycle controls the standard workloads lack: scheduled deletion, pausing, and resuming. The core object is deliberately small; the interesting parts sit in the extensions.

SandboxTemplate defines a reusable blueprint, so you manage a fleet of similar agents from one spec.
SandboxWarmPool keeps pre-provisioned replicas ready, which is how you beat the cold-start budget.
SandboxClaim hands a caller a ready sandbox from that pool without exposing the underlying configuration.

A minimal Sandbox is close to a pod spec:

apiVersion: agents.x-k8s.io/v1beta1
kind: Sandbox
metadata:
  name: research-agent-7f2a
spec:
  podTemplate:
    spec:
      containers:
      - name: runtime
        image: registry.example.com/agent-runtime:2026.7

The Sandbox controller gives that object a stable network identity at the hostname research-agent-7f2a, so session traffic always returns to the same instance. On GKE the same API drives a warm pool that allocates 300 sandboxes per second per cluster at sub-second latency, with 90% of allocations completing in 200 milliseconds, and Pod Snapshots suspend an idle agent and resume it in seconds instead of paying to keep it running. Google reported more than 16x growth in GKE sandboxes in under five months after the November 2025 preview, with Langchain and Lovable among the named early users. Security is built in rather than bolted on: Agent Sandbox natively supports gVisor and a default-deny network policy, with pluggable Kata Containers for stronger kernel isolation, which matters when the workload is code a model wrote seconds ago. If session isolation is your main concern, we covered the provider-level trade-offs in our guide to agent session isolation across AWS, Azure, Google and Anthropic.

GPU scheduling with Dynamic Resource Allocation

Agents that run inference need GPUs, and the old nvidia.com/gpu integer request cannot express "give me 10 GiB of a shared card" or "I need two GPUs on the same NVLink domain." Dynamic Resource Allocation (DRA) fixes that, and it reached general availability in Kubernetes 1.34, with the feature gate locked on in 1.35. DRA lives in the resource.k8s.io/v1 API group and adds four objects: DeviceClass, ResourceClaim, ResourceClaimTemplate, and ResourceSlice. Its consumable-capacity model lets a driver expose a card's total memory and slice it safely, so a scheduler can hand 10 GiB out of a 40 GiB GPU to one claim and the rest to another, enforcing that the sum never exceeds the device. EKS, GKE and AKS all shipped DRA-capable control planes on their 1.34 releases through the first half of 2026, and Red Hat OpenShift 4.21 marked it GA on 25 March 2026.

For agents, fractional allocation is the point. Most agent steps are reasoning and tool calls, not dense matrix math, so pinning a whole H100 to one agent wastes most of the card. A ResourceClaimTemplate that requests a slice lets many agents share a GPU while the scheduler keeps them from overcommitting. We walk through the migration in detail in our Kubernetes DRA GA and GPU ResourceClaims migration guide.

Approach	Best for	State handling	GPU model	Cold-start control
Deployment	Stateless inference APIs	External only	Whole-GPU request	Poor for agents
StatefulSet (size 1)	Legacy single-pod services	Stable per ordinal	Whole-GPU request	Manual
Agent Sandbox	Isolated, bursty agent runtimes	Stable identity plus snapshots	Works with DRA slices	Warm pool, sub-second
Agent Substrate	Millions of sub-second tool calls	Snapshot-driven	Density-optimised	Custom control plane
Serverless GPU	Simple scale-to-zero jobs	None	Opaque	Provider-managed

Patterns that hold up in production

Once you stop treating agents as web services, a few patterns repeat.

Model the workload as a job queue with a chat interface. Agents are bursty and often idle, waiting on a slow external API or a human. A queue plus workers that pull tasks fits far better than a request-per-pod service, and it gives you a real scaling signal: queue depth.

Scale on the right metric with KEDA. Because the HPA cannot see GPU pressure or queue depth, event-driven autoscaling with KEDA lets you scale on the queue itself, or on GPU metrics you export, instead of on CPU that stays flat.

Keep a warm pool, and snapshot the idle ones. A warm pool absorbs the cold-start cost so the agent loop never waits minutes for a pod. Snapshots and standby capacity buffers (suspended VMs) then hold that pool cheaply, which is the difference between a warm pool that is affordable and one that is not.

Make long-running work durable. A durable workflow engine such as Temporal checkpoints each step, so an agent that runs for an hour survives a pod restart without redoing the first 55 minutes. Pair that with the Sandbox's persistent storage and you have crash-safe state, not best-effort memory.

Google is pushing the density argument further with Agent Substrate, a new open-source project it announced alongside the GKE GA. The team framed the split directly: "While standard Kubernetes is optimized to handle thousands of long-running services, Agent Substrate is designed for the chatter of millions of sub-second tool calls that would otherwise overwhelm a standard control plane," wrote Brandon Royal, Product Manager for GKE, and Tim Hockin, Software Engineer at Google Cloud. For most teams that is a signal, not a shopping list: you do not need a custom control plane yet, but you should design as if agent volume will grow.

The cost angle you cannot ignore

Every one of these choices is a cost decision. A warm pool of ready H100 sandboxes is fast, but idle GPUs still bill. Across AWS, GCP and Azure an on-demand H100 costs roughly $3.00 to $6.98 per GPU-hour as of April 2026, with AWS p5 near the top once you normalise its 8-GPU instance to a per-GPU rate and GCP A3-High around $3.00; the median on-demand rate sits near $2.99. Keep ten GPUs warm and idle and you are spending on the order of $700 a day for capacity nobody is using. This is why Pod Snapshots and standby buffers matter more than they first appear: suspending an idle sandbox to a cheaper cold pool, then resuming in seconds, is what keeps the warm-pool pattern from becoming a budget hole. Google also claims up to 30% better price-performance for Agent Sandbox on its Arm-based Axion processors than comparable hyperscaler options, which is worth testing against your own numbers rather than accepting on faith. The real cost is usually the idle time, not the busy time.

India-specific considerations

For teams building in India, GPU access and price dominate the decision. Hyperscaler H100 capacity in Indian regions is thinner than in the United States, and per-hour rates track the global $3 to $7 band, so a warm pool that keeps GPUs idle hurts a rupee-denominated budget faster than a dollar one. Fractional GPU sharing through DRA is often the single biggest lever an Indian team has, because it lets several agents share one scarce card. Data residency adds a second constraint: if agents process personal data, the Digital Personal Data Protection Act 2023 (DPDP) pushes toward keeping agent state and logs in-country, which favours a self-managed cluster with a Sandbox primitive over an opaque serverless GPU service you cannot inspect. We compare rental economics in our note on India GPU cloud rental pricing for H100 and B200.

FAQ

What is Agent Sandbox in Kubernetes?

Agent Sandbox is an open-source project under Kubernetes SIG Apps that adds a Sandbox Custom Resource Definition and controller. It gives each AI agent a single stateful pod with a stable hostname, persistent storage that survives restarts, and lifecycle controls like pause and resume, which standard Deployments and StatefulSets do not provide.

Why can't I just run AI agents as a normal Deployment?

A Deployment treats pods as interchangeable, but an agent is a singleton that holds conversation and plan state across many calls. Requests must return to the same instance, and killing a pod loses the reasoning context. Deployments also scale on CPU, which stays flat while a GPU-bound agent is fully loaded.

When did Agent Sandbox become generally available?

Google Kubernetes Engine announced general availability of Agent Sandbox on 20 May 2026, after a preview at KubeCon North America in November 2025. Google reported more than 16-fold growth in GKE sandboxes in under five months following that preview, with Langchain and Lovable among the named early adopters.

How does Kubernetes DRA help GPU-bound agents?

Dynamic Resource Allocation reached general availability in Kubernetes 1.34 and lets you request a slice of a GPU rather than a whole card. A driver can expose a 40 GiB GPU and hand 10 GiB to one claim, so several agents share one device while the scheduler prevents overcommitment. Most agent steps do not need a full GPU.

What is the cold-start problem for AI agents?

Loading model weights into VRAM takes 30 to 120 seconds, and a cold GPU replica can take 3 to 10 minutes to become ready. An agent's reasoning loop needs a fresh sandbox in under two seconds, so without a warm pool the loop stalls. Warm pools and pod snapshots close that gap.

How much does it cost to keep GPUs warm for agents?

An on-demand NVIDIA H100 costs roughly $3.00 to $6.98 per GPU-hour across AWS, GCP and Azure as of April 2026, with a median near $2.99. Idle GPUs still bill at that rate, so ten warm, unused cards can cost about $700 a day. Pod snapshots and standby buffers cut that idle spend.

Is Agent Sandbox secure enough for untrusted agent code?

Agent Sandbox natively supports gVisor and a default-deny Kubernetes network policy, and offers pluggable Kata Containers for stronger kernel isolation. That matters because agents often run code a language model generated seconds earlier. Isolation choices are configurable per sandbox, so you can match the runtime to your risk level.

Do I need Agent Substrate as well as Agent Sandbox?

Most teams do not. Agent Substrate is a newer open-source project for ultra-scale workloads with millions of sub-second tool calls that would overwhelm a standard control plane. Agent Sandbox covers typical agent runtimes today. Design for growth, but adopt Substrate only when a standard cluster becomes the bottleneck.

How eCorpIT can help

eCorpIT is a Gurugram-based, senior-led engineering organisation, founded in 2021 and assessed at CMMI Level 5, that designs and runs Kubernetes platforms for AI workloads. We help teams pick the right primitive for agent runtimes, set up DRA-based GPU scheduling and warm pools, and keep idle-GPU spend under control with snapshots and autoscaling. As an AWS, Microsoft and Google technology partner, we work across clouds and design aligned with DPDP requirements for Indian data residency. To scope an agent platform, talk to our engineering team or read more about our managed Kubernetes AI platform service.

References

Last updated: 23 July 2026.

DEV Community

Stateful AI agents on Kubernetes: the 2026 Agent Sandbox and GPU scheduling playbook

Stateful AI agents on Kubernetes: the 2026 Agent Sandbox and GPU scheduling playbook

Why Kubernetes fights AI agents

The autoscaling and cold-start problem

Agent Sandbox: a Kubernetes primitive for agents

GPU scheduling with Dynamic Resource Allocation

Patterns that hold up in production

The cost angle you cannot ignore

India-specific considerations

FAQ

How eCorpIT can help

References

Top comments (0)