Ant(on) Weiss

Posted on Oct 13

Google Built It. I Ran It on AWS.

#kubernetes #ai #platformengineering #aws

The agent runtime is more portable than you think — mostly.

Last week I got Google's Agent Substrate running on Amazon EKS, with S3 snapshots, RDS state and gVisor sandboxing. It works!

When Google introduced Agent Substrate alongside GKE Agent Sandbox in May, the coverage focused on the obvious: Kubernetes was never built for agents, so Google built a runtime on top of it. The more interesting question came at the end. Will this new agent control plane settle on one open project, the way container orchestration settled on Kubernetes? Or will every hyperscaler ship its own, and bring back the fragmentation one layer up?

So this became the first thing I did when diving into Agent Substrate. Check how actually portable it is.

Short version: the core is portable. The edges are still Google-shaped. Knowing exactly where those edges are is what your platform team needs before it decides where its agents will live for the next five years.

Agents are processes, not services

Kubernetes was designed for a modest number of long-lived, replicated services. An agent is something else: a stateful session that sleeps most of the time, wakes up to run a burst of model-generated code, then goes quiet again. (And BTW - this isn't about agents alone - this reflects the overall change in software runtime patterns we see evolving alongside the agentic shift. We are entering the age of new software ephemerality. But that deserves a standalone post.)

Picture a coding or a troubleshooting agent left open all afternoon. It works for ten seconds, then waits twenty minutes for the next prompt. Multiply that by every engineer in the company and you have thousands of sessions that are alive on paper and idle in practice. Give each one a Pod and you pay for reserved memory that does nothing. Make each one a Kubernetes object and you ask the API server to hold millions of resources it was never sized for.

Agent Substrate's answer is an old OS trick. It treats agents like processes and Pods like CPU cores:

A large registry of actors (agents) is multiplexed onto a small pool of pre-warmed workers (Pods).
Idle actors are snapshotted, RAM and filesystem included, to object storage and resumed on demand. (So agents are both ephemeral and stateful!)
Its own control plane and router sit beside Kubernetes, so waking an actor never waits on the Kubernetes scheduler.
Kubernetes still does what it is good at: provisioning nodes and Pods underneath.

The project reports 30x+ oversubscription with sub-second resume. Those are its numbers, measured on GKE. The question I cared about was simpler: does any of this depend on GKE?

Side note: who "ate" what? Open the Substrate repo and nearly everything is prefixed ate. The CLI is kubectl ate, the API group lives under ate.dev, and the components are named after it. The docs never say where the name comes from. My guess: it's the tail of substr-ate. Here are the pieces this post refers to:

ate-api-server: the control plane. It owns each actor's lifecycle, decides which worker an actor runs on, and coordinates snapshots. Its state lives in Postgres, not etcd, because it changes too fast for Kubernetes to store.

atelet: a per-node DaemonSet, the Substrate counterpart of the kubelet. It pulls images, drives the sandbox on its node and streams snapshots to and from object storage. That's why it needs its own cloud identity.

ateom: a small coordinator inside each worker Pod. It runs the sandbox (gVisor or a microVM) on atelet's behalf, so the agent process can come and go while the Pod stays up.

atenet: the networking layer. Its router wakes a suspended actor when a request arrives and sends traffic to the right worker.

atecontroller: the Kubernetes controller that turns a WorkerPool resource into a Deployment of warm worker Pods.

atespace: Substrate's isolation and naming boundary for actors. It's similar to a namespace, but it isn't a Kubernetes namespace.

The port: one table and one script

The Substrate control plane itself needed almost nothing. Everything I touched lived at the infrastructure edge, where the upstream install assumes GCP. The mapping is mostly one-to-one:

Concern	GKE (upstream)	EKS (my port)
Workload identity	GKE Workload Identity	IRSA (EKS Pod Identity also works)
Snapshot storage	GCS	S3: versioned, SSE-S3, public access blocked
Control-plane state	Cloud SQL Postgres	RDS Postgres 16, Multi-AZ, IAM auth
Image registry	Artifact Registry	ECR with scan-on-push
Telemetry	GKE Managed OTel / Prometheus	ADOT collector via the existing OTLP endpoint flag
Sandbox	gVisor	gVisor, unchanged

The result has two parts. The first is a one-shot provisioner: run AWS_REGION=us-west-2 ./setup-aws.sh, and in about 25–30 minutes you have a VPC, an EKS cluster, an S3 bucket, ECR, RDS and a bootstrapped database schema. The second is an aws kustomize overlay for the Substrate installer, which wires in S3, IRSA and the EKS image-credential layout. Actors suspend to S3, resume onto any worker, and keep their memory across the round trip.

The provisioner lives at antweiss/agent-substrate-on-aws, as the AWS counterpart of upstream's GCP setup tooling. The installer overlay is in PR #2241 while the maintainers decide where cloud-specific installers should live.

Where it's still Google-shaped

The table makes it look easy. The interesting part is what failed on the way, because each failure shows an assumption baked into today's install:

The Kubernetes version floor is higher than you think. On EKS 1.33 the install hangs with the pod-certificate controller crash-looping. 1.33 serves PodCertificateRequest only as alpha, and EKS won't let you flip alpha feature gates. GKE lets you enable the needed beta APIs at cluster creation. On EKS, 1.37 is the real minimum.
Image pulls run as the wrong identity. The kubelet image-credential provider runs as a subprocess of atelet. So actor image pulls use atelet's IAM role, not the node role. The first golden snapshot fails with a 401 from ECR until you grant atelet read access to the registry.
A monitoring resource that only exists on GKE. The router ships a PodMonitoring custom resource from GKE Managed Prometheus. EKS has no such CRD, so kubectl apply fails. The overlay drops it until there's an ADOT or Prometheus Operator equivalent.
Managed Postgres isn't real Postgres superuser. RDS gives you rds_superuser, not superuser. Creating a schema owned by another role fails until you grant that role to yourself first.
Cloud URLs baked into specs. gVisor and microVM asset locations are gs:// URLs inside CRD spec strings, and a gcloud shim still lives in the credentials code. You mirror the assets to S3 and steer around the rest.

None of this is a design flaw. It's what happens when a project is born inside one cloud and moves fast. The maintainers already agree: they want cloud-specific installers out of the core repo, with GKE bits moving to a separate substrate-gke repository. That's the healthy pattern. Vanilla Kubernetes in core, clouds at the edges.

The lesson for platform teams is broader than Substrate. Portability isn't a property of the architecture diagram. It's a property of the install path, the identity model and the URLs hiding in your CRDs. Check those before you trust the word "open".

The decision this actually forces

The question I already hear AWS practitioners asking: do we really need this?!

Bedrock AgentCore Runtime gives each session its own Firecracker microVM, with routing, identity and lifecycle handled for you. It's a good product. Why would anyone run Google's runtime on EKS instead?

Because the two make opposite bets about what an idle agent should cost.

	AgentCore Runtime (managed)	Agent Substrate on EKS (self-run)
Unit of isolation	One microVM per session	gVisor sandbox (microVM backend in progress) in a shared worker pool
Idle sessions	Live until idle timeout or lifetime cap	Snapshotted to S3, using no compute
Session lifetime	Capped (8 hours on microVMs)	As long as the snapshot exists
Cost driver	Per-session vCPU and GB-hours	Worker nodes, S3 and RDS you size yourself
Ops burden	Near zero	Yours: upgrades, capacity, security
Portability	AWS only	Any conformant Kubernetes

The managed model wins when sessions are short, traffic is spiky and you have no platform team to spare. The density model wins when you have many long-lived, mostly idle sessions, like coding agents, per-customer assistants and background workers. There, paying per session for compute that is mostly waiting adds up quickly. That matters even more when agentic ROI is as unclear as it is nowadays in most places.

This isn't really a technology choice. It's the same build-vs-buy call platform teams made about Kubernetes ten years ago, made one layer up. Make it on purpose, with your own session profile, not by default.

I didn't measure overall costs of running this on AWS, neither did I compare it to doing the same thing on AgentCore. Interested in something like this? Drop me a note.

What's next

Two things make the next few months interesting.

First, microVMs without bare metal. EC2 now supports nested virtualization on regular Intel instance families such as C8i, M8i and R8i. That puts Substrate's microVM backend within reach of ordinary EKS node groups, without paying for .metal instances. That's my next experiment.

Second, governance. The CNCF sandbox vote for Substrate has passed, and the transfer is in progress. Once the project sits in a neutral foundation with cloud installers as peers, "Google's runtime" becomes "the community's runtime." Then the fragmentation question gets a real answer.

If you run agents on AWS, try it and tell me what breaks. The PR and the discussion with the maintainers are linked below. If your team is weighing AgentCore against running its own agent platform, I'd like to hear how you're thinking about it. That decision is coming for every platform team, whether they plan for it or not.

I hope to be around KubeCon in Salt Lake City (November 9–12) if you want to chat about it in person.

DEV Community